CN110968419A - Data receiving method and device - Google Patents
Data receiving method and device Download PDFInfo
- Publication number
- CN110968419A CN110968419A CN201811162010.XA CN201811162010A CN110968419A CN 110968419 A CN110968419 A CN 110968419A CN 201811162010 A CN201811162010 A CN 201811162010A CN 110968419 A CN110968419 A CN 110968419A
- Authority
- CN
- China
- Prior art keywords
- receiving
- server
- data
- servers
- performance parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0277—Online advertisement
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Development Economics (AREA)
- Accounting & Taxation (AREA)
- General Engineering & Computer Science (AREA)
- Finance (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Game Theory and Decision Science (AREA)
- Economics (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Computer And Data Communications (AREA)
Abstract
The application discloses a data receiving method and device. The method comprises the following steps: monitoring performance parameters of a receiving server, wherein the performance parameters are used for expressing the capability of the receiving server for receiving data, and the receiving server is a virtual server virtualized by a physical cluster server; determining whether the number of the receiving servers needs to be adjusted or not according to the performance parameters; under the condition that the number of the receiving servers is determined to be required to be adjusted, adjusting the number of the receiving servers; and receiving and processing the data according to the receiving servers with the adjusted number. Through the method and the device, the problems that partial data cannot be received or resources are wasted due to the fact that adjustment cannot be performed when the data size is changed greatly in the related art are solved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data receiving method and apparatus.
Background
The internet advertisement exposure means that an advertiser puts advertisements in a certain mode, such as characters, pictures, videos and the like, through the internet, and a user can see the advertisements so as to expose the advertisements. For example, a user may consider that an advertisement is exposed when the user accesses the internet through a browser and loads an advertisement picture in some place of the page, and the exposure amount is calculated according to the number of times the picture is loaded. Because the internet advertisement is usually delivered in a very wide range, the exposure of one kind of advertisement can reach millions or even tens of millions in a short time, and for monitoring the exposure times of the huge internet advertisement, a receiving device with excellent concurrency is required to process the huge internet advertisement.
The current common solution is to use fixed receivers to form a cluster to overcome the problem that a single receiver is not enough to respond to millions of concurrencies. In practice, a receiving device is a server, a request is sent to a load balancer, and the load balancer averagely forwards the request to each server, so that the pressure of the original server is distributed to a cluster. The cluster type receiver is relatively inferior in scalability. The advertisement exposure amount is different according to different time intervals, and the number of servers in the cluster cannot bear the multiplied advertisement traffic in a fixed condition. The bottleneck of hardware on the reception of a single server still occurs at this time. The cluster-type receiving device is relatively large in cost, and particularly, when the advertisement exposure is low, the performance of the whole cluster is still used for responding, which is undoubtedly a waste of resources. In principle, the receiving device in the cluster manner is a load balancer which is relatively dependent on the entrance of the receiving device, and once the load balancer fails, the following server cluster cannot respond to any advertisement request, so that the disaster tolerance is very poor.
In the prior art, an effective solution is not provided at present for the problem that partial data cannot be received or resources are wasted due to the fact that adjustment cannot be performed when the data size is changed greatly in the related art.
Disclosure of Invention
The present application mainly aims to provide a data receiving method and apparatus, so as to solve the problem in the related art that when the data amount changes greatly, adjustment cannot be performed, so that part of data cannot be received, or resources are wasted.
In order to achieve the above object, according to an aspect of the present application, there is provided a data receiving method including: monitoring performance parameters of a receiving server, wherein the performance parameters are used for expressing the capability of the receiving server for receiving data, and the receiving server is a virtual server virtualized by a physical cluster server; determining whether the number of the receiving servers needs to be adjusted or not according to the performance parameters; adjusting the number of receiving servers in case of determining that the number of receiving servers needs to be adjusted; and receiving and processing the data according to the receiving servers with the adjusted number.
Optionally, the performance parameter of the receiving server includes at least one of: the utilization rate of a Central Processing Unit (CPU), the utilization rate of a memory and the utilization rate of network bandwidth.
Optionally, determining whether the number of receiving servers needs to be adjusted according to the performance parameter includes: judging whether the monitored monitoring value of the performance parameter exceeds a first preset threshold value, wherein the first preset threshold value is a value of the performance parameter corresponding to the receiving server needing to be increased; and determining that the number of receiving servers needs to be increased in the case that the judgment result is yes.
Optionally, determining whether the number of receiving servers needs to be adjusted according to the performance parameter includes: judging whether the monitored monitoring value of the performance parameter is lower than a second preset threshold value, wherein the second preset threshold value is a value of the performance parameter corresponding to the receiving server needing to be reduced; in the case where the judgment result is yes, it is determined that the number of receiving servers needs to be reduced.
Optionally, the physical cluster servers are distributed in different physical areas and controlled by a load balancer, and the load balancer distributes data to a plurality of receiving servers virtualized based on the physical cluster servers in a polling manner.
Optionally, the method further includes: distributing and binding a plurality of public network IP addresses for the load balancer of the physical area to which the receiving server belongs; and under the condition that the bound public network IP address cannot be used, unbinding the unusable public network IP address and rebinding other usable public network IP addresses.
Optionally, the method further includes: and under the condition that the load balancer of the physical area to which the receiving server belongs is interrupted, forwarding data to the receiving server through the load balancer of the adjacent area of the physical area to which the receiving server belongs.
In order to achieve the above object, according to another aspect of the present application, there is provided a data receiving apparatus including: the system comprises a monitoring module, a data processing module and a data processing module, wherein the monitoring module is used for monitoring performance parameters of a receiving server, the performance parameters are used for representing the data processing capacity of the receiving server, and the receiving server is a virtual server virtualized by a physical cluster server; a determining module, configured to determine whether the number of receiving servers needs to be adjusted according to the performance parameter; the adjusting module is used for adjusting the number of the receiving servers under the condition that the number of the receiving servers needs to be adjusted; and the receiving module is used for receiving and processing the data according to the receiving servers with the adjusted number.
In order to achieve the above object, according to another aspect of the present application, there is provided a storage medium including a stored program, wherein the program performs the method of any one of the above.
To achieve the above object, according to another aspect of the present application, there is provided a processor for executing a program, wherein the program executes to perform the method of any one of the above.
Through the application, the following steps are adopted: monitoring performance parameters of a receiving server, wherein the performance parameters are used for expressing the capability of the receiving server for receiving data, and the receiving server is a virtual server virtualized by a physical cluster server; determining whether the number of the receiving servers needs to be adjusted or not according to the performance parameters; adjusting the number of receiving servers in case of determining that the number of receiving servers needs to be adjusted; the data are received and processed according to the receiving servers with the adjusted number, so that the problem that all data cannot be received at the same time in the related technology and part of data cannot be received at the moment is solved. And then the effect of changing the receiving capability of the receiving device according to the data size of the data and enabling the receiving device to effectively receive the data is achieved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
fig. 1 is a flowchart of a data receiving method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a receiving apparatus provided according to an embodiment of the present application;
FIG. 3 is a flow chart of a scaling method for receiving data according to an embodiment of the present application;
FIG. 4 is a flow chart of a method for IP address pooling dynamic binding provided according to an embodiment of the present application;
FIG. 5 is a flow chart of cross-regional distribution of traffic provided according to an embodiment of the present application;
fig. 6 is a schematic diagram of a data receiving apparatus according to an embodiment of the present application.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of description, some terms or expressions referred to in the embodiments of the present application are explained below:
load balancing: the method is a technology for transmitting the traffic to a plurality of servers according to a certain rule to disperse the load.
IP address: refers to an internet protocol address, and is a unique identifier for the computers to communicate with each other.
According to an embodiment of the present application, there is provided a data receiving method.
Fig. 1 is a flowchart of a data receiving method according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:
step S101, monitoring performance parameters of a receiving server, wherein the performance parameters are used for representing the capability of the receiving server for receiving data, and the receiving server is a virtual server virtualized by a physical cluster server;
step S102, determining whether the number of the receiving servers needs to be adjusted or not according to the performance parameters;
step S103, under the condition that the number of the receiving servers is determined to be required to be adjusted, adjusting the number of the receiving servers;
and step S104, performing receiving processing on the data according to the receiving servers with the adjusted number.
The monitoring of the performance parameter of the receiving server may be monitoring the performance parameter of the receiving server when the receiving server receives the data. When the performance of the receiving server is constant, the performance parameter of the receiving server changes to a certain extent when dynamically changing data is received, for example, when the data volume of the received data increases, the load of the receiving server increases accordingly, and the performance parameter also increases/decreases accordingly; when the data amount of the received data becomes smaller, the load of the receiving server becomes smaller accordingly, and the performance parameter becomes smaller/larger accordingly. When the type of the performance parameter is different, the variation of the performance parameter may be different from or the same as the variation of the data amount received by the receiving server, depending on the type of the performance parameter.
The performance parameters of the receiving server are performance parameters indicating the receiving capability of the receiving server, and the performance parameters of the receiving server have many different functions, including performance parameters indicating the stability of the receiving server, performance parameters indicating the processing capability of the receiving server, performance parameters indicating the computing capability of the receiving server, and the like. In this embodiment, the purpose of monitoring the performance parameter is to determine whether the receiving server is capable of receiving continuously changing data, that is, to monitor whether the receiving capability of the receiving server meets the condition of receiving data, and therefore, the performance parameter in this embodiment is a performance parameter used for representing the receiving capability of the receiving server.
The receiving server is a virtual server virtualized by a physical cluster server. Since the receiving capability of the server depends on the real server cluster when receiving data, the receiving capability of the server cluster is not changed under the general condition, but the received data is changed dynamically. When the receiving capability of the server cluster is far greater than the data volume of the received data, the resource waste of the service cluster is caused, and the server cluster with the receiving capability is not required to receive the data. When the receiving capability of the server cluster is not enough to receive data with a large data volume, partial loss of data which cannot be received occurs, which causes data loss, even overload of the server cluster. Therefore, in this embodiment, the virtual server that can be used to receive data is virtualized by using the physical cluster server, and the receiving capability of the receiving server can be adjusted at any time according to the data volume of the received data. Therefore, the situation that resources are not wasted and data loss is not caused is achieved.
The performance parameters of the receiving server determine whether the number of the receiving servers needs to be adjusted according to the performance parameters, that is, the receiving capability of the receiving server is adjusted, the receiving capability of the receiving server can be adjusted in various ways, and the receiving capability of the receiving server can be changed by changing the basic parameters of the receiving server. The method has the advantages of low speed, low efficiency, large computation amount and heavy load. In the embodiment, the receiving capacity of the receiving servers is increased by increasing the number of the receiving servers, and under the condition that the receiving capacity of the original receiving server is insufficient, a new receiving server is added for continuously receiving, so that the received data is not influenced, and the integrity of the received data can be ensured; when the data volume is small and the receiving capacity of the receiving server is excessive, part of the servers are deleted, so that the receiving capacity of the receiving server is matched with the data volume of the received data, and the resource waste of the receiving server can be effectively prevented.
Therefore, in the present embodiment, it is determined whether or not the adjustment of the number of receiving servers is required, based on the performance parameter, and in the case where it is determined that the adjustment of the number of receiving servers is required, the number of receiving servers is adjusted. And after the receiving capacity of the receiving server is matched with the data volume of the received data, receiving the data according to the receiving server with the adjusted number.
Optionally, in the data receiving method provided in the embodiment of the present application, the performance parameter of the receiving server includes at least one of: the utilization rate of a Central Processing Unit (CPU), the utilization rate of a memory and the utilization rate of network bandwidth.
The performance parameter of the receiving server is used to indicate the capability of the receiving server to receive data, and may be various performance parameters of the receiving server, for example, the utilization rate of the CPU may reflect the data size of the data received by the receiving server from the front, when the data size of the received data increases, the utilization rate of the CPU also increases, and when the data size of the received data decreases, the utilization rate of the CPU also decreases. For example, the usage rate of the memory may reflect the size of the data amount of the data received by the receiving server from the front, and the usage rate of the memory may increase when the data amount of the received data increases and decrease when the data amount of the received data decreases. For another example, the usage rate of the network bandwidth may reflect the size of the data amount of the data received by the receiving server from the front, and when the data amount of the received data increases, the usage rate of the memory may also increase, and when the data amount of the received data decreases, the usage rate of the memory may also decrease.
Optionally, in the data receiving method provided in the embodiment of the present application, determining whether the number of receiving servers needs to be adjusted according to the performance parameter includes: judging whether the monitored value of the monitored performance parameter exceeds a first preset threshold value, wherein the first preset threshold value is the value of the performance parameter corresponding to the receiving server needing to be increased; and determining that the number of receiving servers needs to be increased in the case that the judgment result is yes.
The above-mentioned determining whether to need to adjust the number of the receiving servers according to the performance parameter may increase the data volume, and if the data volume is increased, the performance parameter of the receiving server is increased/decreased accordingly, and the performance parameter needs to be monitored, and if the data volume is increased to the extent that the receiving capability is insufficient, the number of the receiving servers is controlled to be increased. The method can be used for judging whether the monitored value of the monitored performance parameter exceeds a first preset threshold value, wherein the first preset threshold value is the value of the performance parameter corresponding to the receiving server needing to be increased; and determining that the number of the receiving servers needs to be increased to realize the judgment if the judgment result is yes.
Optionally, in the data receiving method provided in the embodiment of the present application, determining whether the number of receiving servers needs to be adjusted according to the performance parameter includes: judging whether the monitored value of the performance parameter is lower than a second preset threshold value, wherein the second preset threshold value is a value of the performance parameter corresponding to the receiving server needing to be reduced; in the case where the judgment result is yes, it is determined that the number of receiving servers needs to be reduced.
The determination of whether or not the number of receiving servers needs to be adjusted based on the performance parameters may result in a decrease in the data amount, and the performance parameters of the receiving servers may also be decreased/increased accordingly when the data amount decreases. The performance parameters need to be monitored, and the number of control receiving servers is reduced when the data size is reduced to an excessive receiving capacity. The monitoring method can be used for judging whether the monitored value of the performance parameter is lower than a second preset threshold value, wherein the second preset threshold value is a value of the performance parameter corresponding to the receiving server needing to be reduced; and in the case that the judgment result is yes, determining that the number of the receiving servers needs to be reduced to realize the judgment.
Optionally, in the data receiving method provided in this embodiment of the present application, the physical cluster servers are distributed in different physical areas and controlled by a load balancer, and the load balancer distributes data to a plurality of receiving servers obtained based on virtualization of the physical cluster servers in a polling manner.
The number of the physical cluster servers can be multiple, the physical cluster servers are distributed in different physical areas, the physical cluster servers are connected through a special line, and the physical cluster servers are all connected with a data center used for processing received data through the special line. For example, 4 physical cluster servers can be respectively arranged in Beijing, Shanghai, Guangzhou and Shenzhen. Each physical area is provided with a load balancing device and a set of physical cluster servers, and the number of virtual receiving servers of the physical cluster servers can be controlled through the load balancing device, so that the receiving capacity of the receiving servers is controlled.
The physical cluster servers of each physical area are connected by a dedicated line. Received data is subjected to domain name resolution, and is subjected to polling resolution to IP addresses of physical cluster servers of 4 physical areas. And the load balance receives data according to the number of the receiving servers in the physical cluster servers, so that each receiving server can uniformly receive the data corresponding to the receiving capacity of the receiving server. Thereby achieving a high availability of service.
The load balancer may also be used to distribute data to a virtual plurality of receiving servers. In this embodiment, in order to ensure that each receiving server receives data corresponding to its receiving capability, the data may be sequentially distributed according to the sequence of the establishment time of the receiving servers. For example, there are four receiving servers, i.e., a server a, a server b, a server c, and a server d, respectively, according to the set-up time, when distributing data, data is distributed to the server a first, and when the receiving capability of the server a is equivalent to the received data, data is distributed to the server b, and the data is arranged in this way until the received data is received completely. The resource of the receiving server can be ensured not to be wasted, and the receiving capacity change of the receiving server can be observed through the receiving record of the receiving server.
Optionally, in the data receiving method provided in the embodiment of the present application, the method further includes: distributing and binding a plurality of public network IP addresses for a load balancer of a physical area to which a receiving server belongs; and under the condition that the bound public network IP address cannot be used, unbinding the unusable public network IP address and rebinding other usable public network IP addresses.
In the receiving method, the load balancer in each physical area has a plurality of public network IP addresses, and a possible disaster is that the public network IP address connected to the load balancer cannot be used for various reasons, and at this time, data cannot reach the receiving server, so that the domain name of the physical cluster server needs to be unbound from the public network IP address and bound to a new public network IP address again. Thus, disaster tolerance of the receiving server in the physical area can be ensured.
Optionally, in the data receiving method provided in the embodiment of the present application, the method further includes: and under the condition that the load balancer of the physical area to which the receiving server belongs is interrupted in a network, forwarding the data to the receiving server through the load balancer of the adjacent area of the physical area to which the receiving server belongs.
In the receiving method, it may also happen that a network in a certain physical area is interrupted and cannot be accessed, the domain name resolution directly deletes the public network IP address mapping of the physical area, the physical cluster server in which the domain name resolution is located is taken over by the load balancer of the previous physical area, and the internal resource pool in which the domain name resolution is located is still unchanged.
The data receiving method provided by the embodiment of the application monitors the performance parameters of the receiving server, wherein the performance parameters are used for indicating the data receiving capacity of the receiving server, and the receiving server is a virtual server virtualized by a physical cluster server; determining whether the number of the receiving servers needs to be adjusted or not according to the performance parameters; under the condition that the number of the receiving servers is determined to be required to be adjusted, adjusting the number of the receiving servers; the data are received and processed according to the receiving servers with the adjusted quantity, and the problems that in the related technology, when the data quantity is changed greatly, adjustment cannot be carried out, partial data cannot be received, or resources are wasted are solved. And then the effect of changing the receiving capability of the receiving device according to the data size of the data and enabling the receiving device to effectively receive the data is achieved.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
The embodiment of the present application further provides a highly available device and a disaster recovery method for highly concurrent internet advertisement exposure reception as a preferred embodiment, which will be described in detail below.
The embodiment mainly aims at solving the problem of high availability of advertisement reception, ensures that all advertisement exposure requests can be responded by a receiving device, and simultaneously solves the problems of resource cost and poor disaster tolerance. The embodiment mainly introduces a high-availability device and a disaster recovery method for receiving internet advertisement exposure under high concurrency. The device consists of scalable server clusters and load balancers distributed on different physical area nodes. Meanwhile, two disaster recovery methods aiming at the device are provided: an IP address pooling dynamic binding technique and a cross-regional distribution traffic technique. The method comprises the following specific steps:
fig. 2 is a schematic structural diagram of a receiving apparatus according to an embodiment of the present application, and as shown in fig. 2, the apparatus includes a plurality of clusters of physical area nodes, each cluster includes a non-fixed number of nodes, the number of nodes is automatically scaled according to an actual concurrency number, hardware resource consumption of a server, and other comprehensive judgment, and each cluster is connected in series to a data center. However, each load balancing device is only associated with the cluster of the physical node, and the load balancing device binds a unique IP address, and normally preferentially forwards the request to the cluster of the physical area. Once a certain IP address cannot be used, the system automatically eliminates the reception of the IP address, and meanwhile, the cluster server of the node is forwarded by the adjacent load balancing equipment.
This embodiment is described below as an alternative embodiment. The receiving device is deployed in 4 physical areas, Beijing, Shanghai, Guangzhou, Chengdu and the like, each area is deployed with a load balancing device and a set of server cluster, and each area is connected by a special line. All ad requests are polled to 4 physical zone IP addresses via domain name resolution. The load balancing forwards the advertisement exposure request according to the number of the servers in the cluster, and each server can uniformly receive the request and make a response. The high availability of service is achieved by forming the device from multiple sets of clusters in different places.
The server cluster is actually a resource pool consisting of a set of physical servers. The resource pool is monitored by a main service, and the monitored indexes mainly comprise a memory, a CPU, a disk and a network bandwidth. The receiving server is a server virtualized in the resource pool, and the virtual server is also managed and configured by the resource pool main service. The number of virtual servers in practice is dynamically adjusted according to the request amount of advertisement exposure, a threshold value needs to be added to indexes such as a memory, a CPU (central processing unit), a disk, network bandwidth and the like, and a new receiving server is virtualized from a resource pool after the threshold value is exceeded.
A typical feature of internet advertisement is that a very large amount of exposure is achieved in a short time, for example, an advertiser puts an advertisement at a certain time, and the advertisement has a very large amount of exposure in a short time, so that a receiving device may have a rapid increase of requests in a short time. At this time, the main service in the resource pool observes that the CPU, the memory, or the network bandwidth of each virtual server has a sharp increase in resource utilization rate, and at this time, it is necessary to start increasing the number of virtual servers to expand the receiving capability of the whole receiving apparatus. Similarly, after a period of time, some internet advertisement is not released, and the part of advertisement does not send a request to the receiving device any more, and if the number of virtual servers remains unchanged at this time, resource waste will inevitably occur, and the cost will be affected by the wear and tear of the physical equipment. Therefore, when the resource utilization rate of the resource pool master service is low, a reduction strategy is adopted to reduce the number of cluster servers, so that the resource utilization rate of each virtual server is improved. The reduction strategy is once in 10 minutes.
Because the reduction server can directly delete the server, the advertisement exposure data of the data disk can be firstly backed up in the deletion server, the data is directly transmitted to the data center by a special line, compression and transmission are firstly carried out to save bandwidth, and after MD5 checks are carried out remotely and locally, all resources of the virtual server can be directly deleted and released to the resource pool.
The determination of the number of cluster servers is realized by a set of monitoring algorithm, the number of clusters is determined by the CPU occupation and the memory occupation of each preset server, the number of clusters in the physical area is expanded and contracted, the cluster scale is intelligently adjusted, the expansion strategy flow is shown in fig. 3, and fig. 3 is a flow chart of the expansion method for receiving data provided according to the embodiment of the present application.
In the aspect of disaster tolerance, an IP address pooling dynamic binding technology and a cross-regional flow distribution technology are mainly adopted. Fig. 4 is a flowchart of an IP address pooling dynamic binding method according to an embodiment of the present application, and as shown in fig. 4, the IP address pooling dynamic binding technique refers to that a load of each area is balanced by a plurality of public network IP addresses, and a general disaster is that the IP address cannot be used (for example, a Distributed Denial of Service (DDos) attack, specifically, an attack is initiated on one or more targets by combining a plurality of computers as an attack platform by using a client/server technique), at this time, an exposure request cannot reach a receiving device, and therefore, a domain name needs to be rapidly unbound to the IP address and newly bound to a new IP address. Thus, disaster tolerance of the receiving server in the physical area can be ensured.
Fig. 5 is a flow chart of cross-regional traffic distribution provided according to an embodiment of the present application, and as shown in fig. 5, when a certain physical region network is interrupted and cannot be accessed, domain name resolution directly deletes the public network IP address mapping of the physical region, the cluster server where the cluster server is located is taken over by the last physical region load balancer, and the internal resource pool where the cluster server is located remains unchanged, and because there is a private line intercommunication between clusters in different physical regions, a request is forwarded to a cluster in a failure region by a normal load balancer in an adjacent region through a private line, so that cross-regional traffic forwarding can be implemented without affecting the load of each server in each physical region, and stability of a receiving apparatus is ensured.
The embodiment realizes the high-concurrency internet advertisement receiving device, can stably and efficiently receive millions or even tens of millions of requests in unit time, has good expansion and contraction characteristics for short-time request surge, not only considers the high availability of production, but also considers the cost control. The dynamic IP address pooling dynamic binding technology and the cross-region distribution technology provided by the disaster recovery scheme can enable the device to achieve excellent node stability.
The embodiment of the present application further provides a data receiving apparatus, and it should be noted that the data receiving apparatus in the embodiment of the present application may be used to execute the method for receiving data provided in the embodiment of the present application. The following describes a data receiving apparatus according to an embodiment of the present application.
Fig. 6 is a schematic diagram of a data receiving apparatus according to an embodiment of the present application. As shown in fig. 6, the apparatus includes: a monitoring module 61, a determination module 62, an adjustment module 63 and a receiving module 64.
The monitoring module 61 is configured to monitor a performance parameter of the receiving server, where the performance parameter is used to indicate a capability of the receiving server to process data, and the receiving server is a virtual server virtualized by a physical cluster server; a determining module 62, connected to the detecting module 61, for determining whether the number of receiving servers needs to be adjusted according to the performance parameters; an adjusting module 63, connected to the determining module 62, for adjusting the number of receiving servers when it is determined that the number of receiving servers needs to be adjusted; and a receiving module 64 connected to the adjusting module 63, for receiving and processing the data according to the adjusted number of receiving servers.
Optionally, in the data receiving apparatus provided in the embodiment of the present application, the determining module 62 includes: the first judging unit is used for judging whether the monitored value of the monitored performance parameter exceeds a first preset threshold value, wherein the first preset threshold value is the value of the performance parameter corresponding to the receiving server needing to be increased; a first determination unit configured to determine that the number of receiving servers needs to be increased in a case where a determination result of the first determination unit is yes.
Optionally, in the data receiving apparatus provided in the embodiment of the present application, the determining module 62 includes: the second judging unit is used for judging whether the monitored value of the monitored performance parameter is lower than a second preset threshold value, wherein the second preset threshold value is a value required to reduce the performance parameter corresponding to the receiving server; and a second determination unit configured to determine that the number of receiving servers needs to be reduced in a case where the determination result is yes.
In the data receiving apparatus provided in the embodiment of the present application, a monitoring module 61 is used to monitor performance parameters of a receiving server, where the performance parameters are used to indicate a capability of the receiving server to receive data, and the receiving server is a virtual server virtualized by a physical cluster server; the determining module 62 determines whether the number of receiving servers needs to be adjusted according to the performance parameter; the adjusting module 63 adjusts the number of the receiving servers when determining that the number of the receiving servers needs to be adjusted; the receiving module 64 receives and processes the data according to the receiving servers with the adjusted number, so that the problem that in the related art, when the data amount changes greatly, the adjustment cannot be performed, so that part of the data cannot be received, or resources are wasted is solved. And then the effect of changing the receiving capability of the receiving device according to the data size of the data and enabling the receiving device to effectively receive the data is achieved.
The data receiving device comprises a processor and a memory, wherein the monitoring module 61, the determining module 62, the adjusting module 63, the receiving module 64 and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more, and the receiving capability of the receiving device is changed according to the data size of the data by adjusting the kernel parameters, so that the receiving device can effectively receive the data.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
An embodiment of the present invention provides a storage medium having a program stored thereon, which when executed by a processor implements the data receiving method.
The embodiment of the invention provides a processor, which is used for running a program, wherein the data receiving method is executed when the program runs.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein the processor executes the program and realizes the following steps: monitoring performance parameters of a receiving server, wherein the performance parameters are used for expressing the capability of the receiving server for receiving data, and the receiving server is a virtual server virtualized by a physical cluster server; determining whether the number of the receiving servers needs to be adjusted or not according to the performance parameters; under the condition that the number of the receiving servers is determined to be required to be adjusted, adjusting the number of the receiving servers; and receiving and processing the data according to the receiving servers with the adjusted number.
Optionally, the performance parameter of the receiving server includes at least one of: the utilization rate of a Central Processing Unit (CPU), the utilization rate of a memory and the utilization rate of network bandwidth.
Optionally, determining whether the number of receiving servers needs to be adjusted according to the performance parameter includes: judging whether the monitored value of the monitored performance parameter exceeds a first preset threshold value, wherein the first preset threshold value is the value of the performance parameter corresponding to the receiving server needing to be increased; and determining that the number of receiving servers needs to be increased in the case that the judgment result is yes.
Optionally, determining whether the number of receiving servers needs to be adjusted according to the performance parameter includes: judging whether the monitored value of the performance parameter is lower than a second preset threshold value, wherein the second preset threshold value is a value of the performance parameter corresponding to the receiving server needing to be reduced; in the case where the judgment result is yes, it is determined that the number of receiving servers needs to be reduced.
Optionally, the physical cluster servers are distributed in different physical areas and controlled by a load balancer, and the load balancer distributes data to a plurality of receiving servers virtualized based on the physical cluster servers in a polling manner.
Optionally, the method further comprises: distributing and binding a plurality of public network IP addresses for a load balancer of a physical area to which a receiving server belongs; and under the condition that the bound public network IP address cannot be used, unbinding the unusable public network IP address and rebinding other usable public network IP addresses.
Optionally, the method further comprises: and under the condition that the load balancer of the physical area to which the receiving server belongs is interrupted in a network, forwarding the data to the receiving server through the load balancer of the adjacent area of the physical area to which the receiving server belongs.
The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: monitoring performance parameters of a receiving server, wherein the performance parameters are used for expressing the capability of the receiving server for receiving data, and the receiving server is a virtual server virtualized by a physical cluster server; determining whether the number of the receiving servers needs to be adjusted or not according to the performance parameters; under the condition that the number of the receiving servers is determined to be required to be adjusted, adjusting the number of the receiving servers; and receiving and processing the data according to the receiving servers with the adjusted number.
Optionally, the performance parameter of the receiving server includes at least one of: the utilization rate of a Central Processing Unit (CPU), the utilization rate of a memory and the utilization rate of network bandwidth.
Optionally, determining whether the number of receiving servers needs to be adjusted according to the performance parameter includes: judging whether the monitored value of the monitored performance parameter exceeds a first preset threshold value, wherein the first preset threshold value is the value of the performance parameter corresponding to the receiving server needing to be increased; and determining that the number of receiving servers needs to be increased in the case that the judgment result is yes.
Optionally, determining whether the number of receiving servers needs to be adjusted according to the performance parameter includes: judging whether the monitored value of the performance parameter is lower than a second preset threshold value, wherein the second preset threshold value is a value of the performance parameter corresponding to the receiving server needing to be reduced; in the case where the judgment result is yes, it is determined that the number of receiving servers needs to be reduced.
Optionally, the physical cluster servers are distributed in different physical areas and controlled by a load balancer, and the load balancer distributes data to a plurality of receiving servers virtualized based on the physical cluster servers in a polling manner.
Optionally, the method further comprises: distributing and binding a plurality of public network IP addresses for a load balancer of a physical area to which a receiving server belongs; and under the condition that the bound public network IP address cannot be used, unbinding the unusable public network IP address and rebinding other usable public network IP addresses.
Optionally, the method further comprises: and under the condition that the load balancer of the physical area to which the receiving server belongs is interrupted in a network, forwarding the data to the receiving server through the load balancer of the adjacent area of the physical area to which the receiving server belongs.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A data receiving method, comprising:
monitoring performance parameters of a receiving server, wherein the performance parameters are used for expressing the capability of the receiving server for receiving data, and the receiving server is a virtual server virtualized by a physical cluster server;
determining whether the number of the receiving servers needs to be adjusted or not according to the performance parameters;
adjusting the number of receiving servers in case of determining that the number of receiving servers needs to be adjusted;
and receiving and processing the data according to the receiving servers with the adjusted number.
2. The method of claim 1, wherein the performance parameters of the receiving server comprise at least one of:
the utilization rate of a Central Processing Unit (CPU), the utilization rate of a memory and the utilization rate of network bandwidth.
3. The method of claim 2, wherein determining whether an adjustment to the number of receiving servers is required based on the performance parameter comprises:
judging whether the monitored monitoring value of the performance parameter exceeds a first preset threshold value, wherein the first preset threshold value is a value of the performance parameter corresponding to the receiving server needing to be increased;
and determining that the number of receiving servers needs to be increased in the case that the judgment result is yes.
4. The method of claim 2, wherein determining whether an adjustment to the number of receiving servers is required based on the performance parameter comprises:
judging whether the monitored monitoring value of the performance parameter is lower than a second preset threshold value, wherein the second preset threshold value is a value of the performance parameter corresponding to the receiving server needing to be reduced;
in the case where the judgment result is yes, it is determined that the number of receiving servers needs to be reduced.
5. The method of claim 1, wherein the physical cluster servers are distributed in different physical areas and are controlled by a load balancer that distributes data in a round robin fashion to a plurality of receiving servers virtualized based on the physical cluster servers.
6. The method of claim 5, further comprising:
distributing and binding a plurality of public network IP addresses for the load balancer of the physical area to which the receiving server belongs; and under the condition that the bound public network IP address cannot be used, unbinding the unusable public network IP address and rebinding other usable public network IP addresses.
7. The method of claim 5, further comprising:
and under the condition that the load balancer of the physical area to which the receiving server belongs is interrupted, forwarding data to the receiving server through the load balancer of the adjacent area of the physical area to which the receiving server belongs.
8. A data receiving device, comprising:
the system comprises a monitoring module, a data processing module and a data processing module, wherein the monitoring module is used for monitoring performance parameters of a receiving server, the performance parameters are used for representing the data processing capacity of the receiving server, and the receiving server is a virtual server virtualized by a physical cluster server;
a determining module, configured to determine whether the number of receiving servers needs to be adjusted according to the performance parameter;
the adjusting module is used for adjusting the number of the receiving servers under the condition that the number of the receiving servers needs to be adjusted;
and the receiving module is used for receiving and processing the data according to the receiving servers with the adjusted number.
9. A storage medium, characterized in that the storage medium comprises a stored program, wherein the program performs the method of any one of claims 1 to 7.
10. A processor, characterized in that the processor is configured to run a program, wherein the program when running performs the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811162010.XA CN110968419A (en) | 2018-09-30 | 2018-09-30 | Data receiving method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811162010.XA CN110968419A (en) | 2018-09-30 | 2018-09-30 | Data receiving method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110968419A true CN110968419A (en) | 2020-04-07 |
Family
ID=70029283
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811162010.XA Pending CN110968419A (en) | 2018-09-30 | 2018-09-30 | Data receiving method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110968419A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114285832A (en) * | 2021-05-11 | 2022-04-05 | 鸬鹚科技(深圳)有限公司 | Disaster recovery system, method, computer device and medium for multiple data centers |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105162883A (en) * | 2015-09-25 | 2015-12-16 | 网宿科技股份有限公司 | Network load balancing processing system, methods and devices |
| CN105159775A (en) * | 2015-08-05 | 2015-12-16 | 浪潮(北京)电子信息产业有限公司 | Load balancer based management system and management method for cloud computing data center |
| CN107329827A (en) * | 2017-06-22 | 2017-11-07 | 平安科技（深圳）有限公司 | Support lvs dispatching methods, equipment and the storage medium of Hash scheduling strategy |
-
2018
- 2018-09-30 CN CN201811162010.XA patent/CN110968419A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105159775A (en) * | 2015-08-05 | 2015-12-16 | 浪潮(北京)电子信息产业有限公司 | Load balancer based management system and management method for cloud computing data center |
| CN105162883A (en) * | 2015-09-25 | 2015-12-16 | 网宿科技股份有限公司 | Network load balancing processing system, methods and devices |
| CN107329827A (en) * | 2017-06-22 | 2017-11-07 | 平安科技（深圳）有限公司 | Support lvs dispatching methods, equipment and the storage medium of Hash scheduling strategy |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114285832A (en) * | 2021-05-11 | 2022-04-05 | 鸬鹚科技(深圳)有限公司 | Disaster recovery system, method, computer device and medium for multiple data centers |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11102290B2 (en) | Peer-to-peer network prioritizing propagation of objects through the network | |
| CN110602156A (en) | Load balancing scheduling method and device | |
| US9116752B1 (en) | Systems, methods, devices and arrangements for server load distribution | |
| WO2015096681A1 (en) | Method and device for processing service access | |
| US20130007253A1 (en) | Method, system and corresponding device for load balancing | |
| KR20010088742A (en) | Parallel Information Delievery Method Based on Peer-to-Peer Enabled Distributed Computing Technology | |
| JP2013525931A (en) | Dynamic binding used for content delivery | |
| CN107105013B (en) | File processing method, server, terminal and system | |
| CN108259603B (en) | Load balancing method and device | |
| CN105007337A (en) | Cluster system load balancing method and system thereof | |
| US9819626B1 (en) | Placement-dependent communication channels in distributed systems | |
| US20200050479A1 (en) | Blockchain network and task scheduling method therefor | |
| US20180227139A1 (en) | Method of Terminal-Based Conference Load-Balancing, and Device and System Utilizing Same | |
| JP2014035717A (en) | Load distribution method taking account of node of each rank of multi-rank | |
| CN111092921A (en) | Data acquisition method, device and storage medium | |
| CN105068755A (en) | Data duplicate storage method facing cloud computing content distribution network | |
| CN106302640A (en) | Data request processing method and device | |
| US9760370B2 (en) | Load balancing using predictable state partitioning | |
| US12058210B2 (en) | Multichannel virtual internet protocol address affinity | |
| CN108924203B (en) | Data copy self-adaptive distribution method, distributed computing system and related equipment | |
| Hsieh et al. | The incremental load balance cloud algorithm by using dynamic data deployment | |
| CN105592134B (en) | A kind of method and apparatus of load balancing | |
| KR101089509B1 (en) | Cloud computing network system and file distrubuting method of the same | |
| JP2005501335A (en) | Method and system for distributing the load of computer resources among computers | |
| US9544371B1 (en) | Method to discover multiple paths to disk devices cluster wide |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200407 |
|
| RJ01 | Rejection of invention patent application after publication |