CN113556402B - Data downloading scheduling method and device and electronic equipment - Google Patents
Data downloading scheduling method and device and electronic equipment Download PDFInfo
- Publication number
- CN113556402B CN113556402B CN202110862294.9A CN202110862294A CN113556402B CN 113556402 B CN113556402 B CN 113556402B CN 202110862294 A CN202110862294 A CN 202110862294A CN 113556402 B CN113556402 B CN 113556402B
- Authority
- CN
- China
- Prior art keywords
- task
- downloading
- log data
- download
- configuration information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000012546 transfer Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 abstract description 9
- 230000008569 process Effects 0.000 description 18
- 238000010586 diagram Methods 0.000 description 15
- 238000004590 computer program Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 8
- 238000012986 modification Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 238000007405 data analysis Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000004140 cleaning Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000011068 loading method Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/084—Configuration by using pre-existing information, e.g. using templates or copying from other elements
- H04L41/0846—Configuration by using pre-existing information, e.g. using templates or copying from other elements based on copy from other elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The disclosure provides a data downloading scheduling method, a data downloading scheduling device and electronic equipment, relates to the technical field of data processing, and particularly relates to the technical field of content distribution networks. The specific implementation scheme is as follows: displaying a task control interface in a webpage, wherein the task control interface is used for controlling a first downloading task of target log data, the target log data comprises log data stored in M distributed nodes of a content distribution network, and M is a positive integer; and transmitting the first configuration information of the first downloading task to K second electronic devices through a webpage interface, wherein the K second electronic devices are used for downloading log data corresponding to the first downloading task from the M distributed nodes based on the first configuration information.
Description
Technical Field
The disclosure relates to the technical field of data processing, in particular to the technical field of content distribution networks, and specifically relates to a data downloading scheduling method, a data downloading scheduling device and electronic equipment.
Background
The content delivery network (Content Delivery Network, CDN) is an intelligent virtual network constructed on the basis of the existing network, and by means of the edge servers deployed in various places, a user can obtain required content nearby through load balancing, content delivery, scheduling and other functional modules of the center platform, network congestion is reduced, and response speed and hit rate of the user when accessing the content are improved.
To implement CDN services, a large number of distributed nodes are usually required to be built at edges of each place, and the distributed nodes may generate a large amount of log data during the service process, so that to implement analysis of the log data, the log data needs to be downloaded from the distributed nodes.
At present, a downloading scheme of log data usually uses a script to process a required data path in advance, and then downloads the log data by a single node.
Disclosure of Invention
The disclosure provides a data downloading scheduling method, a data downloading scheduling device and electronic equipment.
According to a first aspect of the present disclosure, there is provided a data download scheduling method, which is applied to a first electronic device, including:
displaying a task control interface in a webpage, wherein the task control interface is used for controlling a first downloading task of target log data, the target log data comprises log data stored in M distributed nodes of a content distribution network, and M is a positive integer;
and transmitting the first configuration information of the first downloading task to K second electronic devices through a webpage interface, wherein the K second electronic devices are used for downloading log data corresponding to the first downloading task from the M distributed nodes based on the first configuration information, and K is a positive integer.
According to a second aspect of the present disclosure, there is provided a data download scheduling apparatus, the apparatus being applied to a first electronic device, comprising:
the system comprises a display module, a task control interface and a content distribution network, wherein the display module is used for displaying a task control interface in a webpage, the task control interface is used for controlling a first downloading task of target log data, the target log data comprises log data stored in M distributed nodes of the content distribution network, and M is a positive integer;
the first sending module is used for sending the first configuration information of the first downloading task to K second electronic devices through a webpage interface, the K second electronic devices are used for downloading log data corresponding to the first downloading task from the M distributed nodes based on the first configuration information, and K is a positive integer.
According to a third aspect of the present disclosure, there is provided an electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any one of the methods of the first aspect.
According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any of the methods of the first aspect.
According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements any of the methods of the first aspect.
According to the technology disclosed by the invention, the problem that the log data downloading effect generated in the CDN service process is relatively poor is solved, and the log data downloading effect generated in the CDN service process is improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
fig. 1 is a flow chart of a data download scheduling method according to a first embodiment of the present disclosure;
FIG. 2 is an interface schematic diagram of a new task;
FIG. 3 is a schematic diagram of a task control interface;
FIG. 4 is a schematic diagram of sub-task splitting and scheduling;
FIG. 5 is a schematic diagram of a distributed download of a second electronic device;
FIG. 6 is a schematic diagram of a data transfer flow
FIG. 7 is a schematic diagram of a distributed download scheduling system for log data for CDN services;
FIG. 8 is a schematic diagram of a distributed download scheduling flow of log data for CDN services;
fig. 9 is a schematic structural view of a data download scheduling apparatus according to a second embodiment of the present disclosure;
fig. 10 is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
First embodiment
As shown in fig. 1, the present disclosure provides a data download scheduling method, which is applied to a first electronic device, and includes the following steps:
Step S101: and displaying a task control interface in the webpage, wherein the task control interface is used for controlling a first downloading task of target log data, the target log data comprises log data stored in M distributed nodes of a content distribution network, and M is a positive integer.
In this embodiment, the data downloading scheduling method relates to the technical field of data processing, in particular to the technical field of content distribution network, and can be widely applied to CDN scenarios.
In the CDN scenario, a large amount of log data may be generated in the CDN service process by the distributed nodes built in each place, where the log data is the basis for data analysis in the CDN scenario. For example, a user of the CDN may download various log data generated after the relevant service accesses the CDN, and an operation and maintenance person and a data analysis person may download log data of all node dimensions for analysis.
In this step, the task control interface may be displayed on the web page, and the display manner may be that after the user creates the first download task on the web page, the task control interface may also be a default display, which is not specifically limited herein.
When the task control interface is displayed by default, after the user opens the webpage, the task control interface may be displayed by default in the webpage, where the task control interface may control the first download task created when the webpage is started last time, may control the first download task of all or part of the log data of the part of the distributed nodes in the CDN by default, and may also control the first download task of all or part of the log data of all of the distributed nodes in the CDN by default, which is not specifically limited herein.
When the user creates the first download task in the web page and then displays the task control interface, the user can create a download task in the web page, as shown in fig. 2, the user can set the download range of the target log data in the interface of the newly created download task, including but not limited to the generation time of the log data, the service and the service corresponding to the log data, the type of the log data, the distributed node corresponding to the log data, the log data generation interval and time delay, and the user can set the task name of the first download task and the storage path of the target log data. Accordingly, the first electronic device may generate the first download task based on the user setting information.
Then, a task control interface may be displayed based on the first download task, where the task control interface may be a visual operation interface of the first download task, where user setting information of the first download task may be displayed, and at the same time, an operation state of the first download task, a download progress, an operation control, and the like may be displayed, where the operation control is used to be operated by a user to control the operation state of the first download task, as shown in fig. 3.
In the step, the task control interface is displayed in the webpage, so that the receiving and the storage of the control target log data which can be configured through the webpage interface can be realized, and the task condition can be accurately checked in the task control interface.
Step S102: and transmitting the first configuration information of the first downloading task to K second electronic devices through a webpage interface, wherein the K second electronic devices are used for downloading log data corresponding to the first downloading task from the M distributed nodes based on the first configuration information, and K is a positive integer.
In this step, the web interface is a web interface, which may be defined as a remote maintenance web interface, and the remote maintenance function is a very important function of the web page, and may perform a management maintenance operation of a specific service item of the first electronic device through the web interface.
When the first downloading task is configured, the first electronic device and the second electronic device can be connected through the web interface, so that the first electronic device can send the first configuration information of the first downloading task to the second electronic device through the web interface.
The first configuration information may define a downloading situation of the first downloading task, including but not limited to a task downloading state, a task downloading trigger time, a task downloading link, a downloading manner of the K second electronic devices to the task, and the like. The downloading manner of the K second electronic devices to the task may include at least one of the following:
Different second electronic devices download different log data in parallel;
different downloading devices in the same second electronic equipment download different log data in parallel;
each downloader downloads different log data serially.
On the basis of connecting the first electronic device and the second electronic devices through the web interface, when the task control interface is displayed, the first configuration information of the first download task may be issued to the K second electronic devices through the web interface, or when the control state (i.e., the running state) of the first download task is the download start state, the first configuration information of the first download task may be issued to the K second electronic devices through the web interface, which is not particularly limited herein.
The running state of the first download task may have various situations, including a task deleted state, a task initialized state, a task suspended state and a task restored state, and the running state may be controlled based on an operation control in the task control interface, or may be controlled by timing or periodic triggering.
The download starting state refers to a task initialization state, and when the first electronic device monitors that the running state of the first download task is the download starting state, the first electronic device can automatically configure the first download task to obtain first configuration information and send the first configuration information to the K electronic devices, or send preset first configuration information to the K electronic devices, or obtain the first configuration information based on modification operation of a user on basic configuration and send the first configuration information to the K electronic devices.
The first electronic device may issue the first configuration information of the first download task to each second electronic device, or split the first download task according to one or more of a service, a generation time, a type and a distributed node corresponding to the log data, and correspondingly split the first configuration information to obtain N second configuration information of N second download tasks, and issue the N second configuration information to each second electronic device, or issue the N second configuration information to K second electronic devices, so that each second electronic device may be allocated to obtain a part of the second download task.
For example, in the distributed nodes 1 and 2, the generation time is the last month, and log data of the access type is downloaded, so that the first download task can be split according to the distributed node and the generation time corresponding to the log data. The method specifically includes splitting a downloading task of log data in a distributed node 1 into one major class, splitting the downloading task of log data in a distributed node 2 into another major class, and splitting the downloading task again according to the sequence of the generation time of the log data for each major class and taking one day as a unit to finally obtain N second downloading tasks.
Correspondingly, splitting the first configuration information to obtain N pieces of second configuration information of N pieces of second downloading tasks, and then, issuing the N pieces of second configuration information to K pieces of second electronic equipment.
The number of the second electronic devices may be greater than the number of the distributed nodes, may be less than the number of the distributed nodes, or may be equal to the number of the distributed nodes.
Under the condition that the number of the second electronic devices is equal to the number of the distributed nodes, the first downloading tasks can be split according to the distributed nodes, and second configuration information of the second downloading tasks corresponding to each distributed node is issued to the second electronic device, so that each second electronic device downloads log data corresponding to the second downloading tasks from one distributed node based on the received second configuration information of the second downloading tasks, namely, one second electronic device is responsible for downloading the log data from the corresponding distributed node.
For example, the first downloading task is to download log data in the distributed node 1 and the distributed node 2, and the two second electronic devices may be used to execute the first downloading task to download the log data, which are the second electronic device 1 and the second electronic device 2 respectively. The downloading task of the log data in the distributed node 1 can be split into a large class, and the second configuration information of the second downloading task in the large class is issued to a second electronic device, such as the second electronic device 1; splitting the task of downloading the log data in the distributed node 2 into another major class, and transmitting the second configuration information of the second downloading task in the major class to another second electronic device, such as the second electronic device 2.
Further, each second electronic device receives configuration information sent by the first electronic device through the webpage interface, and downloads log data corresponding to the first downloading task from the corresponding distributed node based on the received configuration information. In an alternative embodiment, the first downloading task may be split according to a single distributed node, and the K second electronic devices may download log data from the M distributed nodes in parallel based on second configuration information of the split second downloading task.
The second electronic device can download log data through the downloader, and the second electronic device can comprise one or more downloaders, and the second electronic device can download log data of the distributed node in parallel by adopting the plurality of downloaders at the same time based on the downloading mode of the second electronic device to the task in the configuration information, for example, log data of different types in the same distributed node can be downloaded in parallel, so that the downloading efficiency can be improved. For each downloader, the downloader can be adopted to download the log data of the distributed nodes in series at different moments, for example, the log data of different times can be downloaded in series according to the sequence of the log data generation time, and thus the concurrency and controllability of the log data downloading of the distributed nodes can be improved.
In this embodiment, a task control interface is displayed in a web page, where the task control interface is configured to control a first download task of target log data, where the target log data includes log data stored in M distributed nodes of a content distribution network; and the first configuration information of the first downloading task is sent to K second electronic devices through a webpage interface, wherein the K second electronic devices are used for downloading log data corresponding to the first downloading task from the M distributed nodes based on the first configuration information. Therefore, K second electronic devices can be scheduled to download log data from M distributed nodes through the webpage interface, and visual control of log data downloading tasks is achieved through the task control interface, so that a user perceives the downloading process and state, and the downloading effect of log data generated in the CDN service process can be improved.
Optionally, before the step S102, the method further includes:
splitting the first downloading task based on the distribution condition of the target log data under the condition that the control state of the first downloading task in the task control interface is a downloading starting state, so as to obtain N second downloading tasks, wherein N is an integer larger than 1;
Splitting the first configuration information to obtain N pieces of second configuration information of the N pieces of second downloading tasks;
the step S102 specifically includes:
and transmitting the N pieces of second configuration information to the K pieces of second electronic equipment through the webpage interface.
In this embodiment, since the log data generated in the distributed node is very much, in order to improve the downloading efficiency, when it is monitored that the control state of the first downloading task in the task control interface is a downloading start state, the first downloading task may be split to obtain a plurality of second downloading tasks.
The first downloading task can be split according to one or more of the service, the generation time, the type and the distributed node corresponding to the log data, and in the splitting process, the principle is that no intersection exists between log data corresponding to the second downloading task obtained through splitting.
For example, the first downloading task is split to obtain 3 second downloading tasks, which are respectively a second downloading task 1, a second downloading task 2 and a second downloading task 3, the log data required to be downloaded for the second downloading task 1 is the log data with the generation time of 9:00-12:00 in the morning in the distributed node 1, the log data required to be downloaded for the second downloading task 2 is the log data with the generation time of 12:00-15:00 in the distributed node 1, and the log data required to be downloaded for the second downloading task 3 is the log data with the generation time of 9:00-12:00 in the morning in the distributed node 2. Therefore, each second downloading task is independent of the execution of other second downloading tasks, does not affect each other, and acquires resources equally, so that the situation of repeatedly downloading log data can be avoided.
In an alternative embodiment, the first downloading task may be split according to a single distributed node to obtain a downloading task corresponding to the single distributed node, and then split again according to one or more of a service, a generation time and a type corresponding to the log data, so as to obtain N second downloading tasks.
After the N second download tasks are split, second configuration information of each second download task needs to be acquired, so that the second electronic device can download log data corresponding to the second download tasks from the distributed node based on the second configuration information. Specifically, the first configuration information may be split, where the splitting manner is similar to that of the first downloading task, and details are not described herein, so as to obtain N pieces of second configuration information of N pieces of second downloading tasks.
Correspondingly, the first configuration information of the first downloading task is issued to the K second electronic devices through the web interface, which may mean that the N second configuration information is issued to the K second electronic devices through the web interface. In this way, the K second electronic devices can download log data corresponding to the N second download tasks in parallel from the M distributed nodes based on the N second configuration information, so that the downloading efficiency can be improved, and the downloading time period can be reduced.
In an example, a schematic diagram of sub-task splitting and scheduling is shown in fig. 4, where the first downloading task may be started when the running state of the first downloading task is monitored to be the task initialization state, and in the process of the first downloading task, the first downloading task may be split to obtain a plurality of sub-tasks, and configuration information of the plurality of sub-tasks is distributed to distribute and schedule the sub-tasks.
Under the condition of the initial state of the subtask, the subtask can be triggered to be carried out at fixed time through the configuration information, so as to download the log data corresponding to the subtask. When the single subtask is completed, the second electronic device can circularly execute the next subtask based on the configuration information until all the subtasks are completed, and at the moment, the first downloading task is completed; or, the first electronic device may monitor the completion of the subtasks by configuring the cache control, and when it is monitored that one subtask is completed, may issue another cached configuration information to the second electronic device again, where the configuration information may be used to schedule the second electronic device to circularly execute the next subtask, that is, in this example, the first electronic device may send the configuration information to the second electronic device multiple times, so that the scheduling of the subtasks by the first electronic device is controllable.
In addition, the user can trigger an operation control in the task control interface to control the running state of the first downloading task, for example, in the process of the first downloading task, the user can trigger a pause operation control to control the first downloading task to pause downloading, for example, in the process of the task pause, the user can trigger a resume operation control to control the first downloading task to resume downloading.
Optionally, in the case where K is greater than 1, the issuing, by the web interface, the N pieces of second configuration information to the K pieces of second electronic devices includes:
distributing the N pieces of second configuration information to the K pieces of second electronic equipment through the webpage interface;
and K is equal to M, the K second electronic devices are used for downloading log data from the M distributed nodes in parallel, and each second electronic device is used for downloading the log data corresponding to the second downloading task from one distributed node based on the received second configuration information of the second downloading task.
In this embodiment, when K is greater than 1, the N pieces of second configuration information may be distributed to the K pieces of second electronic devices through the web interface, so that the K pieces of second electronic devices may respectively download log data corresponding to the second download task in parallel from the M distributed nodes based on the received second configuration information.
And the first downloading task is split according to a single distributed node, and all second downloading tasks corresponding to one distributed node are distributed to one second electronic device according to the distributed node corresponding to the second downloading task, so that the distributed nodes and the second electronic device are in one-to-one correspondence.
The distributed downloading schematic diagram of the second electronic device is shown in fig. 5, and the distributed downloading schematic diagram of the second electronic device comprises two second electronic devices, each second electronic device comprises n downloaders, the two second electronic devices respectively download log data from different distributed nodes in parallel, acquire resources equally, and the n downloaders in each second electronic device download log data from the same distributed node in parallel, and acquire resources equally.
For one downloader, the downloader 1 may be used to serially download log data of the distributed node, for example, the log data may be serially downloaded in a queuing manner according to the sequence of the log data generation time by using a plurality of second download tasks corresponding to the downloader 1. The downloading process comprises the following steps: and acquiring a second downloading task currently arranged at the head of the team, acquiring a downloading link from second configuration information of the second downloading task, and downloading log data from a position corresponding to the distributed node according to the downloading link.
The set of the second downloading tasks currently being executed by the K second electronic devices can be defined as an active task pool, the downloading states of the second downloading tasks in the active task pool are downloading active states, and each second electronic device can feed back the number of the second downloading tasks currently being executed and the number of the second downloading tasks currently being downloaded in parallel to the first electronic device, so that the first electronic device can monitor the downloading condition, and can also perform flow control based on the information fed back by the second electronic device, so that the flow cost is reduced.
In this embodiment, the N second configuration information is distributed to the K second electronic devices through the web interface, so that the K second electronic devices may download log data in parallel from the M distributed nodes, and each second electronic device is configured to download log data from one distributed node based on the received second configuration information of the second download task, and the single downloader in the second electronic device is connected in series, which may further improve the downloading efficiency and reduce the downloading time period for the concurrency of each distributed node.
Optionally, after the sending the N pieces of second configuration information to the K pieces of second electronic devices through the web interface, the method further includes:
The method comprises the steps of receiving active task information fed back by the K second electronic devices, wherein the active task information is used for representing the number of target downloading tasks corresponding to the K second electronic devices, and the target downloading tasks are second downloading tasks with downloading states in downloading active states in the N second downloading tasks;
and sending state control information to the K second electronic devices based on the active task information, wherein the state control information is used for adjusting the downloading states of the N second downloading tasks.
In this embodiment, the download states of the second download task may include a download active state, a download waiting state, and a download completion state, where the download waiting state refers to that the second download task has not yet been executed, is being queued, and the download completion state refers to that the second download task has been executed and completed.
The first electronic device may receive the active task information fed back by each second electronic device, and count active task information fed back by the K second electronic devices, where the active task information may represent the number of target download tasks corresponding to the K second electronic devices, where the target download task is a second download task whose download state is in a download active state in the N second download tasks, that is, a second download task being executed by the second electronic device.
The active task information may include only one digital symbol, i.e. the number of target download tasks, e.g. 1000, that characterizes the feedback of the second electronic device, and may also include other information, not specifically limited herein, of the second download task whose download status is in download active status.
The first electronic device may perform flow control based on the active task information, and specifically, may send status control information to the K second electronic devices, where the status control information is used to adjust download statuses of the N second download tasks.
In an alternative embodiment, the state control information is used to increase the number of second download tasks in the active task pool, for example, when the first electronic device monitors, based on the active task information, that the number of second download tasks in the active task pool is 500, and the traffic is relatively cheaper according to the current time, such as at night, in which case the number of second download tasks in the active task pool may be increased, such as up to 1000, to more fully utilize the traffic.
Specifically, the state control information may update the download state of the second download task from the download waiting state to the download active state, so as to schedule the second download task to perform distributed download with other second download tasks in the download active state.
In another alternative embodiment, the state control information is used to reduce the number of second download tasks in the active task pool, for example, when the first electronic device monitors that the number of second download tasks in the active task pool is 1000 based on the active task information, and according to the current time, such as the morning, the flow rate is relatively expensive, in which case, the number of second download tasks in the active task pool may be reduced, such as to 500, so as to avoid consuming excessive flow rate at the current time.
Specifically, the state control information may update the download state of the second download task from the download active state to the download waiting state, so as to schedule the second download task to exit from executing.
That is, the first electronic device may schedule the second electronic device to perform tasks as much as possible at a time when the traffic is relatively inexpensive to fully utilize the current traffic, and schedule the second electronic device to perform tasks as little as possible at a time when the traffic is relatively expensive to avoid consuming too much traffic at the current time, so that the cost of downloading the log data may be reduced.
Optionally, after the sending the N pieces of second configuration information to the K pieces of second electronic devices through the web interface, the method further includes:
Acquiring a file name of data to be stored from a download catalog of the first download task, wherein the file name comprises a task name of a second download task corresponding to the data to be stored;
acquiring target task information of a second download task corresponding to the task name from task information corresponding to the download catalog, wherein the target task information comprises position information stored in log data corresponding to the second download task;
and controlling the log data stored in the position information to be restored.
In this embodiment, the first electronic device may schedule the second electronic device to store the downloaded log data to a corresponding location, and may schedule the log data stored offline to be restored.
Specifically, after the first electronic device splits the first download task, a download catalog may be created according to the split second download task. For example, a primary folder may be created for a first download task, where the folder is used to store N second download tasks obtained by splitting, then M secondary folders are created in the folder, and for each secondary folder, it is used to store task information of all second download tasks corresponding to a distributed node, then in each secondary folder, a tertiary folder, even a quaternary folder, may be created according to the service, type, or generation time of log data, which will not be specifically described herein.
In the download directory, task information of each second download task may be stored, for example, the task information may include location information stored in log data corresponding to the second download task, where the location information may be a specific location in a file storage cluster, where the file storage cluster may include a distributed file system (Andrew File System, AFS) cluster or a smart cloud object storage cluster, and so on. The folders in the download directory may be named by information such as task names, service names of log data, and time stamps.
The download catalog of the first download task is created, and after the task information of each second download task is stored in the download catalog, the download catalog and the task information stored therein may be stored in a database, or stored locally, which is not particularly limited herein.
Accordingly, after the first download task is completed, the log data stored in the file storage cluster is not directly usable due to the large amount of data, and the format problem, so that the required log data can be restored for data analysis.
As shown in fig. 6, when log data is transferred, a file name of the data to be stored may be obtained from a download directory of the first download task, where the file name may include a task name of the second download task corresponding to the data to be stored, a service name of the log data, a timestamp, and the like.
Task information corresponding to the download catalog can be obtained from the database, target task information of a second download task corresponding to the task name can be obtained from the task information, and the task information stored in the second download task can be obtained from a folder corresponding to the second download task.
A dump path may then be created and the log data stored by the location information is transcoded and then dumped to other data modules, such as a cloud data warehouse (UCloud Data Warehouse, UDW), via a web interface scheduling file storage cluster. Meanwhile, the distributed file system can be scheduled to delete the log data stored by the position information under the condition that the transfer is successful.
In this embodiment, the log data stored in the file storage cluster is transferred to other data modules, so that the log data can be directly used, and thus the convenience of log data analysis can be improved.
In order to illustrate aspects of embodiments of the present disclosure in more detail, this is illustrated in fig. 7 and 8.
A schematic diagram of a distributed download scheduling system of log data of a CDN service is shown in fig. 7, where the distributed download scheduling system may include a first electronic device, a second electronic device, a distributed node, a file storage cluster, and a cloud data repository.
The first electronic device may perform task control through the web interface, where the task control involves multiple processes, namely, main task start control and task loop execution control, where the main task start control involves main task state inspection modification and main task decomposition, and may decompose the main task when it is detected that the running state of the main task, that is, the first download task, is a task initialization state.
After the main task is decomposed into a plurality of subtasks, the cyclic execution of the subtasks can be controlled, and the method can specifically relate to the state inspection of the main task, the inspection of the active task, the acquisition of the situation of the subtasks, the acquisition of the concurrency situation and the issuing of the subtasks, and can decompose the main task into the subtasks and issue the subtasks under the condition that the running state of the main task is the task initialization state, and can issue the subtasks under the condition that the situation of the subtasks is acquired (such as the active situation of the subtasks) and the concurrency situation (such as the concurrency situation of a downloader of the second electronic equipment).
In addition, the first electronic device may further perform concurrency control (e.g., control the number of subtasks currently executed by the second electronic device) on the second electronic device, configure cache control (e.g., send configuration information of the cache periodically to trigger cyclic execution of the subtasks), visualization of the downloading progress of the log data (e.g., view the downloading progress of the current subtask), cleaning of the history data (e.g., clean backup data in the file storage cluster), and monitoring of the downloading catalog.
For each second electronic device, the second electronic device comprises n downloaders, and each downloader is used for acquiring subtasks, modifying the download state of the subtasks, and serially downloading log data from the distributed nodes according to the links to obtain the file of the log data.
And then, file dumping is carried out, and bad file identification and cleaning and uploading of good file clustering to a distributed file system cluster or an intelligent cloud object storage cluster can be involved in the dumping process. And (3) transferring the files in the distributed file system cluster or the intelligent cloud object storage cluster to a cloud data warehouse after extraction, conversion and loading processes.
A distributed downloading scheduling flow diagram of log data of CDN service is shown in FIG. 8, a first electronic device newly adds a main task in a web interface, stores related information of the newly added main task in a database, displays a task control interface in the web interface for task viewing, and a user can perform task pause or deletion operation in the task control interface and correspondingly feed back to the database through the web interface to modify the running state of the main task.
The user can also perform starting operation in the task control interface, correspondingly, the first electronic device can split the main task into sub-tasks, distribute the sub-tasks and distribute configuration information of the sub-tasks to perform sub-task scheduling, the user can also perform configuration modification in the web page, distribute the cached basic configuration information after modification, and meanwhile can periodically update the basic configuration information through auxiliary configuration analysis.
After the subtasks are distributed, the subtasks corresponding to each distributed node can be sent to a database, and an active task pool can be generated or adjusted regularly according to time limit or other limit in the process of executing the subtasks by the second electronic device.
The plurality of second electronic devices can execute the subtasks in the active task pool in parallel, store the downloaded log data into the file storage cluster, transcode the log data and then transfer the log data into the cloud data warehouse.
Second embodiment
As shown in fig. 9, the present disclosure provides a data download scheduling apparatus 900, which is applied to a first electronic device, including:
the display module 901 is configured to display a task control interface in a web page, where the task control interface is configured to control a first download task of target log data, where the target log data includes log data stored in M distributed nodes of a content distribution network, and M is a positive integer;
the first sending module 902 is configured to send the first configuration information of the first downloading task to K second electronic devices through a web interface, where K is a positive integer, and the K second electronic devices are configured to download log data corresponding to the first downloading task from the M distributed nodes based on the first configuration information.
Optionally, the method further comprises:
the first splitting module is used for splitting the first downloading task based on the distribution condition of the target log data under the condition that the control state of the first downloading task in the task control interface is monitored to be a downloading starting state, so that N second downloading tasks are obtained, wherein N is an integer larger than 1;
the second splitting module is used for splitting the first configuration information to obtain N pieces of second configuration information of the N pieces of second downloading tasks;
the first sending module 902 is specifically configured to send the N pieces of second configuration information to the K pieces of second electronic devices through the web interface.
Optionally, in the case where K is greater than 1, the first sending module 902 is specifically configured to:
distributing the N pieces of second configuration information to the K pieces of second electronic equipment through the webpage interface;
and K is equal to M, the K second electronic devices are used for downloading log data from the M distributed nodes in parallel, and each second electronic device is used for downloading the log data corresponding to the second downloading task from one distributed node based on the received second configuration information of the second downloading task.
Optionally, the method further comprises:
the receiving module is used for receiving active task information fed back by the K second electronic devices, wherein the active task information is used for representing the number of target downloading tasks corresponding to the K second electronic devices, and the target downloading tasks are second downloading tasks with downloading states in downloading active states in the N second downloading tasks;
and the second sending module is used for sending state control information to the K second electronic devices based on the active task information, wherein the state control information is used for adjusting the downloading states of the N second downloading tasks.
Optionally, the method further comprises:
the first acquisition module is used for acquiring the file name of the data to be stored from the download catalog of the first download task, wherein the file name comprises the task name of the second download task corresponding to the data to be stored;
the second acquisition module is used for acquiring target task information of a second download task corresponding to the task name from the task information corresponding to the download catalog, wherein the target task information comprises position information stored in log data corresponding to the second download task;
and the control transfer module is used for controlling the log data stored by the position information to transfer.
The data download scheduling apparatus 900 provided in the present disclosure can implement each process implemented by the data download scheduling method embodiment, and can achieve the same beneficial effects, so that repetition is avoided, and no further description is given here.
In the technical scheme of the disclosure, the acquisition, storage, application and the like of the related user personal information all conform to the regulations of related laws and regulations, and the public sequence is not violated.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
Fig. 10 shows a schematic block diagram of an example electronic device 1000 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 10, the apparatus 1000 includes a computing unit 1001 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a Random Access Memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other by a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.
Various components in device 1000 are connected to I/O interface 1005, including: an input unit 1006 such as a keyboard, a mouse, and the like; an output unit 1007 such as various types of displays, speakers, and the like; a storage unit 1008 such as a magnetic disk, an optical disk, or the like; and communication unit 1009 such as a network card, modem, wireless communication transceiver, etc. Communication unit 1009 allows device 1000 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks.
The computing unit 1001 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 1001 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the respective methods and processes described above, such as a data download scheduling method. For example, in some embodiments, the data download scheduling method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 1000 via ROM 1002 and/or communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the data download scheduling method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the data download scheduling method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (10)
1. A data download scheduling method, the method being applied to a first electronic device, comprising:
displaying a task control interface in a webpage, wherein the task control interface is used for controlling a first downloading task of target log data, the target log data comprises log data stored in M distributed nodes of a content distribution network, and M is an integer greater than 1;
the first configuration information of the first downloading task is issued to K second electronic devices through a webpage interface, the K second electronic devices are used for downloading log data corresponding to the first downloading task from the M distributed nodes based on the first configuration information, and K is a positive integer;
Before the first configuration information of the first downloading task is sent to the K second electronic devices through the web interface, the method further includes:
splitting the first downloading task based on the distribution condition of the target log data under the condition that the control state of the first downloading task in the task control interface is a downloading starting state, so as to obtain N second downloading tasks, wherein N is an integer larger than 1; the distribution condition of the target log data comprises the condition of a distributed node generating the target log data;
splitting the first configuration information to obtain N pieces of second configuration information of the N pieces of second downloading tasks;
the step of issuing the first configuration information of the first downloading task to the K second electronic devices through the web interface includes:
and transmitting the N pieces of second configuration information to the K pieces of second electronic equipment through the webpage interface.
2. The method of claim 1, wherein in a case where K is greater than 1, the issuing, by the web interface, the N second configuration information to the K second electronic devices includes:
distributing the N pieces of second configuration information to the K pieces of second electronic equipment through the webpage interface;
And K is equal to M, the K second electronic devices are used for downloading log data from the M distributed nodes in parallel, and each second electronic device is used for downloading the log data corresponding to the second downloading task from one distributed node based on the received second configuration information of the second downloading task.
3. The method of claim 1, further comprising, after the issuing, by the web interface, the N second configuration information to the K second electronic devices:
the method comprises the steps of receiving active task information fed back by the K second electronic devices, wherein the active task information is used for representing the number of target downloading tasks corresponding to the K second electronic devices, and the target downloading tasks are second downloading tasks with downloading states in downloading active states in the N second downloading tasks;
and sending state control information to the K second electronic devices based on the active task information, wherein the state control information is used for adjusting the downloading states of the N second downloading tasks.
4. The method of claim 1, further comprising, after the issuing, by the web interface, the N second configuration information to the K second electronic devices:
Acquiring a file name of data to be stored from a download catalog of the first download task, wherein the file name comprises a task name of a second download task corresponding to the data to be stored;
acquiring target task information of a second download task corresponding to the task name from task information corresponding to the download catalog, wherein the target task information comprises position information stored in log data corresponding to the second download task;
and controlling the log data stored in the position information to be restored.
5. A data download scheduling apparatus, the apparatus being applied to a first electronic device, comprising:
the system comprises a display module, a task control interface and a control module, wherein the display module is used for displaying a task control interface in a webpage, the task control interface is used for controlling a first downloading task of target log data, the target log data comprises log data stored in M distributed nodes of a content distribution network, and M is an integer larger than 1;
the first sending module is used for sending the first configuration information of the first downloading task to K second electronic devices through a webpage interface, the K second electronic devices are used for downloading log data corresponding to the first downloading task from the M distributed nodes based on the first configuration information, and K is a positive integer;
The apparatus further comprises:
the first splitting module is used for splitting the first downloading task based on the distribution condition of the target log data under the condition that the control state of the first downloading task in the task control interface is monitored to be a downloading starting state, so that N second downloading tasks are obtained, wherein N is an integer larger than 1; the distribution condition of the target log data comprises the condition of a distributed node generating the target log data;
the second splitting module is used for splitting the first configuration information to obtain N pieces of second configuration information of the N pieces of second downloading tasks;
the first sending module is specifically configured to send the N pieces of second configuration information to the K pieces of second electronic devices through the web interface.
6. The apparatus of claim 5, wherein in the case where K is greater than 1, the first transmitting module is specifically configured to:
distributing the N pieces of second configuration information to the K pieces of second electronic equipment through the webpage interface;
and K is equal to M, the K second electronic devices are used for downloading log data from the M distributed nodes in parallel, and each second electronic device is used for downloading the log data corresponding to the second downloading task from one distributed node based on the received second configuration information of the second downloading task.
7. The apparatus of claim 5, further comprising:
the receiving module is used for receiving active task information fed back by the K second electronic devices, wherein the active task information is used for representing the number of target downloading tasks corresponding to the K second electronic devices, and the target downloading tasks are second downloading tasks with downloading states in downloading active states in the N second downloading tasks;
and the second sending module is used for sending state control information to the K second electronic devices based on the active task information, wherein the state control information is used for adjusting the downloading states of the N second downloading tasks.
8. The apparatus of claim 5, further comprising:
the first acquisition module is used for acquiring the file name of the data to be stored from the download catalog of the first download task, wherein the file name comprises the task name of the second download task corresponding to the data to be stored;
the second acquisition module is used for acquiring target task information of a second download task corresponding to the task name from the task information corresponding to the download catalog, wherein the target task information comprises position information stored in log data corresponding to the second download task;
And the control transfer module is used for controlling the log data stored by the position information to transfer.
9. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.
10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110862294.9A CN113556402B (en) | 2021-07-29 | 2021-07-29 | Data downloading scheduling method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110862294.9A CN113556402B (en) | 2021-07-29 | 2021-07-29 | Data downloading scheduling method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113556402A CN113556402A (en) | 2021-10-26 |
| CN113556402B true CN113556402B (en) | 2024-03-19 |
Family
ID=78133207
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110862294.9A Active CN113556402B (en) | 2021-07-29 | 2021-07-29 | Data downloading scheduling method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113556402B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101958837A (en) * | 2010-09-30 | 2011-01-26 | 北京世纪互联工程技术服务有限公司 | Log processing system, log processing method, node server and center server |
| CN102567097A (en) * | 2011-12-29 | 2012-07-11 | Tcl王牌电器(惠州)有限公司 | Method and terminal for downloading multiple tasks |
| CN103595782A (en) * | 2013-11-11 | 2014-02-19 | 中安消技术有限公司 | Distributed storage system and method for downloading files thereof |
| CN104462400A (en) * | 2014-12-10 | 2015-03-25 | 北京奇虎科技有限公司 | Method, device and browser client for downloading files in mobile terminals |
| CN105634845A (en) * | 2014-10-30 | 2016-06-01 | 任子行网络技术股份有限公司 | Method and system for carrying out multi-dimensional statistic analysis on large number of DNS journals |
| WO2017050141A1 (en) * | 2015-09-24 | 2017-03-30 | 网宿科技股份有限公司 | Distributed storage-based file delivery system and method |
| CN107332719A (en) * | 2017-08-16 | 2017-11-07 | 北京云端智度科技有限公司 | A kind of method that daily record is analyzed in real time in CDN system |
| CN109327351A (en) * | 2018-09-12 | 2019-02-12 | 拉扎斯网络科技（上海）有限公司 | Real-time collecting method, device, electronic equipment and the storage medium of daily record data |
| CN111953551A (en) * | 2020-08-27 | 2020-11-17 | 网易（杭州）网络有限公司 | Log data transmission method and device, electronic equipment and storage medium |
| CN112685270A (en) * | 2020-12-24 | 2021-04-20 | 恒安嘉新（北京）科技股份公司 | System monitoring log acquisition method and device, electronic equipment and medium |
| WO2021077263A1 (en) * | 2019-10-21 | 2021-04-29 | 深圳市欢太科技有限公司 | Power consumption log processing method and apparatus, electronic device, and storage medium |
| CN112800017A (en) * | 2021-01-21 | 2021-05-14 | 网易（杭州）网络有限公司 | Distributed log collection method, device, medium and electronic equipment |
-
2021
- 2021-07-29 CN CN202110862294.9A patent/CN113556402B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101958837A (en) * | 2010-09-30 | 2011-01-26 | 北京世纪互联工程技术服务有限公司 | Log processing system, log processing method, node server and center server |
| CN102567097A (en) * | 2011-12-29 | 2012-07-11 | Tcl王牌电器(惠州)有限公司 | Method and terminal for downloading multiple tasks |
| CN103595782A (en) * | 2013-11-11 | 2014-02-19 | 中安消技术有限公司 | Distributed storage system and method for downloading files thereof |
| CN105634845A (en) * | 2014-10-30 | 2016-06-01 | 任子行网络技术股份有限公司 | Method and system for carrying out multi-dimensional statistic analysis on large number of DNS journals |
| CN104462400A (en) * | 2014-12-10 | 2015-03-25 | 北京奇虎科技有限公司 | Method, device and browser client for downloading files in mobile terminals |
| WO2017050141A1 (en) * | 2015-09-24 | 2017-03-30 | 网宿科技股份有限公司 | Distributed storage-based file delivery system and method |
| CN107332719A (en) * | 2017-08-16 | 2017-11-07 | 北京云端智度科技有限公司 | A kind of method that daily record is analyzed in real time in CDN system |
| CN109327351A (en) * | 2018-09-12 | 2019-02-12 | 拉扎斯网络科技（上海）有限公司 | Real-time collecting method, device, electronic equipment and the storage medium of daily record data |
| WO2021077263A1 (en) * | 2019-10-21 | 2021-04-29 | 深圳市欢太科技有限公司 | Power consumption log processing method and apparatus, electronic device, and storage medium |
| CN111953551A (en) * | 2020-08-27 | 2020-11-17 | 网易（杭州）网络有限公司 | Log data transmission method and device, electronic equipment and storage medium |
| CN112685270A (en) * | 2020-12-24 | 2021-04-20 | 恒安嘉新（北京）科技股份公司 | System monitoring log acquisition method and device, electronic equipment and medium |
| CN112800017A (en) * | 2021-01-21 | 2021-05-14 | 网易（杭州）网络有限公司 | Distributed log collection method, device, medium and electronic equipment |
Non-Patent Citations (3)
| Title |
|---|
| A. Valentine ; Hughes Network ; Systems Ltd ; .DVB Cable Network Interface Unit MIBfor EuroModem compliant Cable Modems.IETF .2000,全文. * |
| 基于OSGi的分布式系统集中日志管理方案;王宇飞;刘丹;吴嘉生;;计算机系统应用;20170615(06);全文 * |
| 大规模软件系统日志汇集服务平台设计与实现;汤网祥;王金华;赫凌俊;李敏敬;;计算机应用与软件;20181112(11);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113556402A (en) | 2021-10-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110413294B (en) | Service release system, method, device and equipment | |
| CN111506412B (en) | Airflow-based distributed asynchronous task construction and scheduling system and method | |
| US11514304B2 (en) | Continuously provisioning large-scale machine learning models | |
| CN113742031B (en) | Node state information acquisition method and device, electronic equipment and readable storage medium | |
| CN112165691B (en) | Content delivery network scheduling method, device, server and medium | |
| CN108932157B (en) | Method, system, electronic device and readable medium for distributed processing of tasks | |
| CN111290768B (en) | Updating method, device, equipment and medium of containerized application system | |
| CN103678133A (en) | Task scheduling system for application software cloud testing | |
| CN109245908B (en) | Method and device for switching master cluster and slave cluster | |
| CN110651256B (en) | System and method for preventing service interruption during software update | |
| US10409638B2 (en) | Digital processing system for event and/or time based triggering management, and control of tasks | |
| CN111510480B (en) | Request sending method and device and first server | |
| CN113377665A (en) | Container technology-based testing method and device, electronic equipment and storage medium | |
| CN112398669B (en) | Hadoop deployment method and device | |
| CN114968283A (en) | Container mirror image pulling method, device and equipment and readable storage medium | |
| CN111782229A (en) | Applet starting method and device and electronic equipment | |
| CN114490048A (en) | Task execution method and device, electronic equipment and computer storage medium | |
| CN114492834A (en) | Training method, training device, training equipment, training system and storage medium | |
| CN114064438A (en) | Database fault processing method and device | |
| CN113608765A (en) | Data processing method, device, equipment and storage medium | |
| CN113556402B (en) | Data downloading scheduling method and device and electronic equipment | |
| CN113556387A (en) | Edge gateway control method, system, device, electronic equipment and storage medium | |
| CN112817992A (en) | Method, device, electronic equipment and readable storage medium for executing change task | |
| CN112866985A (en) | Flow control method, resource downloading method, device, equipment and storage medium | |
| CN106657195B (en) | Task processing method and relay device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |