CN112685192A - Data acquisition method and device, client and computer readable storage medium - Google Patents
Data acquisition method and device, client and computer readable storage medium Download PDFInfo
- Publication number
- CN112685192A CN112685192A CN201910995284.5A CN201910995284A CN112685192A CN 112685192 A CN112685192 A CN 112685192A CN 201910995284 A CN201910995284 A CN 201910995284A CN 112685192 A CN112685192 A CN 112685192A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- pulling
- pull
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 230000005540 biological transmission Effects 0.000 claims abstract description 40
- 238000004458 analytical method Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Landscapes
- Mobile Radio Communication Systems (AREA)
Abstract
The application relates to a data acquisition method, a data acquisition device, a client and a computer-readable storage medium, wherein the method comprises the following steps: receiving a data pulling request sent by a client, wherein the data pulling request carries the type and the identification of the client; analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks; determining a task pulling strategy corresponding to the task flow according to the type of the client; based on the task pulling strategy, executing a data pulling task in the task flow to obtain pulling data; and sending the pull data to the client corresponding to the identifier. According to the method and the device, the task pulling strategy of the task flow can be flexibly configured according to the type of the client, the data pulling task is executed based on the configured task pulling strategy, the pulling data is obtained, the data pulling requirements of different clients are met, and the data transmission performance is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for acquiring data, a client, and a computer-readable storage medium.
Background
In the oil and gas group, because the staff is many, the pulling tasks in the corresponding various task flows are also many, and because the clients corresponding to the staff are distributed all over the country, because the types of the clients are many, various data need to be pulled in the task flows, and in the related technology, because the data pulling strategy configured in the data pulling process is single, the data pulling requirements of various different clients cannot be adapted.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, embodiments of the present application provide a method, an apparatus, a client, and a computer-readable storage medium for acquiring data.
In view of this, in a first aspect, an embodiment of the present application provides a data acquisition method, where the method includes the following steps:
receiving a data pulling request sent by a client, wherein the data pulling request carries the type and the identification of the client;
analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks;
determining a task pulling strategy corresponding to the task flow according to the type of the client;
based on the task pulling strategy, executing a data pulling task in the task flow to obtain pulling data;
and sending the pull data to the client corresponding to the identifier.
With reference to the first aspect, in a first possible implementation manner of the first aspect, the determining, according to the type of the client, a task pull policy corresponding to the task flow includes:
if the type of the client is a mobile terminal, determining that a task pull strategy corresponding to the task flow is a first pull strategy, wherein the first pull strategy is at least used for determining a data pull task which is actually required to be pulled in data pull tasks of the task flow; and/or the presence of a gas in the gas,
determining the data pulling task actually required to be pulled in the data pulling tasks of the task flow through the first pulling strategy, wherein the data pulling task comprises the following steps:
obtaining a historical task pulling record of the mobile terminal, wherein the historical task pulling record comprises a first historical task which is successfully pulled and a second historical task which is unsuccessfully pulled;
deleting a first task corresponding to the first historical task from the data pulling tasks of the task flow, and/or adding a second task corresponding to the second historical task to obtain an updated data pulling task;
determining the updated data pulling task as the data pulling task which needs to be actually pulled; and/or the presence of a gas in the gas,
based on the task pull strategy, executing a data pull task in the task flow to obtain pull data, including:
and executing the data pulling task which is actually required to be pulled in the task flow based on the first pulling strategy to obtain the pulled data.
With reference to the first aspect, in a second possible implementation manner of the first aspect, the determining, according to the type of the client, a task pull policy corresponding to the task flow includes:
if the type of the client is a non-mobile terminal, determining that a task pull strategy corresponding to the task flow is a second pull strategy, wherein the second pull strategy is at least used for determining a data pull task which can be executed once in the data pull tasks of the task flow; and/or the presence of a gas in the gas,
determining a data pulling task which can be executed in a single time in the data pulling tasks of the task flow through the second pulling strategy, wherein the data pulling task comprises the following steps:
acquiring a predetermined first packet threshold, wherein the first packet threshold is the maximum data volume allowed to be pulled when a first data pulling task is executed;
acquiring the data volume corresponding to each data pulling task in the task flow;
calculating an accumulated value of the data quantity of each data pulling task;
when the accumulated value does not exceed the first packet threshold value, determining all data pulling tasks in the task flow as data pulling tasks which can be executed once, so that when the first data pulling task is executed, all data pulling tasks in the task flow are executed; and/or the presence of a gas in the gas,
based on the task pull strategy, executing a data pull task in the task flow to obtain pull data, including:
and based on the second pulling strategy, performing data pulling on all the data pulling tasks in the task flow to obtain pulling data.
With reference to the first aspect, in a third possible implementation manner of the first aspect, when the accumulated value exceeds the top packet threshold, the method further includes:
acquiring the priority corresponding to each data pulling task;
and determining a first data pulling task according to the priority and the first packet threshold, wherein the data pulling tasks in the first data pulling task are selected according to the sequence of the priority from high to low, the data volume corresponding to the first data pulling task does not exceed the first packet threshold, and the sum of the data volume corresponding to the first data pulling task and the data volume corresponding to a next data pulling task adjacent to the priority exceeds the first packet threshold.
With reference to the first aspect, in a fourth possible implementation manner of the first aspect, after determining the first data pull task, the method further includes:
acquiring a predetermined back packet threshold, wherein the back packet threshold is the maximum data volume allowed to be pulled when a non-primary data pulling task is executed, and the back packet threshold is larger than the primary packet threshold;
determining at least one non-primary data pulling task according to the priority and the post-packet threshold, wherein the data pulling tasks in the non-primary data pulling tasks are selected from high to low in sequence of priority, the priority of the data pulling tasks in the non-primary data pulling tasks is lower than the priority of the data pulling tasks in the primary data pulling tasks, the data quantity corresponding to the non-primary data pulling tasks does not exceed the post-packet threshold, and the sum of the data quantity corresponding to the non-primary data pulling tasks and the data quantity corresponding to the post-data pulling tasks adjacent to the priority exceeds the post-packet threshold; and/or the presence of a gas in the gas,
the executing a data pulling task in the task flow based on the task pulling strategy to obtain pulling data includes:
and respectively executing the first data pulling task and the non-first data pulling task in the task flow according to the sequence of the priority from high to low based on the second pulling strategy.
With reference to the first aspect, in a fifth possible implementation manner of the first aspect, the first packet threshold and the second packet threshold are determined by:
determining the historical average transmission speed of a historical data packet, and the historical transmission time length of the first packet and the historical transmission time length of the later packet of the historical data packet according to the historical task pull record of the non-mobile terminal;
calculating to obtain the first packet threshold according to the historical average transmission speed and the historical transmission duration of the first packet;
and calculating to obtain the rear packet threshold according to the historical average transmission speed and the historical transmission time of the rear packet.
With reference to the first aspect, in a sixth possible implementation manner of the first aspect, the method further includes:
receiving feedback information, wherein the feedback information is sent by the client when the client sends the data pulling request next time after the data is successfully pulled for the non-last time; or, the feedback information is sent by the client in an idle time period without a data pulling task after the data is successfully pulled for the last time;
and/or the presence of a gas in the gas,
and encrypting the pull data by using an encryption algorithm.
In a second aspect, an embodiment of the present application provides an apparatus for acquiring data, where the apparatus includes:
the system comprises a receiving unit, a sending unit and a receiving unit, wherein the receiving unit is used for receiving a data pulling request sent by a client, and the data pulling request carries the type and the identification of the client;
the analysis unit is used for analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks;
a determining unit, configured to determine a task pull policy corresponding to the task flow according to the type of the client;
the execution unit is used for executing a data pulling task in the task flow based on the task pulling strategy to obtain pulling data; and
and the sending unit is used for sending the pull data to the client corresponding to the identifier.
In a third aspect, an embodiment of the present application provides a client, where the client includes: at least one processor, memory, at least one network interface, and a user interface;
the at least one processor, memory, at least one network interface, and user interface are coupled together by a bus system;
the processor is configured to execute the steps of the data acquisition method according to the first aspect by calling a program or instructions stored in the memory.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a data acquisition program is stored, and when executed by a processor, the data acquisition program implements the steps of the data acquisition method according to the first aspect.
Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages:
according to the data acquisition method, the data acquisition device, the client and the computer readable storage medium, the data acquisition method receives a data pulling request sent by the client, and the data pulling request carries the type and the identification of the client; analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks; determining a task pulling strategy corresponding to the task flow according to the type of the client; based on the task pulling strategy, executing a data pulling task in the task flow to obtain pulling data; and sending the pull data to the client corresponding to the identifier. According to the embodiment of the application, the task pulling strategy of the task flow can be flexibly configured according to the type of the client, the data pulling task is executed based on the configured task pulling strategy, the pulling data is obtained, the data pulling requirements of different clients are met, and the data transmission performance is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a data acquisition method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a data acquisition method according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a data acquisition method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a data acquisition method according to an embodiment of the present application;
fig. 5 is a schematic flowchart of a data acquisition method according to an embodiment of the present application;
fig. 6 is a schematic flowchart of a data acquisition method according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a data acquisition apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a client according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
A server implementing various embodiments of the present invention will now be described with reference to the accompanying drawings. In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in themselves. Thus, "module" and "component" may be used in a mixture.
In order to solve the problem that a data pulling strategy configured in a data pulling process in the related art is relatively single and cannot adapt to data pulling requirements of various different clients, an embodiment of the present application provides a data obtaining method, in which a task pulling strategy corresponding to a task flow is determined according to a type of a client (for example, a data pulling task that actually needs to be pulled in the data pulling task of the task flow is determined, or a data pulling task that can be executed at a single time in the data pulling task of the task flow is determined); and executing the data pulling task in the task flow based on the task pulling strategy to obtain the pulled data. According to the embodiment of the application, the task pulling strategy of the task flow can be flexibly configured according to the type of the client, the data pulling task is executed based on the configured task pulling strategy, the pulling data is obtained, the data pulling requirements of different clients are met, and the data transmission performance is improved.
It should be noted that, although the task pull policy is exemplified in the foregoing, it should be understood by those skilled in the art that the task pull policy may also be configured according to actual requirements, as long as it is ensured that the pull data obtained based on the configured task pull policy meets the data pull requirements of different clients, and the embodiment of the present application is not limited to this.
An embodiment of the present application provides a data obtaining method, as shown in fig. 1, the method may include the following steps:
s101, receiving a data pulling request sent by a client, wherein the data pulling request carries the type and the identification of the client.
Optionally, the types of the client in the embodiment of the present application include a mobile terminal and a non-mobile terminal, where the mobile terminal includes a mobile phone, a tablet computer, a notebook computer, a palm top computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a navigation device, a wearable device, and the like, and the non-mobile terminal includes a fixed terminal such as a Digital TV, a desktop computer, and the like. The identification of the client is used to distinguish between a plurality of different clients, and includes, but is not limited to, ID information of the client, and the like.
S102, analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks.
Optionally, the data pull request is used to request corresponding pull data in the task flow, and the pull data is obtained by executing one or more data pull tasks in the task flow.
S103, determining a task pulling strategy corresponding to the task flow according to the type of the client.
Optionally, when the type of the client is a mobile terminal, determining that a task pull strategy corresponding to the task flow is a first pull strategy, where the first pull strategy is at least used to determine a data pull task that actually needs to be pulled in data pull tasks of the task flow; and when the type of the client is a non-mobile terminal, determining that the task pull strategy corresponding to the task flow is a second pull strategy, wherein the second pull strategy is at least used for determining the data pull task which can be executed in a single time in the data pull tasks of the task flow. In the following, how to determine the task pull policies corresponding to the different types of clients will be described in detail, which is not described herein again.
And S104, executing the data pulling task in the task flow based on the task pulling strategy to obtain the pulling data.
And S105, sending the pull data to the client corresponding to the identifier.
According to the embodiment of the application, the task pulling strategy of the task flow can be flexibly configured according to the type of the client, the data pulling task is executed based on the configured task pulling strategy, the pulling data is obtained, the data pulling requirements of different clients are met, and the data transmission performance is improved.
It should be noted that, although fig. 1 describes a related scheme of the data acquisition method, those skilled in the art should understand that other related data may also be continuously acquired by configuring other task pull policies.
In order to facilitate understanding of the embodiments of the present application, specific examples are described below.
Optionally, referring to fig. 2, fig. 2 shows a data obtaining method, where the client is a mobile terminal, and the method includes the following steps:
s201, receiving a data pulling request sent by a mobile terminal, wherein the data pulling request carries the type and the identification of the mobile terminal.
S202, analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks.
S203, determining that the task pull strategy corresponding to the task flow is a first pull strategy, wherein the first pull strategy is at least used for determining a data pull task which is actually required to be pulled in the data pull tasks of the task flow.
Optionally, when the type of the client is a mobile terminal, in order to save traffic consumption of the mobile terminal and improve data transmission efficiency, the first pulling policy adopted in the embodiment of the present application is pulling on demand, that is, pulling a data pulling task that the mobile terminal actually needs to pull, so that a user can quickly see a task list in which the data pulling task is stored at the mobile terminal.
And S204, executing the data pulling task actually required to be pulled in the task flow based on the first pulling strategy to obtain the pulled data.
S205, sending the pull data to the mobile terminal corresponding to the identifier.
Optionally, referring to fig. 3, fig. 3 shows a data obtaining method, where in step S203, determining, by the first pull policy, a data pull task actually required to be pulled in data pull tasks of the task flow includes the following steps:
s301, historical task pulling records of the mobile terminal are obtained, wherein the historical task pulling records comprise a first historical task which is successfully pulled and a second historical task which is unsuccessfully pulled.
S302, deleting a first task corresponding to the first historical task from the data pulling tasks of the task flow, and/or adding a second task corresponding to the second historical task to obtain an updated data pulling task.
And S303, determining the updated data pulling task as the data pulling task which needs to be actually pulled.
In the embodiment of the application, after a user logs in a mobile terminal, a data pulling task stored in a task flow of the user at a server end is automatically pulled, a historical task pulling record stored at the server end or the historical task pulling record stored at the mobile terminal is compared, the aim of comparison is not to do repeated pulling, a normal pulling process is realized, the server end records the data pulling task which is successfully pulled by the user, but under an abnormal condition, for example, when the data pulling task is failed to pull, the server is informed, the current mobile terminal fails to pull, the pulling data which is failed to pull is possibly lost, under the condition, the data pulling task in the task flow of the user needs to be compared with the historical task pulling record, then, a first task corresponding to a first historical task which is successfully pulled is deleted from the data pulling task in the task flow of the user, and/or adding a second task corresponding to a second historical task which fails to be pulled in the data pulling task in the task flow of the user, so that the safe and efficient transmission of the data can be ensured, and the probability of data loss is reduced.
Optionally, the history task pull record needs to be determined according to feedback information sent by the mobile terminal, and details of sending the feedback information by the mobile terminal will be described later, which is not described herein again.
Optionally, referring to fig. 4, fig. 4 shows a data obtaining method, where the client is a non-mobile terminal, and the method includes the following steps:
s401, receiving a data pulling request sent by a non-mobile terminal, wherein the data pulling request carries the type and the identification of the non-mobile terminal.
S402, analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks.
And S403, determining that the task pull strategy corresponding to the task flow is a second pull strategy, wherein the second pull strategy is at least used for determining a data pull task which can be executed in a single time in the data pull tasks of the task flow.
Optionally, determining a data pulling task that can be executed at a single time in the data pulling tasks of the task flow includes the following two ways: according to the first scheme, the sum of the data quantity of all the data pulling tasks does not exceed the first packet threshold value, and the pulling can be finished at one time, so that only the first data pulling task needs to be determined; in the second scheme, the sum of the data amounts of all the data pulling tasks exceeds the first packet threshold, and the data pulling cannot be completed at one time, so that the first data pulling task and at least one subsequent data pulling task need to be determined. These two schemes will be described in detail later, and will not be described in detail here.
S404, based on the second pulling strategy, data pulling is carried out on all data pulling tasks in the task flow, and pulling data are obtained.
S405, sending the pull data to the non-mobile terminal corresponding to the identifier.
According to the embodiment of the application, when the type of the client is the non-mobile terminal, under the condition that the flow is not considered, in order to reduce the quantity of the request data and save the time for pulling the task data, the adopted second pulling strategy is batch pulling, namely, the data pulling task which can be executed once in the data pulling tasks of the task flow is determined, so that a user can pull the most data pulling tasks at one time at the non-mobile terminal, and the task transmission speed is ensured.
The first scheme, that is, the sum of the data amounts of all the data pulling tasks does not exceed the first packet threshold, is described below.
Optionally, referring to fig. 5, fig. 5 shows a data obtaining method, where in step S403, determining, by using the second pull policy, a data pull task that can be executed at a single time in the data pull tasks of the task flow includes the following steps:
s501, obtaining a predetermined initial packet threshold value, wherein the initial packet threshold value is the maximum data volume allowed to be pulled when a first data pulling task is executed.
Optionally, the header packet threshold is a maximum data amount allowed to be transmitted by the header packet.
And S502, acquiring the data volume corresponding to each data pulling task in the task flow.
And S503, calculating an accumulated value of the data quantity of each data pulling task.
S504, when the accumulated value does not exceed the first packet threshold value, all data pulling tasks in the task flow are determined to be data pulling tasks which can be executed once, so that when the first data pulling task is executed, all data pulling tasks in the task flow are executed.
According to the data pulling method and device, when the accumulated value corresponding to each data pulling task in the task flow does not exceed the first packet threshold value, data pulling is performed on all the data pulling tasks in the task flow, and therefore the time for pulling task data is saved in order to reduce the quantity of requested data.
The second scheme, in which the sum of the data amounts of all the data pulling tasks exceeds the first packet threshold, is described below.
Optionally, referring to fig. 6, fig. 6 shows a data obtaining method, which, based on the embodiment shown in fig. 5, includes the following steps:
s601, obtaining a predetermined first packet threshold value, wherein the first packet threshold value is the maximum data volume allowed to be pulled when a first data pulling task is executed.
S602, acquiring the data volume corresponding to each data pulling task in the task flow.
And S603, calculating an accumulated value of the data quantity of each data pulling task.
S604, judging whether the accumulated value exceeds a first packet threshold value, and if not, executing the step S605; if yes, go to step S606.
S605, determining all the data pulling tasks in the task flow as data pulling tasks which can be executed once, so that when the first data pulling task is executed, all the data pulling tasks in the task flow are executed with data pulling.
And S606, acquiring the corresponding priority of each data pulling task.
Optionally, the priority of the data pulling task is determined according to the urgency and importance degree of the data pulling task, and the priority order of the data pulling task from high to low, which is determined according to the urgency and importance degree of the data pulling task, is as follows: urgent and important, non-urgent and important, urgent not important, not urgent not important, the priority of the urgent and important data pull task may be set to be highest in the above order.
The priority may also be determined only according to the urgency of the data pulling task, for example, the order of the data pulling task priority from high to low is: urgent, not urgent.
The priority may also be determined according to the importance of the data pulling task, for example, the order of the priority of the data pulling task from high to low is: important, not important.
Optionally, the urgency of the data pull task is determined according to an deadline of the data pull task and a current date, for example, if the current date is 2016, 10 and 8 days, the priority of the data pull task with the deadline of 2016, 10 and 10 days is higher than that of the data pull task with the deadline of 2016, 11 and 1 days; the importance of the data pull task is determined by the nature of the task itself.
S607, determining a first data pulling task according to the priority and the first packet threshold, wherein the data pulling tasks in the first data pulling task are selected according to the sequence of the priority from high to low, the data volume corresponding to the first data pulling task does not exceed the first packet threshold, and the sum of the data volume corresponding to the first data pulling task and the data volume corresponding to a next data pulling task adjacent to the priority exceeds the first packet threshold.
For example, there are multiple data pull tasks (hereinafter referred to as tasks) in a task flow, and it is assumed that the priorities of the tasks are ordered from high to low as follows: task 1 > task 2 > task 3 > task 4 > task 5, the data size corresponding to task 1 is 300M, the data sizes corresponding to task 2 and task 3 are all 80M, and the data size corresponding to task 4 and task 5 is 200M. When the first packet threshold is set to 500M, when the first data pull task is determined, since it is calculated that the total amount of data corresponding to tasks 1, 2, and 3 is 300+80+80 ═ 460 < 500M (first packet threshold), and the total amount of data corresponding to tasks 1, 2, 3, and 4 is 300+80 +200 ═ 660 > 500M (first packet threshold), tasks 1, 2, and 3 are determined as the first data pull task, and when data pull is performed for the first time, data corresponding to tasks 1, 2, and 3 are pulled at once.
Optionally, after determining the first data pulling task, the method further includes:
acquiring a predetermined back packet threshold, wherein the back packet threshold is the maximum data volume allowed to be pulled when a non-primary data pulling task is executed, and the back packet threshold is larger than the primary packet threshold;
and determining at least one non-primary data pulling task according to the priority and the post-packet threshold, wherein the data pulling tasks in the non-primary data pulling tasks are selected from high to low in sequence, the priority of the data pulling tasks in the non-primary data pulling tasks is lower than that of the data pulling tasks in the primary data pulling tasks, the data quantity corresponding to the non-primary data pulling tasks does not exceed the post-packet threshold, and the sum of the data quantity corresponding to the non-primary data pulling tasks and the data quantity corresponding to the post-data pulling tasks adjacent to the priority exceeds the post-packet threshold. The determination process of the non-primary data pull task may refer to the above determination example of the primary data pull task, and is not described herein again.
In another embodiment, the executing a data pulling task in the task flow based on the task pulling policy to obtain pulled data includes:
and respectively executing the first data pulling task and the non-first data pulling task in the task flow according to the sequence of the priority from high to low based on the second pulling strategy.
Optionally, for example, the first packet threshold is M, the last packet threshold is N, and N > M, when a data pulling task is packed, a data pulling task cannot cross a packet, if the size of a packet exceeds the first packet threshold M, special processing needs to be performed on the packet, for example, the data pulling task is packed separately, and if the data size of the separately packed data pulling task still exceeds the first packet threshold M, the data pulling task is directly packed.
Optionally, the first packet threshold and the last packet threshold are determined by the following method:
determining the historical average transmission speed of a historical data packet, and the historical transmission time length of the first packet and the historical transmission time length of the later packet of the historical data packet according to the historical task pull record of the non-mobile terminal;
calculating to obtain the first packet threshold according to the historical average transmission speed and the historical transmission duration of the first packet;
and calculating to obtain the rear packet threshold according to the historical average transmission speed and the historical transmission time of the rear packet.
For example, the historical task pull record records the speed of the historical data packet for the past transmission data, the historical average transmission speed of the historical data packet is calculated to be x/per second according to the speed of the past transmission data, the historical transmission time of the first packet is 2 seconds, the historical average transmission speed is multiplied by the historical transmission time of the first packet, and the size of the threshold value M of the first packet is 2 x; the historical transmission time length of the back packet is 3 seconds, the historical average transmission speed is multiplied by the historical transmission time length of the back packet, and the size of the obtained threshold value N of the back packet is 3 x.
It should be noted that, although the historical average transmission speed, the historical transmission duration of the first packet, and the historical transmission duration of the last packet are exemplified in the foregoing, it should be understood by those skilled in the art that the first packet threshold and the last packet threshold may also be calculated by continuously setting other values that meet the calculation requirement, and this is not limited in the embodiment of the present application.
Optionally, the method further includes:
and receiving feedback information, wherein the feedback information is sent by the client when the client sends the data pulling request next time after the data is successfully pulled for the non-last time. That is, for a task flow, under the condition that data pulling is not performed for the last time, information feedback is not specially and independently performed, but feedback information is carried in the data pulling request and is jointly transmitted when the data pulling request is transmitted next time, so that the interaction times with a server are effectively reduced, and the resource occupation is reduced.
In another embodiment, the feedback information may be sent by the client in an idle period without a data pull task after the last successful data pull. That is, for the last data pull of a task flow, after the data pull is successful, the feedback information is not sent immediately, but whether the client currently has data pull task processing, such as sending a data pull request, receiving feedback pull data, and the like, is detected.
In one embodiment, the pull data may be encrypted using an encryption algorithm.
Optionally, the feedback information is used to inform the server of a receiving state of the pull data, so as to avoid data loss, and a sending opportunity of the feedback information includes the following two situations: in the process of pulling data, after the data is pulled successfully for the last time, the client performs task analysis on the acquired information of part of tasks locally, then prepares the task number, and sends the feedback information serving as additional information to the server when sending a data pulling request for the next time, so that the feedback information is prevented from being sent immediately, the interaction between the client and the server is reduced, and the data processing pressure of the server is reduced; and after the client pulls data for the last time, waiting for a time period, and if no other data pulling tasks exist in the time period, independently sending feedback data to the server.
Optionally, the encryption algorithm in the embodiment of the present application includes but is not limited to: base64 (binary data represented based on 64 printable characters), AES128, RSA encryption algorithm, etc.
Referring to fig. 7, fig. 7 shows a data acquisition apparatus, which includes:
a receiving unit 71, configured to receive a data pulling request sent by a client, where the data pulling request carries a type and an identifier of the client;
an analyzing unit 72, configured to analyze the data pulling request to obtain a task stream, where the task stream includes one or more data pulling tasks;
a determining unit 73, configured to determine a task pull policy corresponding to the task flow according to the type of the client;
the execution unit 74 is configured to execute the data pulling task in the task flow based on the task pulling policy to obtain pulled data; and
a sending unit 75, configured to send the pull data to the client corresponding to the identifier.
Fig. 8 is a schematic structural diagram of a client according to another embodiment of the present invention. The client 800 shown in fig. 8 includes: at least one processor 801, memory 802, at least one network interface 804, and other user interfaces 803. The various components in client 800 are coupled together by a bus system 805. It is understood that the bus system 805 is used to enable communications among the components connected. The bus system 805 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 805 in fig. 8.
The user interface 803 may include, among other things, a display, a keyboard, or a pointing device (e.g., a mouse, trackball, touch pad, or touch screen, among others.
It will be appreciated that the memory 802 in embodiments of the invention may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be a Read-only memory (ROM), a programmable Read-only memory (PROM), an erasable programmable Read-only memory (erasabprom, EPROM), an electrically erasable programmable Read-only memory (EEPROM), or a flash memory. The volatile memory may be a Random Access Memory (RAM) which functions as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (staticiram, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (syncronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM ), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and direct memory bus RAM (DRRAM). The memory 802 described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
In some embodiments, memory 802 stores elements, executable units or data structures, or a subset thereof, or an expanded set thereof as follows: an operating system 8021 and application programs 8022.
The operating system 8021 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used for implementing various basic services and processing hardware-based tasks. The application 8022 includes various applications, such as a media player (MediaPlayer), a Browser (Browser), and the like, for implementing various application services. A program implementing a method according to an embodiment of the present invention may be included in application program 8022.
In the embodiment of the present invention, the processor 801 is configured to execute the method steps provided by each method embodiment by calling the program or instruction stored in the memory 802, specifically, the program or instruction stored in the application 8022, and for example, includes:
receiving a data pulling request sent by a client, wherein the data pulling request carries the type and the identification of the client;
analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks;
determining a task pulling strategy corresponding to the task flow according to the type of the client;
based on the task pulling strategy, executing a data pulling task in the task flow to obtain pulling data;
and sending the pull data to the client corresponding to the identifier.
The methods disclosed in the embodiments of the present invention described above may be implemented in the processor 801 or implemented by the processor 801. The processor 801 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 801. The processor 801 may be a general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software elements in the decoding processor. The software elements may be located in ram, flash, rom, prom, or eprom, registers, among other storage media that are well known in the art. The storage medium is located in the memory 802, and the processor 801 reads the information in the memory 802, and combines the hardware to complete the steps of the method.
It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the processing units may be implemented within one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units configured to perform the functions described herein, or a combination thereof.
For a software implementation, the techniques described herein may be implemented by means of units performing the functions described herein. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.
An embodiment of the present application further provides a computer-readable storage medium, where a data obtaining program is stored on the computer-readable storage medium, and when the data obtaining program is executed by a processor, the steps of the data obtaining method according to the method embodiments are implemented, for example, including:
receiving a data pulling request sent by a client, wherein the data pulling request carries the type and the identification of the client;
analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks;
determining a task pulling strategy corresponding to the task flow according to the type of the client;
based on the task pulling strategy, executing a data pulling task in the task flow to obtain pulling data;
and sending the pull data to the client corresponding to the identifier.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the units may be implemented in the same software and/or hardware or in a plurality of software and/or hardware when implementing the invention.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, they are described in relative terms, as long as they are described in partial descriptions of method embodiments. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or client that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or client. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or client that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for obtaining data, the method comprising the steps of:
receiving a data pulling request sent by a client, wherein the data pulling request carries the type and the identification of the client;
analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks;
determining a task pulling strategy corresponding to the task flow according to the type of the client;
based on the task pulling strategy, executing a data pulling task in the task flow to obtain pulling data;
and sending the pull data to the client corresponding to the identifier.
2. The method according to claim 1, wherein the determining a task pull policy corresponding to the task flow according to the type of the client includes:
if the type of the client is a mobile terminal, determining that a task pull strategy corresponding to the task flow is a first pull strategy, wherein the first pull strategy is at least used for determining a data pull task which is actually required to be pulled in data pull tasks of the task flow; and/or the presence of a gas in the gas,
determining the data pulling task actually required to be pulled in the data pulling tasks of the task flow through the first pulling strategy, wherein the data pulling task comprises the following steps:
obtaining a historical task pulling record of the mobile terminal, wherein the historical task pulling record comprises a first historical task which is successfully pulled and a second historical task which is unsuccessfully pulled;
deleting a first task corresponding to the first historical task from the data pulling tasks of the task flow, and/or adding a second task corresponding to the second historical task to obtain an updated data pulling task;
determining the updated data pulling task as the data pulling task which needs to be actually pulled; and/or the presence of a gas in the gas,
based on the task pull strategy, executing a data pull task in the task flow to obtain pull data, including:
and executing the data pulling task which is actually required to be pulled in the task flow based on the first pulling strategy to obtain the pulled data.
3. The method according to claim 1, wherein the determining a task pull policy corresponding to the task flow according to the type of the client includes:
if the type of the client is a non-mobile terminal, determining that a task pull strategy corresponding to the task flow is a second pull strategy, wherein the second pull strategy is at least used for determining a data pull task which can be executed once in the data pull tasks of the task flow; and/or the presence of a gas in the gas,
determining a data pulling task which can be executed in a single time in the data pulling tasks of the task flow through the second pulling strategy, wherein the data pulling task comprises the following steps:
acquiring a predetermined first packet threshold, wherein the first packet threshold is the maximum data volume allowed to be pulled when a first data pulling task is executed;
acquiring the data volume corresponding to each data pulling task in the task flow;
calculating an accumulated value of the data quantity of each data pulling task;
when the accumulated value does not exceed the first packet threshold value, determining all data pulling tasks in the task flow as data pulling tasks which can be executed once, so that when the first data pulling task is executed, all data pulling tasks in the task flow are executed; and/or the presence of a gas in the gas,
based on the task pull strategy, executing a data pull task in the task flow to obtain pull data, including:
and based on the second pulling strategy, performing data pulling on all the data pulling tasks in the task flow to obtain pulling data.
4. The method of claim 3, wherein when the accumulated value exceeds the top packet threshold, the method further comprises:
acquiring the priority corresponding to each data pulling task;
and determining a first data pulling task according to the priority and the first packet threshold, wherein the data pulling tasks in the first data pulling task are selected according to the sequence of the priority from high to low, the data volume corresponding to the first data pulling task does not exceed the first packet threshold, and the sum of the data volume corresponding to the first data pulling task and the data volume corresponding to a next data pulling task adjacent to the priority exceeds the first packet threshold.
5. The method of claim 4, wherein after determining a first data pull task, the method further comprises:
acquiring a predetermined back packet threshold, wherein the back packet threshold is the maximum data volume allowed to be pulled when a non-primary data pulling task is executed, and the back packet threshold is larger than the primary packet threshold;
determining at least one non-primary data pulling task according to the priority and the post-packet threshold, wherein the data pulling tasks in the non-primary data pulling tasks are selected from high to low in sequence of priority, the priority of the data pulling tasks in the non-primary data pulling tasks is lower than the priority of the data pulling tasks in the primary data pulling tasks, the data quantity corresponding to the non-primary data pulling tasks does not exceed the post-packet threshold, and the sum of the data quantity corresponding to the non-primary data pulling tasks and the data quantity corresponding to the post-data pulling tasks adjacent to the priority exceeds the post-packet threshold; and/or the presence of a gas in the gas,
the executing a data pulling task in the task flow based on the task pulling strategy to obtain pulling data includes:
and respectively executing the first data pulling task and the non-first data pulling task in the task flow according to the sequence of the priority from high to low based on the second pulling strategy.
6. The method of claim 5, wherein the first packet threshold and the second packet threshold are determined by:
determining the historical average transmission speed of a historical data packet, and the historical transmission time length of the first packet and the historical transmission time length of the later packet of the historical data packet according to the historical task pull record of the non-mobile terminal;
calculating to obtain the first packet threshold according to the historical average transmission speed and the historical transmission duration of the first packet;
and calculating to obtain the rear packet threshold according to the historical average transmission speed and the historical transmission time of the rear packet.
7. The method according to any one of claims 3-6, further comprising:
receiving feedback information, wherein the feedback information is sent by the client when the client sends the data pulling request next time after the data is successfully pulled for the non-last time; or, the feedback information is sent by the client in an idle time period without a data pulling task after the data is successfully pulled for the last time;
and/or the presence of a gas in the gas,
and encrypting the pull data by using an encryption algorithm.
8. An apparatus for acquiring data, the apparatus comprising:
the system comprises a receiving unit, a sending unit and a receiving unit, wherein the receiving unit is used for receiving a data pulling request sent by a client, and the data pulling request carries the type and the identification of the client;
the analysis unit is used for analyzing the data pulling request to obtain a task flow, wherein the task flow comprises one or more data pulling tasks;
a determining unit, configured to determine a task pull policy corresponding to the task flow according to the type of the client;
the execution unit is used for executing a data pulling task in the task flow based on the task pulling strategy to obtain pulling data; and
and the sending unit is used for sending the pull data to the client corresponding to the identifier.
9. A client, the client comprising: at least one processor, memory, at least one network interface, and a user interface;
the at least one processor, memory, at least one network interface, and user interface are coupled together by a bus system;
the processor is adapted to execute the steps of the method of acquiring data according to any one of claims 1 to 7 by calling a program or instructions stored in the memory.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon an acquisition program of data, which when executed by a processor implements the steps of the acquisition method of data according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910995284.5A CN112685192A (en) | 2019-10-18 | 2019-10-18 | Data acquisition method and device, client and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910995284.5A CN112685192A (en) | 2019-10-18 | 2019-10-18 | Data acquisition method and device, client and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112685192A true CN112685192A (en) | 2021-04-20 |
Family
ID=75445670
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910995284.5A Pending CN112685192A (en) | 2019-10-18 | 2019-10-18 | Data acquisition method and device, client and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112685192A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342381A (en) * | 2021-06-29 | 2021-09-03 | 中国平安人寿保险股份有限公司 | Data configuration method and device for application update, computer equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106484805A (en) * | 2016-09-22 | 2017-03-08 | 联动优势电子商务有限公司 | A kind of data export method and data handling system |
| CN107222426A (en) * | 2016-03-21 | 2017-09-29 | 阿里巴巴集团控股有限公司 | The method of flow control, apparatus and system |
| CN107819579A (en) * | 2017-12-13 | 2018-03-20 | 西安Tcl软件开发有限公司 | A kind of processing method, server and the computer-readable recording medium of user's request |
| CN108269104A (en) * | 2016-12-30 | 2018-07-10 | 腾讯科技（深圳）有限公司 | Media information put-on method launches engine server and media information jettison system |
| CN109194647A (en) * | 2018-08-30 | 2019-01-11 | 北京金山安全软件有限公司 | Data transmission method and device, electronic equipment and storage medium |
| US20190182168A1 (en) * | 2017-12-11 | 2019-06-13 | International Business Machines Corporation | Dynamic throttling thresholds |
| CN109981779A (en) * | 2019-03-27 | 2019-07-05 | 掌阅科技股份有限公司 | Service providing method, server and computer storage medium |
-
2019
- 2019-10-18 CN CN201910995284.5A patent/CN112685192A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107222426A (en) * | 2016-03-21 | 2017-09-29 | 阿里巴巴集团控股有限公司 | The method of flow control, apparatus and system |
| CN106484805A (en) * | 2016-09-22 | 2017-03-08 | 联动优势电子商务有限公司 | A kind of data export method and data handling system |
| CN108269104A (en) * | 2016-12-30 | 2018-07-10 | 腾讯科技（深圳）有限公司 | Media information put-on method launches engine server and media information jettison system |
| US20190182168A1 (en) * | 2017-12-11 | 2019-06-13 | International Business Machines Corporation | Dynamic throttling thresholds |
| CN107819579A (en) * | 2017-12-13 | 2018-03-20 | 西安Tcl软件开发有限公司 | A kind of processing method, server and the computer-readable recording medium of user's request |
| CN109194647A (en) * | 2018-08-30 | 2019-01-11 | 北京金山安全软件有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN109981779A (en) * | 2019-03-27 | 2019-07-05 | 掌阅科技股份有限公司 | Service providing method, server and computer storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 王子光;王子明;: "高并发数据共享系统基于Push-Pull混合模式的性能优化", 现代信息科技, no. 04, 25 April 2018 (2018-04-25) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342381A (en) * | 2021-06-29 | 2021-09-03 | 中国平安人寿保险股份有限公司 | Data configuration method and device for application update, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110489447B (en) | Data query method and device, computer equipment and storage medium | |
| US9785664B2 (en) | Gathering transaction data associated with locally stored data files | |
| US8949258B2 (en) | Techniques to manage file conversions | |
| JP5768870B2 (en) | PROGRAM DISTRIBUTION DEVICE, PROGRAM DISTRIBUTION METHOD, PROGRAM DISTRIBUTION SYSTEM, AND COMPUTER PROGRAM | |
| CN109040227B (en) | Service request response method and device based on block chain and computer equipment | |
| US20200334078A1 (en) | Scheduling requests based on resource information | |
| CN111770157B (en) | Business processing method and device, electronic equipment and storage medium | |
| CN110943934A (en) | Service request processing method, system, terminal and readable storage medium | |
| WO2020147403A1 (en) | Cloud storage based file processing method, system and computer device | |
| CN113411404A (en) | File downloading method, device, server and storage medium | |
| US10642585B1 (en) | Enhancing API service schemes | |
| CN109428926B (en) | Method and device for scheduling task nodes | |
| CN111193789B (en) | Subscription information pushing method, device, computer equipment and readable storage medium | |
| CN110413610B (en) | Method and system for improving export efficiency of business data report forms and database server | |
| CN110008187B (en) | File transmission scheduling method, device, equipment and computer readable storage medium | |
| CN106933673B (en) | Method and device for adjusting number of logical threads of component | |
| CN112685192A (en) | Data acquisition method and device, client and computer readable storage medium | |
| CN113742389A (en) | Service processing method and device | |
| CN103746843A (en) | Method and equipment for processing RPC (Remote Procedure Call Protocol) command | |
| CN115278786A (en) | Service access method, device, electronic equipment and storage medium | |
| CN107632893B (en) | Message queue processing method and device | |
| CN114048059A (en) | Method and device for adjusting timeout time of interface, computer equipment and storage medium | |
| CN114422530A (en) | Flow control method and device, computer equipment and storage medium | |
| CN114374657A (en) | Data processing method and device | |
| US20110213815A1 (en) | Minimizing bandwidth in file path-centric protocol message |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |