CN110209690B - Data acquisition method and device and electronic equipment - Google Patents
Data acquisition method and device and electronic equipment Download PDFInfo
- Publication number
- CN110209690B CN110209690B CN201910461150.5A CN201910461150A CN110209690B CN 110209690 B CN110209690 B CN 110209690B CN 201910461150 A CN201910461150 A CN 201910461150A CN 110209690 B CN110209690 B CN 110209690B
- Authority
- CN
- China
- Prior art keywords
- data
- group
- limit value
- acquired
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2425—Iterative querying; Query formulation based on the results of a preceding query
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
Abstract
The embodiment of the invention provides a data acquisition method, a data acquisition device and electronic equipment, wherein the method comprises the following steps: determining the id minimum value and the id maximum value of the data id of the previous group of data and the number of the data of the previous group of data aiming at the previous group of data which is obtained in all the data to be obtained; calculating an id lower limit value and an id upper limit value of the data id of a group of data to be acquired at each time according to a data id growth trend represented by an id minimum value, an id maximum value and a data number and based on the maximum data number which can be acquired at each time; a set of data consisting of data of an id range represented by an id lower limit value and an id upper limit value is obtained. Because the data id range of the data to be acquired is calculated according to the method before the data is acquired every time, the number of the acquired data is not less than that of the data acquired every time in the prior art, the memory space is more fully utilized, the total time consumption of data query is shortened, and the query efficiency is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data acquisition method and apparatus, and an electronic device.
Background
With the continuous development of informatization, the data generation speed is increasing, the data volume required to be processed expands rapidly, and when the data volume required to be processed is large, all the data required to be processed cannot be acquired at one time due to the limitation of the size of the device memory, so that the data can be acquired in batches through condition query in the related art.
In the prior art, the condition query batch acquisition data specifically may be: assuming that the number of data which can be queried by the memory is 50, and the query condition supports querying according to id, the data range of the first query can be set to 1 or more and less than or equal to 50, the data range of the second query is 51 or more and less than or equal to 100, the data range of the third query is 101 or more and less than or equal to 150, and so on until all data are queried. Due to the fact that the id is discontinuous, the memory space of each query may not be fully utilized, the total time consumption of data query is long, and the query efficiency is low.
Disclosure of Invention
The embodiment of the invention aims to provide a data acquisition method, a data acquisition device and electronic equipment, so that the memory space can be more fully utilized in the data acquisition process, the total time consumption of data query is shortened, and the data query efficiency is improved. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a data acquisition method, including:
determining the id minimum value and the id maximum value of the data id of the previous group of data and the number of the data of the previous group of data aiming at the previous group of data which is obtained in all data to be obtained;
according to the data id growth trend represented by the id minimum value, the id maximum value and the data number, and based on the maximum data number which can be obtained each time, calculating the id lower limit value and the id upper limit value of the data id of a group of data to be obtained at the time;
and acquiring a set of data consisting of data of an id range represented by the id lower limit value and the id upper limit value.
In one embodiment, the calculating the id lower limit value and the id upper limit value of the data id of the group of data to be acquired at this time based on the data id growth trend expressed according to the id minimum value, the id maximum value and the data number and the maximum data number that can be acquired at each time includes:
determining the maximum value of the id as the lower limit value of the id of the data id of the group of data to be acquired;
calculating the id upper limit value of the data id of a group of data to be acquired at this time by adopting the following relational expression:
wherein y represents the id upper limit value of the data id of a group of data to be acquired at this time, and y1Represents the id minimum, y2And the lower limit value of the id of the data id of the group of data to be acquired at this time is represented, a represents the maximum number of data which can be acquired at each time, and b represents the number of the data.
In one embodiment, the method further comprises:
calculating the id lower limit value and the id upper limit value of the data id of a group of data to be acquired for the first time based on the id minimum value of the data id of all the data and the maximum data number;
and acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the group of data to be acquired for the first time.
In one embodiment, the calculating the id lower limit value and the id upper limit value of the data id of the group of data to be acquired for the first time based on the id minimum value of the data ids of all the data and the maximum number of data pieces includes:
determining the minimum value of the ids of the data of all the data as the lower limit value of the ids of the data of a group of data to be acquired for the first time;
and calculating the sum of the lower limit value of the id of the group of data to be acquired for the first time and the maximum number of the data to be acquired as the upper limit value of the id of the group of data to be acquired for the first time.
In one embodiment, after said determining the id minimum and the id maximum of the data id of the previous set of data, the method further comprises:
and stopping acquiring the data when the maximum value of the id of the data id of the previous group of data is equal to the maximum value of the id of the data id of all the data.
In one embodiment, the maximum number of data pieces is calculated according to the memory occupancy and the total memory capacity of a piece of data, and the memory occupancy of the piece of data is determined by the following steps:
acquiring a preset number of pieces of data, and calculating the average memory occupation amount of the preset number of pieces of data to be used as the memory occupation amount of one piece of data; alternatively, the first and second electrodes may be,
the method comprises the steps of obtaining a preset number of pieces of data, selecting the data with the largest memory occupation amount in the preset number of pieces of data, and taking the memory occupation amount of the data as the memory occupation amount of one piece of data.
In one embodiment, the method further comprises:
if the number of the data of one group of data to be acquired at this time is larger than the maximum number of the data, calculating an id lower limit value and an id upper limit value of the data id of another group of data to be acquired at this time according to the id maximum value and the maximum number of the data;
and acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the other group of data.
In one embodiment, the method further comprises:
and if the number of the data of the group of data to be acquired is larger than the maximum number of the data, displaying reminding information, wherein the reminding information indicates that the data acquisition fails.
In a second aspect, an embodiment of the present invention provides a data acquisition apparatus, including:
the determining module is used for determining the id minimum value and the id maximum value of the data id of the previous group of data and the number of the data of the previous group of data aiming at the previous group of data which is obtained in all the data to be obtained;
the calculation module is used for calculating the id lower limit value and the id upper limit value of the data id of the group of data to be acquired at this time based on the maximum data number which can be acquired at each time according to the data id growth trend represented by the id minimum value, the id maximum value and the data number;
and the acquisition module is used for acquiring a group of data consisting of the data of the id range represented by the id lower limit value and the id upper limit value.
In an embodiment, the calculation module is specifically configured to determine the maximum id value as an id lower limit value of a data id of a group of data to be acquired this time; and calculating the id upper limit value of the data id of a group of data to be acquired at this time by adopting the following relational expression:
wherein y represents the id upper limit value of the data id of a group of data to be acquired at this time, and y1Represents the id minimum, y2And the lower limit value of the id of the data id of the group of data to be acquired at this time is represented, a represents the maximum number of data which can be acquired at each time, and b represents the number of the data.
In one embodiment, the calculation module is further configured to calculate an id lower limit value and an id upper limit value of a data id of a group of data to be acquired for the first time based on the id minimum value of the data ids of all the data and the maximum number of data pieces;
the acquisition module is further used for acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the group of data to be acquired for the first time.
In an embodiment, the calculation module is specifically configured to determine the minimum value of the ids of the data of all the data as the lower limit value of the ids of the data of a group of data to be acquired for the first time; and calculating the sum of the lower limit value of the id of the group of data to be acquired for the first time and the maximum number of data pieces to serve as the upper limit value of the id of the group of data to be acquired for the first time.
In one embodiment, the obtaining module is further configured to stop obtaining data when the maximum value of the id of the data id of the previous group of data is equal to the maximum value of the id of the data id of all the data after the determining module determines the minimum value and the maximum value of the id of the data id of the previous group of data.
In one embodiment, the maximum number of data is calculated according to the memory occupation amount and the total memory capacity of one piece of data;
the calculation module is further configured to determine the memory footprint of the piece of data by:
acquiring a preset number of pieces of data, and calculating the average memory occupation amount of the preset number of pieces of data to be used as the memory occupation amount of one piece of data; or acquiring a preset number of pieces of data, selecting the data with the largest memory occupation amount in the preset number of pieces of data, and taking the memory occupation amount of the data as the memory occupation amount of one piece of data.
In an embodiment, the calculating module is further configured to calculate, according to the maximum id value and the maximum data number, an id lower limit value and an id upper limit value of a data id of another group of data to be acquired at this time if the data number of the group of data to be acquired at this time is greater than the maximum data number;
the acquisition module is further used for acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the other group of data.
In one embodiment, the apparatus further comprises:
and the display module is used for displaying reminding information if the number of the data of the group of data to be acquired is greater than the maximum number of the data, wherein the reminding information indicates that the data acquisition fails.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing any one of the steps of the data acquisition method when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned data acquisition method steps.
In a fifth aspect, embodiments of the present invention further provide a computer program product including instructions, which when run on a computer, cause the computer to perform any of the steps of the data acquisition method described above.
According to the data acquisition method, the data acquisition device and the electronic equipment, the id upper limit value and the id lower limit value of the data id of the group of data to be acquired at the time can be calculated according to the id minimum value and the id maximum value of the data id of the previous group of data which are acquired and the data id increase trend represented by the number of data pieces, and based on the maximum number of data pieces which can be acquired at each time.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
Fig. 1 is a flowchart of a data acquisition method according to an embodiment of the present invention;
FIG. 2 is another flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data acquisition device according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
An embodiment of the present invention provides a data acquisition method, as shown in fig. 1, including the following steps:
step 101, determining the minimum value and the maximum value of the id of the data id of the previous group of data and the number of the data of the previous group of data according to the previous group of data acquired in all the data to be acquired.
And 102, calculating an id lower limit value and an id upper limit value of the data id of the group of data to be acquired at the time according to the data id growth trend represented by the id minimum value, the id maximum value and the data number, and based on the maximum data number which can be acquired at each time.
The maximum number of data which can be acquired each time can be calculated according to the memory occupation amount and the total memory capacity of one piece of data, and the memory occupation amount of the one piece of data can be determined in the following way:
acquiring a preset number of pieces of data, and calculating the average memory occupation amount of the preset number of pieces of data to be used as the memory occupation amount of one piece of data; or acquiring a preset number of pieces of data, selecting the data with the largest memory occupation amount in the preset number of pieces of data, and taking the memory occupation amount of the data as the memory occupation amount of one piece of data.
For example, assuming that the memory is 200M, if 10 pieces of data are randomly acquired from all the data to be acquired, and the memory occupancy of the largest piece of data among the 10 pieces of data is 1M, the 1M is taken as the memory occupancy of one piece of data, and the maximum number of data that can be acquired each time is the ratio of the total memory capacity to the memory occupancy of one piece of data, that is, in this embodiment, the maximum number of data that can be acquired each time is 200.
In this step, a numerical value that can indicate the increasing trend of the data id can be calculated according to the minimum value and the maximum value of the data id of the previous group of data and the number of data pieces, and the lower limit value and the upper limit value of the data id of the group of data to be acquired at this time are calculated based on the maximum number of data pieces that can be acquired at each time.
Specifically, the maximum value of the id of the data id of the last group of obtained data may be determined as the id lower limit value of the data id of the group of data to be obtained this time; and calculating the id upper limit value of the data id of the group of data to be acquired at this time through the following relational expression:
wherein y represents the id upper limit value of the data id of a group of data to be acquired at this time, and y1Represents the id minimum, y2And the lower limit value of the id of the data id of the group of data to be acquired at this time is represented, a represents the maximum number of data which can be acquired at each time, and b represents the number of the data. For example, if the query condition supports querying by id, and id is integer, if id values of data id of the last group of acquired data are: 25. 20, 26, 32, 28, obviously, in the data id of the group of data, the maximum value of id is 32, the minimum value of id is 20, and the maximum value of id, that is, 32 can be used as the id lower limit value of the data id of the next group of data to be acquired. The id values of the data ids can be arranged from small to large, and then the data with the id value of 20 is the first piece of data, and the data with the id value of 32 is the fifth piece of data. Assuming that the maximum capacity of the memory can accommodate 10 pieces of data, that is, the maximum number of data that can be obtained at a time is 10, then in the above formula, y is1＝20，y 232, 10 and 5. And then, substituting the data into a formula, and obtaining the upper limit value y of the id of the data of the group of data to be acquired at the time as 62.
And 103, acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value.
As can be seen from the embodiment of step 102, the lower limit value of the next set of data to be obtained is 32, and the upper limit value is 62, and in this step, data in the range of 32 < id ≦ 62 can be obtained.
According to the data acquisition method provided by the embodiment of the invention, the id upper limit value and the id lower limit value of the data id of the group of data to be acquired at this time can be calculated according to the id minimum value and the id maximum value of the data id of the previous group of acquired data and the data id increase trend represented by the number of data pieces and based on the maximum number of data pieces which can be acquired at each time.
The method provided by the present invention is described in detail with reference to the accompanying drawings by using a specific embodiment, as shown in fig. 2, another flow chart of the data acquisition method is provided, which comprises the following steps:
step 201, based on the minimum value of the data id of all the data and the maximum number of data which can be obtained each time, calculating the lower limit value and the upper limit value of the data id of a group of data to be obtained for the first time.
In this step, since there is no previous set of data to be referred to before the first set of data is acquired, the id lower limit value and the id upper limit value of the data id of the first set of data to be acquired are calculated based on the id minimum value of the data id of all the data and the maximum number of pieces of data that can be acquired each time.
Specifically, the minimum value of the id in the data ids of all the data may be determined first, and the minimum value of the id or the previous value of the minimum value of the id may be determined as the lower limit value of the id of the data id of the group of data to be acquired for the first time. And calculating the sum of the id lower limit value and the maximum number of data pieces which can be acquired each time as the id upper limit value of the data id of the group of data to be acquired for the first time.
Step 202, acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of a data id of a group of data to be acquired for the first time.
In this step, if the minimum value of id in the data ids of all the data is 3, and the maximum number of data that can be obtained each time is 20, data in the range of 2 < id ≦ 23 may be obtained, and optionally, data in the range of 3 ≦ id ≦ 23 may also be obtained.
In this embodiment, after determining the id minimum value and the id maximum value of the data id of the previous group of data, the method further includes: and stopping acquiring the data when the maximum value of the id of the data id of the last group of data is equal to the maximum value of the id of the data of all the data.
In the above embodiment, when the maximum value of the id of the data id of the previous group of data is equal to the maximum value of the id of the data id of all the data, it indicates that all the data to be acquired has been acquired, and at this time, the data acquisition may be stopped.
In this embodiment, if the number of the group of data to be obtained this time is greater than the maximum number of data that can be obtained each time, for example, if the maximum number of data that can be obtained each time is 15 and the number of the group of data to be obtained this time is 20, the memory capacity occupied by the data to be obtained this time exceeds the total memory capacity, at this time, a prompt message may be displayed, where the prompt message indicates that the data obtaining this time fails, for example, "data obtaining fails", "exceeding the memory |", and the prompt message may be a message indicating that the data obtaining this time fails! And reminding information.
In this embodiment, after the data acquisition fails, an id lower limit value and an id upper limit value of another set of data id to be acquired at this time may be calculated according to the maximum value of the id of the data id of the previous set of data and the maximum number of data pieces. The maximum value of the id of the data id of the previous group of data may be used as the lower limit value of the id of the data id of another group of data to be acquired this time, and the calculation method of the upper limit value of the id of the data id of another group of data to be acquired this time may be the same as the calculation method of the upper limit value of the id of the data id of the group of data to be acquired for the first time in step 201, and is not described here again. And after the id lower limit value and the id upper limit value of the data id of the other group of data to be acquired at this time are confirmed, acquiring a group of data consisting of data in the id range represented by the id lower limit value and the id upper limit value.
Based on the same inventive concept, according to the above data acquisition method provided in the embodiments of the present invention, an embodiment of the present invention further provides a data acquisition apparatus, a schematic structural diagram of which is shown in fig. 3, the apparatus including:
the determining module 30 is configured to determine, for a previous group of acquired data in all data to be acquired, a minimum value id and a maximum value id of data id of the previous group of data, and a number of data pieces of the previous group of data;
the calculation module 31 is configured to calculate an id lower limit value and an id upper limit value of a data id of a group of data to be acquired at this time based on the maximum number of data that can be acquired at each time according to the data id growth trend represented by the id minimum value, the id maximum value, and the number of data pieces;
and the obtaining module 32 is used for obtaining a group of data consisting of data of the id range represented by the id lower limit value and the id upper limit value.
In an embodiment, the calculating module 31 is specifically configured to determine the maximum id value as an id lower limit value of a data id of a group of data to be acquired this time; and calculating the id upper limit value of the data id of a group of data to be acquired at this time by adopting the following relational expression:
wherein y represents the id upper limit value of the data id of a group of data to be acquired at this time, and y1Represents the id minimum, y2And the lower limit value of the id of the data id of the group of data to be acquired at this time is represented, a represents the maximum number of data which can be acquired at each time, and b represents the number of the data.
In one embodiment, the calculating module 31 is further configured to calculate an id lower limit value and an id upper limit value of a data id of a group of data to be acquired for the first time based on the id minimum value of the data ids of all the data and the maximum number of data pieces;
the obtaining module 32 is further configured to obtain a group of data including data in an id range represented by an id lower limit value and an id upper limit value of a data id of the group of data to be obtained for the first time.
In an embodiment, the calculating module 31 is specifically configured to determine the minimum value of the ids of the data ids of all the data as the lower limit value of the ids of the data ids of a group of data to be acquired for the first time; and calculating the sum of the lower limit value of the id of the group of data to be acquired for the first time and the maximum number of data pieces to serve as the upper limit value of the id of the group of data to be acquired for the first time.
In one embodiment, the obtaining module 32 is further configured to stop obtaining data when the maximum id value of the data id of the previous group of data is equal to the maximum id value of the data id of all the data after the determining module 30 determines the minimum id value and the maximum id value of the data id of the previous group of data.
In one embodiment, the maximum number of data is calculated according to the memory occupation amount and the total memory capacity of one piece of data;
the calculating module 31 is further configured to determine the memory usage amount of the piece of data by:
acquiring a preset number of pieces of data, and calculating the average memory occupation amount of the preset number of pieces of data to be used as the memory occupation amount of one piece of data; or acquiring a preset number of pieces of data, selecting the data with the largest memory occupation amount in the preset number of pieces of data, and taking the memory occupation amount of the data as the memory occupation amount of one piece of data.
In an embodiment, the calculating module 31 is further configured to calculate, according to the maximum value of the id and the maximum number of data pieces, a lower limit value and an upper limit value of the id of data id of another group of data to be acquired at this time if the number of data pieces of the group of data to be acquired at this time is greater than the maximum number of data pieces;
the obtaining module 32 is further configured to obtain a set of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the another set of data.
In one embodiment, the apparatus further comprises:
and the display module 33 is configured to display a reminding message if the number of the data of the group of data to be acquired at this time is greater than the maximum number of the data, where the reminding message indicates that the data acquisition at this time fails.
An embodiment of the present invention further provides an electronic device, as shown in fig. 4, including a processor 41, a communication interface 42, a memory 43, and a communication bus 44, where the processor 41, the communication interface 42, and the memory 43 complete mutual communication through the communication bus 44,
a memory 43 for storing a computer program;
the processor 41, when executing the program stored in the memory 53, implements the following steps:
determining the id minimum value and the id maximum value of the data id of the previous group of data and the number of the data of the previous group of data aiming at the previous group of data which is obtained in all the data to be obtained;
calculating an id lower limit value and an id upper limit value of the data id of a group of data to be acquired at each time according to a data id growth trend represented by an id minimum value, an id maximum value and a data number and based on the maximum data number which can be acquired at each time;
a set of data consisting of data of an id range represented by an id lower limit value and an id upper limit value is obtained.
In one embodiment, the method for calculating the id lower limit value and the id upper limit value of the data id of a group of data to be acquired at the time based on the data id growth trend expressed by the id minimum value, the id maximum value and the data number and the maximum data number capable of being acquired at each time comprises the following steps:
determining the maximum value of the id as the lower limit value of the id of the data id of a group of data to be acquired;
calculating the id upper limit value of the data id of a group of data to be acquired at this time by adopting the following relational expression:
wherein y represents the id upper limit value of the data id of a group of data to be acquired at this time, and y1Represents the id minimum, y2And the lower limit value of the id of the data id of the group of data to be acquired at this time is represented, a represents the maximum number of data which can be acquired at each time, and b represents the number of the data.
In one embodiment, the method further comprises:
calculating the id lower limit value and the id upper limit value of the data id of a group of data to be acquired for the first time based on the id minimum value and the maximum data number of the data id of all the data;
and acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the group of data to be acquired for the first time.
In one embodiment, calculating the id lower limit value and the id upper limit value of the data id of a group of data to be acquired for the first time based on the id minimum value of the data id of all the data and the maximum data number comprises:
determining the minimum value of the ids of the data of all the data as the lower limit value of the ids of the data of a group of data to be acquired for the first time;
and calculating the sum of the lower limit value of the id of the group of data to be acquired for the first time and the maximum number of data pieces as the upper limit value of the id of the group of data to be acquired for the first time.
In one embodiment, after determining the id minimum and the id maximum of the data id of the previous set of data, the method further comprises:
and stopping acquiring the data when the maximum value of the id of the data id of the last group of data is equal to the maximum value of the id of the data of all the data.
In one embodiment, the maximum number of data pieces is calculated according to the memory occupancy and the total memory capacity of a piece of data, and the memory occupancy of a piece of data is determined by the following steps:
acquiring a preset number of pieces of data, and calculating the average memory occupation amount of the preset number of pieces of data to be used as the memory occupation amount of one piece of data; alternatively, the first and second electrodes may be,
the method comprises the steps of obtaining a preset number of pieces of data, selecting the data with the largest memory occupation amount from the preset number of pieces of data, and taking the memory occupation amount of the data as the memory occupation amount of one piece of data.
In one embodiment, the method further comprises:
if the number of data of one group of data to be acquired at this time is larger than the maximum number of data, calculating an id lower limit value and an id upper limit value of data id of another group of data to be acquired at this time according to the id maximum value and the maximum number of data;
a set of data composed of data of an id range represented by an id lower limit value and an id upper limit value of data id of another set of data is acquired.
In one embodiment, the method further comprises:
and if the number of the data of the group of data to be acquired is larger than the maximum number of the data, displaying reminding information, wherein the reminding information shows that the data acquisition fails.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program realizes the steps of any one of the above data acquisition methods when executed by a processor.
In a further embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the steps of any of the data acquisition methods described in the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, the electronic device, the computer-readable storage medium, and the computer program product embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (15)
1. A method of data acquisition, comprising:
determining the id minimum value and the id maximum value of the data id of the previous group of data and the number of the data of the previous group of data aiming at the previous group of data which is obtained in all data to be obtained;
according to the data id growth trend represented by the id minimum value, the id maximum value and the data number, and based on the maximum data number which can be obtained each time, calculating the id lower limit value and the id upper limit value of the data id of a group of data to be obtained at the time;
acquiring a group of data consisting of data of an id range represented by the id lower limit value and the id upper limit value;
the method for calculating the id lower limit value and the id upper limit value of the data id of the group of data to be acquired at the time based on the data id growth trend represented by the id minimum value, the id maximum value and the data number and the maximum data number which can be acquired at each time comprises the following steps:
determining the maximum value of the id as the lower limit value of the id of the data id of the group of data to be acquired;
calculating the id upper limit value of the data id of a group of data to be acquired at this time by adopting the following relational expression:
wherein y represents the id upper limit value of the data id of a group of data to be acquired at this time, and y1Represents the id minimum, y2And the lower limit value of the id of the data id of the group of data to be acquired at this time is represented, a represents the maximum number of data which can be acquired at each time, and b represents the number of the data.
2. The method of claim 1, further comprising:
calculating the id lower limit value and the id upper limit value of the data id of a group of data to be acquired for the first time based on the id minimum value of the data id of all the data and the maximum data number;
and acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the group of data to be acquired for the first time.
3. The method according to claim 2, wherein the calculating of the id lower limit value and the id upper limit value of the data id of the group of data to be acquired for the first time based on the id minimum value of the data id of all the data and the maximum data number comprises:
determining the minimum value of the ids of the data of all the data as the lower limit value of the ids of the data of a group of data to be acquired for the first time;
and calculating the sum of the lower limit value of the id of the group of data to be acquired for the first time and the maximum number of the data to be acquired as the upper limit value of the id of the group of data to be acquired for the first time.
4. The method according to claim 1, further comprising, after said determining id minimum and id maximum of data id of said previous set of data:
and stopping acquiring the data when the maximum value of the id of the data id of the previous group of data is equal to the maximum value of the id of the data id of all the data.
5. The method of claim 1, wherein the maximum number of data pieces is calculated according to a memory footprint and a total memory capacity of a piece of data, and the memory footprint of the piece of data is determined by:
acquiring a preset number of pieces of data, and calculating the average memory occupation amount of the preset number of pieces of data to be used as the memory occupation amount of one piece of data; alternatively, the first and second electrodes may be,
the method comprises the steps of obtaining a preset number of pieces of data, selecting the data with the largest memory occupation amount in the preset number of pieces of data, and taking the memory occupation amount of the data as the memory occupation amount of one piece of data.
6. The method of claim 1, further comprising:
if the number of the data of one group of data to be acquired at this time is larger than the maximum number of the data, calculating an id lower limit value and an id upper limit value of the data id of another group of data to be acquired at this time according to the id maximum value and the maximum number of the data;
and acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the other group of data.
7. The method of claim 6, further comprising:
and if the number of the data of the group of data to be acquired is larger than the maximum number of the data, displaying reminding information, wherein the reminding information indicates that the data acquisition fails.
8. A data acquisition apparatus, comprising:
the determining module is used for determining the id minimum value and the id maximum value of the data id of the previous group of data and the number of the data of the previous group of data aiming at the previous group of data which is obtained in all the data to be obtained;
the calculation module is used for calculating the id lower limit value and the id upper limit value of the data id of the group of data to be acquired at this time based on the maximum data number which can be acquired at each time according to the data id growth trend represented by the id minimum value, the id maximum value and the data number;
the acquisition module is used for acquiring a group of data consisting of data of an id range represented by the id lower limit value and the id upper limit value;
the computing module is specifically used for determining the maximum value of the id as the id lower limit value of the data id of the group of data to be acquired; and calculating the id upper limit value of the data id of a group of data to be acquired at this time by adopting the following relational expression:
wherein y represents the id upper limit value of the data id of a group of data to be acquired at this time, and y1Represents the id minimum, y2And the lower limit value of the id of the data id of the group of data to be acquired at this time is represented, a represents the maximum number of data which can be acquired at each time, and b represents the number of the data.
9. The apparatus according to claim 8, wherein the calculating module is further configured to calculate an id lower limit value and an id upper limit value of a data id of a group of data to be acquired for the first time based on the id minimum value of the data ids of all the data and the maximum number of data pieces;
the acquisition module is further used for acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the group of data to be acquired for the first time.
10. The apparatus according to claim 9, wherein the computing module is specifically configured to determine a minimum id value of data ids of all the data as a lower id value of a data id of a group of data to be acquired for the first time; and calculating the sum of the lower limit value of the id of the group of data to be acquired for the first time and the maximum number of data pieces to serve as the upper limit value of the id of the group of data to be acquired for the first time.
11. The apparatus according to claim 8, wherein the obtaining module is further configured to stop obtaining data when the maximum id value of the data id of the previous group of data is equal to the maximum id value of the data id of all the data after the determining module determines the minimum id value and the maximum id value of the data id of the previous group of data.
12. The apparatus of claim 8, wherein the maximum number of data is calculated according to a memory footprint and a total memory capacity of a piece of data;
the calculation module is further configured to determine the memory footprint of the piece of data by:
acquiring a preset number of pieces of data, and calculating the average memory occupation amount of the preset number of pieces of data to be used as the memory occupation amount of one piece of data; or acquiring a preset number of pieces of data, selecting the data with the largest memory occupation amount in the preset number of pieces of data, and taking the memory occupation amount of the data as the memory occupation amount of one piece of data.
13. The apparatus according to claim 8, wherein the calculating module is further configured to calculate, according to the maximum id value and the maximum number of data pieces, an id lower limit value and an id upper limit value of a data id of another set of data to be acquired at this time if the number of data pieces of one set of data to be acquired at this time is greater than the maximum number of data pieces;
the acquisition module is further used for acquiring a group of data consisting of data of an id range represented by an id lower limit value and an id upper limit value of the data id of the other group of data.
14. The apparatus of claim 13, further comprising:
and the display module is used for displaying reminding information if the number of the data of the group of data to be acquired is greater than the maximum number of the data, wherein the reminding information indicates that the data acquisition fails.
15. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910461150.5A CN110209690B (en) | 2019-05-30 | 2019-05-30 | Data acquisition method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910461150.5A CN110209690B (en) | 2019-05-30 | 2019-05-30 | Data acquisition method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110209690A CN110209690A (en) | 2019-09-06 |
| CN110209690B true CN110209690B (en) | 2021-07-20 |
Family
ID=67789557
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910461150.5A Active CN110209690B (en) | 2019-05-30 | 2019-05-30 | Data acquisition method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110209690B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1575484A (en) * | 2001-09-24 | 2005-02-02 | 尼奥普泰克技术公司 | Method for anonymous computerized processing of documents or objects |
| CN104781170A (en) * | 2012-10-15 | 2015-07-15 | 杰富意钢铁株式会社 | Shipping work planning system, shipping work planning method, and shipping work method |
| CN106844690A (en) * | 2017-01-23 | 2017-06-13 | 北京齐尔布莱特科技有限公司 | A kind of data distributing method, device and computing device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DK2858318T3 (en) * | 2013-09-09 | 2018-02-12 | Nex Services North America Llc Corp Trust Center | Process and apparatus for order entry in an electronic trading system |
-
2019
- 2019-05-30 CN CN201910461150.5A patent/CN110209690B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1575484A (en) * | 2001-09-24 | 2005-02-02 | 尼奥普泰克技术公司 | Method for anonymous computerized processing of documents or objects |
| CN104781170A (en) * | 2012-10-15 | 2015-07-15 | 杰富意钢铁株式会社 | Shipping work planning system, shipping work planning method, and shipping work method |
| CN106844690A (en) * | 2017-01-23 | 2017-06-13 | 北京齐尔布莱特科技有限公司 | A kind of data distributing method, device and computing device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110209690A (en) | 2019-09-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI777010B (en) | Prediction of information conversion rate, information recommendation method and device | |
| CN110061930B (en) | Method and device for determining data flow limitation and flow limiting values | |
| CN110825977A (en) | Data recommendation method and related equipment | |
| CN110764986A (en) | Page fluency detection method and device | |
| CN110020383B (en) | Page data request processing method and device | |
| CN111898380A (en) | Text matching method and device, electronic equipment and storage medium | |
| CN107656918A (en) | Obtain the method and device of targeted customer | |
| CN110209690B (en) | Data acquisition method and device and electronic equipment | |
| CN112527615A (en) | Equipment determination method and device, electronic equipment and storage medium | |
| CN111079061A (en) | Data processing method and electronic equipment | |
| CN113205578B (en) | Layer processing method, layer rendering method and related equipment | |
| CN109615465B (en) | Service order processing method and device and electronic equipment | |
| CN112527538B (en) | Device updating method and device, electronic device and storage medium | |
| CN110309421B (en) | UGC content quality evaluation method and device and electronic equipment | |
| CN113766333B (en) | Method and device for determining video heat value, electronic equipment and storage medium | |
| CN112328081A (en) | File searching method and device, electronic equipment and readable storage medium | |
| CN112069164A (en) | Data query method and device, electronic equipment and computer readable storage medium | |
| CN111832964B (en) | User rating method and device and electronic equipment | |
| CN113849133B (en) | Processing method and device of privacy data, electronic equipment and storage medium | |
| CN109255077B (en) | Account push method and device | |
| CN113572824B (en) | Dynamic updating method and device for timeout duration, electronic equipment and storage medium | |
| CN113806737B (en) | Malicious process risk level assessment method, terminal equipment and storage medium | |
| CN109190352B (en) | Method and device for verifying accuracy of authorization text | |
| CN109324797B (en) | Desktop icon generation method, computer readable storage medium and terminal equipment | |
| CN113704237A (en) | Abnormal data detection method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |