CN111046077A - Data acquisition method and device, storage medium and terminal - Google Patents
Data acquisition method and device, storage medium and terminal Download PDFInfo
- Publication number
- CN111046077A CN111046077A CN201911040235.2A CN201911040235A CN111046077A CN 111046077 A CN111046077 A CN 111046077A CN 201911040235 A CN201911040235 A CN 201911040235A CN 111046077 A CN111046077 A CN 111046077A
- Authority
- CN
- China
- Prior art keywords
- data
- time period
- preset
- matched
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses a data acquisition method and device, a storage medium and a terminal, relates to the technical field of data processing, and mainly aims to process and store historical data and real-time data respectively through two calculation modes of batch processing and stream processing. The method comprises the following steps: identifying whether a data acquisition time period carried in the received data acquisition request is matched with a preset data node time length; if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation; and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing. The invention is suitable for data acquisition.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for acquiring data, a storage medium, and a terminal.
Background
With the development of internet information technology, the role of business data in the expansion of daily business and business maintenance is increasing day by day, and the core index data is used for conducting business guidance and driving business development, thereby playing a vital role in the process of daily business operation of internet companies.
At present, the existing data acquisition method needs a system background to store all detail data in a period of time, and when a user queries data in any period of time, all the stored detail data are summarized to obtain the data. However, since the user needs to perform summary calculation on all the data again each time the user acquires the data, the data acquisition cost is increased, the data acquisition efficiency is reduced, and the calculation resources are greatly wasted.
Disclosure of Invention
In view of the above, the present invention provides a data acquisition method and apparatus, a storage medium, and a terminal, and mainly aims to process and store historical data and real-time data in two calculation manners, i.e., batch processing and stream processing, respectively, and when acquiring data, the data acquisition method and apparatus can directly correspond to an acquisition processing result to reduce the amount of repeated calculation, thereby improving the data acquisition efficiency, reducing the data acquisition cost, and saving the calculation resources.
According to an aspect of the present invention, there is provided a data acquisition method, including:
identifying whether a data acquisition time period carried in the received data acquisition request is matched with a preset data node time length;
if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation;
and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing.
Optionally, the identifying whether a data acquisition time period carried in the received data acquisition request matches a preset data node duration includes:
extracting a data acquisition time period carried in the received data acquisition request;
and comparing the data acquisition time period with a preset data node time length, and judging whether the data time period is matched with the preset data node time length.
Optionally, the method further comprises:
and according to the preset time interval, obtaining historical data synchronously, carrying out batch calculation processing on different service indexes, and storing the processing result.
Further, the storing the processing result includes:
and storing the processing result into different partitions of the first preset storage position by taking the statistical time batch as an identifier.
Further, before identifying whether the data acquisition time period carried in the received data acquisition request matches a preset data node duration, the method further includes:
and according to the real-time data obtained by real-time synchronization, performing stream calculation processing by using the synchronous time batch as an identifier, and storing a processing result into a second preset storage position.
Further, the acquiring data matched with the acquisition time period from the data obtained through batch calculation processing includes:
acquiring data matched with the data acquisition time period from the first preset storage position according to the statistical time batch as an identifier;
the acquiring of the data matching the acquisition time period from the data obtained through the stream calculation processing includes:
and acquiring data matched with the data acquisition time period from the second preset storage position according to the synchronous time batch as the identifier.
Further, the method further comprises:
and respectively sending a synchronous historical data request and a synchronous real-time data request to a service database so as to update data according to the historical data and the real-time data acquired from the service database.
According to another aspect of the present invention, there is provided an apparatus for acquiring data, comprising:
the identification unit is used for identifying whether a data acquisition time period carried in the received data acquisition request is matched with the duration of a preset data node;
the first acquisition unit is used for acquiring data matched with the acquisition time period from the data obtained by batch calculation if the data acquisition time period is not matched with the preset data node duration;
and the second acquisition unit is used for acquiring data matched with the acquisition time period from the data obtained by the flow calculation processing if the data acquisition time period is matched with the preset data node time period.
Further, the identification unit includes:
the extraction module is used for extracting the data acquisition time period carried in the received data acquisition request;
and the comparison module is used for comparing the data acquisition time period with the preset data node duration and judging whether the data time period is matched with the preset data node duration.
Optionally, the apparatus further comprises:
and the first processing unit is used for synchronously obtaining historical data according to a preset time interval, carrying out batch calculation processing on different service indexes and storing processing results.
Further, the first processing unit is specifically configured to store the processing result in different partitions of the first preset storage location by using the statistical time batch as an identifier.
Optionally, the apparatus further comprises:
and the second processing unit is used for performing stream calculation processing by taking the synchronous time batch as an identifier according to the real-time data obtained by real-time synchronization and storing a processing result into a second preset storage position.
Further, the first obtaining unit is specifically configured to obtain, from the first preset storage location, data that matches the data obtaining time period according to the statistical time batch as an identifier;
further, the second obtaining unit is specifically configured to obtain, from the second preset storage location, data that matches the data obtaining time period according to the synchronization time batch as an identifier.
Optionally, the apparatus further comprises:
and the sending unit is used for respectively sending a synchronous historical data request and a synchronous real-time data request to a service database so as to update data according to the historical data and the real-time data acquired from the service database.
According to another aspect of the present invention, a storage medium is provided, and the storage medium stores at least one executable instruction, which causes a processor to execute operations corresponding to the data acquisition method.
According to still another aspect of the present invention, there is provided a terminal including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data acquisition method.
By the technical scheme, the technical scheme provided by the embodiment of the invention at least has the following advantages:
compared with the prior art that a background of a system stores all detail data in a period of time, and when a user inquires data in any time period, all the stored detail data are collected to obtain data, the invention identifies whether the data acquisition time period carried in a received data acquisition request is matched with the duration of a preset data node or not; if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation; and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing. The historical data and the real-time data can be respectively processed and stored through two calculation modes of batch processing and stream processing, and when the data are acquired, the processing results can be directly and correspondingly acquired so as to reduce repeated calculation amount, thereby improving the data acquisition efficiency, reducing the data acquisition cost and saving calculation resources.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 is a flowchart illustrating a data acquisition method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data acquisition method provided by the embodiment of the invention;
FIG. 3 is a block diagram of a data acquisition module according to an embodiment of the present invention;
FIG. 4 is a block diagram of an apparatus for acquiring data according to an embodiment of the present invention;
FIG. 5 is a block diagram of another data acquisition apparatus according to an embodiment of the present invention;
fig. 6 shows a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
As described in the background art, in the existing data acquisition method, a system background is required to store all detail data in a period of time, and when a user queries data in any period of time, all the stored detail data are summarized to obtain the data. However, since the user needs to perform summary calculation on all the data again each time the user acquires the data, the processing cost of the data is increased, the processing efficiency of the data is reduced, and the calculation resources are greatly wasted.
In order to solve the above problem, an embodiment of the present invention provides a data acquisition method, as shown in fig. 1, the method includes:
101. and identifying whether the data acquisition time period carried in the received data acquisition request is matched with the preset data node time length.
The data acquisition request may be sent from the front end to the server, and the data may be service index data, which may specifically include order number, passenger flow volume, and the like. After receiving the data acquisition request, a data acquisition time period carried in the data acquisition request may be extracted, and the data acquisition time period is compared with a preset data node duration, where the data acquisition time period may be a time interval in which data to be acquired is located, for example, the time period may be 9 months and 1 days, or 9 months and 1 days 07:00 to 15: 00, the preset data node time length may be an interval time length from the current system time, and may be set according to a service requirement, for example, may be specifically set within 2 days from the current system time, or within 12 hours from the current system time, and the like. If the data acquisition time period is within the preset data node time length range, it can be determined that the data acquisition time period is matched with the preset data node time length.
102. And if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained by batch calculation.
The batch calculation processing may be to process the historical data according to a preset calculation rule at a preset time interval, and the processing result may be stored in a first preset storage location, where the first preset storage location may include a data list or other specified data structures. For example, in the case of 9/month/1/day, all the order data of 8/month/31/day may be summed up to obtain the total order amount of 8/month/31/day, and the total order amount data may be stored in the first preset storage location. Specifically, if the data obtaining time period does not match the preset data node duration, data matching the obtaining time period may be obtained from data obtained through batch processing according to a corresponding relationship, and the first preset storage location may store the corresponding relationship between the data obtaining time period and the data, and may store data counted in different time periods.
103. And if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing.
The stream calculation processing may be processing real-time data according to a preset calculation rule when receiving the real-time data, where the processing result may be stored in a second preset storage location, and the second preset storage location may include a data list or other specified data structures. For example, every time a new order data is received in 9 months and 1 days, all the order data received in 9 months and 1 days can be added to obtain the order total data in 9 months and 1 days, and the order total data is stored in the second preset storage position. Specifically, if the data obtaining time period is matched with the duration of the preset data node, the data matched with the data obtaining time period may be searched in the second data list according to the corresponding relationship, and the second data list may store data synchronized in real time at different time periods.
The invention provides a data acquisition method, which is characterized in that whether a data acquisition time period carried in a received data acquisition request is matched with a preset data node time length is identified; if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation; and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing. The historical data and the real-time data can be respectively processed and stored through two calculation modes of batch processing and stream processing, and when the data are acquired, the processing results can be directly and correspondingly acquired so as to reduce repeated calculation amount, thereby improving the data acquisition efficiency, reducing the data acquisition cost and saving calculation resources.
An embodiment of the present invention provides another data acquisition method, as shown in fig. 2, the method includes:
and 201a, synchronously obtaining historical data according to a preset time interval, carrying out batch calculation processing on different service indexes, and storing a processing result.
For the embodiment of the present invention, the execution subject of the service index statistics may be a batch calculation engine, and specifically may include an offline ETL technology, which may be used to perform statistics on different service indexes according to the synchronized historical data, and store the statistical result in the first preset storage location. The preset time interval may be set to be synchronized once a day, or once every 12 hours, and the like, and the data synchronized in the preset time interval may be determined as historical data; the service index may be data obtained by processing the historical data according to a calculation algorithm of different service indexes, and specifically may include the number of orders, the passenger flow volume, and the like; the first preset storage location may specifically be a batch partition table. Specifically, data are synchronized to a service database according to a preset time interval, the historical data are processed by using calculation algorithms of different service indexes, and a processing result is stored in the first preset storage position.
For further limitation and extension, the storing the statistical result to the first preset storage location in the embodiment of the present invention includes: and storing the processing result into different partitions of the first preset storage position by taking the statistical time batch as an identifier.
For the embodiment of the present invention, in order to facilitate searching for corresponding data in the first preset storage location, the statistical time batches are used as identifiers to store the statistical results in different partitions of the first preset storage location. The first preset storage location may be a batch partition table, and the statistical time batches of the data in the list are used as identifiers for partition storage. When data is searched, the corresponding time batch identifier can be searched in the first preset storage location according to the data acquisition time period, so as to acquire data quickly.
For example, after statistics is performed according to the historical data of 2019, month and 1, the order number of the 2019, month and 1 is obtained, the order number of the month and 1 can be stored in a partition marked as 20190901 in a batch partition table, and when data is searched, the time period of 2019, month and 1 can be obtained according to the data, a corresponding time batch mark 20190901 is searched in the batch partition table, and then the corresponding data is searched.
For the embodiment of the present invention, step 201b, which is parallel to step 201 a: and updating to the second preset storage position by taking the synchronous time batch as an identifier according to the real-time data obtained by real-time synchronization.
For the embodiment of the present invention, the execution subject of the synchronous update may be a stream computing engine, and may be configured to synchronize the real-time data and update the real-time data to the second preset storage location. Specifically, data is synchronized in real time to a service database, the data can be determined as real-time data, the real-time data is stored in the second preset storage location, and a synchronization time batch is used as an identifier. When data is searched, the corresponding synchronous time batch identifier can be searched in the second preset storage location according to the data acquisition time period, so as to acquire data quickly. The second preset storage location may be a time index table, and the synchronized time batch may be used as an index of the time index table.
202. And identifying whether the data acquisition time period carried in the received data acquisition request is matched with the preset data node time length.
This step is the same as step 101 shown in fig. 1, and is not described herein again.
For the embodiment of the present invention, the step 201 may specifically include: extracting a data acquisition time period carried in the received data acquisition request; and comparing the data acquisition time period with a preset data node time length, and judging whether the data time period is matched with the preset data node time length.
For example, if the current system time is 2019, month 9 and day 3, the data acquisition time period is 2019, month 9 and day 1, and the preset data node duration is within 2 days of the current system time, the data acquisition time period is compared with the preset data node duration, so that the data acquisition time period is not matched with the preset data node; and if the data acquisition time period is 2019, 9, month and 2, the data acquisition time period can be matched with the preset data node.
203a, if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained by batch calculation.
The first preset storage location stores data counted in different time periods, which can be service indexes counted in different time periods, and each time period only stores one service index data processed in the time period. For example, in the 20190901 partition in the data list, only one service index data of 2019, 9, 1 and the like is stored, and the data can be directly extracted for feedback when the data needs to be acquired. Therefore, the service index data can be prevented from being recalculated every time data is acquired, the data acquisition efficiency can be improved, and the calculation resources are saved.
203b, if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing.
For example, as shown in fig. 3, which shows a data acquisition module structure diagram, the service database may synchronize real-time data to a stream computation engine, and the stream computation engine may store the data in a real-time index table after data processing. After receiving a data acquisition request sent by a front end, the real-time index table may search for corresponding data according to a matching result of the data acquisition time period and the data node duration, and respond to the data acquisition request by using the data.
For the embodiment of the invention, in order to improve the efficiency of searching data in the first preset storage position and the second preset storage position, data matched with the data acquisition time period is searched in the second preset storage position according to the statistical time batch as the identifier, and feedback is performed; and searching data matched with the data acquisition time period in the second preset storage position according to the synchronous time batch as the identifier, and feeding back.
For example, if the data obtaining time period is 2019, 9, 1, a storage partition with a statistical time batch identifier of 20190901 may be correspondingly searched in the second preset storage location according to the data obtaining time period, data in the partition may be extracted, and the data may be fed back.
It should be noted that, after the data matched with the data acquisition time period is searched in the second preset storage location according to the synchronization time batch as the identifier, the data may be processed according to the calculation methods of different service indexes, and the processing result is fed back.
For example, the current system time is 2019, month and 1 day, the preset time node duration is 2 days, if the order number of the 2019, month and 1 day needs to be acquired, a storage partition with a synchronous time batch identifier of 20190901 is searched in the second preset storage location according to the synchronous time batch, real-time data in the partition is extracted, the real-time data is summarized according to a calculation algorithm of the order number, and the order number of the 2019, month and 1 day is acquired and fed back.
Further, in order to accurately acquire the service index data, the embodiment of the present invention may further include: and respectively sending a synchronous historical data request and a synchronous real-time data request to a service database so as to update the first preset storage position and the second preset storage position according to the historical data and the real-time data acquired from the service database.
Specifically, a synchronous historical data request may be sent to the service database according to a preset time interval to obtain historical data; sending a synchronous implementation data request to the database to acquire real-time data; processing the historical data according to calculation algorithms of different service indexes, and updating the first preset storage position by using the processing result; and updating the first preset storage position by using the real-time data.
The invention provides another data acquisition method, which is characterized in that whether a data acquisition time period carried in a received data acquisition request is matched with the duration of a preset data node is identified; if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation; and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing. The historical data and the real-time data can be respectively processed and stored through two calculation modes of batch processing and stream processing, and when the data are acquired, the processing results can be directly and correspondingly acquired so as to reduce repeated calculation amount, thereby improving the data acquisition efficiency, reducing the data acquisition cost and saving calculation resources.
Further, as an implementation of the method shown in fig. 1, an embodiment of the present invention provides an apparatus for acquiring data, as shown in fig. 4, the apparatus includes: a recognition unit 31, a first acquisition unit 32, and a second acquisition unit 33.
The identifying unit 31 may be configured to identify whether a data acquisition time period carried in the received data acquisition request matches a preset data node duration;
the first obtaining unit 32 may be configured to, if the data obtaining time period does not match the preset data node duration, obtain data matching the obtaining time period from data obtained through batch processing;
the second obtaining unit 33 may be configured to, if the data obtaining time period matches the preset data node time period, obtain data matching the obtaining time period from data obtained through stream calculation processing.
Compared with the prior art that a background of a system stores all detail data in a period of time, and when a user inquires data in any time period, all the stored detail data are summarized to obtain the data, the data acquisition device identifies whether the data acquisition time period carried in a received data acquisition request is matched with the duration of a preset data node; if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation; and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing. The historical data and the real-time data can be respectively processed and stored through two calculation modes of batch processing and stream processing, and when the data are acquired, the processing results can be directly and correspondingly acquired so as to reduce repeated calculation amount, thereby improving the data acquisition efficiency, reducing the data acquisition cost and saving calculation resources.
Further, as an implementation of the method shown in fig. 2, an embodiment of the present invention provides another data obtaining apparatus, as shown in fig. 5, where the apparatus includes: a recognition unit 41, a first acquisition unit 42, a second acquisition unit 43, a first processing unit 44, a second processing unit 45, a transmission unit 46.
The identifying unit 41 may be configured to identify whether a data acquisition time period carried in the received data acquisition request matches a preset data node duration;
a first obtaining unit 42, configured to obtain, if the data obtaining time period does not match the preset data node time length, data that matches the obtaining time period from data obtained through batch processing;
the second obtaining unit 43 may be configured to, if the data obtaining time period matches the preset data node time period, obtain data matching the obtaining time period from data obtained through stream calculation processing.
Further, the identification unit 41 includes:
the extracting module 411 may be configured to extract a data acquisition time period carried in the received data acquisition request;
the comparing module 412 may be configured to compare the data obtaining time period with a preset data node time length, and determine whether the data obtaining time period is matched with the preset data node time length.
Further, the apparatus further comprises:
the first processing unit 44 may be configured to synchronously obtain historical data according to a preset time interval, perform batch calculation processing on different service indicators, and store a processing result.
Further, the first processing unit 44 is specifically configured to store the processing result in different partitions of the first preset storage location by using the statistical time batch as an identifier.
Further, the apparatus further comprises:
the second processing unit 45 may be configured to perform stream calculation processing with the synchronization time batch as an identifier according to the real-time data obtained by real-time synchronization, and store a processing result in a second preset storage location.
For the embodiment of the present invention, the first obtaining unit 42 may be specifically configured to obtain, from the first preset storage location, data matched with the data obtaining time period according to the statistical time batch as an identifier;
for the embodiment of the present invention, the second obtaining unit 43 may be specifically configured to obtain, from the second preset storage location, data matched with the data obtaining time period according to the synchronization time batch as an identifier.
Further, the apparatus further comprises:
the sending unit 46 may be configured to send a synchronous historical data request and a synchronous real-time data request to the service database, respectively, so as to update data according to the historical data and the real-time data acquired from the service database.
The invention provides another data acquisition device, and the embodiment of the invention identifies whether the data acquisition time period carried in the received data acquisition request is matched with the duration of a preset data node; if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation; and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing. The historical data and the real-time data can be respectively processed and stored through two calculation modes of batch processing and stream processing, and when the data are acquired, the processing results can be directly and correspondingly acquired so as to reduce repeated calculation amount, thereby improving the data acquisition efficiency, reducing the data acquisition cost and saving calculation resources.
According to an embodiment of the present invention, a storage medium is provided, where at least one executable instruction is stored, and the computer executable instruction can execute the data obtaining method in any of the above method embodiments.
Fig. 6 is a schematic structural diagram of a terminal according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the terminal.
As shown in fig. 6, the terminal may include: a processor (processor)502, a communication interface 504, a memory 506, and a communication bus 508.
Wherein: the processor 502, communication interface 504, and memory 506 communicate with one another via a communication bus 508.
A communication interface 504 for communicating with network elements of other devices, such as clients or other servers.
The processor 502 is configured to execute the program 510, and may specifically perform relevant steps in the above-described embodiment of the entity information tagging processing method.
In particular, program 510 may include program code that includes computer operating instructions.
The processor 502 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement an embodiment of the invention. The terminal comprises one or more processors, which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 506 for storing a program 510. The memory 506 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 510 may specifically be used to cause the processor 502 to perform the following operations:
identifying whether a data acquisition time period carried in the received data acquisition request is matched with a preset data node time length; if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation; and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for obtaining data, comprising:
identifying whether a data acquisition time period carried in the received data acquisition request is matched with a preset data node time length;
if the data acquisition time period is not matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through batch calculation;
and if the data acquisition time period is matched with the preset data node time length, acquiring data matched with the acquisition time period from the data obtained through flow calculation processing.
2. The method according to claim 1, wherein the identifying whether the data acquisition time period carried in the received data acquisition request matches a preset data node duration includes:
extracting a data acquisition time period carried in the received data acquisition request;
and comparing the data acquisition time period with a preset data node time length, and judging whether the data time period is matched with the preset data node time length.
3. The method according to claim 1, wherein before identifying whether the data acquisition time period carried in the received data acquisition request matches a preset data node time period, the method further comprises:
and according to the preset time interval, obtaining historical data synchronously, carrying out batch calculation processing on different service indexes, and storing the processing result.
4. The method of claim 3, wherein storing the processing results comprises:
and storing the processing result into different partitions of the first preset storage position by taking the statistical time batch as an identifier.
5. The method according to claim 4, wherein before identifying whether the data acquisition time period carried in the received data acquisition request matches a preset data node time duration, the method further comprises:
and according to the real-time data obtained by real-time synchronization, performing stream calculation processing by using the synchronous time batch as an identifier, and storing a processing result into a second preset storage position.
6. The method of claim 5, wherein said obtaining data from the batch process that matches the time period of the obtaining comprises:
acquiring data matched with the data acquisition time period from the first preset storage position according to the statistical time batch as an identifier;
the acquiring of the data matching the acquisition time period from the data obtained through the stream calculation processing includes:
and acquiring data matched with the data acquisition time period from the second preset storage position according to the synchronous time batch as the identifier.
7. The method according to any one of claims 1-6, characterized in that the method further comprises:
and respectively sending a synchronous historical data request and a synchronous real-time data request to a service database so as to update data according to the historical data and the real-time data acquired from the service database.
8. An apparatus for processing data, comprising:
the identification unit is used for identifying whether a data acquisition time period carried in the received data acquisition request is matched with the duration of a preset data node;
the first acquisition unit is used for acquiring data matched with the acquisition time period from the data obtained by batch calculation if the data acquisition time period is not matched with the preset data node duration;
and the second acquisition unit is used for acquiring data matched with the acquisition time period from the data obtained by the flow calculation processing if the data acquisition time period is matched with the preset data node time period.
9. A storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the method of acquiring data according to any one of claims 1 to 7.
10. A terminal, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the data acquisition method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911040235.2A CN111046077A (en) | 2019-10-29 | 2019-10-29 | Data acquisition method and device, storage medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911040235.2A CN111046077A (en) | 2019-10-29 | 2019-10-29 | Data acquisition method and device, storage medium and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111046077A true CN111046077A (en) | 2020-04-21 |
Family
ID=70231805
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911040235.2A Pending CN111046077A (en) | 2019-10-29 | 2019-10-29 | Data acquisition method and device, storage medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111046077A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112256734A (en) * | 2020-10-20 | 2021-01-22 | 中国农业银行股份有限公司 | Big data processing method, device, system, equipment and storage medium |
| CN113312367A (en) * | 2021-06-25 | 2021-08-27 | 上海致景信息科技有限公司 | Yield efficiency recalculation method, apparatus, storage medium and processor |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761309A (en) * | 2014-01-23 | 2014-04-30 | 中国移动（深圳）有限公司 | Operation data processing method and system |
| CN106528717A (en) * | 2016-10-26 | 2017-03-22 | 中国电子产品可靠性与环境试验研究所 | Data processing method and system |
| WO2017166713A1 (en) * | 2016-03-30 | 2017-10-05 | 乐视控股（北京）有限公司 | Service request processing method and device |
| CN108629029A (en) * | 2018-05-09 | 2018-10-09 | 北京京东尚科信息技术有限公司 | A kind of data processing method and device applied to data warehouse |
| CN109597842A (en) * | 2018-12-14 | 2019-04-09 | 深圳前海微众银行股份有限公司 | Data real-time computing technique, device, equipment and computer readable storage medium |
| CN109829009A (en) * | 2018-12-28 | 2019-05-31 | 北京邮电大学 | Configurable isomeric data real-time synchronization and visual system and method |
| CN109951463A (en) * | 2019-03-07 | 2019-06-28 | 成都古河云科技有限公司 | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column |
-
2019
- 2019-10-29 CN CN201911040235.2A patent/CN111046077A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761309A (en) * | 2014-01-23 | 2014-04-30 | 中国移动（深圳）有限公司 | Operation data processing method and system |
| WO2017166713A1 (en) * | 2016-03-30 | 2017-10-05 | 乐视控股（北京）有限公司 | Service request processing method and device |
| CN106528717A (en) * | 2016-10-26 | 2017-03-22 | 中国电子产品可靠性与环境试验研究所 | Data processing method and system |
| CN108629029A (en) * | 2018-05-09 | 2018-10-09 | 北京京东尚科信息技术有限公司 | A kind of data processing method and device applied to data warehouse |
| CN109597842A (en) * | 2018-12-14 | 2019-04-09 | 深圳前海微众银行股份有限公司 | Data real-time computing technique, device, equipment and computer readable storage medium |
| CN109829009A (en) * | 2018-12-28 | 2019-05-31 | 北京邮电大学 | Configurable isomeric data real-time synchronization and visual system and method |
| CN109951463A (en) * | 2019-03-07 | 2019-06-28 | 成都古河云科技有限公司 | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112256734A (en) * | 2020-10-20 | 2021-01-22 | 中国农业银行股份有限公司 | Big data processing method, device, system, equipment and storage medium |
| CN113312367A (en) * | 2021-06-25 | 2021-08-27 | 上海致景信息科技有限公司 | Yield efficiency recalculation method, apparatus, storage medium and processor |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110321387B (en) | Data synchronization method, equipment and terminal equipment | |
| CN106997431B (en) | Data processing method and device | |
| CN111586126A (en) | Method, device and equipment for pre-downloading small program and storage medium | |
| CN110012049B (en) | Information push method, system, server and computer readable storage medium | |
| CN111046077A (en) | Data acquisition method and device, storage medium and terminal | |
| CN108733790B (en) | Data sorting method, device, server and storage medium | |
| CN114647698A (en) | Data synchronization method and device and computer storage medium | |
| CN110990350B (en) | Log analysis method and device | |
| CN110460668B (en) | File uploading method and device, computer equipment and storage medium | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN107871055B (en) | Data analysis method and device | |
| CN111475532A (en) | Data processing optimization method and device, storage medium and terminal | |
| CN110807035A (en) | Serial number generation method, device, equipment and storage medium | |
| CN113901037A (en) | Data management method, device and storage medium | |
| CN110515979B (en) | Data query method, device, equipment and storage medium | |
| CN115658750A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN110363515B (en) | Rights and interests card account information inquiry method, system, server and readable storage medium | |
| CN109086279B (en) | Report caching method and device | |
| CN109783580A (en) | Method of data synchronization and device | |
| CN114547184A (en) | Personnel information synchronization method, terminal device and storage medium | |
| CN112532414A (en) | Method, device and equipment for determining ISP attribution and computer storage medium | |
| CN111061719A (en) | Data collection method, device, equipment and storage medium | |
| CN114153830B (en) | Data verification method and device, computer storage medium and electronic equipment | |
| CN112738207B (en) | Keyword data transmission method and device, storage medium and electronic device | |
| CN110633430A (en) | Event discovery method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |