CN112764988B - Data segment acquisition method and device - Google Patents
Data segment acquisition method and device Download PDFInfo
- Publication number
- CN112764988B CN112764988B CN202110021400.0A CN202110021400A CN112764988B CN 112764988 B CN112764988 B CN 112764988B CN 202110021400 A CN202110021400 A CN 202110021400A CN 112764988 B CN112764988 B CN 112764988B
- Authority
- CN
- China
- Prior art keywords
- data
- acquisition
- record
- node
- segment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 230000001419 dependent effect Effects 0.000 claims description 15
- 238000013480 data collection Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 abstract description 9
- 230000000875 corresponding effect Effects 0.000 description 25
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data segment collection method, which can be used for collecting source data of a plurality of platform servers, dividing the source data to be collected into a plurality of segments of segment data for segment collection, recording log records corresponding to each segment collection task by using a log table so as to accurately position segment nodes of the segment data, ensuring that each collection is continuously collected from the segment nodes of the segment data collected last time, and avoiding missing collection and repeated collection of the data; and the scheme processes the dependency relationship of the segmented data, so that the integrity of the acquired data is ensured.
Description
Technical Field
The present invention relates to the field of data acquisition, and in particular, to a method and apparatus for data segment acquisition.
Background
In order to make a system administrator know the service condition of the software system in time, specific data generated by the operation of the software system needs to be collected so as to count and analyze the specific data. In other words, data collection is the process of extracting valuable data from the target system software and placing the data in a database in a structured format, where consistency and integrity of the collected data will greatly affect the statistics and analysis results of the data.
However, the current data acquisition method, in particular, the data acquisition method for mass data has the following problems:
first, for a data acquisition task that is fully acquired once, the data acquisition task is easily broken due to various reasons, such as network interruption, network delay, etc., which can cause the interruption of the data acquisition task, and once the data acquisition task is broken, the phenomenon that data acquisition is missed or repeated is easily caused when the data acquisition task is re-acquired, so that the acquired data and the source data are inconsistent, and the consistency of the image data is caused. Particularly, for mass data, the duration of the data acquisition task is longer, and correspondingly, the situation that the data acquisition is interrupted can occur with higher probability in the process of the data acquisition task.
Secondly, the data come from a plurality of software systems, and a certain data dependency relationship exists among the plurality of data, if the phenomenon of data acquisition interruption occurs, or the phenomenon of incomplete acquired data is easy to occur by adopting a method for acquiring the data of different system software in a segmented way.
Disclosure of Invention
The invention aims to provide a data segment acquisition method and a device, wherein the data segment acquisition method can be used for multi-system data acquisition, can ensure the consistency and the integrity of the data acquisition, and can reduce the pressure of a server and improve the efficiency of subsequent statistical analysis compared with the traditional one-time full acquisition mode.
In order to achieve the above purpose, the present technical solution provides a data segment collection method, including the following steps:
acquiring a previous segmented acquisition log record corresponding to a previous acquisition node in an acquisition database, wherein the previous segmented acquisition log record at least records a segmented record starting node and a segmented record ending node of segmented data acquired by the previous acquisition node, and an acquisition node segment is arranged between the current acquisition node and the previous acquisition node;
acquiring the first generated data record and the last generated data record in a platform database in the acquisition node section, and determining the acquisition starting point position in the following manner:
if the previous segment acquisition log record does not record the data of the data record, taking the data acquisition node of the data record generated at first as the acquisition starting point position;
if the data of the previous acquisition log record and the last generated data record are intersected, taking the segment record ending node in the previous acquisition log record as the acquisition starting point position;
and the current acquisition node starts to acquire segmented data by using the acquisition starting point position, and records segmented acquisition log records corresponding to the current acquisition node.
In a second aspect, an application scenario of a data segment collection method is provided.
In a third aspect, a data segment collection apparatus for running a data segment collection method is provided, including:
the acquisition database is used for storing segmented acquisition log records, wherein the segmented acquisition log records at least record segmented record starting nodes and segmented ending nodes of segmented data;
the platform database is used for storing data records;
the log acquisition unit is used for acquiring a previous segmented acquisition log record corresponding to a previous acquisition node in an acquisition database, wherein the previous segmented acquisition log record at least records a segmented record starting node and a segmented record ending node of segmented data acquired by the previous acquisition node, and an acquisition node segment is arranged between the current acquisition node and the previous acquisition node;
the data record acquisition unit acquires the data record generated first and the data record generated last in the platform database in the acquisition node section, and determines the acquisition starting point position in the following manner:
if the previous segment acquisition log record does not record the data of the data record, taking the data acquisition node of the data record generated at first as the acquisition starting point position;
if the data of the previous acquisition log record and the last generated data record are intersected, taking the segment record ending node in the previous acquisition log record as the acquisition starting point position;
and the current acquisition node starts to acquire segmented data by using the acquisition starting point position, and records segmented acquisition log records corresponding to the current acquisition node.
Compared with the prior art, the technical scheme has the following characteristics and beneficial effects;
1) And consistency of the acquired data and the source data is ensured. According to the scheme, the data are collected in a segmented mode, the segmented data are rapidly positioned in a mode of recording the node positions of the segmented data, the situation that each time of collection starts from the position where the last collection ends is avoided, the data are prevented from being collected in a missing mode or the situation that the data are collected in a repeated mode, meanwhile, the data among the multiple platform servers are stored in a correlated mode, and the data consistency is guaranteed.
2) And ensuring the data integrity of the acquired data. The scheme processes the dependency relationship among the platform data, ensures the sequence of data acquisition, and avoids the unilateral performance of data statistics.
3) The server pressure is reduced, and the subsequent data analysis efficiency is improved. Compared with the traditional one-time acquisition of all data, the data segmented acquisition can properly lighten the pressure of a server side, can collect the data of each platform, is convenient for the summarizing and counting task of the follow-up data, and improves the counting efficiency.
Drawings
Fig. 1 is a diagram of an operating system architecture of a data segment collection method according to the present invention.
Fig. 2 is a schematic diagram of segmented acquisition.
Fig. 3 is an application view showing a data segment collection method of the present invention.
Fig. 4 is a schematic diagram of a method for confirming a collection start node by using the data segment collection method of the present invention.
In the figure: 101-client, 102-platform server, 103-platform data, 104-acquisition server, 105-acquisition data.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which are derived by a person skilled in the art based on the embodiments of the invention, fall within the scope of protection of the invention.
According to the first aspect of the invention, a data segment collection method is provided, the method can be used for collecting source data of one or more platform servers, the source data to be collected is divided into a plurality of segments of segment data for segment collection, log records corresponding to each segment collection task are recorded by using a log table so as to accurately position segment nodes of the segment data, the fact that each collection is continuously carried out from the segment nodes of the segment data collected last time is ensured, and missing collection and repeated collection of the data are avoided; and the scheme processes the dependency relationship of the segmented data, so that the integrity of the acquired data is ensured.
Because the scheme performs segmented acquisition on the source data, the acquisition time of each segment is shorter, and compared with the traditional scheme of one-time data full acquisition, the probability of interruption of the data acquisition task is reduced; and even if the phenomenon of data acquisition interruption occurs in the scheme, the interruption position of the data can be accurately positioned when the data acquisition is resumed, so that the data is continuously acquired, and the consistency of the acquired data and the source data is ensured. In addition, the scheme determines the collected segmented data nodes based on the dependency relationship between the source data, fully considers the dependency relationship between the source data and ensures the integrity of the collected data.
As shown in fig. 1, a service framework of a data segment collection method provided by an embodiment of the present solution is shown, where the service framework includes a plurality of clients 101, a plurality of platform servers 102, a platform database 103, a collection server 104, and a collection database 105; the plurality of clients 101 and the platform server 102 establish communication connection, the platform server 102 and the acquisition server 104 establish communication connection, the platform server 102 stores data in the platform database 103, and the acquisition server 104 stores data in the acquisition data 105.
It will be appreciated that in other embodiments of the present application, the service framework may include one client 101, a plurality of platform servers 102, a platform database 103, an acquisition server 104, and an acquisition database 105; or a plurality of clients 101, a platform server 102, a platform database 103, an acquisition server 104 and an acquisition database 105; or a client 101, a platform server 102, a platform database 103, an acquisition server 104, and an acquisition database 105, which is not limited in this application.
Wherein the client 101 corresponds to a user terminal, the user may use the client 101 to interact with the platform service terminal 102 through a network or other communication protocols to receive or send messages, and the client 101 may be various electronic devices with a display screen and supporting information interaction, including, but not limited to, tablet computers, desktop computers, and the like. The platform server 102 may be a server providing various service support, the platform server 102 returns data to the client 101 based on a request of a user, and source data generated by interaction between the platform server 102 and the client 101 is stored in the platform database 103, where the source data is recorded in a form of a data record, and the data record at least includes: the data name of the source data, the data collection node, and the data content. The collection server 104 is a server for implementing data segment collection, acquires the source data in the platform database 103 according to the feature data feature segment to obtain the collection data, and stores the collection data and the corresponding collection log record in the collection database 105, where the segment collection log record at least includes: the collection node corresponds to the segmented collection task name of the collection node, the segmented record starting node and the segmented record ending node of the segmented data.
Correspondingly, the collection database 105 has built-in collection log tables, and the collection log records are stored in the collection log tables.
The structure of the log table is shown in the following table one:
table structure of table-log table
| Field name | Field type | Field interpretation |
| name | String | Segmented acquisition task name |
| startTime | Date | Segment recording start time |
| endTime | Date | Segment recording end time |
| operationTime | Date | Acquisition node |
Based on the above service framework, the data segment collection method provided by the scheme comprises the following steps:
acquiring a previous segmented acquisition log record corresponding to a previous acquisition node in an acquisition database 105, wherein the previous segmented acquisition log record comprises a segmented acquisition task name corresponding to the previous acquisition node, a segmented record starting node and a segmented record ending node of the segmented data, the previous acquisition node is the node closest to the current acquisition node, and an acquisition node segment is arranged between the current acquisition node and the previous acquisition node;
acquiring a first generated data record and a last generated data record in the platform database 103 in the acquisition node section, wherein the data records comprise a data name of source data, a data acquisition node and data content, and determining an acquisition starting point position in the following manner:
if the previous segment acquisition log record does not record the data of the data record, taking the data acquisition node of the data record generated at first as the acquisition starting point position;
if the data of the previous acquisition log record and the last generated data record are intersected, taking the segment record ending node in the previous acquisition log record as the acquisition starting point position;
and the current acquisition node starts to acquire segmented data by using the acquisition starting point position, and records segmented acquisition log records corresponding to the current acquisition node.
In this scheme, the platform database 103 stores source data generated by a plurality of clients 101, and the collection server 104 collects the source data in the platform database 103 in segments in the form of segment data at intervals of collection nodes. And in an embodiment of the present solution, the acquisition server 104 acquires the acquired data from the platform database 103 by acquiring the source data satisfying specific data characteristics in the manner mentioned above, so as to facilitate subsequent data analysis on the acquired data.
Specifically, the collection server 104 collects corresponding segment data at each collection node, and all data satisfying specific data characteristics are collected by the segment data collected by a plurality of collection nodes. In this scheme, the acquisition node is selected as an acquisition time node or an acquisition data capacity node.
When the acquisition node is an acquisition time node, the time is taken as the node to perform segmented acquisition. Illustratively, the duration of source data a is 10:00-11:00, 10:30 can be taken as the segmentation node of the source data A, and 10 is collected firstly: 00-10:30 source data, and collecting 10:30-11:00 source data.
When the acquisition node is a node for acquiring data capacity, the data capacity is taken as the node for segmented acquisition. For example, if the data capacity of the source data a is 100kb, 50kb is taken as a segment of the source data a to receive you, and 0-50kb of source data is collected first, and then 50-100kb of source data is collected.
In addition, after the acquisition server 104 completes the segmented acquisition task of each acquisition node, a segmented acquisition log record is generated and stored in the acquisition database 105, where the segmented acquisition log record includes an acquisition node, a segmented acquisition task name corresponding to the acquisition node, a segmented record start node and a segmented record end node of the segmented data. The segmented acquisition task names comprise task names of segmented acquisition tasks and data names of source data, so that data in the source data can be correspondingly identified through the data names, and the data acquisition tasks are correspondingly identified through the task names. The data collection task is set to collect source data meeting specific data characteristics. For example, if the data collection task a is set to collect source data meeting the characteristic data feature X, the collection server 104 collects all source data B1, B2/B3 meeting the characteristic data feature X, and the data names of the source data B1, B2, B3 are recorded in the collection process.
In an embodiment of the present solution, before "acquiring the previous segmented acquisition log record corresponding to the previous acquisition node in the acquisition database 105" further includes triggering a data acquisition task at the current acquisition node, where the data acquisition task includes a task name and an acquisition instruction. The data collection task is to collect source data with specific data characteristics from the platform database 103, and the collection server 105 resumes collecting source data after receiving the collection instruction.
The "acquire previous segment acquisition log record corresponding to previous acquisition node in acquisition database 105" includes: and based on the task name and the segmented acquisition task name in the acquisition log record, performing matching search based on the previous acquisition node and the acquisition node in the segmented acquisition log record, and acquiring the segmented acquisition log record which is matched with two conditions simultaneously as the previous segmented log record.
And in the scheme, the quick matching inquiry of the previous segmented acquisition log records can be performed through the SQL structure language.
When the data record generated first and the data record generated last in the platform database 103 in the collection node section are acquired, if the data corresponding to the data record has a dependency relationship, the data record generated last is determined according to the data-dependent data-generated last node, and at this time, the data is dependent data. Specifically, judging whether the data record generated at the last has a dependency relationship, if so, determining a last generation node of the dependent data by using the last generation node of the dependent data, and acquiring the data record generated at the last; and if the last generated data record has no dependency relationship, acquiring the last generated data record corresponding to the last generated node in the acquisition node section.
In this scheme, the platform servers 102 and the clients 101 communicate and interactively acquire source data, so that a dependency relationship exists in the source data of the internal part of the platform servers 102. And if the attribute source of the data corresponding to the data record comes from the dependent data, the data corresponding to the data record is considered to have a dependency relationship. That is, the dependency refers to: when collecting dependent data in the collection database 105, where the value of a certain attribute of the dependent data is to be derived from the dependent data, it is necessary to ensure that the dependent data has collected the data associated with the dependent data before the dependent data is stored, which is the relation of the dependent data to the dependent data. If a dependency exists, the time of the last collection node of the data being relied upon needs to be referenced.
If the previous segment acquisition log record does not record the data of the data record, taking the data acquisition node of the data record generated first as the acquisition starting point position comprises the following steps: and if the segment acquisition task name in the previous segment acquisition log record is not matched with the data name of the source data, taking a data acquisition node of the data record generated first as the acquisition starting point position. At this time, the acquisition server 104 has not previously acquired the source data, so the source data is acquired from the generation location of the source data.
The "if the data of the previous acquisition log record and the last generated data record intersect, and the segment record ending node in the previous acquisition log record is used as the acquisition starting point position" includes if the segment record ending node in the previous acquisition log record is smaller than the current acquisition node and not larger than the data acquisition node of the last data record.
If the segment record ending node in the previous acquisition log record is smaller than the current acquisition node, the segment record ending node indicates that the acquisition server 104 has acquired part or all of the source data; and if the segment record ending node in the previous acquisition log record is not greater than the data acquisition node of the latest data record, the acquisition server 104 only acquires part of the source data, and the segment record ending node in the previous acquisition log record is used as the acquisition starting point position.
Correspondingly, "if the segment record ending node in the previous collection log record is smaller than the current collection node and equal to the data collection node of the latest data record, the source data is not collected" is implied in the judgment rule, and this indicates that the collection server 104 collects all the source data.
The above manner of determining the "determination of the acquisition start position" can be visualized as shown in fig. 2: for the case that source data a belongs to "the segment record end node within the previous acquisition log record is smaller than the current acquisition node and equal to the data acquisition node of the most recent data record"; for the case that source data B and source data C belong to "the segment record end node within the previous acquisition log record is smaller than the current acquisition node and not greater than the data acquisition node of the latest data record"; for the case that the source data D belongs to "if the previous segment acquisition log record does not record the data of the data record", the data acquisition node of the data record generated first is used as the acquisition starting point position ".
The "the current acquisition node starts to acquire the segmented data with the acquisition start position" includes: the collection server 104 stores the collected data in the collection database 105, and stores the collection log corresponding to the current collection node into the collection database 105. It should be noted that the collected data and the segmented data collected by the collection server 104 previously are summarized to obtain data consistent with the source data.
In a second aspect, an application scheme of a data segment collection method is provided, where the data segment collection method can be applied to data collection of a platform server with multiple application terminals participating, and an application scenario of the data segment collection method applied to "managing use conditions of an online conference system" is provided in the scheme.
Specifically, the application scene is: the manager needs to acquire the "meeting scale situation of last week of each enterprise in the use process of the meeting system platform". That is, the number of participants of each enterprise on a conference system platform within a set period of time needs to be acquired, and the required source data are: and in the set time period, meeting information of enterprises and meeting information corresponding to each meeting.
In this application scenario, the platform server 102 is a conference system platform, the users log in on their respective clients 101 to use an online conference system, the participant information and the conference information are stored as source data in the platform database 103, and the acquisition server 104 acquires source data satisfying specific data characteristics from the platform database 103.
The scheme comprises the following steps:
setting a log table for storing segmented acquisition log records, wherein the segmented acquisition log record information comprises: the acquisition time corresponds to the segmented acquisition task name, the segmented recording start time and the segmented recording end time of the segmented data of the acquisition time;
triggering a data acquisition task, wherein the data acquisition task is used for acquiring the number of the participants of each enterprise of the online conference in the last week, and the data acquisition task starts to be executed and acquires data at intervals of acquisition time;
searching a segmented acquisition log record corresponding to the latest acquisition time before the current acquisition time in a log table by executing SQL, and defining as lastLog;
taking startTime in lastLog as start time and last time endTime of lastData as end time, acquiring earliest meeting record between current acquisition time and latest acquisition time from meeting system platform, defining as first data and latest meeting record as lastData. Because the conference records herein need to correlate the participant data on the video device platform, this approach ensures that the conference data collected in segments is complete and contains the participant data.
If the lastLog has no meeting record corresponding to the first data, starting to acquire the meeting record at the meeting starting time of the first data; if the endTime of the lastLog is smaller than the current acquisition time and the endTime of the lastLog is not greater than the startTime of the lastData, starting to acquire data from the endTime of the lastLog; storing the acquired data in the acquisition database 105 completes the acquisition of the segmented data.
The data acquisition task is repeated through the timing task polling, and the original data in the conference system are all acquired into the acquisition database, so that the consistency of the data on two sides of the data is ensured. And finally, counting meeting data according to the enterprise group through SQL to obtain the relationship data of the enterprise and the participants, returning to the front end and rendering the page. Thus, the statistical diagram shown in FIG. 3 is obtained. Specifically, since the collection database 105 stores all the collection data meeting the specific data characteristics in the set period (set by the collection time), the SQL is used to collect the source data corresponding to the specific enterprise, so that the number of participants in the enterprise can be obtained, and the relationship data can be displayed in a visual form.
In a third aspect, the present solution provides a data segment collection device, which uses the above data segment collection method to perform data segment collection, including:
the acquisition database is used for storing segmented acquisition log records, wherein the segmented acquisition log records at least record segmented record starting nodes and segmented ending nodes of segmented data;
the platform database is used for storing data records;
the log acquisition unit is used for acquiring a previous segmented acquisition log record corresponding to a previous acquisition node in an acquisition database, wherein the previous segmented acquisition log record at least records a segmented record starting node and a segmented record ending node of segmented data acquired by the previous acquisition node, and an acquisition node segment is arranged between the current acquisition node and the previous acquisition node;
the data record acquisition unit acquires the data record generated first and the data record generated last in the platform database in the acquisition node section, and determines the acquisition starting point position in the following manner:
if the previous segment acquisition log record does not record the data of the data record, taking the data acquisition node of the data record generated at first as the acquisition starting point position;
if the data of the previous acquisition log record and the last generated data record are intersected, taking the segment record ending node in the previous acquisition log record as the acquisition starting point position;
and the current acquisition node starts to acquire segmented data by using the acquisition starting point position, and records segmented acquisition log records corresponding to the current acquisition node.
It should be noted that the content of the data segment collection method operated by the data segment collection device is as described in the embodiment of the first aspect, and is not described in detail.
The computer system of the server for implementing the data segment collection method of the embodiment includes a central processing unit CPU) that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) or a program loaded from a storage section into a Random Access Memory (RAM). In the RAM, various programs and data required for the system operation are also stored. The CPU, ROM and RAM are connected to each other by a bus. An input/output (I/O) interface is also connected to the bus.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the data segment collection method shown in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via a communication portion, and/or installed from a removable medium. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. The modules involved in the embodiments of the present invention may be implemented in software, or may be implemented in hardware, and the described modules may also be disposed in a processor.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by one of the devices, cause the device to perform the flow steps corresponding to the data segment collection method. An electronic device is also provided; comprises a processor; and a memory in which is stored computer program instructions which, when executed by the processor, cause the processor to perform the data segment collection method mentioned in the first aspect above.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. The data segment acquisition method is characterized by comprising the following steps of: acquiring a previous segmented acquisition log record corresponding to a previous acquisition node in an acquisition database, wherein the previous segmented acquisition log record at least records a segmented record starting node and a segmented record ending node of segmented data acquired by the previous acquisition node, and an acquisition node segment is arranged between a current acquisition node and the previous acquisition node; acquiring the first generated data record and the last generated data record in a platform database in the acquisition node section, and determining the acquisition starting point position in the following manner: if the previous segment acquisition log record does not record the data of the data record, taking the data acquisition node of the data record generated at first as the acquisition starting point position; if the segment record ending node in the previous acquisition log record is smaller than the current acquisition node and not larger than the data acquisition node of the latest data record, taking the segment record ending node in the previous acquisition log record as the acquisition starting point position; if the segment record ending node in the previous acquisition log record is smaller than the current acquisition node and is equal to the data acquisition node of the latest data record, not acquiring data; and the current acquisition node starts to acquire segmented data by using the acquisition starting point position, and records segmented acquisition log records corresponding to the current acquisition node.
2. The method of claim 1, wherein if the previous segment acquisition log record does not record the data of the data record comprises: the segment collection task name in the previous segment collection log record and the data name of the data record are not matched.
3. The method of claim 1, wherein obtaining a previous segment collection log record corresponding to a previous collection node in a collection database comprises: triggering a data acquisition task at a current acquisition node, and searching the previous segmented acquisition log record based on the data acquisition task and the current acquisition node.
4. A data segment collection method according to claim 3, wherein the previous segment log record is obtained by performing a matching search based on the task name of the data collection task and the segment collection task name in the segment collection log record and based on the previous collection node and the collection node in the segment collection log record.
5. The method for data segment collection according to claim 1, wherein in "obtaining the first generated data record and the last generated data record in the platform database in the collection node segment", if the data corresponding to the data record has a dependency relationship, determining the last generated data record according to the last generated node of the dependent data on which the data depends.
6. The method of claim 5, wherein if the attribute of the data corresponding to the data record is derived from the dependent data, the data corresponding to the data record is considered to have a dependency relationship.
7. The method of claim 1, wherein the acquisition node is an acquisition time node or an acquisition data capacity node.
8. The method for data segment collection according to claim 1, wherein the method is applied to data collection of platform servers with multiple application terminals participating.
9. A data segment collection device, comprising: the acquisition database is used for storing segmented acquisition log records, wherein the segmented acquisition log records at least record segmented record starting nodes and segmented ending nodes of segmented data; the platform database is used for storing data records; the log acquisition unit acquires a previous segmented acquisition log record corresponding to a previous acquisition node in an acquisition database, wherein the previous segmented acquisition log record at least records a segmented record starting node and a segmented record ending node of segmented data acquired by the previous acquisition node, and an acquisition node segment is arranged between a current acquisition node and the previous acquisition node; the data record acquisition unit acquires the data record generated first and the data record generated last in the platform database in the acquisition node section, and determines the acquisition starting point position in the following manner: if the previous segment acquisition log record does not record the data of the data record, taking the data acquisition node of the data record generated at first as the acquisition starting point position; if the segment record ending node in the previous acquisition log record is smaller than the current acquisition node and not larger than the data acquisition node of the latest data record, taking the segment record ending node in the previous acquisition log record as the acquisition starting point position; if the segment record ending node in the previous acquisition log record is smaller than the current acquisition node and is equal to the data acquisition node of the latest data record, not acquiring data; and the current acquisition node starts to acquire segmented data by using the acquisition starting point position, and records segmented acquisition log records corresponding to the current acquisition node.
10. An electronic device, comprising: a processor; and a memory in which is stored computer program instructions which, when executed by the processor, cause the processor to perform the data segment collection method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110021400.0A CN112764988B (en) | 2021-01-08 | 2021-01-08 | Data segment acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110021400.0A CN112764988B (en) | 2021-01-08 | 2021-01-08 | Data segment acquisition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112764988A CN112764988A (en) | 2021-05-07 |
| CN112764988B true CN112764988B (en) | 2024-02-23 |
Family
ID=75700912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110021400.0A Active CN112764988B (en) | 2021-01-08 | 2021-01-08 | Data segment acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112764988B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114189490B (en) * | 2021-11-26 | 2023-03-31 | 广州市百果园信息技术有限公司 | User list processing method, system, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105468764A (en) * | 2015-12-02 | 2016-04-06 | 广州华多网络科技有限公司 | Data processing method and apparatus and cluster service system |
| CN107528864A (en) * | 2016-06-20 | 2017-12-29 | 中国科学院微电子研究所 | Heterogeneous network data processing method and system |
| CN110647421A (en) * | 2018-06-27 | 2020-01-03 | 阿里巴巴集团控股有限公司 | Database processing method, device and system and electronic equipment |
| CN111198853A (en) * | 2018-11-16 | 2020-05-26 | 北京微播视界科技有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN112182025A (en) * | 2020-10-28 | 2021-01-05 | 深圳前海微众银行股份有限公司 | Log analysis method, device, equipment and computer readable storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5471400B2 (en) * | 2009-12-17 | 2014-04-16 | 富士通株式会社 | Job analysis program and method, and job analysis apparatus |
| US11144532B2 (en) * | 2016-06-27 | 2021-10-12 | Aveva Software, Llc | Transactional integrity in a segmented database architecture |
-
2021
- 2021-01-08 CN CN202110021400.0A patent/CN112764988B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105468764A (en) * | 2015-12-02 | 2016-04-06 | 广州华多网络科技有限公司 | Data processing method and apparatus and cluster service system |
| CN107528864A (en) * | 2016-06-20 | 2017-12-29 | 中国科学院微电子研究所 | Heterogeneous network data processing method and system |
| CN110647421A (en) * | 2018-06-27 | 2020-01-03 | 阿里巴巴集团控股有限公司 | Database processing method, device and system and electronic equipment |
| CN111198853A (en) * | 2018-11-16 | 2020-05-26 | 北京微播视界科技有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN112182025A (en) * | 2020-10-28 | 2021-01-05 | 深圳前海微众银行股份有限公司 | Log analysis method, device, equipment and computer readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| Efficient_Collection_of_Connected_Vehicles_Data_With_Precision_Guarantees;Negin Alemazkoor;《IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS》;20191003;第4637-4645 * |
| 应用软件运行日志的收集与服务处理框架;张骁;应时;张韬;;计算机工程与应用;20171101(第10期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112764988A (en) | 2021-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109104336B (en) | Service request processing method and device, computer equipment and storage medium | |
| US10282481B2 (en) | Providing link to portion of media object in real time in social networking update | |
| CN111597257A (en) | Database synchronization method and device, storage medium and terminal | |
| CN110636340B (en) | Video file uploading method, storage device, terminal device and storage medium | |
| CN108073625B (en) | System and method for metadata information management | |
| US11816172B2 (en) | Data processing method, server, and computer storage medium | |
| CN106815254B (en) | Data processing method and device | |
| CN110505495B (en) | Multimedia resource frame extraction method, device, server and storage medium | |
| CN111274256B (en) | Resource management and control method, device, equipment and storage medium based on time sequence database | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| WO2021068113A1 (en) | Method and apparatus for compiling duration statistics, electronic device, and computer-readable medium | |
| CN112559813B (en) | Internet of things gateway data processing method and device based on instruction association pushing | |
| WO2020237898A1 (en) | Personalized recommendation method for online education system, terminal and storage medium | |
| CN113051460A (en) | Elasticissearch-based data retrieval method and system, electronic device and storage medium | |
| CN111541684B (en) | Signaling sending method and device of live broadcast room, server and storage medium | |
| CN112491609A (en) | System application architecture discovery method and system based on business service | |
| CN112764988B (en) | Data segment acquisition method and device | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN106487653B (en) | Message processing method and server | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| CN110647448A (en) | Mobile application operation log data real-time analysis method, server and system | |
| CN113254335B (en) | Test data processing method and device, server and storage medium | |
| CN109063201B (en) | Impala online interactive query method based on mixed storage scheme | |
| CN114756901B (en) | Operational risk monitoring method and device | |
| CN118035511A (en) | Recommendation platform, recommendation method, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |