CN112015627A - Data acquisition method and related device - Google Patents
Data acquisition method and related device Download PDFInfo
- Publication number
- CN112015627A CN112015627A CN202010888919.4A CN202010888919A CN112015627A CN 112015627 A CN112015627 A CN 112015627A CN 202010888919 A CN202010888919 A CN 202010888919A CN 112015627 A CN112015627 A CN 112015627A
- Authority
- CN
- China
- Prior art keywords
- acquisition
- acquired
- source system
- data source
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to the technical field of computers, and provides a data acquisition method and a related device, wherein the method is applied to a data acquisition platform and comprises the following steps: acquiring system information of at least one data source system to be acquired; configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired; docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired; configuring the acquisition task parameters of each data source system to be acquired; and acquiring data of the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired. According to the technical scheme, the data acquisition efficiency can be improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data acquisition method and a related apparatus.
Background
Data acquisition is the primary step in data analysis, and currently, the data acquisition mode is to customize and develop data acquisition logic for different data sources.
When data are acquired from different data sources, a corresponding acquisition tool needs to be customized and developed for each data source, so that the development difficulty is high, and the data acquisition efficiency is reduced. Moreover, since the collection tasks collected for each data source are distributed in the independent collection tools, when a plurality of collection tasks need to be managed, the collection tasks need to be managed in the independent collection tools, which also results in low data collection efficiency.
Disclosure of Invention
The application provides a data acquisition method and a related device, which can improve the data acquisition efficiency.
The application provides a data acquisition method in a first aspect, wherein the method is applied to a data acquisition platform, and the method comprises the following steps:
acquiring system information of at least one data source system to be acquired;
configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired;
configuring the acquisition task parameters of each data source system to be acquired;
and acquiring data of the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
The second aspect of the present application provides a data acquisition device, which is applied to a data acquisition platform, the device comprising:
the acquisition module is used for acquiring system information of at least one data source system to be acquired;
the first configuration module is used for configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
the docking module is used for docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired;
the second configuration module is used for configuring the acquisition task parameters of each data source system to be acquired;
and the acquisition module is used for acquiring data of the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
A third aspect of the present application provides an electronic device comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, the programs comprising instructions for performing the steps of the method of any of the first aspects of the present application.
A fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon for execution by a processor to perform some or all of the steps described in any of the methods of the first aspect of the present application.
It can be seen that, according to the data acquisition method and the related device provided by the application, firstly, system information of at least one data source system to be acquired is obtained, secondly, acquisition interface parameters of each data source system to be acquired are configured according to the system information of each data source system to be acquired in the at least one data source system to be acquired, secondly, the at least one data source system to be acquired is docked according to the acquisition interface parameters of each data source system to be acquired, secondly, acquisition task parameters of each data source system to be acquired are configured, and finally, data acquisition is carried out on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired. Therefore, when data are required to be acquired from different data source systems, the data source systems are in butt joint through the unified data acquisition platform, and an acquisition tool does not need to be developed for each data source system, so that the development difficulty is reduced, and the data acquisition efficiency is improved. Moreover, the collection tasks of each data source system are distributed in a unified data collection platform, and can be uniformly managed, so that the data collection efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic diagram of a data acquisition system according to an embodiment of the present application;
fig. 2 is a schematic flow chart of a data acquisition method according to an embodiment of the present application;
fig. 3 is a schematic flow chart of another data acquisition method provided in the embodiments of the present application;
FIG. 4 is a schematic diagram of a data acquisition device according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device in a hardware operating environment according to an embodiment of the present application.
Detailed Description
The data acquisition method and the related device provided by the embodiment of the application can improve the data acquisition efficiency.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "first," "second," "third," "fourth," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The following describes embodiments of the present application in detail.
Referring to fig. 1, fig. 1 is a schematic diagram of a data acquisition system according to an embodiment of the present disclosure. As shown in fig. 1, the data acquisition system 100 includes a data acquisition platform 101 and at least one data source system to be acquired 102.
When data acquisition needs to be performed on the at least one data source system 102 to be acquired, the data acquisition platform 101 acquires system information of the at least one data source system 102 to be acquired; the data acquisition platform 101 configures acquisition interface parameters of each data source system to be acquired according to system information of each data source system to be acquired in the at least one data source system to be acquired 102; the data acquisition platform 101 is in butt joint with the at least one data source system to be acquired 102 according to the acquisition interface parameters of each data source system to be acquired; the data acquisition platform 101 configures acquisition task parameters of each data source system to be acquired; the data acquisition platform 101 acquires data of the at least one data source system to be acquired 102 according to the acquisition task parameters of each data source system to be acquired.
Therefore, when data are required to be acquired from different data source systems, the data source systems are in butt joint through the unified data acquisition platform, and an acquisition tool does not need to be developed for each data source system, so that the development difficulty is reduced, and the data acquisition efficiency is improved. Moreover, the collection tasks of each data source system are distributed in a unified data collection platform, and can be uniformly managed, so that the data collection efficiency is improved.
Referring to fig. 2, fig. 2 is a schematic flowchart of a data acquisition method provided in an embodiment of the present application, where the data acquisition method may be applied to the data acquisition platform shown in fig. 1. As shown in fig. 2, a data acquisition method provided in an embodiment of the present application may include:
201. and acquiring system information of at least one data source system to be acquired.
When data acquisition needs to be carried out on at least one data source system to be acquired, the data acquisition platform can provide a uniform visual configuration interface without using a plurality of acquisition tools. Specifically, system information of the data source system is input on a visual configuration interface provided by the data acquisition platform, wherein the system information comprises the construction degree of the data source system to be acquired, and different interfaces can be selected for acquisition according to the system information. And after the input is finished, the data acquisition platform acquires the system information of the at least one data source system to be acquired.
Optionally, after the system information is entered, the data management platform may manage, including modifying and/or deleting, the at least one data source system to be collected. For example, the data management platform may modify the system information of the entered data source system to be collected, and may also delete the system information of the entered data source system to be collected. For another example, system information of the data source system to be acquired may be newly added to the data management platform.
202. And configuring the acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired.
Specifically, the system information of the data source system to be acquired includes the construction degree of the data source system to be acquired, and different interfaces can be selected for acquisition according to the construction degree of the data source system to be acquired.
After the system information of each data source system to be acquired is acquired, the data acquisition platform adds an acquisition interface to each data source system to be acquired, and then configures acquisition interface parameters of each data source system to be acquired, wherein the acquisition interface parameters are used for determining an acquisition interface adaptive to the data source system to be acquired.
Optionally, the data acquisition platform may modify the added acquisition interface, and may also delete the added acquisition interface.
203. And docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired.
The data acquisition platform comprises an interface resolver and is used for realizing automatic interface adaptation. Specifically, the acquisition interface parameters of each data source system to be acquired are analyzed through an interface analyzer to determine an acquisition interface adapted to each data source system to be acquired, wherein the acquisition interface includes at least one of a file acquisition interface, a database acquisition interface and a network service acquisition interface. And docking the at least one data source system to be acquired through an acquisition interface adaptive to each data source system to be acquired.
Optionally, after the at least one data source system to be acquired is docked, the data acquisition platform may further configure database information of each data source system to be acquired; and the data acquisition platform determines a database corresponding to each data source system to be acquired according to the database information, wherein the database is used for storing the data acquired from the corresponding data source system to be acquired.
Specifically, after the data acquisition platform is docked with the at least one data source system to be acquired, for the acquired data from different data source systems, corresponding databases need to be configured for storing the acquired data, for example, different databases are established for storing the data from different data source systems.
204. And configuring the acquisition task parameters of each data source system to be acquired.
The data acquisition platform can provide a uniform visual configuration interface, and data acquisition tasks and scheduling of data processing figures are uniformly distributed by the data acquisition platform. Specifically, when data acquisition is performed on at least one to-be-acquired data source system, a data acquisition task can be added on a visual configuration interface provided by a data acquisition platform, and then acquisition task parameters of each to-be-acquired data source system are configured, so that unified task management operation is realized.
Optionally, the data acquisition platform may further modify the data acquisition task of the data source system to be acquired, and/or delete the data acquisition task of the data source system to be acquired.
Optionally, when configuring the acquisition task parameters of the data source system to be acquired, the task execution time may be specifically configured, so as to implement the timing execution of the data acquisition task.
205. And acquiring data of the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
Specifically, when the collection task parameter is configured, the task execution time may be specifically configured. And the data acquisition platform acquires the current time, and if the current time reaches the task execution time, the data acquisition platform acquires data of at least one data source system to be acquired.
Optionally, the data acquisition platform configures database information of each to-be-acquired data source system, the data acquisition platform determines a database corresponding to each to-be-acquired data source system according to the database information, and the data acquired by the data acquisition platform from each to-be-acquired data source system is stored in the corresponding database.
It can be seen that, according to the data acquisition method provided by the application, firstly, system information of at least one data source system to be acquired is obtained, secondly, acquisition interface parameters of each data source system to be acquired are configured according to the system information of each data source system to be acquired in the at least one data source system to be acquired, secondly, the at least one data source system to be acquired is docked according to the acquisition interface parameters of each data source system to be acquired, secondly, acquisition task parameters of each data source system to be acquired are configured, and finally, data acquisition is performed on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired. Therefore, when data are required to be acquired from different data source systems, the data source systems are in butt joint through the unified data acquisition platform, and an acquisition tool does not need to be developed for each data source system, so that the development difficulty is reduced, and the data acquisition efficiency is improved. Moreover, the collection tasks of each data source system are distributed in a unified data collection platform, and can be uniformly managed, so that the data collection efficiency is improved.
Referring to fig. 3, fig. 3 is a schematic flow chart of another data acquisition method provided in the embodiment of the present application, where the data acquisition method can be applied to the data acquisition platform shown in fig. 1. As shown in fig. 3, another data acquisition method provided in the embodiment of the present application may include:
301. system information to a data source system to be acquired is acquired.
When data acquisition needs to be carried out on at least one data source system to be acquired, the data acquisition platform can provide a uniform visual configuration interface without using a plurality of acquisition tools. Specifically, system information of the data source system is input on a visual configuration interface provided by the data acquisition platform, wherein the system information comprises the construction degree of the data source system to be acquired, and different interfaces can be selected for acquisition according to the system information. And after the input is finished, the data acquisition platform acquires the system information of the at least one data source system to be acquired.
Optionally, after the system information is entered, the data management platform may manage, including modifying and/or deleting, the at least one data source system to be collected. For example, the data management platform may modify the system information of the entered data source system to be collected, and may also delete the system information of the entered data source system to be collected. For another example, system information of the data source system to be acquired may be newly added to the data management platform.
302. And configuring the acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired.
Specifically, the system information of the data source system to be acquired includes the construction degree of the data source system to be acquired, and different interfaces can be selected for acquisition according to the construction degree of the data source system to be acquired.
After the system information of each data source system to be acquired is acquired, the data acquisition platform adds an acquisition interface to each data source system to be acquired, and then configures acquisition interface parameters of each data source system to be acquired, wherein the acquisition interface parameters are used for determining an acquisition interface adaptive to the data source system to be acquired.
Optionally, the data acquisition platform may modify the added acquisition interface, and may also delete the added acquisition interface.
303. And analyzing the acquisition interface parameters of each data source system to be acquired through an interface analyzer so as to determine an acquisition interface adaptive to each data source system to be acquired.
The data acquisition platform comprises an interface resolver and is used for realizing automatic interface adaptation. Specifically, the acquisition interface parameters of each data source system to be acquired are analyzed through an interface analyzer to determine an acquisition interface adapted to each data source system to be acquired, wherein the acquisition interface includes at least one of a file acquisition interface, a database acquisition interface and a network service acquisition interface.
304. And docking the at least one data source system to be acquired through an acquisition interface adaptive to each data source system to be acquired.
Optionally, after the at least one data source system to be acquired is docked, the data acquisition platform may further configure database information of each data source system to be acquired; and the data acquisition platform determines a database corresponding to each data source system to be acquired according to the database information, wherein the database is used for storing the data acquired from the corresponding data source system to be acquired.
Specifically, after the data acquisition platform is docked with the at least one data source system to be acquired, for the acquired data from different data source systems, corresponding databases need to be configured for storing the acquired data, for example, different databases are established for storing the data from different data source systems.
305. And configuring collection task parameters of each data source system to be collected, wherein the collection task parameters comprise task execution time.
The data acquisition platform can provide a uniform visual configuration interface, and data acquisition tasks and scheduling of data processing figures are uniformly distributed by the data acquisition platform. Specifically, when data acquisition is performed on at least one to-be-acquired data source system, a data acquisition task can be added on a visual configuration interface provided by a data acquisition platform, and then acquisition task parameters of each to-be-acquired data source system are configured, so that unified task management operation is realized.
In a possible implementation manner, when configuring the collection task parameters of the data source system to be collected, the task execution time may be specifically configured, so as to implement the timing execution of the data collection task.
Optionally, the data acquisition platform may further modify the data acquisition task of the data source system to be acquired, and/or delete the data acquisition task of the data source system to be acquired.
306. And acquiring the current time.
307. And if the current time reaches the task execution time, performing data acquisition on the at least one data source system to be acquired.
Specifically, when the collection task parameter is configured, the task execution time may be specifically configured. And the data acquisition platform acquires the current time, and if the current time reaches the task execution time, the data acquisition platform acquires data of at least one data source system to be acquired.
Optionally, the data acquisition platform configures database information of each to-be-acquired data source system, the data acquisition platform determines a database corresponding to each to-be-acquired data source system according to the database information, and the data acquired by the data acquisition platform from each to-be-acquired data source system is stored in the corresponding database.
308. And detecting the data acquired from the at least one data source system to be acquired so as to judge whether the data acquisition fails.
Specifically, after data acquisition is performed on the at least one data source system to be acquired, the data acquisition platform detects the acquired data to judge whether the data acquisition fails. In a possible implementation, the detecting the collected data includes performing integrity detection on the collected data, and if the integrity detection fails, it may be determined that the data collection failed.
309. And if the data acquisition fails, acquiring an acquisition failure type.
If the data acquisition fails, the data acquisition platform acquires an acquisition failure type, wherein the acquisition failure type comprises at least one of interface call failure, network exception and service exception, and different strategies can be configured for different acquisition failure types.
310. And according to the acquisition failure type, performing data acquisition on the at least one data source system to be acquired again.
Specifically, according to the acquisition failure type, the data acquisition platform acquires data of the at least one data source system to be acquired again so as to ensure the data quality. For different acquisition failure types, the data acquisition platform can set different acquisition strategies.
In a possible implementation manner, the data acquisition platform acquires the acquisition times for acquiring the data of the at least one data source system to be acquired again, and when the acquisition times are smaller than a preset time threshold, continues to acquire the data of the at least one data source system to be acquired again until the data acquisition is successful; and when the acquisition times are not less than the preset time threshold, sending prompt information to a preset manager, wherein the prompt information is used for prompting the preset manager to perform manual acquisition.
For example, for the acquisition failure type of the network anomaly, the set acquisition policy includes: and the data acquisition platform automatically acquires again, knows that the data acquisition is successful again, and when the number of times of the data acquisition reaches a preset number threshold value, if the data acquisition is not successful, the data acquisition platform does not automatically acquire any more, but sends prompt information to preset management personnel to prompt the preset management personnel to manually acquire the data.
In one possible embodiment, in the data collection platform, the automatic collection is in a default on state, that is, when the data collection fails, the data collection platform automatically collects again by default. Meanwhile, the data acquisition platform also supports data acquisition in a manual operation mode, for example, when the number of times of re-acquisition reaches a preset number threshold, the data acquisition platform does not automatically acquire any more, and then the data acquisition can be performed again through manual operation.
In another possible implementation manner, in the data acquisition process, the data acquisition platform can also detect the timeliness of the acquired data, and automatic or manual additional acquisition is performed through data timeliness detection.
Specifically, when the data acquisition task is executed, the data acquisition platform acquires first update time of acquired data, compares the first update time with second update time of latest update of the data in the acquired data source system, continues to execute the data acquisition task if the first update time is the same as the second update time, and re-supplements and acquires the data if the first update time is less than the second update time, and similarly, can perform automatic or manual supplementation and acquisition. Therefore, after the original data in the data source system to be acquired is updated, the updated latest data can be acquired in time, and the timeliness of the data is ensured.
Optionally, the data acquisition platform monitors the data acquisition process while performing the data acquisition task. The data acquisition platform can monitor a data acquisition task, a data processing task and a data acquisition process, and if the data acquisition task, the data processing task and the data acquisition process are monitored to be abnormal, the data acquisition platform processes the data acquisition task, the data processing task and the data acquisition process through a preset processing strategy. For example, an alarm is triggered for some abnormal acquisition events, and is presented in an acquisition front page of the data acquisition platform, or highlighted, or displayed in a pop-up window, or the like. For abnormal acquisition events which seriously affect data acquisition, the data acquisition platform can inform maintenance personnel to process the events in time.
In a possible implementation manner, the data acquisition platform monitors a data acquisition process of the at least one to-be-acquired data source system, when at least one acquisition abnormal event is detected, acquires a type of each acquisition abnormal event in the at least one acquisition abnormal event, classifies the at least one acquisition abnormal event according to the type of each acquisition abnormal event to obtain n types of acquisition abnormal events, where n is a positive integer, selects m types of acquisition abnormal events from the n types of acquisition abnormal events according to an index threshold corresponding to each acquisition abnormal event in the n types of acquisition abnormal events, where m is a positive integer not greater than n, and gives an alarm to the m types of acquisition abnormal events.
Specifically, when the data acquisition platform monitors the data acquisition process of the at least one data source system to be acquired, alarm filtering is performed. For large-batch abnormal event collection, filtering is carried out through a preset filtering rule, and then the abnormal events are stored and uploaded to a manager in real time. That is, after a large number of abnormal acquisition events are detected, the data acquisition platform firstly acquires the types of the abnormal acquisition events, classifies the abnormal acquisition events according to the types of the abnormal acquisition events, filters the abnormal acquisition events of different types through a preset index threshold value after classification, thereby determining the abnormal acquisition events needing to be alarmed and giving an alarm, and takes other measures such as re-acquisition and the like for the abnormal acquisition events needing no alarm.
For example, when the detected abnormal acquisition event includes field mismatch, file absence and network abnormality, acquiring the number of records with field mismatch, the number of times of file absence and the number of times of network abnormality, and presetting index thresholds for field mismatch, file absence and network abnormality, wherein the index thresholds are respectively a first index threshold, a second index threshold and a third index threshold, when the number of records with field mismatch reaches the first index threshold, an alarm is triggered, otherwise, the alarm is filtered, similarly, when the number of times of file absence reaches the second index threshold, an alarm is triggered, otherwise, the alarm is filtered, when the number of times of network abnormality reaches the third index threshold, an alarm is also triggered, and otherwise, the alarm is filtered.
In another possible embodiment, during the monitoring process, the data collection platform further supports recording a data collection log for the user to view.
It can be seen that, with the data acquisition method provided by the embodiment of the application, when data are required to be acquired from different data source systems, a plurality of data source systems are docked through the unified data acquisition platform, and an acquisition tool does not need to be developed for each data source system, so that the development difficulty is reduced, and the data acquisition efficiency is improved. Moreover, the collection tasks of each data source system are distributed in a unified data collection platform, and can be uniformly managed, so that the data collection efficiency is improved.
Referring to fig. 4, fig. 4 is a schematic diagram of a data acquisition device according to an embodiment of the present application, where the data acquisition device is applied to a data acquisition platform. As shown in fig. 4, a data acquisition device provided in an embodiment of the present application may include:
the obtaining module 401 is configured to obtain system information of at least one data source system to be collected.
A first configuration module 402, configured to configure an acquisition interface parameter of each to-be-acquired data source system according to system information of each to-be-acquired data source system in the at least one to-be-acquired data source system.
The docking module 403 is configured to dock the at least one data source system to be acquired according to the acquisition interface parameter of each data source system to be acquired.
A second configuration module 404, configured to configure the collection task parameter of each data source system to be collected.
And the acquisition module 405 is configured to acquire data of the at least one data source system to be acquired according to the acquisition task parameter of each data source system to be acquired.
For specific implementation of the data acquisition device in the embodiment of the present application, reference may be made to each embodiment of the data acquisition method, which is not described herein again.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device in a hardware operating environment according to an embodiment of the present application. As shown in fig. 5, an electronic device of a hardware operating environment according to an embodiment of the present application may include:
a processor 501, such as a CPU.
The memory 502 may alternatively be a high speed RAM memory or a stable memory such as a disk memory.
A communication interface 503 for implementing connection communication between the processor 501 and the memory 502.
Those skilled in the art will appreciate that the configuration of the electronic device shown in fig. 5 does not constitute a limitation of the electronic device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 5, the memory 502 may include an operating system, a network communication module, and a data collection program. The operating system is a program that manages and controls the hardware and software resources of the electronic device, supporting the operation of the data collection program and other software or programs. The network communication module is used to implement communication between the components in the memory 502 and with other hardware and software in the electronic device.
In the electronic device shown in fig. 5, the processor 501 is configured to execute the data acquisition program stored in the memory 502, and implement the following steps:
acquiring system information of at least one data source system to be acquired;
configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired;
configuring the acquisition task parameters of each data source system to be acquired;
and acquiring data of the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
For specific implementation of the electronic device in the embodiment of the present application, reference may be made to each embodiment of the data acquisition method, which is not described herein again.
Another embodiment of the present application provides a computer-readable storage medium storing a computer program for execution by a processor to perform the steps of:
acquiring system information of at least one data source system to be acquired;
configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired;
configuring the acquisition task parameters of each data source system to be acquired;
and acquiring data of the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
For specific implementation of the computer-readable storage medium in the embodiment of the present application, reference may be made to the embodiments of the data acquisition method, which are not described herein again.
It is also noted that while for simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present application is not limited by the order of acts, as some steps may, in accordance with the present application, occur in other orders and concurrently. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application. In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A data acquisition method is applied to a data acquisition platform, and comprises the following steps:
acquiring system information of at least one data source system to be acquired;
configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired;
configuring the acquisition task parameters of each data source system to be acquired;
and acquiring data of the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
2. The method of claim 1, wherein the data acquisition platform comprises an interface parser, and the interfacing the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired comprises:
analyzing the acquisition interface parameters of each data source system to be acquired through the interface analyzer to determine an acquisition interface adapted to each data source system to be acquired, wherein the acquisition interface comprises at least one of a file acquisition interface, a database acquisition interface and a network service acquisition interface;
and docking the at least one data source system to be acquired through an acquisition interface adaptive to each data source system to be acquired.
3. The method according to claim 1 or 2, wherein after said interfacing said at least one data source system to be acquired according to the acquisition interface parameters of said each data source system to be acquired, the method further comprises:
configuring database information of each data source system to be acquired;
and determining a database corresponding to each data source system to be acquired according to the database information, wherein the database is used for storing the data acquired from the corresponding data source system to be acquired.
4. The method according to claim 1, wherein the collection task parameters include task execution time, and the collecting data of the at least one to-be-collected data source system according to the collection task parameters of each to-be-collected data source system includes:
acquiring current time;
and if the current time reaches the task execution time, performing data acquisition on the at least one data source system to be acquired.
5. The method of claim 1, further comprising:
detecting data collected from the at least one data source system to be collected to judge whether the data collection fails;
if the data acquisition fails, acquiring an acquisition failure type;
and according to the acquisition failure type, performing data acquisition on the at least one data source system to be acquired again.
6. The method according to claim 5, wherein after the data acquisition of the at least one data source system to be acquired is resumed according to the acquisition failure type, the method further comprises:
acquiring the acquisition times of acquiring data of the at least one data source system to be acquired again;
when the acquisition times are smaller than a preset time threshold value, continuing to perform data acquisition on the at least one data source system to be acquired again until the data acquisition is successful;
and when the acquisition times are not less than the preset time threshold, sending prompt information to preset management personnel, wherein the prompt information is used for prompting the preset management personnel to carry out manual acquisition.
7. The method of claim 1, further comprising:
monitoring the data acquisition process of the at least one data source system to be acquired;
when at least one acquisition abnormal event is detected, acquiring the type of each acquisition abnormal event in the at least one acquisition abnormal event;
classifying the at least one acquisition abnormal event according to the type of each acquisition abnormal event to obtain n types of acquisition abnormal events, wherein n is a positive integer;
selecting m types of acquisition abnormal events from the n types of acquisition abnormal events according to the index threshold corresponding to each acquisition abnormal event in the n types of acquisition abnormal events, wherein m is a positive integer not greater than n;
and carrying out alarm prompt on the m types of abnormal acquisition events.
8. A data acquisition device, characterized in that the device is applied to a data acquisition platform, the device comprises:
the acquisition module is used for acquiring system information of at least one data source system to be acquired;
the first configuration module is used for configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
the docking module is used for docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired;
the second configuration module is used for configuring the acquisition task parameters of each data source system to be acquired;
and the acquisition module is used for acquiring data of the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
9. An electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, the programs comprising instructions for performing the steps of the method of any of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which is executed by a processor to implement the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010888919.4A CN112015627B (en) | 2020-08-28 | 2020-08-28 | Data acquisition method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010888919.4A CN112015627B (en) | 2020-08-28 | 2020-08-28 | Data acquisition method and related device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112015627A true CN112015627A (en) | 2020-12-01 |
| CN112015627B CN112015627B (en) | 2023-04-28 |
Family
ID=73503866
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010888919.4A Active CN112015627B (en) | 2020-08-28 | 2020-08-28 | Data acquisition method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112015627B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114697204A (en) * | 2022-03-21 | 2022-07-01 | 昭通亮风台信息科技有限公司 | Log collection method and system for distributed system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678726A (en) * | 2012-09-05 | 2014-03-26 | 亿阳信通股份有限公司 | Data complementary collection method and system |
| WO2014101487A1 (en) * | 2012-12-28 | 2014-07-03 | 中兴通讯股份有限公司 | Data acquisition method and device |
| CN104111983A (en) * | 2014-06-30 | 2014-10-22 | 中国科学院信息工程研究所 | Open-type multi-source data collection system and method |
| CN110032587A (en) * | 2019-04-17 | 2019-07-19 | 上海圣剑网络科技股份有限公司 | Data summarization method, device, computer equipment and storage medium |
| CN110874265A (en) * | 2019-11-15 | 2020-03-10 | 厦门物通博联网络科技有限公司 | Method, device and equipment for rapidly acquiring data and storage medium |
| CN111459924A (en) * | 2020-03-24 | 2020-07-28 | 上海携程商务有限公司 | Data acquisition method, system, electronic device, and medium |
| CN111475556A (en) * | 2020-03-19 | 2020-07-31 | 平安国际智慧城市科技股份有限公司 | Data acquisition method, data acquisition equipment, server and readable storage medium |
-
2020
- 2020-08-28 CN CN202010888919.4A patent/CN112015627B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678726A (en) * | 2012-09-05 | 2014-03-26 | 亿阳信通股份有限公司 | Data complementary collection method and system |
| WO2014101487A1 (en) * | 2012-12-28 | 2014-07-03 | 中兴通讯股份有限公司 | Data acquisition method and device |
| CN104111983A (en) * | 2014-06-30 | 2014-10-22 | 中国科学院信息工程研究所 | Open-type multi-source data collection system and method |
| CN110032587A (en) * | 2019-04-17 | 2019-07-19 | 上海圣剑网络科技股份有限公司 | Data summarization method, device, computer equipment and storage medium |
| CN110874265A (en) * | 2019-11-15 | 2020-03-10 | 厦门物通博联网络科技有限公司 | Method, device and equipment for rapidly acquiring data and storage medium |
| CN111475556A (en) * | 2020-03-19 | 2020-07-31 | 平安国际智慧城市科技股份有限公司 | Data acquisition method, data acquisition equipment, server and readable storage medium |
| CN111459924A (en) * | 2020-03-24 | 2020-07-28 | 上海携程商务有限公司 | Data acquisition method, system, electronic device, and medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114697204A (en) * | 2022-03-21 | 2022-07-01 | 昭通亮风台信息科技有限公司 | Log collection method and system for distributed system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112015627B (en) | 2023-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3798846B1 (en) | Operation and maintenance system and method | |
| US9652316B2 (en) | Preventing and servicing system errors with event pattern correlation | |
| CN108521339B (en) | Feedback type node fault processing method and system based on cluster log | |
| EP3849142A1 (en) | Automatic root cause diagnosis in networks | |
| CN111629043B (en) | Cross-platform health management system based on cloud mode | |
| US20220050765A1 (en) | Method for processing logs in a computer system for events identified as abnormal and revealing solutions, electronic device, and cloud server | |
| CN109413642B (en) | Terminal safety detection and monitoring systematization method | |
| CN111262750B (en) | Method and system for evaluating baseline model | |
| CN107104848B (en) | Information technology system monitoring method and device | |
| CN107566172B (en) | Active management method and system based on storage system | |
| WO2015187001A2 (en) | System and method for managing resources failure using fast cause and effect analysis in a cloud computing system | |
| CN116204385A (en) | Computer log monitoring method and system | |
| CN112988537A (en) | Server fault diagnosis method and device and related equipment | |
| CN117422434A (en) | Wisdom fortune dimension dispatch platform | |
| CN105207797A (en) | Fault locating method and fault locating device | |
| CN112015627A (en) | Data acquisition method and related device | |
| CN116932232B (en) | Data processing method of development platform based on BS architecture | |
| CN110609761B (en) | Method and device for determining fault source, storage medium and electronic equipment | |
| CN113760634A (en) | Data processing method and device | |
| CN116594840A (en) | Log fault acquisition and analysis method, system, equipment and medium based on ELK | |
| CN116302989A (en) | Pressure testing method and system, storage medium and computer equipment | |
| CN115766768A (en) | Method and device for designing sensing center in computational power network operating system | |
| CN115408271A (en) | One-stop closed loop test method, system, equipment and medium | |
| CN113722135A (en) | Error log acquisition system, method, device and medium | |
| CN113065001A (en) | Fault loss stopping method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210202 Address after: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.) Applicant after: Shenzhen saiante Technology Service Co.,Ltd. Address before: 1-34 / F, Qianhai free trade building, 3048 Xinghai Avenue, Mawan, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong 518000 Applicant before: Ping An International Smart City Technology Co.,Ltd. |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |