CN112015627B - Data acquisition method and related device - Google Patents
Data acquisition method and related device Download PDFInfo
- Publication number
- CN112015627B CN112015627B CN202010888919.4A CN202010888919A CN112015627B CN 112015627 B CN112015627 B CN 112015627B CN 202010888919 A CN202010888919 A CN 202010888919A CN 112015627 B CN112015627 B CN 112015627B
- Authority
- CN
- China
- Prior art keywords
- acquisition
- acquired
- data
- source system
- data source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to the technical field of computers, and provides a data acquisition method and a related device, wherein the method is applied to a data acquisition platform and comprises the following steps: acquiring system information of at least one data source system to be acquired; configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired; according to the acquisition interface parameters of each data source system to be acquired, docking the at least one data source system to be acquired; configuring acquisition task parameters of each data source system to be acquired; and carrying out data acquisition on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired. According to the technical scheme, the data acquisition efficiency can be improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data acquisition method and a related device.
Background
Data collection is the primary step in data analysis, and currently, the data collection logic is developed in a customized manner for different data sources.
When data are collected from different data sources, corresponding collection tools need to be customized and developed for each data source, so that development difficulty is high, and data collection efficiency is reduced. Moreover, because the collection tasks for collecting each data source are distributed in independent collection tools, when a plurality of collection tasks need to be managed, the collection tasks need to be managed in the independent collection tools respectively, and therefore the data collection efficiency is low.
Disclosure of Invention
The application provides a data acquisition method and a related device, which can improve data acquisition efficiency.
The first aspect of the application provides a data acquisition method, which is applied to a data acquisition platform, and comprises the following steps:
acquiring system information of at least one data source system to be acquired;
configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
according to the acquisition interface parameters of each data source system to be acquired, docking the at least one data source system to be acquired;
configuring acquisition task parameters of each data source system to be acquired;
and carrying out data acquisition on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
A second aspect of the present application provides a data acquisition device for use in a data acquisition platform, the device comprising:
the acquisition module is used for acquiring system information of at least one data source system to be acquired;
the first configuration module is used for configuring the acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
the docking module is used for docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired;
the second configuration module is used for configuring the acquisition task parameters of each data source system to be acquired;
and the acquisition module is used for carrying out data acquisition on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
A third aspect of the present application provides an electronic device comprising a processor, a memory, a communication interface and one or more programs, wherein the one or more programs are stored in the memory and configured for execution by the processor, the programs comprising instructions for performing the steps of any of the methods of the first aspect of the present application.
A fourth aspect of the present application provides a computer readable storage medium storing a computer program for execution by a processor to perform some or all of the steps described in any of the methods of the first aspect of the present application.
It can be seen that, through the data acquisition method and the related device provided by the present application, firstly, system information of at least one data source system to be acquired is acquired, secondly, according to the system information of each data source system to be acquired in the at least one data source system to be acquired, an acquisition interface parameter of each data source system to be acquired is configured, secondly, according to the acquisition interface parameter of each data source system to be acquired, the at least one data source system to be acquired is docked, secondly, an acquisition task parameter of each data source system to be acquired is configured, and finally, according to the acquisition task parameter of each data source system to be acquired, data acquisition is performed on the at least one data source system to be acquired. Therefore, when data are required to be acquired from different data source systems, a plurality of data source systems are butted through a unified data acquisition platform, and an acquisition tool is not required to be developed for each data source system, so that development difficulty is reduced, and data acquisition efficiency is improved. Moreover, the acquisition tasks of each data source system are distributed in a unified data acquisition platform, so that the acquisition tasks can be managed uniformly, and the data acquisition efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic diagram of a data acquisition system according to an embodiment of the present application;
fig. 2 is a schematic flow chart of a data acquisition method according to an embodiment of the present application;
fig. 3 is a flow chart of another data collection method according to an embodiment of the present application;
fig. 4 is a schematic diagram of a data acquisition device according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device in a hardware running environment according to an embodiment of the present application.
Detailed Description
The data acquisition method and the related device can improve the data acquisition efficiency.
In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims of this application and in the above-described figures, are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.
The embodiments of the present application are described in detail below.
Referring first to fig. 1, fig. 1 is a schematic diagram of a data acquisition system according to an embodiment of the present application. As shown in fig. 1, the data acquisition system 100 includes a data acquisition platform 101 and at least one data source system 102 to be acquired.
When data acquisition is required to be performed on the at least one data source system to be acquired 102, the data acquisition platform 101 acquires system information of the at least one data source system to be acquired 102; the data acquisition platform 101 configures acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired 102; the data acquisition platform 101 interfaces with the at least one data source system 102 to be acquired according to the acquisition interface parameters of each data source system to be acquired; the data acquisition platform 101 configures acquisition task parameters of each data source system to be acquired; the data acquisition platform 101 performs data acquisition on the at least one data source system 102 according to the acquisition task parameters of each data source system to be acquired.
It can be seen that when data needs to be acquired from different data source systems, a plurality of data source systems are docked through a unified data acquisition platform, and an acquisition tool does not need to be developed for each data source system, so that development difficulty is reduced, and data acquisition efficiency is improved. Moreover, the acquisition tasks of each data source system are distributed in a unified data acquisition platform, so that the acquisition tasks can be managed uniformly, and the data acquisition efficiency is improved.
Referring to fig. 2, fig. 2 is a flow chart of a data collection method according to an embodiment of the present application, where the data collection method may be applied to the data collection platform shown in fig. 1. As shown in fig. 2, a data acquisition method provided in an embodiment of the present application may include:
201. system information of at least one data source system to be acquired is acquired.
When data acquisition is required to be performed on at least one data source system to be acquired, the data acquisition platform can provide a uniform visual configuration interface without using a plurality of acquisition tools. Specifically, system information of a data source system is recorded on a visual configuration interface provided by a data acquisition platform, wherein the system information comprises the construction degree of the data source system to be acquired, and different interfaces can be selected for acquisition according to the system information. After the input is completed, the data acquisition platform acquires the system information of the at least one data source system to be acquired.
Optionally, after entering the system information, the data management platform may manage, including modify and/or delete, the at least one data source system to be collected. For example, the data management platform may modify the entered system information of the data source system to be collected, and may also delete the entered system information of the data source system to be collected. For another example, system information of the data source system to be collected may also be newly added in the data management platform.
202. And configuring the acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired.
Specifically, the system information of the data source system to be collected includes the construction degree of the data source system to be collected, and different interfaces can be selected for collection according to the construction degree of the data source system to be collected.
After the system information of each data source system to be acquired is acquired, the data acquisition platform adds an acquisition interface to each data source system to be acquired, and then configures the acquisition interface parameters of each data source system to be acquired, wherein the acquisition interface parameters are used for determining an acquisition interface matched with the data source system to be acquired.
Optionally, the data collection platform may modify the collection interface that has been added, and may delete the collection interface that has been added.
203. And docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired.
The data acquisition platform comprises an interface parser for realizing automatic interface adaptation. Specifically, the collection interface parameters of each data source system to be collected are analyzed through an interface analyzer to determine the collection interface matched with each data source system to be collected, wherein the collection interface comprises at least one of a file collection interface, a database collection interface and a network service collection interface. The at least one data source system to be acquired is docked through an acquisition interface adapted to each data source system to be acquired.
Optionally, after docking the at least one data source system to be collected, the data collection platform may further configure database information of each data source system to be collected; and the data acquisition platform determines a database corresponding to each data source system to be acquired according to the database information, and the database is used for storing the data acquired from the corresponding data source system to be acquired.
Specifically, after the data collection platform is docked with the at least one data source system to be collected, a corresponding database needs to be configured for the collected data from different data source systems, so as to store the collected data, for example, different databases are established for storing the data from different data source systems.
204. And configuring the acquisition task parameters of each data source system to be acquired.
The data acquisition platform can provide a unified visual configuration interface, and the scheduling of data acquisition tasks and data processing figures is uniformly distributed by the data acquisition platform. Specifically, when data is acquired for at least one data source system to be acquired, a data acquisition task can be added on a visual configuration interface provided by a data acquisition platform, and then acquisition task parameters of each data source system to be acquired are configured, so that unified task management operation is realized.
Optionally, the data acquisition platform may also modify the data acquisition task of the data source system to be acquired and/or delete the data acquisition task of the data source system to be acquired.
Optionally, when the acquisition task parameters of the data source system to be acquired are configured, the task execution time can be specifically configured, so that the timing execution of the data acquisition task is realized.
205. And carrying out data acquisition on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
Specifically, when the acquisition task parameters are configured, the task execution time can be specifically configured. The data acquisition platform acquires the current time, and if the current time reaches the task execution time, the data acquisition platform performs data acquisition on at least one data source system to be acquired.
Optionally, the data acquisition platform configures database information of each data source system to be acquired, the data acquisition platform determines a database corresponding to each data source system to be acquired according to the database information, and the data acquired by the data acquisition platform from each data source system to be acquired is stored in the corresponding database.
It can be seen that, by the data acquisition method provided by the present application, firstly, system information of at least one data source system to be acquired is acquired, secondly, according to the system information of each data source system to be acquired in the at least one data source system to be acquired, an acquisition interface parameter of each data source system to be acquired is configured, secondly, according to the acquisition interface parameter of each data source system to be acquired, at least one data source system to be acquired is docked, secondly, an acquisition task parameter of each data source system to be acquired is configured, and finally, according to the acquisition task parameter of each data source system to be acquired, data acquisition is performed on at least one data source system to be acquired. Therefore, when data are required to be acquired from different data source systems, a plurality of data source systems are butted through a unified data acquisition platform, and an acquisition tool is not required to be developed for each data source system, so that development difficulty is reduced, and data acquisition efficiency is improved. Moreover, the acquisition tasks of each data source system are distributed in a unified data acquisition platform, so that the acquisition tasks can be managed uniformly, and the data acquisition efficiency is improved.
Referring to fig. 3, fig. 3 is a flow chart of another data collection method according to an embodiment of the present application, where the data collection method may be applied to the data collection platform shown in fig. 1. As shown in fig. 3, another data collection method provided in the embodiment of the present application may include:
301. and acquiring system information of a data source system of the to-be-sampled less set.
When data acquisition is required to be performed on at least one data source system to be acquired, the data acquisition platform can provide a uniform visual configuration interface without using a plurality of acquisition tools. Specifically, system information of a data source system is recorded on a visual configuration interface provided by a data acquisition platform, wherein the system information comprises the construction degree of the data source system to be acquired, and different interfaces can be selected for acquisition according to the system information. After the input is completed, the data acquisition platform acquires the system information of the at least one data source system to be acquired.
Optionally, after entering the system information, the data management platform may manage, including modify and/or delete, the at least one data source system to be collected. For example, the data management platform may modify the entered system information of the data source system to be collected, and may also delete the entered system information of the data source system to be collected. For another example, system information of the data source system to be collected may also be newly added in the data management platform.
302. And configuring the acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired.
Specifically, the system information of the data source system to be collected includes the construction degree of the data source system to be collected, and different interfaces can be selected for collection according to the construction degree of the data source system to be collected.
After the system information of each data source system to be acquired is acquired, the data acquisition platform adds an acquisition interface to each data source system to be acquired, and then configures the acquisition interface parameters of each data source system to be acquired, wherein the acquisition interface parameters are used for determining an acquisition interface matched with the data source system to be acquired.
Optionally, the data collection platform may modify the collection interface that has been added, and may delete the collection interface that has been added.
303. Analyzing the acquisition interface parameters of each data source system to be acquired through an interface analyzer so as to determine an acquisition interface matched with each data source system to be acquired.
The data acquisition platform comprises an interface parser for realizing automatic interface adaptation. Specifically, the collection interface parameters of each data source system to be collected are analyzed through an interface analyzer to determine the collection interface matched with each data source system to be collected, wherein the collection interface comprises at least one of a file collection interface, a database collection interface and a network service collection interface.
304. The at least one data source system to be acquired is docked through an acquisition interface adapted to each data source system to be acquired.
Optionally, after docking the at least one data source system to be collected, the data collection platform may further configure database information of each data source system to be collected; and the data acquisition platform determines a database corresponding to each data source system to be acquired according to the database information, and the database is used for storing the data acquired from the corresponding data source system to be acquired.
Specifically, after the data collection platform is docked with the at least one data source system to be collected, a corresponding database needs to be configured for the collected data from different data source systems, so as to store the collected data, for example, different databases are established for storing the data from different data source systems.
305. And configuring acquisition task parameters of each data source system to be acquired, wherein the acquisition task parameters comprise task execution time.
The data acquisition platform can provide a unified visual configuration interface, and the scheduling of data acquisition tasks and data processing figures is uniformly distributed by the data acquisition platform. Specifically, when data is acquired for at least one data source system to be acquired, a data acquisition task can be added on a visual configuration interface provided by a data acquisition platform, and then acquisition task parameters of each data source system to be acquired are configured, so that unified task management operation is realized.
In one possible implementation manner, when the acquisition task parameters of the data source system to be acquired are configured, the task execution time can be specifically configured, so that the timing execution of the data acquisition task is realized.
Optionally, the data acquisition platform may also modify the data acquisition task of the data source system to be acquired and/or delete the data acquisition task of the data source system to be acquired.
306. The current time is obtained.
307. And if the current time reaches the task execution time, carrying out data acquisition on the at least one data source system to be acquired.
Specifically, when the acquisition task parameters are configured, the task execution time can be specifically configured. The data acquisition platform acquires the current time, and if the current time reaches the task execution time, the data acquisition platform performs data acquisition on at least one data source system to be acquired.
Optionally, the data acquisition platform configures database information of each data source system to be acquired, the data acquisition platform determines a database corresponding to each data source system to be acquired according to the database information, and the data acquired by the data acquisition platform from each data source system to be acquired is stored in the corresponding database.
308. And detecting the data acquired from the at least one data source system to be acquired so as to judge whether the data acquisition fails.
Specifically, after data acquisition is performed on the at least one data source system to be acquired, the data acquisition platform detects the acquired data to judge whether the data acquisition fails. In one possible embodiment, detecting the collected data includes detecting the integrity of the collected data, and if the integrity detection is not passed, determining that the data collection has failed.
309. And if the data acquisition fails, acquiring the acquisition failure type.
If the data acquisition fails, the data acquisition platform acquires an acquisition failure type, wherein the acquisition failure type comprises at least one of interface call failure, network abnormality and service abnormality, and for different acquisition failure types, the data acquisition platform can be configured with different strategies.
310. And re-acquiring the data of the at least one data source system to be acquired according to the acquisition failure type.
Specifically, according to the acquisition failure type, the data acquisition platform carries out data acquisition on the at least one data source system to be acquired again so as to ensure the data quality. For different acquisition failure types, the data acquisition platform can set different acquisition strategies.
In one possible implementation manner, the data acquisition platform acquires the acquisition times of carrying out data acquisition on the at least one data source system to be acquired again, and when the acquisition times are smaller than a preset time threshold, the data acquisition is carried out on the at least one data source system to be acquired again until the data acquisition is successful; and when the acquisition times are not less than the preset times threshold, sending prompt information to preset management personnel, wherein the prompt information is used for prompting the preset management personnel to carry out manual acquisition.
For example, for the acquisition failure type of network anomalies, the set acquisition policy includes: the data acquisition platform automatically re-acquires, knows that re-acquisition is successful, and when the number of re-acquisition reaches a preset number threshold, if the re-acquisition is not successful, the data acquisition platform does not acquire automatically any more, and sends prompt information to preset management personnel to prompt the preset management personnel to acquire manually.
In one possible embodiment, in the data acquisition platform, the automatic acquisition is in a default on state, that is, when the data acquisition fails, the data acquisition platform defaults to re-automatically acquire. Meanwhile, the data acquisition platform also supports the data acquisition by a manual operation mode, for example, when the number of times of re-acquisition reaches a preset number of times threshold, the data acquisition platform does not automatically acquire any more, and at this time, the data acquisition can be performed again by manual operation.
In another possible implementation manner, in the process of collecting data, the data collection platform can also detect the timeliness of the collected data, and automatic or manual supplementary collection is performed through the data timeliness detection.
Specifically, when the data acquisition task is executed, the data acquisition platform acquires a first update time of acquired data, compares the first update time with a second update time of latest update of the data in the acquired data source system, continues to execute the data acquisition task if the first update time is the same as the second update time, and re-acquires the data if the first update time is less than the second update time, and can also automatically or manually acquire the data in a complementary manner. Therefore, after the original data in the data source system to be acquired is updated, the updated latest data can be acquired in time, so that the timeliness of the data is ensured.
Optionally, the data acquisition platform monitors the data acquisition process while performing the data acquisition task. The data acquisition platform can monitor the data acquisition task, the data processing task and the data acquisition process, and if the data acquisition platform is abnormal, the data acquisition platform can process the data through a preset processing strategy. For example, alarms are triggered for certain acquisition anomalies and presented in the acquisition front of the data acquisition platform, or may be highlighted, or a pop-up display or the like. For the acquisition abnormal event which seriously affects the data acquisition, the data acquisition platform can timely inform maintenance personnel to process.
In a possible implementation manner, the data acquisition platform monitors a data acquisition process of the at least one data source system to be acquired, when at least one acquisition abnormal event is detected, the type of each acquisition abnormal event in the at least one acquisition abnormal event is acquired, the at least one acquisition abnormal event is classified according to the type of each acquisition abnormal event to obtain n types of acquisition abnormal events, n is a positive integer, m types of acquisition abnormal events are selected from the n types of acquisition abnormal events according to an index threshold value corresponding to each acquisition abnormal event in the n types of acquisition abnormal events, m is a positive integer not greater than n, and alarm prompt is carried out on the m types of acquisition abnormal events.
Specifically, when the data acquisition platform monitors the data acquisition process of the at least one data source system to be acquired, alarm filtering is required. For a large number of acquisition abnormal events, filtering is performed through preset filtering rules, and then the acquisition abnormal events are stored and uploaded to management staff in real time. That is, when a large number of acquisition abnormal events are detected, the data acquisition platform firstly acquires the types of the acquisition abnormal events, classifies the acquisition abnormal events according to the type of each acquisition abnormal event, and filters the acquisition abnormal events of different types through preset index thresholds after classifying, so as to determine the acquisition abnormal event needing to be warned and warn, and other measures, such as re-acquisition and the like, are taken for the acquisition abnormal event which does not need to be warned.
For example, when the detected acquisition abnormal event includes field mismatch, file absence and network abnormality, the number of records of field mismatch, the number of times of file absence and the number of times of network abnormality are obtained, index thresholds are preset for the field mismatch, the file absence and the network abnormality, which are a first index threshold, a second index threshold and a third index threshold respectively, when the number of records of field mismatch reaches the first index threshold, an alarm is triggered, otherwise filtered, when the number of times of file absence reaches the second index threshold, an alarm is triggered, otherwise filtered, and when the number of times of network abnormality reaches the third index threshold, an alarm is triggered, otherwise filtered.
In another possible embodiment, the data acquisition platform also supports recording data acquisition logs for viewing by the user during monitoring.
It can be seen that, according to the data acquisition method provided by the embodiment of the application, when data needs to be acquired from different data source systems, the plurality of data source systems are in butt joint through the unified data acquisition platform, and an acquisition tool does not need to be developed for each data source system, so that development difficulty is reduced, and data acquisition efficiency is improved. Moreover, the acquisition tasks of each data source system are distributed in a unified data acquisition platform, so that the acquisition tasks can be managed uniformly, and the data acquisition efficiency is improved.
Referring to fig. 4, fig. 4 is a schematic diagram of a data acquisition device according to an embodiment of the present application, where the data acquisition device is applied to a data acquisition platform. As shown in fig. 4, a data acquisition device provided in an embodiment of the present application may include:
the acquiring module 401 is configured to acquire system information of at least one data source system to be acquired.
The first configuration module 402 is configured to configure an acquisition interface parameter of each data source system to be acquired according to system information of each data source system to be acquired in the at least one data source system to be acquired.
And the docking module 403 is configured to dock the at least one data source system to be collected according to the collection interface parameter of each data source system to be collected.
A second configuration module 404, configured to configure the acquisition task parameters of each data source system to be acquired.
And the acquisition module 405 is configured to perform data acquisition on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
The specific implementation of the data acquisition device in this embodiment may refer to each embodiment of the data acquisition method, and will not be described herein.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device of a hardware running environment according to an embodiment of the present application. As shown in fig. 5, an electronic device of a hardware running environment according to an embodiment of the present application may include:
a processor 501, such as a CPU.
The memory 502 may alternatively be a high-speed RAM memory or a stable memory, such as a disk memory.
A communication interface 503 for enabling a connected communication between the processor 501 and the memory 502.
It will be appreciated by those skilled in the art that the configuration of the electronic device shown in fig. 5 is not limiting of the electronic device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 5, the memory 502 may include an operating system, a network communication module, and a data collection program. An operating system is a program that manages and controls the hardware and software resources of an electronic device, supporting the execution of data collection programs and other software or programs. The network communication module is used to enable communication between components within the memory 502 and with other hardware and software in the electronic device.
In the electronic device shown in fig. 5, the processor 501 is configured to execute a data acquisition program stored in the memory 502, and implement the following steps:
acquiring system information of at least one data source system to be acquired;
configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
according to the acquisition interface parameters of each data source system to be acquired, docking the at least one data source system to be acquired;
configuring acquisition task parameters of each data source system to be acquired;
and carrying out data acquisition on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
The specific implementation of the electronic device in the embodiment of the present application may refer to each embodiment of the data acquisition method, which is not described herein.
Another embodiment of the present application provides a computer readable storage medium storing a computer program for execution by a processor to:
acquiring system information of at least one data source system to be acquired;
configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
according to the acquisition interface parameters of each data source system to be acquired, docking the at least one data source system to be acquired;
configuring acquisition task parameters of each data source system to be acquired;
and carrying out data acquisition on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired.
The specific implementation of the computer readable storage medium in the embodiments of the present application may refer to each embodiment of the data acquisition method, which is not described herein.
It should also be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood and appreciated by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required in the present application. In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.
The above embodiments are merely for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims (7)
1. A data acquisition method, wherein the method is applied to a data acquisition platform, the method comprising:
acquiring system information of at least one data source system to be acquired;
configuring acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
according to the acquisition interface parameters of each data source system to be acquired, docking the at least one data source system to be acquired;
configuring database information of each data source system to be acquired;
determining a database corresponding to each data source system to be acquired according to the database information, wherein the database is used for storing the data acquired from the corresponding data source system to be acquired;
configuring acquisition task parameters of each data source system to be acquired; according to the acquisition task parameters of each data source system to be acquired, data acquisition is carried out on at least one data source system to be acquired;
detecting the data acquired from the at least one data source system to be acquired to judge whether the data acquisition fails or not;
if the data acquisition fails, acquiring an acquisition failure type; the acquisition failure types comprise interface call failure, network abnormality and service abnormality, and acquisition strategies corresponding to different acquisition failure types are different;
according to the acquisition failure type, carrying out data acquisition on the at least one data source system to be acquired again;
when data acquisition is carried out, acquiring first updating time of acquired data, comparing the first updating time with second updating time of latest updating of the data in an acquired data source system, and if the first updating time is the same as the second updating time, continuing to execute a data acquisition task; if the first updating time is smaller than the second updating time, the data are re-acquired;
monitoring a data acquisition process of the at least one data source system to be acquired;
when at least one acquisition abnormal event is detected, acquiring the type of each acquisition abnormal event in the at least one acquisition abnormal event; classifying the at least one acquisition abnormal event according to the type of each acquisition abnormal event to obtain n categories of acquisition abnormal events, wherein n is a positive integer; determining m types of acquisition abnormal events from the n types of acquisition abnormal events according to index thresholds corresponding to each type of acquisition abnormal event in the n types of acquisition abnormal events, wherein m is a positive integer not greater than n; alarming and prompting the m types of acquisition abnormal events; for the acquisition abnormal event which does not need to be alarmed, other measures are taken;
when the detected n types of acquisition abnormal events comprise field mismatching and file nonexistence, triggering an alarm if the number of records with the field mismatching reaches a first index threshold corresponding to the field mismatching, otherwise, filtering the alarm; and triggering an alarm if the number of times that the file does not exist reaches a second index threshold corresponding to the file does not exist, otherwise, filtering the alarm.
2. The method of claim 1, wherein the data acquisition platform includes an interface parser, the interfacing the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired, comprising:
analyzing the acquisition interface parameters of each data source system to be acquired through the interface analyzer to determine an acquisition interface matched with each data source system to be acquired, wherein the acquisition interface comprises at least one of a file acquisition interface, a database acquisition interface and a network service acquisition interface;
and docking the at least one data source system to be acquired through an acquisition interface matched with each data source system to be acquired.
3. The method according to claim 1, wherein the acquisition task parameters include task execution time, and the performing data acquisition on the at least one data source system according to the acquisition task parameters of each data source system to be acquired includes:
acquiring the current time;
and if the current time reaches the task execution time, carrying out data acquisition on the at least one data source system to be acquired.
4. The method of claim 1, wherein after the re-data acquisition of the at least one data source system to be acquired according to the acquisition failure type, the method further comprises:
acquiring acquisition times of carrying out data acquisition on the at least one data source system to be acquired again;
when the acquisition times are smaller than a preset time threshold, continuing to acquire data of the at least one data source system to be acquired again until the data acquisition is successful;
and when the acquisition times are not smaller than the preset times threshold, sending prompt information to preset management personnel, wherein the prompt information is used for prompting the preset management personnel to acquire manually.
5. A data acquisition device, the device being applied to a data acquisition platform, the device comprising:
the acquisition module is used for acquiring system information of at least one data source system to be acquired;
the first configuration module is used for configuring the acquisition interface parameters of each data source system to be acquired according to the system information of each data source system to be acquired in the at least one data source system to be acquired;
the docking module is used for docking the at least one data source system to be acquired according to the acquisition interface parameters of each data source system to be acquired;
the second configuration module is used for configuring the acquisition task parameters and database information of each data source system to be acquired; determining a database corresponding to each data source system to be acquired according to the database information, wherein the database is used for storing the data acquired from the corresponding data source system to be acquired;
the acquisition module is used for carrying out data acquisition on the at least one data source system to be acquired according to the acquisition task parameters of each data source system to be acquired;
the acquisition module is further used for detecting the data acquired from the at least one data source system to be acquired so as to judge whether the data acquisition fails; if the data acquisition fails, acquiring an acquisition failure type; the acquisition failure types comprise interface call failure, network abnormality and service abnormality, and acquisition strategies corresponding to different acquisition failure types are different; according to the acquisition failure type, carrying out data acquisition on the at least one data source system to be acquired again;
the acquisition module is specifically configured to acquire a first update time of acquired data when the data is acquired, compare the first update time with a second update time of a latest update of the data in the acquired data source system, and if the first update time is the same as the second update time, continue to execute a data acquisition task; if the first updating time is smaller than the second updating time, the data are re-acquired;
the acquisition module is further used for monitoring the data acquisition process of the at least one data source system to be acquired; when at least one acquisition abnormal event is detected, acquiring the type of each acquisition abnormal event in the at least one acquisition abnormal event; classifying the at least one acquisition abnormal event according to the type of each acquisition abnormal event to obtain n categories of acquisition abnormal events, wherein n is a positive integer; determining m types of acquisition abnormal events from the n types of acquisition abnormal events according to index thresholds corresponding to each type of acquisition abnormal event in the n types of acquisition abnormal events, wherein m is a positive integer not greater than n; alarming and prompting the m types of acquisition abnormal events; for the acquisition abnormal event which does not need to be alarmed, other measures are taken;
when the detected n types of acquisition abnormal events comprise field mismatching and file nonexistence, triggering an alarm if the number of records with the field mismatching reaches a first index threshold corresponding to the field mismatching, otherwise, filtering the alarm; and triggering an alarm if the number of times that the file does not exist reaches a second index threshold corresponding to the file does not exist, otherwise, filtering the alarm.
6. An electronic device comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured for execution by the processor, the programs comprising instructions for performing the steps of the method of any of claims 1-4.
7. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, which is executed by a processor to implement the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010888919.4A CN112015627B (en) | 2020-08-28 | 2020-08-28 | Data acquisition method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010888919.4A CN112015627B (en) | 2020-08-28 | 2020-08-28 | Data acquisition method and related device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112015627A CN112015627A (en) | 2020-12-01 |
| CN112015627B true CN112015627B (en) | 2023-04-28 |
Family
ID=73503866
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010888919.4A Active CN112015627B (en) | 2020-08-28 | 2020-08-28 | Data acquisition method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112015627B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114697204A (en) * | 2022-03-21 | 2022-07-01 | 昭通亮风台信息科技有限公司 | Log collection method and system for distributed system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678726A (en) * | 2012-09-05 | 2014-03-26 | 亿阳信通股份有限公司 | Data complementary collection method and system |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103095492A (en) * | 2012-12-28 | 2013-05-08 | 中兴通讯股份有限公司 | Data collection method and data collection device |
| CN104111983B (en) * | 2014-06-30 | 2017-12-19 | 中国科学院信息工程研究所 | A kind of open multi-source data acquiring system and method |
| CN110032587A (en) * | 2019-04-17 | 2019-07-19 | 上海圣剑网络科技股份有限公司 | Data summarization method, device, computer equipment and storage medium |
| CN110874265A (en) * | 2019-11-15 | 2020-03-10 | 厦门物通博联网络科技有限公司 | Method, device and equipment for rapidly acquiring data and storage medium |
| CN111475556A (en) * | 2020-03-19 | 2020-07-31 | 平安国际智慧城市科技股份有限公司 | Data acquisition method, data acquisition equipment, server and readable storage medium |
| CN111459924A (en) * | 2020-03-24 | 2020-07-28 | 上海携程商务有限公司 | Data acquisition method, system, electronic device, and medium |
-
2020
- 2020-08-28 CN CN202010888919.4A patent/CN112015627B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678726A (en) * | 2012-09-05 | 2014-03-26 | 亿阳信通股份有限公司 | Data complementary collection method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112015627A (en) | 2020-12-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10901727B2 (en) | Monitoring code sensitivity to cause software build breaks during software project development | |
| CN107886238B (en) | Business process management system and method based on mass data analysis | |
| WO2021179574A1 (en) | Root cause localization method, device, computer apparatus, and storage medium | |
| US10310968B2 (en) | Developing software project plans based on developer sensitivity ratings detected from monitoring developer error patterns | |
| CN111629043B (en) | Cross-platform health management system based on cloud mode | |
| CN107612779B (en) | Secondary safety protection network equipment of dispatching data network and service operation monitoring system | |
| CN108521339B (en) | Feedback type node fault processing method and system based on cluster log | |
| US20220050765A1 (en) | Method for processing logs in a computer system for events identified as abnormal and revealing solutions, electronic device, and cloud server | |
| CN104335056B (en) | It is used to separate and control the inserter of the different requests of multiple entities in test cell operation between tester and material processing apparatuses | |
| CN104583789B (en) | The decision-making of test cell controller and the establishment and scheduling for performing tree | |
| CN113836044B (en) | Method and system for collecting and analyzing software faults | |
| CN101848477A (en) | Method and system for diagnosing fault | |
| CN110088744B (en) | Database maintenance method and system | |
| CN109165138A (en) | A kind of method and apparatus of monitoring equipment fault | |
| US20120246520A1 (en) | Monitoring method, information processing apparatus, and computer-readable medium storing monitoring program | |
| CN116880412A (en) | Visual production management platform based on cloud | |
| CN112000806A (en) | Abnormal log monitoring and analyzing method, system, equipment and storage medium | |
| CN112015627B (en) | Data acquisition method and related device | |
| CN117422434A (en) | Wisdom fortune dimension dispatch platform | |
| CN105207797A (en) | Fault locating method and fault locating device | |
| CN116204385A (en) | Computer log monitoring method and system | |
| CN105786679B (en) | A kind of automatic test monitoring system and method and mobile terminal | |
| CN116560893B (en) | Computer application program operation data fault processing system | |
| CN116594840A (en) | Log fault acquisition and analysis method, system, equipment and medium based on ELK | |
| CN116303040A (en) | Inspection reminding method, device, equipment and storage medium for report sending |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210202 Address after: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.) Applicant after: Shenzhen saiante Technology Service Co.,Ltd. Address before: 1-34 / F, Qianhai free trade building, 3048 Xinghai Avenue, Mawan, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong 518000 Applicant before: Ping An International Smart City Technology Co.,Ltd. |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |