CN111475556A - Data acquisition method, data acquisition equipment, server and readable storage medium - Google Patents
Data acquisition method, data acquisition equipment, server and readable storage medium Download PDFInfo
- Publication number
- CN111475556A CN111475556A CN202010199714.5A CN202010199714A CN111475556A CN 111475556 A CN111475556 A CN 111475556A CN 202010199714 A CN202010199714 A CN 202010199714A CN 111475556 A CN111475556 A CN 111475556A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- acquired
- parameters
- acquiring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Bioethics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a data acquisition method, data acquisition equipment, a server and a readable storage medium, wherein the method comprises the following steps: detecting whether the current time is matched with the acquisition time of the task to be acquired; if the current time is matched with the acquisition time of the task to be acquired, acquiring task parameters corresponding to the task to be acquired, and acquiring data corresponding to the task to be acquired according to the task parameters; if the abnormality is monitored in the data acquisition process, determining the abnormality type according to the abnormality information, and processing the abnormality according to the abnormality type, wherein the abnormality type comprises data abnormality and/or network abnormality; and if the integrity verification of the collected data fails, checking the collected data so as to ensure that the integrity verification of the checked data is successful. By the implementation mode, the timeliness of the collected data is ensured, the integrity and the quality of the data are ensured, and the data accuracy and the data safety are improved.
Description
Technical Field
The present invention relates to the field of data acquisition technologies, and in particular, to a data acquisition method, a device, a server, and a readable storage medium.
Background
Data acquisition is the primary step of data analysis, data acquisition logic is customized and developed for different data source types in a mode used for data aggregation in the industry at present, management of data sources, monitoring of acquisition tasks, mapping relations of target databases and the like are distributed in different tool environments in the data acquisition process, the whole data acquisition life cycle is difficult to manage, and a good scheme is not provided on the aspect of guaranteeing the integrity of acquired data.
Disclosure of Invention
The embodiment of the invention provides a data acquisition method, data acquisition equipment, a server and a readable storage medium, wherein the data acquisition task is triggered when the current time is matched with the acquisition time of a task to be acquired, the timeliness of the acquired data is ensured, the integrity and the quality of the data are ensured through data verification, the data accuracy is improved, and the user is prompted to process abnormal information through alarm information, so that the data safety is further improved.
In a first aspect, an embodiment of the present invention provides a data acquisition method, including:
detecting whether the current time is matched with the acquisition time of the task to be acquired;
if the current time is matched with the acquisition time of the task to be acquired, acquiring task parameters corresponding to the task to be acquired, and acquiring data corresponding to the task to be acquired according to the task parameters;
if an exception is monitored in the data acquisition process, determining an exception type according to exception information, and processing the exception according to the exception type, wherein the exception type comprises data exception and/or network exception;
and carrying out integrity verification on the data acquired in the data acquisition process, and if the integrity verification on the acquired data fails, carrying out verification processing on the acquired data so as to ensure that the integrity verification on the data subjected to the verification processing is successful.
Further, the task parameters include acquisition interface parameters, acquiring the task parameters corresponding to the task to be acquired, and acquiring data corresponding to the task to be acquired according to the task parameters, including:
acquiring acquisition interface parameters corresponding to the tasks to be acquired;
and determining a corresponding target acquisition interface according to the acquisition interface parameters, and acquiring data corresponding to the task to be acquired from the target acquisition interface.
Further, the task parameters include database parameters, and acquiring data corresponding to the task to be acquired according to the task parameters corresponding to the task to be acquired includes:
acquiring database parameters corresponding to the tasks to be acquired, wherein the database parameters comprise database identifiers and/or database interfaces;
and determining a corresponding target database according to a database identifier and/or a database interface included in the database parameters, and acquiring data corresponding to the task to be acquired from the target database.
Further, the task parameter includes a file identifier, and acquiring data corresponding to the task to be acquired according to the task parameter corresponding to the task to be acquired includes:
acquiring a file identifier corresponding to the task to be acquired;
and determining a corresponding target file according to the file identifier, and acquiring data corresponding to the task to be acquired from the target file.
Further, the acquiring data corresponding to the task to be acquired according to the task parameter includes:
acquiring the data volume of the data corresponding to the task to be acquired;
if the data volume is detected to be larger than a preset threshold value, splitting data corresponding to the task to be acquired according to the data volume to obtain a plurality of data slices to be acquired;
and parallelly acquiring the plurality of data slices to be acquired according to the task parameters through a plurality of threads.
Further, the processing the exception according to the exception category includes:
if the abnormal type is determined to be abnormal data, modifying the abnormal data according to the abnormal information; and/or the presence of a gas in the gas,
and if the abnormal type is determined to be the network abnormality, generating alarm information, outputting and displaying the alarm information, wherein the alarm information is used for prompting a user of the network abnormality and checking the network.
Further, if the integrity verification of the collected data fails, the checking process of the collected data includes:
if the data corresponding to the task to be collected is detected to include the missed collection data except the collected data, determining that the integrity verification of the collected data fails, and collecting the missed collection data according to the task parameters; alternatively, the first and second electrodes may be,
and if the acquired data is detected to comprise redundant data except the data corresponding to the task to be acquired, determining that the integrity verification of the acquired data fails, and deleting the redundant data.
In a second aspect, an embodiment of the present invention provides a data acquisition device, which includes a unit for executing the data acquisition method of the first aspect.
In a third aspect, an embodiment of the present invention provides a server, including a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, where the memory is used to store a computer program that supports a data acquisition device to execute the method described above, and the computer program includes a program, and the processor is configured to call the program to execute the method described above in the first aspect.
In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program, where the computer program is executed by a processor to implement the method of the first aspect.
In the embodiment of the invention, if the data acquisition equipment detects that the current time is matched with the acquisition time of the task to be acquired, the data acquisition equipment can acquire the task parameter corresponding to the task to be acquired and acquire the data corresponding to the task to be acquired according to the task parameter, if the abnormality is monitored in the data acquisition process, the abnormality type can be determined according to the abnormality information, the abnormality is processed according to the abnormality type, the acquired data is subjected to integrity verification, and if the integrity verification of the acquired data fails, the acquired data can be subjected to verification processing, so that the integrity verification of the data subjected to the verification processing is successful. The data acquisition task is triggered when the current time is matched with the acquisition time of the task to be acquired, the timeliness of acquired data is ensured, the integrity and quality of the data are ensured through data verification, the data accuracy is improved, a user is prompted through the alarm information to process abnormal information, and the data safety is further improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention;
fig. 2 is a schematic block diagram of a data acquisition device according to an embodiment of the present invention;
fig. 3 is a schematic block diagram of a server according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data acquisition method provided by the embodiment of the invention can be executed by data acquisition equipment, wherein the data acquisition equipment can be arranged on an acquisition platform, and the acquisition platform can be a server. In some embodiments, the data acquisition device may be installed on a server; in some embodiments, the data acquisition device may be spatially independent of the server; in certain embodiments, the data acquisition device may be a component of the server, i.e. the server comprises a data acquisition device.
In the embodiment of the invention, if the data acquisition equipment detects that the current time is matched with the acquisition time of the task to be acquired, the data acquisition equipment can acquire the task parameters corresponding to the task to be acquired and acquire the data corresponding to the task to be acquired according to the task parameters, and if the abnormality is monitored in the data acquisition process, the data acquisition equipment determines the abnormality type according to the abnormality information and processes the abnormality according to the abnormality type, wherein the abnormality type comprises data abnormality and/or network abnormality. And carrying out integrity verification on the data acquired in the data acquisition process, and if the integrity verification on the acquired data fails, carrying out verification processing on the acquired data so as to ensure that the integrity verification on the data subjected to the verification processing is successful. The data acquisition task is triggered when the current time is matched with the acquisition time of the task to be acquired, the timeliness of acquired data is ensured, the integrity and quality of the data are ensured through data verification, the data accuracy is improved, a user is prompted through the alarm information to process abnormal information, and the data safety is further improved.
The data acquisition method according to the embodiment of the present invention is schematically described below with reference to the drawings.
Referring to fig. 1, fig. 1 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention, and as shown in fig. 1, the method may be executed by a data acquisition device, and a specific explanation of the data acquisition device is as described above, which is not described herein again. Specifically, the method of the embodiment of the present invention includes the following steps.
S101: and detecting whether the current time is matched with the acquisition time of the task to be acquired.
In the embodiment of the invention, the data acquisition equipment can detect whether the current time is matched with the acquisition time of the task to be acquired.
In one embodiment, when detecting whether the current time matches the acquisition time of the task to be acquired, the data acquisition device may detect whether the current time matches the acquisition time including that the current time matches the acquisition time of the task to be acquired, and if so, may determine that the current time matches the acquisition time of the task to be acquired.
In one embodiment, the data acquisition device may detect whether the current time is within a specified time range before the acquisition time of the task to be acquired when detecting whether the current time matches the acquisition time of the task to be acquired, and may determine that the current time matches the acquisition time of the task to be acquired if detecting that the current time is within the specified time range before the acquisition time of the task to be acquired. In some embodiments, the specified time range may be any time range before the acquisition time, and the embodiments of the present invention are not particularly limited, and in one example, the specified time range may be within 30 minutes before the acquisition time.
Therefore, the data of the tasks to be acquired can be acquired in time by triggering the acquisition of the data of the tasks to be acquired within the specified time range between the acquisition times.
In one embodiment, the data collection device may output a collection task configuration interface prior to detecting whether the current time matches the collection time of the task to be collected. In some embodiments, the collection task configuration interface is configured to configure any one or more of a collection time, a collection environment, and a task parameter of the task to be collected.
In some embodiments, the collection task configuration interface is a visual configuration interface, supports online collection information configuration, and performs automatic collection according to task configuration, and a user can select different adaptation modes (including database collection, interface collection, file collection, and the like) to perform data collection according to the construction degree of a source system; in order to improve the data acquisition performance, parallel acquisition can be supported; the acquisition task can be monitored, and the failed task is retried, so that the integrity of the acquired data is ensured; the visual task configuration interface can enable a user to manage the acquisition process more visually, the acquired data coverage is more comprehensive, the acquisition performance is higher, and the visual task configuration interface is adaptive to various systems and has stronger compatibility.
S102: and if the current time is matched with the acquisition time of the task to be acquired, acquiring task parameters corresponding to the task to be acquired, and acquiring data corresponding to the task to be acquired according to the task parameters.
In the embodiment of the invention, if the data acquisition equipment detects that the current time is matched with the acquisition time of the task to be acquired, the data acquisition equipment can acquire the task parameter corresponding to the task to be acquired and acquire the data corresponding to the task to be acquired according to the task parameter.
In some embodiments, the task parameters include, but are not limited to, any one or more of collection interface parameters, database parameters, file identifications, and the like.
In some embodiments, the interface acquisition is to acquire data through an acquisition interface, and the acquisition interface may include an interface parser of a system built-in main flow, capable of supporting: files, databases, WebService, Representational State Transfer-Application Programming Interface (REST-API), etc., and in some embodiments, Web Service is a platform-independent, low-coupling, self-contained, programmable Web-based Application. In certain embodiments, the collection tasks include data collection and data processing, including the addition, modification, deletion, etc. of data. In certain embodiments, the database collection is collection of data by a database. In certain embodiments, the document collection is collecting data via a document.
In one embodiment, when acquiring the task parameters corresponding to the task to be acquired and acquiring the data corresponding to the task to be acquired according to the task parameters, the data acquisition device may acquire an adaptation mode carried in the task to be acquired and determine the task parameters corresponding to the task to be acquired according to the adaptation mode, so as to acquire the data corresponding to the task to be acquired according to the task parameters. In some embodiments, the adaptation modes carried in the task to be acquired can be determined according to the adaptation identifiers, and different adaptation modes correspond to different adaptation identifiers; in one example, the adaptation identifiers corresponding to the database collection, the interface collection and the file collection are different. In certain embodiments, the adaptation indicator includes, but is not limited to, numbers, letters, words, and the like.
In an embodiment, when the data acquisition device acquires the task parameters corresponding to the task to be acquired and acquires the data corresponding to the task to be acquired according to the task parameters, the data acquisition device may determine the task parameters corresponding to the task to be acquired according to a preset corresponding relationship between the acquisition task and the task parameters.
Therefore, by the implementation mode, the task parameters corresponding to the tasks to be acquired can be automatically determined, and the data acquisition efficiency is improved.
In one embodiment, the task parameters include acquisition interface parameters, and when the data acquisition device acquires the task parameters corresponding to the task to be acquired and acquires the data corresponding to the task to be acquired according to the task parameters, the data acquisition device may acquire the acquisition interface parameters corresponding to the task to be acquired, determine a corresponding target acquisition interface according to the acquisition interface parameters, and acquire the data corresponding to the task to be acquired from the target acquisition interface. In certain embodiments, the acquisition interface parameters include, but are not limited to, an interface identification.
For example, if the data acquisition device determines the corresponding target acquisition interface 1 according to the acquisition interface parameters included in the task parameters of the task to be acquired, the data acquisition device may acquire data corresponding to the task to be acquired through the target acquisition interface 1.
In one embodiment, the task parameters include database parameters, and the data acquisition device may acquire the database parameters corresponding to the task to be acquired when acquiring the task parameters corresponding to the task to be acquired and acquiring data corresponding to the task to be acquired according to the task parameters, where the database parameters include database identifiers and/or database interfaces, determine a corresponding target database according to the database identifiers and/or database interfaces included in the database parameters, and acquire the data corresponding to the task to be acquired from the target database.
For example, if the data acquisition device determines the corresponding database interface 21 in the target database 2 according to the database parameters included in the task parameters of the task to be acquired, the data acquisition device may acquire data corresponding to the task to be acquired through the database interface 21 in the target database 2.
In one embodiment, the task parameters include file identifiers, and when the data acquisition device acquires the task parameters corresponding to the task to be acquired and acquires the data corresponding to the task to be acquired according to the task parameters, the data acquisition device may acquire the file identifiers corresponding to the task to be acquired, determine corresponding target files according to the file identifiers, and acquire the data corresponding to the task to be acquired from the target files. In some embodiments, the file identification includes, but is not limited to, path information of the file, and in some embodiments, the path information of the file includes, but is not limited to, a file name.
For example, if the data collection device determines that the corresponding target file is a file with a file name of the first file in the D disk according to the file identifier included in the task parameter of the task to be collected, the data collection device may collect data corresponding to the task to be collected from the first file.
Therefore, the efficiency and flexibility of data acquisition can be improved by acquiring data in multiple modes.
In an embodiment, when acquiring data corresponding to the task to be acquired according to the task parameter, the data acquisition device may acquire a data volume of the data corresponding to the task to be acquired, and if it is detected that the data volume is greater than a preset threshold, the data corresponding to the task to be acquired may be split according to the data volume to obtain a plurality of data slices to be acquired, and the plurality of data slices to be acquired are acquired in parallel according to the task parameter through a plurality of threads respectively.
In an embodiment, when the data acquisition device splits the data corresponding to the task to be acquired according to the data volume, the data corresponding to the task to be acquired may be split according to a preset splitting rule. In some embodiments, the preset splitting rule may be splitting according to a preset data amount; in some embodiments, the preset splitting rule may be that the data amount of the data corresponding to the task to be collected is split into n pieces on average; in other embodiments, other splitting rules may also be employed, which are not specifically limited herein.
For example, assuming that when data corresponding to the task to be acquired is acquired according to the task parameters by the data acquisition device, the data volume of the data corresponding to the task to be acquired is X, if it is detected that the data volume X is greater than a preset threshold Y, the data corresponding to the task to be acquired may be averagely split according to the data volume X to obtain n data slices to be acquired, and the n data slices to be acquired are acquired in parallel according to the task parameters by a plurality of threads respectively.
Therefore, when data with large data volume is acquired, the data is split into a plurality of data slices, and the data is acquired in parallel through multiple threads, so that the data acquisition efficiency is improved.
S103: if the abnormality is monitored in the data acquisition process, determining the abnormality type according to the abnormality information, and processing the abnormality according to the abnormality type, wherein the abnormality type comprises data abnormality and/or network abnormality.
In the embodiment of the invention, if the data acquisition equipment monitors the abnormity in the data acquisition process, the abnormity category can be determined according to the abnormity information, and the abnormity is processed according to the abnormity category, wherein the abnormity category comprises data abnormity and/or network abnormity.
In an embodiment, after monitoring the exception, the data acquisition device may perform acquisition of data corresponding to the task to be acquired according to the task parameter again before determining the exception type according to the exception information, and after performing acquisition of data corresponding to the task to be acquired according to the task parameter again, monitor whether the exception exists again, if the exception exists, acquire the exception information, and determine the exception type according to the exception information. In some embodiments, the data acquisition device may perform acquisition of data corresponding to the task to be acquired again for a specified number of times according to the task parameter, and monitor whether an exception exists after the specified number of times is performed.
For example, after monitoring the abnormality, the data acquisition device may perform the step of acquiring data corresponding to the task to be acquired according to the task parameter k times again before determining the abnormality type according to the abnormality information, and if it is monitored that there is an abnormality again after acquiring data corresponding to the task to be acquired according to the task parameter k times again, may acquire the abnormality information and determine the abnormality type according to the abnormality information.
In one embodiment, if the data acquisition device determines that the abnormal type is data abnormal, the data acquisition device may modify the abnormal data according to the abnormal information. In some embodiments, the data exception includes, but is not limited to, an exception such as data absence, data carrying virus data, data being tampered with, and the like, and is not limited herein.
In an embodiment, when the data acquisition device modifies the abnormal data according to the abnormal information, the data acquisition device may determine the type of the abnormal data according to the abnormal information, and if it is determined that the type of the abnormal data is data-nonexistent, the data acquisition device may delete the abnormal data. In an embodiment, if it is determined that the type of the abnormal data is data carrying virus data, the abnormal data may be deleted or the virus data carried by the abnormal data may be deleted. In an embodiment, if it is determined that the type of the abnormal data is data tampered, the abnormal data may be restored to data before being tampered according to a history tampering record of the abnormal data.
By the implementation mode, the abnormal data can be automatically processed when the abnormal data is monitored, so that the safety and the effectiveness of the acquired data are improved.
In an embodiment, if the data acquisition device determines that the abnormality type is a network abnormality, it may generate alarm information, and output and display the alarm information, where the alarm information is used to prompt a user to handle the abnormality. In certain embodiments, the network anomalies include, but are not limited to, network disconnection, network slowness, and the like. By means of outputting the alarm information, the method is helpful for reminding the user to check the network in time so as to avoid delaying data acquisition, and therefore the data acquisition efficiency is improved.
In one embodiment, after generating the alarm information according to the abnormal information and before outputting and displaying the alarm information, the data acquisition device may calculate text similarity between the alarm information, determine that the alarm information with the text similarity greater than a preset similarity threshold is repeated alarm information, and filter the repeated alarm information. In some embodiments, the data acquisition device may calculate the text similarity between the alarm information by using a vector inner product, and may also calculate the text similarity between the alarm information by using a cosine similarity, which is not specifically limited herein.
For example, if the preset similarity threshold is 80%, and the data acquisition device acquires 3 pieces of alarm information, the data acquisition device may calculate the text similarity between each piece of alarm information in the 3 pieces of alarm information by using a cosine similarity calculation method, and if the text similarity between 2 pieces of alarm information is 85%, it may be determined that 85% of the text similarity between the 2 pieces of alarm information is greater than the preset similarity threshold 80%, and therefore the 2 pieces of alarm information may be repeated alarm information, and one of the 2 pieces of repeated alarm information may be deleted to perform filtering.
In one embodiment, when the data collection device filters the alarm information, if the existence of the repeated alarm information is detected, one of the repeated alarm information may be retained, and the rest of the repeated alarm information may be deleted.
For example, when the data collection device filters the alarm information, if it is detected that there are 3 repeated alarm information, one of the 3 repeated alarm information may be retained, and the remaining 2 repeated alarm information may be deleted.
In one embodiment, after generating alarm information according to abnormal information and before outputting and displaying the alarm information, the data acquisition device may acquire the number of alarm information generated according to the abnormal information, and if the number of alarm information is greater than a preset number threshold, may filter the alarm information and output and display the filtered alarm information.
For example, assuming that the preset number threshold is 8, if the number of alarm information acquired by the data acquisition device according to the abnormal information is 10, it may be determined that the number of alarm information 10 is greater than the preset number threshold 8, and therefore the alarm information may be filtered, and the filtered alarm information may be output and displayed.
In one embodiment, when the data acquisition device filters the alarm information, the severity of each alarm information may be calculated, and the alarm information with the severity smaller than a preset threshold may be deleted.
In some embodiments, when the data acquisition device calculates the severity of the alarm information, the data acquisition device may calculate the severity of the alarm information according to a keyword related to the severity in the alarm information. In one embodiment, the data acquisition device may calculate the severity of the warning information according to a ratio of the number of keywords related to the severity in the warning information to the number of all the words in the warning information; in one example, assuming that the number of keywords related to the severity in the warning information is 6 and the number of all words in the warning information is 16, the severity of the warning information may be calculated to be 6/16. Of course, the embodiment of the present invention may also calculate the severity of the warning information in other manners, which is not limited herein.
For example, if the preset threshold is 50%, and the data acquisition device calculates the severity of an alarm message to be 60% when filtering the alarm message, it may be determined that the severity of 60% is less than the preset threshold 50%, and therefore the alarm message may be deleted.
In other embodiments, of course, the data acquisition device may also filter the alarm information in other manners, and the embodiments of the present invention are not limited in particular.
Therefore, through the implementation mode of filtering the alarm information, the situation that the repeated or non-processed alarm information occupies the system bandwidth can be avoided, and the data acquisition efficiency is reduced.
S104: and carrying out integrity verification on the data acquired in the data acquisition process, and if the integrity verification on the acquired data fails, carrying out verification processing on the acquired data so as to ensure that the integrity verification on the data subjected to the verification processing is successful.
In the embodiment of the invention, the data acquisition equipment can carry out integrity verification on the data acquired in the data acquisition process, and if the integrity verification on the acquired data fails, the acquired data can be verified, so that the integrity verification on the verified data is successful.
In some embodiments, the checking of the collected data includes adding, modifying, deleting, and the like of the data, where scheduling of data collection and data checking in the task to be collected is uniformly distributed by a collection platform.
In an embodiment, when the data acquisition device performs verification processing on the acquired data if the integrity verification of the acquired data fails, if it is detected that the data corresponding to the task to be acquired includes missed acquired data other than the acquired data, it may be determined that the integrity verification of the acquired data fails, and the missed acquired data other than the acquired data is acquired according to the task parameter. In some embodiments, the specific implementation of collecting the data that is not collected except the collected data according to the task parameter is as described above, and is not described herein again.
For example, if the data acquisition device detects that the data corresponding to the task to be acquired includes the missing acquisition data x other than the acquired data, the missing acquisition data x other than the acquired data may be acquired according to the task parameter.
Therefore, by the implementation mode of data acquisition omission complementation, the integrity and the quality of the acquired data can be ensured, and the accuracy of the data can be improved.
In an embodiment, when the data acquisition device performs verification processing on the acquired data if the integrity verification of the acquired data fails, if it is detected that the acquired data includes redundant data other than the data corresponding to the task to be acquired, it may be determined that the integrity verification of the acquired data fails, and the redundant data other than the data corresponding to the task to be acquired is deleted.
For example, if the data acquisition device detects that the acquired data includes redundant data y other than the data corresponding to the task to be acquired, the data acquisition device may delete the redundant data y other than the data corresponding to the task to be acquired.
Therefore, through the implementation mode of deleting the collected redundant data, the collected data can be prevented from including irrelevant redundant data, so that the accuracy of the data is improved.
In the embodiment of the invention, if the data acquisition equipment detects that the current time is matched with the acquisition time of the task to be acquired, the data acquisition equipment can acquire the task parameter corresponding to the task to be acquired and acquire the data corresponding to the task to be acquired according to the task parameter, and if the abnormality is monitored in the data acquisition process, the abnormality type can be determined according to the abnormality information and the abnormality is processed according to the abnormality type. By carrying out integrity verification on the data acquired in the data acquisition process, if the integrity verification on the acquired data fails, the acquired data can be verified, so that the integrity verification on the verified data is successful. According to the embodiment of the invention, the data acquisition task is triggered when the current time is matched with the acquisition time of the task to be acquired, the timeliness of the acquired data is ensured, the integrity and the quality of the data are ensured through data verification, the data accuracy is improved, and the user is prompted to process abnormal information through the alarm information, so that the data safety is further improved.
The embodiment of the invention also provides data acquisition equipment, which is used for executing the unit of the method in any one of the preceding claims. Specifically, referring to fig. 2, fig. 2 is a schematic block diagram of a data acquisition device according to an embodiment of the present invention. The data acquisition device of the embodiment includes: the device comprises a detection unit 201, an acquisition unit 202, a determination unit 203 and a processing unit 204.
A detecting unit 201, configured to detect whether a current time matches an acquisition time of a task to be acquired;
the acquisition unit 202 is configured to, if it is detected that the current time matches the acquisition time of the task to be acquired, acquire a task parameter corresponding to the task to be acquired, and acquire data corresponding to the task to be acquired according to the task parameter;
the determining unit 203 is configured to determine an exception category according to exception information if an exception is monitored in a data acquisition process, and process the exception according to the exception category, where the exception category includes a data exception and/or a network exception;
the processing unit 204 is configured to perform integrity verification on the data acquired in the data acquisition process, and if the integrity verification on the acquired data fails, perform verification on the acquired data, so that the integrity verification on the data after the verification is successful.
Further, the task parameters include acquisition interface parameters, and the acquisition unit 202 is specifically configured to, when acquiring the task parameters corresponding to the task to be acquired and acquiring the data corresponding to the task to be acquired according to the task parameters:
acquiring acquisition interface parameters corresponding to the tasks to be acquired;
and determining a corresponding target acquisition interface according to the acquisition interface parameters, and acquiring data corresponding to the task to be acquired from the target acquisition interface.
Further, the task parameters include database parameters, and the acquisition unit 202 is specifically configured to, when acquiring the task parameters corresponding to the task to be acquired and acquiring the data corresponding to the task to be acquired according to the task parameters:
acquiring database parameters corresponding to the tasks to be acquired, wherein the database parameters comprise database identifiers and/or database interfaces;
and determining a corresponding target database according to a database identifier and/or a database interface included in the database parameters, and acquiring data corresponding to the task to be acquired from the target database.
Further, the task parameters include file identifiers, and the acquisition unit 202 is specifically configured to, when acquiring the task parameters corresponding to the task to be acquired and acquiring the data corresponding to the task to be acquired according to the task parameters:
acquiring a file identifier corresponding to the task to be acquired;
and determining a corresponding target file according to the file identifier, and acquiring data corresponding to the task to be acquired from the target file.
Further, when the acquiring unit 202 acquires data corresponding to the task to be acquired according to the task parameter, the acquiring unit is specifically configured to:
acquiring the data volume of the data corresponding to the task to be acquired;
if the data volume is detected to be larger than a preset threshold value, splitting data corresponding to the task to be acquired according to the data volume to obtain a plurality of data slices to be acquired;
and parallelly acquiring the plurality of data slices to be acquired according to the task parameters through a plurality of threads.
Further, when the determining unit 203 processes the exception according to the exception type, it is specifically configured to:
if the abnormal type is determined to be abnormal data, modifying the abnormal data according to the abnormal information; and/or the presence of a gas in the gas,
and if the abnormal type is determined to be the network abnormality, generating alarm information, outputting and displaying the alarm information, wherein the alarm information is used for prompting a user of the network abnormality and checking the network.
Further, if the integrity verification of the collected data fails, the processing unit 204 is specifically configured to, when performing the verification process on the collected data:
if the data corresponding to the task to be collected is detected to include the missed collection data except the collected data, determining that the integrity verification of the collected data fails, and collecting the missed collection data according to the task parameters; alternatively, the first and second electrodes may be,
and if the acquired data is detected to comprise redundant data except the data corresponding to the task to be acquired, determining that the integrity verification of the acquired data fails, and deleting the redundant data.
In the embodiment of the invention, a data acquisition device detects whether the current time is matched with the acquisition time of a task to be acquired through a detection unit 201, if the current time is matched with the acquisition time of the task to be acquired, a task parameter corresponding to the task to be acquired is acquired through an acquisition unit 202, data corresponding to the task to be acquired is acquired according to the task parameter, if an abnormality is monitored in the data acquisition process, the abnormality type is determined through a determination unit 203 according to abnormality information, the abnormality is processed according to the abnormality type, the data acquired in the data acquisition process is subjected to integrity verification through a processing unit 204, and if the integrity verification of the acquired data fails, the acquired data is subjected to verification processing. According to the embodiment of the invention, the data acquisition task is triggered when the current time is matched with the acquisition time of the task to be acquired, the timeliness of the acquired data is ensured, the integrity and the quality of the data are ensured through data verification, the data accuracy is improved, and the user is prompted to process abnormal information through the alarm information, so that the data safety is further improved.
Referring to fig. 3, fig. 3 is a schematic block diagram of a server according to an embodiment of the present invention. The server in this embodiment as shown in the figure may include: one or more processors 301; one or more input devices 302, one or more output devices 303, and memory 304. The processor 301, the input device 302, the output device 303, and the memory 304 are connected by a bus 305. The memory 304 is used for storing computer programs, including programs, and the processor 301 is used for executing the programs stored in the memory 304. Wherein the processor 301 is configured to invoke the program to perform:
detecting whether the current time is matched with the acquisition time of the task to be acquired;
if the current time is matched with the acquisition time of the task to be acquired, acquiring task parameters corresponding to the task to be acquired, and acquiring data corresponding to the task to be acquired according to the task parameters;
if an exception is monitored in the data acquisition process, determining an exception type according to exception information, and processing the exception according to the exception type, wherein the exception type comprises data exception and/or network exception;
and carrying out integrity verification on the data acquired in the data acquisition process, and if the integrity verification on the acquired data fails, carrying out verification processing on the acquired data so as to ensure that the integrity verification on the data subjected to the verification processing is successful.
Further, the task parameters include acquisition interface parameters, and the processor 301 is specifically configured to, when acquiring the task parameters corresponding to the task to be acquired and acquiring the data corresponding to the task to be acquired according to the task parameters:
acquiring acquisition interface parameters corresponding to the tasks to be acquired;
and determining a corresponding target acquisition interface according to the acquisition interface parameters, and acquiring data corresponding to the task to be acquired from the target acquisition interface.
Further, the task parameters include database parameters, and the processor 301 is specifically configured to, when acquiring the task parameters corresponding to the task to be acquired and acquiring the data corresponding to the task to be acquired according to the task parameters:
acquiring database parameters corresponding to the tasks to be acquired, wherein the database parameters comprise database identifiers and/or database interfaces;
and determining a corresponding target database according to a database identifier and/or a database interface included in the database parameters, and acquiring data corresponding to the task to be acquired from the target database.
Further, the task parameters include file identifiers, and the processor 301 obtains the task parameters corresponding to the task to be collected, and when collecting data corresponding to the task to be collected according to the task parameters, is specifically configured to:
acquiring a file identifier corresponding to the task to be acquired;
and determining a corresponding target file according to the file identifier, and acquiring data corresponding to the task to be acquired from the target file.
Further, when the processor 301 acquires data corresponding to the task to be acquired according to the task parameter, the processor is specifically configured to:
acquiring the data volume of the data corresponding to the task to be acquired;
if the data volume is detected to be larger than a preset threshold value, splitting data corresponding to the task to be acquired according to the data volume to obtain a plurality of data slices to be acquired;
and parallelly acquiring the plurality of data slices to be acquired according to the task parameters through a plurality of threads.
Further, when the processor 301 processes the exception according to the exception type, it is specifically configured to:
if the abnormal type is determined to be abnormal data, modifying the abnormal data according to the abnormal information; and/or the presence of a gas in the gas,
and if the abnormal type is determined to be the network abnormality, generating alarm information, outputting and displaying the alarm information, wherein the alarm information is used for prompting a user of the network abnormality and checking the network.
Further, if the integrity verification of the collected data fails, the processor 301 is specifically configured to, when performing the verification process on the collected data:
if the data corresponding to the task to be collected is detected to include the missed collection data except the collected data, determining that the integrity verification of the collected data fails, and collecting the missed collection data according to the task parameters; alternatively, the first and second electrodes may be,
and if the acquired data is detected to comprise redundant data except the data corresponding to the task to be acquired, determining that the integrity verification of the acquired data fails, and deleting the redundant data.
In the embodiment of the invention, if the server detects that the current time is matched with the acquisition time of the task to be acquired, the server can acquire the task parameter corresponding to the task to be acquired and acquire the data corresponding to the task to be acquired according to the task parameter, and if the abnormality is monitored in the data acquisition process, the server can determine the abnormality type according to the abnormality information and process the abnormality according to the abnormality type. By carrying out integrity verification on the data acquired in the data acquisition process, if the integrity verification on the acquired data fails, the acquired data can be verified, so that the integrity verification on the verified data is successful. According to the embodiment of the invention, the data acquisition task is triggered when the current time is matched with the acquisition time of the task to be acquired, the timeliness of the acquired data is ensured, the integrity and the quality of the data are ensured through data verification, the data accuracy is improved, and the user is prompted to process abnormal information through the alarm information, so that the data safety is further improved.
It should be understood that, in the embodiment of the present invention, the Processor 301 may be a Central Processing Unit (CPU), and the Processor may also be other general processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input device 302 may include a touch pad, microphone, etc., and the output device 303 may include a display (L CD, etc.), speakers, etc.
The memory 304 may include a read-only memory and a random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.
In a specific implementation, the processor 301, the input device 302, and the output device 303 described in this embodiment of the present invention may execute the implementation described in the method embodiment described in the figures of the data acquisition method provided in this embodiment of the present invention, or may execute the implementation of the data acquisition device described in fig. 2 in this embodiment of the present invention, which is not described herein again.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the data acquisition method described in the embodiment corresponding to fig. 1 may be implemented, or the data acquisition device according to the embodiment corresponding to fig. 2 may also be implemented, which is not described herein again.
The computer readable storage medium may be an internal storage unit of the data acquisition device according to any of the foregoing embodiments, for example, a hard disk or a memory of the data acquisition device. The computer readable storage medium may also be an external storage device of the data acquisition device, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the data acquisition device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the data acquisition device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the data acquisition device. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a computer-readable storage medium, which includes several instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned computer-readable storage media comprise: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only a part of the embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention.
Claims (10)
1. A method of data acquisition, comprising:
detecting whether the current time is matched with the acquisition time of the task to be acquired;
if the current time is matched with the acquisition time of the task to be acquired, acquiring task parameters corresponding to the task to be acquired, and acquiring data corresponding to the task to be acquired according to the task parameters;
if an exception is monitored in the data acquisition process, determining an exception type according to exception information, and processing the exception according to the exception type, wherein the exception type comprises data exception and/or network exception;
and carrying out integrity verification on the data acquired in the data acquisition process, and if the integrity verification on the acquired data fails, carrying out verification processing on the acquired data so as to ensure that the integrity verification on the data subjected to the verification processing is successful.
2. The method according to claim 1, wherein the task parameters include acquisition interface parameters, and the acquiring task parameters corresponding to the task to be acquired and acquiring data corresponding to the task to be acquired according to the task parameters comprises:
acquiring acquisition interface parameters corresponding to the tasks to be acquired;
and determining a corresponding target acquisition interface according to the acquisition interface parameters, and acquiring data corresponding to the task to be acquired from the target acquisition interface.
3. The method of claim 1, wherein the task parameters include database parameters, and wherein collecting data corresponding to the task to be collected according to the task parameters corresponding to the task to be collected comprises:
acquiring database parameters corresponding to the tasks to be acquired, wherein the database parameters comprise database identifiers and/or database interfaces;
and determining a corresponding target database according to a database identifier and/or a database interface included in the database parameters, and acquiring data corresponding to the task to be acquired from the target database.
4. The method according to claim 1, wherein the task parameters include file identifiers, and the acquiring data corresponding to the task to be acquired according to the task parameters corresponding to the task to be acquired comprises:
acquiring a file identifier corresponding to the task to be acquired;
and determining a corresponding target file according to the file identifier, and acquiring data corresponding to the task to be acquired from the target file.
5. The method according to any one of claims 1-4, wherein the collecting data corresponding to the task to be collected according to the task parameters comprises:
acquiring the data volume of the data corresponding to the task to be acquired;
if the data volume is detected to be larger than a preset threshold value, splitting data corresponding to the task to be acquired according to the data volume to obtain a plurality of data slices to be acquired;
and parallelly acquiring the plurality of data slices to be acquired according to the task parameters through a plurality of threads.
6. The method of claim 1, wherein said processing the exception according to the exception category comprises:
if the abnormal type is determined to be abnormal data, modifying the abnormal data according to the abnormal information; and/or the presence of a gas in the gas,
and if the abnormal type is determined to be the network abnormality, generating alarm information, outputting and displaying the alarm information, wherein the alarm information is used for prompting a user of the network abnormality and checking the network.
7. The method of claim 1, wherein the verifying the collected data if the integrity verification of the collected data fails comprises:
if the data corresponding to the task to be collected is detected to include the missed collection data except the collected data, determining that the integrity verification of the collected data fails, and collecting the missed collection data according to the task parameters; alternatively, the first and second electrodes may be,
and if the acquired data is detected to comprise redundant data except the data corresponding to the task to be acquired, determining that the integrity verification of the acquired data fails, and deleting the redundant data.
8. A data acquisition device comprising means for performing the method of any one of claims 1-7.
9. A server comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being interconnected, wherein the memory is configured to store a computer program, the computer program comprising a program, the processor being configured to invoke the program to perform the method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which is executed by a processor to implement the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010199714.5A CN111475556A (en) | 2020-03-19 | 2020-03-19 | Data acquisition method, data acquisition equipment, server and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010199714.5A CN111475556A (en) | 2020-03-19 | 2020-03-19 | Data acquisition method, data acquisition equipment, server and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111475556A true CN111475556A (en) | 2020-07-31 |
Family
ID=71747808
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010199714.5A Pending CN111475556A (en) | 2020-03-19 | 2020-03-19 | Data acquisition method, data acquisition equipment, server and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111475556A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112015627A (en) * | 2020-08-28 | 2020-12-01 | 平安国际智慧城市科技股份有限公司 | Data acquisition method and related device |
| CN113760587A (en) * | 2020-09-23 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for processing data |
-
2020
- 2020-03-19 CN CN202010199714.5A patent/CN111475556A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112015627A (en) * | 2020-08-28 | 2020-12-01 | 平安国际智慧城市科技股份有限公司 | Data acquisition method and related device |
| CN113760587A (en) * | 2020-09-23 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for processing data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110661659B (en) | Alarm method, device and system and electronic equipment | |
| CN110191094B (en) | Abnormal data monitoring method and device, storage medium and terminal | |
| US8245080B2 (en) | Model-based testing of an application program under test | |
| CN110912738B (en) | Business exception processing method, device, platform and electronic equipment | |
| EP3832479A1 (en) | Production line monitoring method and apparatus, and electronic device and readable storage medium | |
| US10657028B2 (en) | Method for replicating production behaviours in a development environment | |
| CN111475556A (en) | Data acquisition method, data acquisition equipment, server and readable storage medium | |
| CN112395156A (en) | Fault warning method and device, storage medium and electronic equipment | |
| CN109344046B (en) | Data processing method, device, medium and electronic equipment | |
| CN112988509A (en) | Alarm message filtering method and device, electronic equipment and storage medium | |
| CN115344416A (en) | Abnormal log screening method, system and device and computer readable storage equipment | |
| CN114968959A (en) | Log processing method, log processing device and storage medium | |
| KR101876629B1 (en) | Apparatus and method for monitoring condition based on bicdata analysis | |
| CN115658443B (en) | Log filtering method and device | |
| CN112612679A (en) | System running state monitoring method and device, computer equipment and storage medium | |
| CN114610560B (en) | System abnormality monitoring method, device and storage medium | |
| JP2006331026A (en) | Message analysis system and message analysis program | |
| CN111835566A (en) | System fault management method, device and system | |
| CN107465534B (en) | Multi-dimensional hot-plug method of system module | |
| US11372839B2 (en) | Anomalous event confirmation assistance apparatus, anomalous event confirmation assistance meithod, and recording medium | |
| CN117707820A (en) | Fault tracing method, device, computer equipment and storage medium | |
| CN114936113B (en) | Task avalanche recovery method and device, electronic equipment and storage medium | |
| CN117376186A (en) | Data detection method, apparatus, device, storage medium and computer program product | |
| CN116841971A (en) | Method, system, memory and electronic equipment for monitoring abnormal behaviors of users of linux operating system | |
| CN112346662A (en) | Disk cleaning method, device and system based on disk analysis and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210202 Address after: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.) Applicant after: Shenzhen saiante Technology Service Co.,Ltd. Address before: 1-34 / F, Qianhai free trade building, 3048 Xinghai Avenue, Mawan, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong 518000 Applicant before: Ping An International Smart City Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |