CN116109322A - Data acquisition method, data acquisition device, and computer-readable storage medium - Google Patents
Data acquisition method, data acquisition device, and computer-readable storage medium Download PDFInfo
- Publication number
- CN116109322A CN116109322A CN202211347426.5A CN202211347426A CN116109322A CN 116109322 A CN116109322 A CN 116109322A CN 202211347426 A CN202211347426 A CN 202211347426A CN 116109322 A CN116109322 A CN 116109322A
- Authority
- CN
- China
- Prior art keywords
- target
- data
- acquisition
- data source
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/018—Certifying business or products
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Finance (AREA)
- Accounting & Taxation (AREA)
- Entrepreneurship & Innovation (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application discloses a data acquisition method, data acquisition equipment and a computer readable storage medium, which are used for improving the feasibility of a data acquisition scheme and acquiring data under the condition of successful data acquisition. The method comprises the following steps: after receiving the data acquisition request, generating an acquisition task according to the data acquisition request, determining a target routing strategy corresponding to the acquisition task, and matching a target data source and a target acquisition executor corresponding to the acquisition task in a preset configuration list according to the target routing strategy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations, the target acquisition executors are triggered to acquire data from the target data sources, and acquired data are converted according to a preset format to obtain target data.
Description
Technical Field
The embodiment of the application relates to the field of data acquisition, in particular to a data acquisition method, data acquisition equipment and a computer readable storage medium.
Background
In order to enable a financial institution to know the credit status of an enterprise, multi-dimensional credit data such as the basic status, the operating status, the judicial business and the like of the enterprise can be collected, the credit data are arranged, analyzed and processed to generate a credit report of the enterprise, and the credit report is obtained by identifying and analyzing the target company including data such as basic status, operation, finance, financing, industry status and the like, so as to obtain a investigation report about the target company and be used for finding opportunities and identifying risks. Therefore, it is important to collect credit data of the business.
The existing data acquisition method is that after a data acquisition request is obtained, the data acquisition request is sent to a target data source to obtain target data returned by the target data source, then the format of the target data is converted, and the target data after the format conversion is sent to a target server. However, the method is a data acquisition method of a single data source, the target data source is easy to have abnormal service conditions, the target data source has small selectivity, and once the target data source has abnormal service conditions, data acquisition cannot be performed from the target data source, so that the existing data acquisition scheme has low realizability and low possibility of successful data acquisition.
Disclosure of Invention
The embodiment of the application provides a data acquisition method, data acquisition equipment and a computer readable storage medium, which are used for improving the feasibility of a data acquisition scheme and carrying out data acquisition under the condition of successful data acquisition.
In a first aspect, an embodiment of the present application provides a data acquisition method, including:
after receiving a data acquisition request, generating an acquisition task according to the data acquisition request;
determining a target routing strategy corresponding to the acquisition task;
according to the target routing strategy, matching a target data source and a target acquisition executor corresponding to the acquisition task in a preset configuration list; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations;
triggering the target acquisition executor to acquire data from the target data source;
and converting the acquired data according to a preset format to obtain target data.
Optionally, the generating an acquisition task according to the data acquisition request includes:
determining a data product code according to the data acquisition request;
and generating the acquisition task according to the data product code.
The determining the target routing policy corresponding to the acquisition task includes:
and inquiring a routing strategy corresponding to the data product code in a preset configuration list as the target routing strategy according to the data product code corresponding to the acquisition task.
Optionally, the matching, in a preset configuration list, the target data source and the target acquisition executor corresponding to the acquisition task according to the target routing policy includes:
inquiring at least one data source corresponding to the data product code in a preset configuration list, and screening a target data source corresponding to the acquisition task from the at least one data source according to the target routing strategy;
inquiring an acquisition executor corresponding to the target data source in the preset configuration list to serve as a target acquisition executor corresponding to the acquisition task.
Optionally, the screening, according to the target routing policy, the target data source corresponding to the collection task from the at least one data source includes:
acquiring acquisition statistics of a historical acquisition task corresponding to each data source in a preset time period according to each data source;
taking the collection statistics condition of each data source as a target screening basis of the routing strategy;
and screening the target data sources in the at least one data source according to the target screening basis of the routing strategy.
Optionally, the triggering the target acquisition executor to acquire data from the target data source includes:
obtaining a session ticket of the target data source;
triggering the target acquisition executor to acquire data from the target data source according to the session ticket.
Optionally, after the triggering the target acquisition executor to acquire data from the target data source, the method further includes:
and if the data is not acquired within the preset duration and the acquisition task meets the retry condition, returning to execute the step of matching the target data source and the target acquisition executor corresponding to the acquisition task in the preset configuration list.
Optionally, the converting the collected data according to a preset format to obtain target data includes:
analyzing the acquired data to obtain a target object;
respectively converting the attribute and the value type of the target object into preset attribute and preset value type to obtain a converted target object;
and generating the target data according to the converted target object.
In a second aspect, an embodiment of the present application provides a data acquisition device, including:
the generating unit is used for generating an acquisition task according to the data acquisition request after receiving the data acquisition request;
the determining unit is used for determining a target routing strategy corresponding to the acquisition task;
the matching unit is used for matching the target data source corresponding to the acquisition task and the target acquisition executor in a preset configuration list according to the target routing strategy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations;
the triggering unit is used for triggering the target acquisition executor to acquire data from the target data source;
the conversion unit is used for converting the acquired data according to a preset format to obtain target data and sending the target data to the target server.
In a third aspect, an embodiment of the present application provides a data acquisition device, including:
the device comprises a central processing unit, a memory, an input/output interface, a wired or wireless network interface and a power supply;
the memory is a short-term memory or a persistent memory;
the central processor is configured to communicate with the memory and to execute the instruction operations in the memory to perform the aforementioned data acquisition method.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the aforementioned data acquisition method.
In a fifth aspect, embodiments of the present application provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the aforementioned data acquisition method.
From the above technical solutions, the embodiments of the present application have the following advantages: after a data acquisition request can be received, generating an acquisition task according to the data acquisition request, determining a target routing strategy corresponding to the acquisition task, and matching a target data source and a target acquisition executor corresponding to the acquisition task in a preset configuration list according to the target routing strategy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations, and acquired data are converted into target data according to a preset format. The target data source and the target acquisition executor corresponding to the acquisition task can be selectively matched in the preset configuration list according to the target routing strategy, the target data source and the target acquisition executor have high selectivity, the realizability of the data acquisition scheme is high, and the possibility of successful data acquisition is high.
Drawings
Fig. 1 is a schematic architecture diagram of a data acquisition system according to an embodiment of the present disclosure;
fig. 2 is a schematic flow chart of a data acquisition method disclosed in an embodiment of the present application;
FIG. 3 is a flow chart of another data collection method disclosed in an embodiment of the present application;
fig. 4 is a schematic diagram of a mapping relationship between data product encoding and routing policies according to an embodiment of the present disclosure;
fig. 5 is a schematic diagram of mapping relationship between data product codes in a preset configuration list and a target data source and a target acquisition executor according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a data acquisition device according to an embodiment of the present disclosure;
FIG. 7 is a schematic structural diagram of another data acquisition device disclosed in an embodiment of the present application;
fig. 8 is a schematic structural diagram of still another data acquisition device according to an embodiment of the present application.
Detailed Description
The embodiment of the application provides a data acquisition method, data acquisition equipment and a computer readable storage medium, which are used for improving the feasibility of a data acquisition scheme and carrying out data acquisition under the condition of successful data acquisition.
Referring to fig. 1, the architecture of the data acquisition system in the embodiment of the present application includes:
a data acquisition device 101, a target server 102 and a target data source 103. When data collection is performed, the data collection device 101 may be connected to the target server 102 and the target data source 103. The data acquisition device 101 may receive the data acquisition request sent by the target server 102 or by the data acquisition device 101, and the data acquisition device 101 may generate an acquisition task according to the data acquisition request, determine a target data source and a target acquisition executor corresponding to the acquisition task according to the acquisition task, and trigger the target acquisition executor to perform data acquisition on the target data source. The target data source 103 may be a source channel of public information of the target company and may be a data service provider.
Referring to fig. 2, referring to the data acquisition system shown in fig. 1, fig. 2 is a schematic flow chart of a data acquisition method disclosed in an embodiment of the present application, where the method includes:
201. and after receiving the data acquisition request, generating an acquisition task according to the data acquisition request.
When data acquisition is performed, an acquisition task can be generated according to the data acquisition request after the data acquisition request is received.
202. And determining a target routing strategy corresponding to the acquisition task.
After the acquisition task is generated according to the data acquisition request, a target routing strategy corresponding to the acquisition task can be determined.
203. According to the target routing strategy, matching a target data source corresponding to the acquisition task and a target acquisition executor in a preset configuration list; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations.
After determining the target routing strategy corresponding to the acquisition task, matching a target data source and a target acquisition executor corresponding to the acquisition task in a preset configuration list according to the target routing strategy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations. It can be appreciated that the screening basis of the target routing policy may be the collection statistics of each data source, or may be other reasonable screening basis, which is not limited herein.
204. Triggering the target acquisition executor to acquire data from the target data source.
After the target data source and the target acquisition executor corresponding to the acquisition task are matched in the preset configuration list, the target acquisition executor can be triggered to acquire data to the target data source.
205. And converting the acquired data according to a preset format to obtain target data, and transmitting the target data to a target server.
After triggering the target acquisition executor to acquire data from the target data source, the acquired data can be converted into target data according to a preset format, and the target data is sent to the target server.
In the embodiment of the application, after a data acquisition request is received, an acquisition task is generated according to the data acquisition request, a target routing strategy corresponding to the acquisition task is determined, and a target data source and a target acquisition executor corresponding to the acquisition task are matched in a preset configuration list according to the target routing strategy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations, and acquired data are converted into target data according to a preset format. The target data source and the target acquisition executor corresponding to the acquisition task can be selectively matched in the preset configuration list according to the target routing strategy, the target data source and the target acquisition executor have high selectivity, the realizability of the data acquisition scheme is high, and the possibility of successful data acquisition is high.
In this embodiment of the present application, according to a target routing policy, there may be multiple methods for matching a target data source corresponding to a collection task and a target collection executor in a preset configuration list, based on the data collection method shown in fig. 2, referring to fig. 3, fig. 3 is a schematic flow diagram of another data collection method disclosed in the embodiment of the present application, where the method includes:
in this embodiment, when data acquisition is performed, an acquisition task may be generated according to the data acquisition request after the data acquisition request is received. It should be understood that when the data acquisition request is received, the data acquisition request may be checked, and when the check result is qualified, the step of generating the acquisition task according to the data acquisition request may be performed, and when the check result is unqualified, the step of generating the acquisition task according to the data acquisition request may not be performed. The method for verifying the data acquisition request may be verifying the necessity, format and value range of the service parameter in the data acquisition request, for example, verifying the length of the user name, the number of bits and correctness of the verification code, or other reasonable methods for verifying the data acquisition request, which is not limited in detail herein.
Specifically, the method for generating the acquisition task according to the data acquisition request may be to determine the data product code according to the data acquisition request, and then generate the acquisition task according to the data product code. More specifically, the data product content and the data product code may be determined according to the data acquisition request, and then the acquisition task may be generated according to the data product content and the data product code. For example, the data product content may be "acquiring basic information of industry and commerce by company name", the data product code is an internally defined code, the unique identifier of the data product content may be "product 1", the acquisition task may be a task obtained by sorting and packaging the data product content and the data product code, or may be a task obtained by sorting and packaging parameters related to the data product content, such as the data product content and the data product code, where the parameters related to the data product content may be a source of the data product content, may be an external system call, may be an internal system call, and the form of the data product code, the content of the acquisition task, and the like are not limited herein. It should be noted that each collection task generated may consume each collection task according to a task queue "first in first out" manner. It should be appreciated that a queue is a linear table of First-In-First-Out (FIFO), a common data structure, typically implemented as a linked list or array In a particular application. The queue allows only insert operations at the back end (called rear) and delete operations at the front end (called front), the operations in the computer, the definition of a task being largely dependent on the object handling the task.
After the acquisition task is generated according to the data acquisition request, a target routing strategy corresponding to the acquisition task can be determined. Specifically, the method for determining the target routing policy corresponding to the acquisition task may be to query the routing policy corresponding to the data product code in the preset configuration list as the target routing policy according to the data product code corresponding to the acquisition task. It can be appreciated that the routing policy can be highly configured in the preset configuration list according to the requirement, that is, the policy mode (Strategy Pattern) is adopted, so that the flexibility of data acquisition is improved. Specifically, policy patterns (Strategy patterns) refer to a class of behavior or its algorithms that can be changed at runtime, and this type of design Pattern belongs to a behavioral Pattern.
It may be understood that, referring to fig. 4, fig. 4 is a schematic diagram of a mapping relationship between a data product code and a routing policy disclosed in the embodiment of the present application, for example, after an acquisition task is generated, it may be determined that the data product code corresponding to the acquisition task is "product 1", and then the target routing policy corresponding to the data product code "product 1" may be queried in the preset configuration list as "cost optimal policy", where "cost optimal policy" represents "always select the least cost data source", and it is understood that, for each data source, a price identifier of the data source may be obtained, a cost required for data acquisition to the data source is determined according to the price identifier, and the data source with the least cost is determined according to the cost of each data source. It may be understood that the mapping relationship between the data product codes configured in the preset configuration list and the routing policy may be configured according to actual requirements, for example, a fixed weight policy, a polling policy, or an intelligent routing policy, where the fixed weight policy represents accessing in a data source weight proportion, the polling policy represents polling the available data source list, the intelligent routing policy represents selecting a data source according to statistical analysis data, and the mapping relationship between the specifically configured data product codes and the routing policy is not limited herein. It should be appreciated that Polling is a way for the CPU to decide how to provide peripheral services, also known as "Programmed I/O". The concept of the polling method is: the CPU sends out inquiry at regular time, and inquires whether each peripheral device needs its service or not in sequence, if so, the service is given, and after the service is finished, the next peripheral is asked, and then the process is repeated.
After determining the target routing strategy corresponding to the acquisition task, matching a target data source and a target acquisition executor corresponding to the acquisition task in a preset configuration list according to the target routing strategy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations.
Specifically, according to the target routing policy, the method for matching the target data source corresponding to the acquisition task and the target acquisition executor in the preset configuration list may be that at least one data source corresponding to the data product code is queried in the preset configuration list, then the target data source corresponding to the acquisition task is screened from the at least one data source according to the target routing policy, and finally the acquisition executor corresponding to the target data source is queried in the preset configuration list to serve as the target acquisition executor corresponding to the acquisition task.
Referring to fig. 5, fig. 5 is a schematic diagram of a mapping relationship between a data product code and a target data source and a target acquisition actuator in a preset configuration list disclosed in the embodiment of the present application, as can be seen from fig. 5, the data product code is "product 1", it can be determined that the data source and the acquisition actuator corresponding to the data product code "product 1" may be "data source a and actuator A1, data source B and actuator B1, data source C and actuator C1", and please refer to the figures and diagrams continuously, for example, the target routing policy of the data product code "product 1" is "cost optimal policy", and it can be determined that the data source with optimal cost is the data source a in the data source "data source a, data source B and data source C", and then the data source a can be used as the target data source, and the acquisition actuator A1 corresponding to the data source a is used as the target acquisition actuator.
The method for screening the target data source corresponding to the acquisition task in the at least one data source according to the target routing strategy includes the steps of firstly obtaining the acquisition statistics condition of the acquisition task corresponding to the data source in a preset time period for each data source, then taking the acquisition statistics condition of each data source as a target screening basis of the routing strategy, and finally screening the target data source in the at least one data source according to the target screening basis of the routing strategy.
It may be understood that the routing policy is the intelligent routing policy described above, specifically, each data source has a corresponding number of acquisition tasks in a preset time period, the acquisition statistics condition of the historical acquisition task corresponding to the data source in the preset time period can be obtained for each data source, related information of the data source acquired can be firstly asynchronously recorded in the log, for example, the time of acquiring the data source, the acquired content, the time of acquiring consumption, the result of acquiring success or failure, etc., then a timing task is established, the related information of the data source acquired by the log is periodically counted to obtain the acquisition statistics condition of the data source, the acquisition statistics condition can be the request success rate of the data source, the request time, the acquired data quality, the cost of the data source, etc.
After the target data source and the target acquisition executor corresponding to the acquisition task are matched in the preset configuration list, the target acquisition executor can be triggered to acquire data to the target data source.
Specifically, the method for triggering the target acquisition executor to acquire data from the target data source may be that a session ticket of the target data source is acquired first, and then the target acquisition executor is triggered to acquire data from the target data source according to the session ticket.
It should be understood that the session ticket of the target data source is an identity authentication identifier required for the data acquisition device to maintain communication with the data source, and the data acquisition can be performed only by having the identity authentication identifier when an API request is performed to perform data acquisition on the target data source. Specifically, the method for obtaining the session ticket of the target data source may be to call an interface (API) for obtaining the session ticket of the target data source, and the method for triggering the target acquisition executor to acquire data from the target data source according to the session ticket may be to assemble the content of the data product and the session ticket, and call the interface (API) of the target data source to acquire data.
After triggering the target acquisition executor to acquire data to the target data source, if the data is not acquired within the preset time period and the acquisition task meets the retry condition, returning to execute the step of matching the target data source corresponding to the acquisition task and the target acquisition executor in the preset configuration list. The scene in which no data is collected within the preset time period can be a scene such as network abnormality, server abnormality, data content abnormality and the like.
It should be understood that, the method for determining whether the acquisition task meets the retry condition may be a method for determining whether the data product code corresponding to the acquisition task is configured with the retry, if so, determining that the acquisition task meets the retry condition, and determining the number of configured retries, or may be other methods for determining whether the acquisition task meets the retry condition, which is not limited in this embodiment. It should be noted that, after each time the target acquisition executor is triggered to acquire data from the target data source, whether the acquisition result is successful or not may be recorded, when the target data source corresponding to the acquisition task is matched with the target acquisition executor in the preset configuration list again, the data source except the data source with the acquisition failure may be matched, for example, the data source configured by the data product code "product 1" is the "data source a, the data source B and the data source C", the data product code is the "cost optimal policy", the first time matches the data source with the lowest cost with the data source data a, but when the trigger executor A1 acquires the data source a, the acquisition failure may be recorded in the log, after the data source acquisition failure is determined to be configured with the "product 1", the data source with the lowest cost may be re-matched, at this time, the data source with the lowest cost except the data source a may be matched, for example, the data source C may be matched, the trigger executor C1 may be triggered to acquire the data source C for the second time, and if the trigger to acquire the data source C for the second time and the second time matches the data source C may be triggered to acquire the data source C. It can be understood that after the target acquisition executor is triggered to acquire data to the target data source, whether data are acquired within a preset duration and whether the acquisition task meets the retry condition can be judged as a degradation mechanism, so that the stability of data acquisition is improved. The degradation mechanism is to process some services and pages in a strategic way without processing or in a simple way according to the actual service conditions and traffic under the condition that the server pressure is increased suddenly, so as to release the server resources to ensure the normal operation or the efficient operation of the core service.
After triggering the target acquisition executor to acquire data from the target data source, the acquired data can be converted into target data according to a preset format, and the target data is sent to the target server. The method for converting the acquired data according to the preset format to obtain the target data may be that the acquired data is analyzed to obtain the target object, then the attribute and the value type of the target object are converted into the preset attribute and the preset value type respectively to obtain the converted target object, and finally the target data is generated according to the converted target object. For example, the target object obtained after parsing may be "JSON object {" attribute 1": "value 1" } ", the converted target object may be" new JSON object { "property 1 defined by standard": "value 1" } ", for another example, the target object obtained after parsing may be" JSON object { "attribute 1": "value 1" } ", the converted target object may be" hierarchical JSON object { "module 1": { "Attribute 1": "value 1" } ", wherein JSON (JavaScript Object Notation) is a lightweight data exchange format. It can be understood that the mode of converting the acquired data according to the preset format to obtain the target data is an Adapter mode (Adapter Pattern), where the Adapter mode is used as a bridge between two incompatible interfaces, and the design mode of this type belongs to a structural mode and combines the functions of two independent interfaces.
After triggering a target acquisition executor to acquire data to a target data source, if data is not acquired within a preset duration and an acquisition task does not meet a retry condition, determining acquired information as acquisition failure information, converting the acquisition failure information according to a preset format to acquire the target data, and sending the target data to a target server, wherein related information of the target data source acquired by the target data source can be asynchronously recorded in a log, such as acquisition time of the target data source, acquisition content, acquisition consumed time, acquisition unsuccessful result and the like, so that a timing task can be established later, and the related information of the acquisition of the target data source recorded by the log can be counted periodically to obtain the acquisition statistics condition of the target data source.
After triggering a target acquisition executor to acquire data from a target data source, if the data is acquired within a preset time length, the acquired information can be converted into target data according to a preset format and sent to a target server, and related information of the target data source acquired can be asynchronously recorded in a log, such as the acquisition time of the target data source, the acquired content, the acquisition consumption time, the acquisition success result and the like, so that a timing task can be established later, and the related information of the target data source acquired by log records can be counted periodically to obtain the acquisition statistics condition of the target data source.
It can be understood that, besides the method of generating the acquisition task according to the data acquisition request, determining the target routing policy corresponding to the acquisition task, matching the target data source corresponding to the acquisition task and the target acquisition executor in the preset configuration list, triggering the target acquisition executor to perform data acquisition on the target data source, and converting the acquired data into the target data according to the preset format, other reasonable methods are also possible, which is not limited herein.
In this embodiment, after a data acquisition request is received, an acquisition task is generated according to the data acquisition request, a target routing policy corresponding to the acquisition task is determined, and a target data source and a target acquisition executor corresponding to the acquisition task are matched in a preset configuration list according to the target routing policy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations, acquired data are converted into target data according to a preset format, and the target data are sent to a target server. The target data source and the target acquisition executor corresponding to the acquisition task can be selectively matched in the preset configuration list according to the target routing strategy, the target data source and the target acquisition executor have high selectivity, the realizability of the data acquisition scheme is high, and the possibility of successful data acquisition is high. And if the data is not acquired within the preset duration and the acquisition task meets the retry condition, the target data source and the target acquisition executor corresponding to the acquisition task matched in the preset configuration list can be returned to be executed, so that the stability of data acquisition is improved. Furthermore, the routing strategy can be highly configured in a preset configuration list according to requirements, so that the flexibility of data acquisition is improved. And finally, no matter which data source is used for collecting the data, the collected data can be converted according to a preset format to obtain target data, and the target data is sent to a target server, so that the scheme of re-developing a set of other data sources for data collection and format conversion is reduced, and the development and operation cost is reduced.
The data acquisition method in the embodiment of the present application is described above, and the data acquisition device in the embodiment of the present application is described below, referring to fig. 6, one embodiment of the data acquisition device in the embodiment of the present application includes:
the generating unit 601 is configured to generate an acquisition task according to a data acquisition request after receiving the data acquisition request;
a determining unit 602, configured to determine a target routing policy corresponding to the acquisition task generated by the generating unit 601;
a matching unit 603, configured to match, in a preset configuration list, a target data source and a target acquisition executor corresponding to the acquisition task according to the target routing policy determined by the determining unit 602; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations;
the triggering unit 604 is configured to trigger the target acquisition executor matched by the matching unit 603 to perform data acquisition on the target data source;
the conversion unit 605 is configured to convert the data acquired by the triggering unit 604 according to a preset format to obtain target data, and send the target data to a target server.
In the embodiment of the application, after a data acquisition request is received, an acquisition task is generated according to the data acquisition request, a target routing strategy corresponding to the acquisition task is determined, and a target data source and a target acquisition executor corresponding to the acquisition task are matched in a preset configuration list according to the target routing strategy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations, acquired data are converted into target data according to a preset format, and the target data are sent to a target server. The target data source and the target acquisition executor corresponding to the acquisition task can be selectively matched in the preset configuration list according to the target routing strategy, the target data source and the target acquisition executor have high selectivity, the realizability of the data acquisition scheme is high, and the possibility of successful data acquisition is high.
Referring to fig. 7, another embodiment of the data acquisition device in the embodiment of the present application includes:
the generating unit 701 is configured to generate an acquisition task according to a data acquisition request after receiving the data acquisition request;
a determining unit 702, configured to determine a target routing policy corresponding to the acquisition task generated by the generating unit 701;
a matching unit 703, configured to match, in a preset configuration list, a target data source and a target acquisition executor corresponding to the acquisition task according to the target routing policy determined by the determining unit 702; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations;
a triggering unit 704, configured to trigger a target acquisition executor matched by the matching unit 703 to perform data acquisition on the target data source;
the conversion unit 705 is configured to convert the data acquired by the triggering unit 704 according to a preset format to obtain target data, and send the target data to a target server.
The generating unit 701 is specifically configured to determine a data product content and a data product code according to the data acquisition request, and generate the acquisition task according to the data product content and the data product code.
The determining unit 702 is specifically configured to query, in a preset configuration list, a routing policy corresponding to the data product code as the target routing policy according to the data product code corresponding to the acquisition task.
The matching unit 703 is specifically configured to query at least one data source corresponding to the data product code in a preset configuration list, screen a target data source corresponding to the acquisition task from the at least one data source according to the target routing policy, and query an acquisition executor corresponding to the target data source in the preset configuration list as a target acquisition executor corresponding to the acquisition task.
The matching unit 703 is specifically configured to obtain, for each data source, an acquisition statistics condition of an acquisition task corresponding to the data source in a preset time period, and use the acquisition statistics condition of each data source as a target screening basis of the routing policy, and screen the target data source in the at least one data source according to the target screening basis of the routing policy.
The triggering unit 704 is specifically configured to obtain a session ticket of the target data source, and trigger the target acquisition executor to acquire data from the target data source according to the session ticket.
The data acquisition device further comprises:
and a return unit 706, configured to return to executing the step of matching the target data source corresponding to the acquisition task and the target acquisition executor in the preset configuration list if the trigger unit 704 does not acquire data within the preset duration and the acquisition task meets the retry condition.
The conversion unit 705 is specifically configured to parse the collected data to obtain a target object, convert an attribute and a value type of the target object into a preset attribute and a preset value type, respectively, obtain a converted target object, and generate the target data according to the converted target object.
In this embodiment, each unit in the data acquisition device performs the operation of the data acquisition device in the embodiment shown in fig. 2, which is not described herein in detail.
Referring now to fig. 8, yet another embodiment of a data acquisition device 800 in an embodiment of the present application includes:
a central processor 801, a memory 805, an input/output interface 804, a wired or wireless network interface 803, and a power supply 802;
the central processor 801 is configured to communicate with the memory 805 and to execute the operations of the instructions in the memory 805 to perform the method of the embodiment shown in fig. 2 described above.
Embodiments of the present application also provide a computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of the embodiment shown in fig. 2 described above.
Embodiments of the present application also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of the embodiment shown in fig. 2 described above.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (RAM, random access memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Claims (10)
1. A method of data acquisition, comprising:
after receiving a data acquisition request, generating an acquisition task according to the data acquisition request;
determining a target routing strategy corresponding to the acquisition task;
according to the target routing strategy, matching a target data source and a target acquisition executor corresponding to the acquisition task in a preset configuration list; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations;
triggering the target acquisition executor to acquire data from the target data source;
and converting the acquired data according to a preset format to obtain target data.
2. The method of claim 1, wherein generating the acquisition task from the data acquisition request comprises:
determining a data product code according to the data acquisition request;
generating the acquisition task according to the data product code;
the determining the target routing policy corresponding to the acquisition task includes:
and inquiring a routing strategy corresponding to the data product code in a preset configuration list as the target routing strategy according to the data product code corresponding to the acquisition task.
3. The method according to claim 2, wherein the matching, according to the target routing policy, the target data source and the target acquisition executor corresponding to the acquisition task in a preset configuration list includes:
inquiring at least one data source corresponding to the data product code in a preset configuration list, and screening a target data source corresponding to the acquisition task from the at least one data source according to the target routing strategy;
inquiring an acquisition executor corresponding to the target data source in the preset configuration list to serve as a target acquisition executor corresponding to the acquisition task.
4. A method according to claim 3, wherein said screening, according to the target routing policy, the target data source corresponding to the acquisition task from the at least one data source comprises:
acquiring acquisition statistics of a historical acquisition task corresponding to each data source in a preset time period according to each data source;
taking the collection statistics condition of each data source as a target screening basis of the routing strategy;
and screening the target data sources in the at least one data source according to the target screening basis of the routing strategy.
5. The method of claim 1, wherein the triggering the target acquisition executor to perform data acquisition to the target data source comprises:
obtaining a session ticket of the target data source;
triggering the target acquisition executor to acquire data from the target data source according to the session ticket.
6. The method of claim 1, wherein after triggering the target acquisition executor to perform data acquisition to the target data source, the method further comprises:
and if the data is not acquired within the preset duration and the acquisition task meets the retry condition, returning to execute the step of matching the target data source and the target acquisition executor corresponding to the acquisition task in the preset configuration list.
7. The method of claim 1, wherein the converting the collected data according to a preset format to obtain target data comprises:
analyzing the acquired data to obtain a target object;
respectively converting the attribute and the value type of the target object into preset attribute and preset value type to obtain a converted target object;
and generating the target data according to the converted target object.
8. A data acquisition device, comprising:
the generating unit is used for generating an acquisition task according to the data acquisition request after receiving the data acquisition request;
the determining unit is used for determining a target routing strategy corresponding to the acquisition task;
the matching unit is used for matching the target data source corresponding to the acquisition task and the target acquisition executor in a preset configuration list according to the target routing strategy; the preset configuration list comprises at least one group of data sources and acquisition executors with corresponding relations;
the triggering unit is used for triggering the target acquisition executor to acquire data from the target data source;
the conversion unit is used for converting the acquired data according to a preset format to obtain target data and sending the target data to the target server.
9. A data acquisition device, comprising:
the device comprises a central processing unit, a memory, an input/output interface, a wired or wireless network interface and a power supply;
the memory is a short-term memory or a persistent memory;
the central processor is configured to communicate with the memory and to execute instruction operations in the memory to perform the method of any of claims 1 to 7.
10. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211347426.5A CN116109322A (en) | 2022-10-31 | 2022-10-31 | Data acquisition method, data acquisition device, and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211347426.5A CN116109322A (en) | 2022-10-31 | 2022-10-31 | Data acquisition method, data acquisition device, and computer-readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116109322A true CN116109322A (en) | 2023-05-12 |
Family
ID=86260474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211347426.5A Pending CN116109322A (en) | 2022-10-31 | 2022-10-31 | Data acquisition method, data acquisition device, and computer-readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116109322A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117251499A (en) * | 2023-11-15 | 2023-12-19 | 山东光合云谷大数据有限公司 | Data acquisition system |
-
2022
- 2022-10-31 CN CN202211347426.5A patent/CN116109322A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117251499A (en) * | 2023-11-15 | 2023-12-19 | 山东光合云谷大数据有限公司 | Data acquisition system |
| CN117251499B (en) * | 2023-11-15 | 2024-02-06 | 山东光合云谷大数据有限公司 | Data acquisition system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109271411B (en) | Report generation method, report generation device, computer equipment and storage medium | |
| CN107992398B (en) | Monitoring method and monitoring system of service system | |
| US8825798B1 (en) | Business event tracking system | |
| CN108509392B (en) | Multi-mechanism account checking method, system, computer equipment and storage medium | |
| US20080065588A1 (en) | Selectively Logging Query Data Based On Cost | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN103023693A (en) | Behaviour log data management system and behaviour log data management method | |
| CN111367760B (en) | Log collection method and device, computer equipment and storage medium | |
| EP3063704A1 (en) | Method and system for document data extraction template management | |
| CN116109322A (en) | Data acquisition method, data acquisition device, and computer-readable storage medium | |
| CN115622906A (en) | Application log capturing system and method | |
| CN112116273A (en) | Employee query behavior risk monitoring method, device, equipment and storage medium | |
| KR20220088958A (en) | Systems and methods for managing connections in a scalable cluster | |
| CN112734227A (en) | Big data decision system and method | |
| CN116881100A (en) | Log detection method, log alarm method, system, equipment and storage medium | |
| KR20210000041A (en) | Method and apparatus for analyzing log data in real time | |
| CN115344633A (en) | Data processing method, device, equipment and storage medium | |
| CN111639936B (en) | Transaction information acquisition method and device, electronic equipment and readable storage medium | |
| CN114648323A (en) | Service scene oriented call chain processing method, device and medium | |
| CN112346938B (en) | Operation auditing method and device, server and computer readable storage medium | |
| CN110309176B (en) | Data processing method and data transfer station | |
| CN111045983A (en) | Nuclear power station electronic file management method and device, terminal equipment and medium | |
| US11811858B1 (en) | SaaS application contract terms benchmarking in a SaaS management platform | |
| CN114205412A (en) | Information pushing method and device, server and storage medium | |
| CN116629870A (en) | Data processing method, device, equipment and storage medium based on distributed technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |