CN114860658A - Data acquisition method and device, electronic equipment and storage medium - Google Patents
Data acquisition method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114860658A CN114860658A CN202210491976.8A CN202210491976A CN114860658A CN 114860658 A CN114860658 A CN 114860658A CN 202210491976 A CN202210491976 A CN 202210491976A CN 114860658 A CN114860658 A CN 114860658A
- Authority
- CN
- China
- Prior art keywords
- data
- text file
- data acquisition
- request parameters
- acquisition request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000012545 processing Methods 0.000 claims abstract description 17
- 230000000977 initiatory effect Effects 0.000 claims abstract description 15
- 238000004458 analytical method Methods 0.000 claims description 17
- 238000013507 mapping Methods 0.000 abstract description 14
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 210000001072 colon Anatomy 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data acquisition method, a data acquisition device, electronic equipment and a storage medium. The method comprises the following steps: determining a text file comprising at least two data acquisition request parameters; analyzing the text file by adopting a data warehouse client tool, and determining the data column indexes to which the at least two data acquisition request parameters belong; and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data. By adopting the technical scheme of the embodiment of the invention, the text file with the mapping relation between the data column index and the data acquisition request parameter is established, and the text file is analyzed by the data warehouse client tool to determine the required target data; the text file is used as a parameter mapping source, and is dynamically analyzed and assembled into request parameters, so that the automatic request of the HTTP interface is realized.
Description
Technical Field
The embodiment of the invention relates to the technical field of computer application, in particular to a data acquisition method and device, electronic equipment and a storage medium.
Background
In the field of large data collection, data sources exist in the form of Web services and provide a passive data query interface to allow data to be retrieved as HTTP requests. When the amount of data to be acquired is too large or the types of data are numerous, all target data cannot be acquired by single access of a single Web service interface, and the data are acquired in batches by dividing the requests into a plurality of times with different parameters; the HTTP interface of the data query limits the query subtype parameter and does not allow all father type data to be queried at one time; when the HTTP request acquires all data, the pressure of the Web service may be increased dramatically by accessing the HTTP interface at too high a frequency due to the huge data size, resulting in a crash.
Therefore, how to acquire a large amount of data is a technical problem to be solved urgently by those skilled in the art.
Disclosure of Invention
The embodiment of the invention provides a data acquisition method, a data acquisition device, electronic equipment and a storage medium, wherein HTTP is automatically executed based on dynamic parameter assembly of a text file, and the problem that a large amount of data is acquired by relying on an HTTP interface is solved.
In a first aspect, an embodiment of the present invention provides a data acquisition method, including:
determining a text file comprising at least two data acquisition request parameters;
analyzing the text file by adopting a data warehouse client tool, and determining the data column indexes to which the at least two data acquisition request parameters belong;
and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data.
In a second aspect, an embodiment of the present invention further provides a data obtaining apparatus, including:
the text file determining module is used for determining a text file comprising at least two data acquisition request parameters;
the data column index determining module is used for analyzing the text file by adopting a data warehouse client tool and determining the data column indexes to which the at least two data acquisition request parameters belong;
and the data acquisition request module is used for organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data and processing the acquired target data.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
one or more processors;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the data acquisition method according to any embodiment of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data obtaining method according to any embodiment of the present invention.
The embodiment of the invention provides a data acquisition method, a data acquisition device, electronic equipment and a storage medium, wherein a text file comprising at least two data acquisition request parameters is determined; analyzing the text file by adopting a data warehouse client tool, and determining the data column indexes to which the at least two data acquisition request parameters belong; and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data. By adopting the technical scheme of the embodiment of the invention, the text file with the mapping relation between the data column index and the data acquisition request parameter is established, and the text file is analyzed by the data warehouse client tool to determine the required target data; the text file is used as a parameter mapping source, and is dynamically analyzed and assembled into request parameters, so that the automatic request of the HTTP interface is realized.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1A is a flowchart of a data acquisition method according to an embodiment of the present invention;
fig. 1B is a schematic diagram illustrating a flow and configuration for requesting data from an HTTP interface according to an embodiment of the present invention;
fig. 1C is a schematic structural diagram of a text file according to an embodiment of the present invention;
fig. 2A is a flowchart of a data acquisition method according to a second embodiment of the present invention;
FIG. 2B is a schematic diagram of a multi-request structure for different URL parameters according to an embodiment of the present invention;
fig. 2C is a schematic structural diagram of a parameter value set according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data acquisition apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present application.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations (or steps) can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1A is a flowchart of a data obtaining method according to an embodiment of the present invention, where this embodiment is applicable to a case of obtaining a large amount of data, and the method of this embodiment may be executed by a data obtaining apparatus, and the apparatus may be implemented in a hardware and/or software manner. The apparatus may be configured in a server for data acquisition. The method specifically comprises the following steps:
s110, determining a text file comprising at least two data acquisition request parameters.
In the field of big data, there are numerous mature and practical tools for data acquisition and transmission, most of the data sources for the tools are databases, log files, message queue services and the like, few tool software for acquiring data sources for HTTP interfaces are available, and even if the tools are available (for example, flash), the functions of the tools are simple, and the tools cannot handle complicated and variable production environments.
Fig. 1B is a schematic diagram of a flow and configuration for requesting data from an HTTP interface according to an embodiment of the present invention, and referring to fig. 1B, the HTTP interface includes, but is not limited to, a method type, a URL address, configuration parameters, request parameters, and a request result; the interface A is a method type, including but not limited to POST and GET; the interface B is a Uniform Resource Locator (URL) address; the interface C is a configuration parameter, such as Headers configuration information and the like; the D interface is a request parameter, including but not limited to a URL parameter, a form parameter and a Body parameter; the E-interface is the request result including, but not limited to, status code, heads information, and Body data for response.
The data obtaining request parameter may refer to a request parameter for obtaining data, and referring to fig. 1B, the data obtaining request parameter may be a Body parameter, including but not limited to name and age.
In an alternative of the embodiment of the present invention, optionally, the determining a text file including at least two data acquisition request parameters includes:
determining a text file according to at least two data acquisition request parameters;
wherein the text file comprises at least two lines of data; the row of data includes a data field and a column separator; the at least two data acquisition request parameters are represented in data fields and separated by a column separator.
The text File may be a case File, fig. 1C is a schematic structural diagram of a text File according to an embodiment of the present invention, and referring to fig. 1C, the text File includes at least two rows of line data, and in the case File, data of each row is referred to as line data; the row of data includes a data field and a column separator; the at least two data acquisition request parameters are represented in data fields and separated by a column separator. Wherein the content of a row of data consists of one or more meaningful values, referred to as data fields, also referred to as data columns or column data; if the line data has a plurality of data fields, each data field is separated by a fixed separator, the separators of the data fields of all the line data are consistent, and the forms of the separators include but are not limited to English colons, English commas, English underlines and the like; each data field has a sequence number according to the position of the data field in the row data, which is called a data column index, the data column index is also called a column index or a field index, and the index value starts from 0; for example, referring to fig. 1C, CCOP is a data field, the index value is 1, and is separated from other data fields by english semicolons.
And S120, analyzing the text file by adopting a data warehouse client tool, and determining the data column index to which the at least two data acquisition request parameters belong.
The Data Warehouse Client tool (DWHCLI) may refer to tool software for HTTP Data source extraction and offline preprocessing of large (e.g., level above GB) text files. And analyzing the text file by adopting a data warehouse client tool, determining a data structure of the text file, and determining the data column index to which the at least two data acquisition request parameters belong according to the data structure of the text file. For example, a data warehouse client tool is adopted to analyze a case File text File, determine a line data structure in the case File text File, and determine index values to which at least two data fields in a line data belong, for example, referring to fig. 1C, a data column index value of a data field CCOP is 1.
S130, organizing the values of the data fields included in the data column index into request parameters, sending a request to a server to acquire target data, and processing the acquired target data. The organizing of the values of the data fields included in the data column index into the request parameter may mean that the organizing of the values of the data fields included in the data column index into the request parameter initiates a data acquisition request to the server according to target data required by a user; for example, to obtain personal information of a corporate employee, including the name, age, and location of the corporate employee, the client needs to send part of the personal information of the corporate employee to the server, where the part of the personal information of the corporate employee refers to a request parameter. For example, the name and age of the enterprise employee are organized into request parameters, and the client sends the request parameters to the server to initiate a data acquisition request to acquire target data. For example, the index value corresponding to the name is 1, the index value corresponding to the age is 2, and the values of the data fields included in the index value 1 and the index value 2 are obtained; if the acquired name "zhang san" and the age "23" are used as request parameters, a data acquisition request is initiated to the server, so that the server determines personal information required by the client according to the name and the age of the enterprise employee.
If the text file does not include the data acquisition request parameters required by the user, the corresponding data acquisition request parameters required by the user need to be added into the text file, and corresponding setting is carried out on the data warehouse client tool, so that the required data can be acquired. By adopting a dynamic parameter assembly automatic HTTP execution mode based on case files, a plurality of problems of acquiring a large amount of data by relying on an HTTP interface are well solved.
The client initiates a data acquisition request to acquire target data and processes the target data, wherein the client initiates the data acquisition request to acquire the target data and stores part of the target data according to information required by a user. For example, the user needs to store the name, age, and household address of the employee in the obtained personal information of the employee, and the user only needs to set a Path of a key corresponding to the data to be retained, which is called key-Path, and then the program automatically analyzes and stores the data structure.
The embodiment of the invention provides a data acquisition method, which comprises the steps of determining a text file comprising at least two data acquisition request parameters; analyzing the text file by adopting a data warehouse client tool, and determining the data column indexes to which the at least two data acquisition request parameters belong; and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data. By adopting the technical scheme of the embodiment of the invention, the text file with the mapping relation between the data column index and the data acquisition request parameter is established, and the text file is analyzed by the data warehouse client tool to determine the required target data. The text file is used as a parameter mapping source, and is dynamically analyzed and assembled into request parameters, so that the automatic request of the HTTP interface is realized.
Example two
Fig. 2A is a flowchart of a data acquisition method according to a second embodiment of the present invention. Embodiments of the present invention are further optimized on the basis of the above-mentioned embodiments, and the embodiments of the present invention may be combined with various alternatives in one or more of the above-mentioned embodiments. As shown in fig. 2A, the data obtaining method provided in the embodiment of the present invention may include the following steps:
s210, determining a text file comprising at least two data acquisition request parameters.
Because of the consideration of access efficiency and computational efficiency, a large amount of dynamic data and static data are stored in the database in the data structure of the atypical relational database, if the data are extracted by directly butting the database, the data can hardly be analyzed, and the data query HTTP interface provided by the middle station system and the service system can allow the HTTP client to query and obtain the data, and the data structure can be paraphrased.
However, this also presents certain problems: the data volume requested by a single HTTP interface is limited, and the HTTP interface is not suitable for acquiring full data by one request; some HTTP interfaces for data query define query subtype parameters, which do not allow all parent type data to be queried at one time, such as interfaces for querying device records, which allow specific device subtypes to be specified for querying, but do not allow device subtypes to be ignored for querying all device records, but there are thousands of device subtypes, which results in thousands of HTTP interfaces being accessed to query all device data; too high frequency of access to the HTTP interface may stress the HTTP server, causing service degradation or paralysis.
In summary, data acquisition is possible by using the HTTP interface, but there are many limitations, and to obtain a certain type of full data, N times of HTTP interface requests are often required, and HTTP request parameters are different for each HTTP request. Generally, such data acquisition strategies need to be implemented by writing customized code programs or scripts, and users who extract data must have certain code writing capability. Therefore, the embodiment of the invention provides a data acquisition method, which determines a text file according to at least two data acquisition request parameters, establishes a mapping relation between a data column index and the data acquisition request parameters, adopts a data warehouse customer service end tool to acquire target data, reduces code writing and is simple to operate.
S220, analyzing the text file by adopting a data warehouse client tool to obtain a data structure of the inline data of the text file.
The data warehouse client tool analyzes the text file through a preset rule, generates parameters of an HTTP request by taking line data as a unit, and then initiates a data request to an HTTP service. In an alternative of the embodiment of the present invention, optionally, the data of the text file may be read and parsed into a two-dimensional aggregate data structure by using the JavaIO technology, where the structure is referred to as caseC, and each element of the caseC is an array with a fixed length, which is referred to as caseC-a. The caseC-A element is a data field of a certain line of data in the text file.
HTTP request parameters (whether URL parameters, form parameters, or Body parameters) can be considered as key-value pair data structures in K-V format, where K (key) is fixed and V (value) is variable. Fig. 2B is a schematic diagram of a multi-request structure of different URL parameters according to an embodiment of the present invention, and referring to fig. 2B, taking the URL parameters as an example, the URL parameters include project and type. The values of these two parameters are a limited set, and all data obtained by multiple HTTP requests composed of this set are target data to be acquired. Fig. 2C is a schematic structural diagram of a parameter value set according to an embodiment of the present invention, and referring to fig. 2C, the text file stores the parameter value set.
In an alternative of the embodiment of the present invention, optionally, the analyzing the text file by using a data warehouse client tool to obtain a data structure of the inline data of the text file further includes:
when the text file is analyzed, if the analysis is wrong and the analysis error is a predicted error, suspending the data acquisition request;
or if the server side has an analysis error, automatically stopping analyzing the text file and recording the analysis error by adopting a recording log; and when the text file is analyzed again, continuing to initiate a data acquisition request according to the position recorded by the recording log.
Analyzing the text file by adopting a data warehouse client tool, and suspending the data acquisition request if the analysis is wrong and the analysis is wrong in advance; if the server side has an analysis error, automatically stopping analyzing the text file and recording the analysis error by adopting a recording log; and when the text file is analyzed again, continuing to initiate a data acquisition request according to the position recorded by the recording log.
And S230, determining the data column indexes to which the at least two data acquisition request parameters belong according to the data structure of the row of data.
In an alternative of the embodiment of the present invention, optionally, the determining, according to the data structure of the row of data, the data column index to which the at least two data acquisition request parameters belong includes:
analyzing the text file by adopting a data warehouse client tool to obtain a data structure of the inline data of the text file;
determining data column indexes to which the at least two data acquisition request parameters belong according to the data structure of the row of data; and the data column index represents the positions of the at least two data acquisition request parameters.
And establishing the relation of the starting state parameters by specifying the corresponding relation between the data column index in the case File line data and the URL data acquisition request parameter K. The mapping rules specified by DWHCL are: keyName1: indexNum 1; keyName2: indexNum 2; .... The mapping between the data acquisition request parameter K and the data column index in the case File is established by English colons, and a plurality of mapping relations are separated by English semicolons. The mapping relation syntax established may be: project is 0; type: 1. After the DWHCLI establishes the mapping between the case File and the URL parameter, the automation of dynamically assembling the HTTP parameter and gradually acquiring data is realized by circularly traversing the case set and executing the HTTP request. In the whole process, only a user needs to perform simple mapping configuration of the case File and the parameter K, and the user of data extraction is not required to have certain code writing capability.
S240, organizing the values of the data fields included in the data column index into request parameters, sending a request to a server to acquire target data, and processing the acquired target data.
In an alternative of the embodiment of the present invention, optionally, the organizing values of the data fields included in the data column index into request parameters, initiating a request to a server to obtain target data, and processing the obtained target data, includes:
determining the value of a data field included in the data column index according to the data column index to which the at least two data acquisition request parameters belong;
and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data.
The method for returning HTTP request data through the body of response is a currently common method, and DWHCLI makes a targeted data structure analysis design for json structure data returned in the method.
The body results for response are exemplified as follows:
for the returned result example, most of the returned result examples only concern data array data, so the DWHCLI provides a configuration according to json data structure key value mapping, and the program can automatically analyze and store the json data structure only by setting a Path of a key corresponding to data to be reserved, namely a key-Path, by a user. In the example, if the data of the data is required to be saved, the value of the key-Path configured by the user is content.data; the data structure of the content.data is selected as an "array".
The embodiment of the invention provides a data acquisition method, which comprises the steps of determining a text file comprising at least two data acquisition request parameters; analyzing the text file by adopting a data warehouse client tool to obtain a data structure of the inline data of the text file; determining data column indexes to which the at least two data acquisition request parameters belong according to the data structure of the row of data; and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data. By adopting the technical scheme of the embodiment of the invention, for a scene that data is acquired by HTTP requests of different parameters for a plurality of times, the data warehouse client tool provides simple configuration content to realize dynamic parameter assembly, automatically executes request logic, and analyzes and stores a return result.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data acquisition apparatus according to a third embodiment of the present invention, where the apparatus includes: a text file determining module 310, a data column index determining module 320 and a data acquisition request module 330. Wherein:
a text file determining module 310, configured to determine a text file including at least two data acquisition request parameters;
a data column index determining module 320, configured to analyze the text file by using a data warehouse client tool, and determine a data column index to which the at least two data acquisition request parameters belong;
the data obtaining request module 330 is configured to organize the values of the data fields included in the data column index into request parameters, initiate a request to the server to obtain target data, and process the obtained target data.
On the basis of the foregoing embodiment, optionally, the text file determining module includes:
determining a text file according to at least two data acquisition request parameters;
wherein the text file comprises at least two lines of data; the row of data includes a data field and a column separator; the at least two data acquisition request parameters are represented in data fields and separated by a column separator.
On the basis of the foregoing embodiment, optionally, the data column index determining module includes:
analyzing the text file by adopting a data warehouse client tool to obtain a data structure of the inline data of the text file;
determining data column indexes to which the at least two data acquisition request parameters belong according to the data structure of the row of data; and the data column index represents the positions of the at least two data acquisition request parameters.
On the basis of the foregoing embodiment, optionally, the data obtaining request result returning module includes:
determining the value of a data field included in the data column index according to the data column index to which the at least two data acquisition request parameters belong;
and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data.
On the basis of the foregoing embodiment, optionally, the data column index determining module further includes:
when the text file is analyzed, if the analysis is wrong and the analysis error is a predicted error, suspending the data acquisition request;
or if the server side has an analysis error, automatically stopping analyzing the text file and recording the analysis error by adopting a recording log; and when the text file is analyzed again, continuing to initiate a data acquisition request according to the position recorded by the recording log.
The device can execute the data acquisition method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects for executing the data acquisition method.
Example four
Fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present application. The embodiment of the application provides electronic equipment, and an interaction device for acquiring data provided by the embodiment of the application can be integrated in the electronic equipment. As shown in fig. 4, the present embodiment provides an electronic device 400, which includes: one or more processors 420; the storage device 410 is configured to store one or more programs, and when the one or more programs are executed by the one or more processors 420, the one or more processors 420 implement the data acquisition method provided in the embodiment of the present application, the method includes:
determining a text file comprising at least two data acquisition request parameters;
analyzing the text file by adopting a data warehouse client tool, and determining the data column indexes to which the at least two data acquisition request parameters belong;
and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data.
Of course, those skilled in the art will understand that the processor 420 also implements the technical solution of the data acquisition method provided in any embodiment of the present application.
The electronic device 400 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 4, the electronic device 400 includes a processor 420, a storage device 410, an input device 430, and an output device 440; the number of the processors 420 in the electronic device may be one or more, and one processor 420 is taken as an example in fig. 4; the processor 420, the storage device 410, the input device 430, and the output device 440 in the electronic apparatus may be connected by a bus or other means, and are exemplified by a bus 450 in fig. 4.
The storage device 410 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and module units, such as program instructions corresponding to the data acquisition method in the embodiment of the present application.
The storage device 410 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the storage 410 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, storage 410 may further include memory located remotely from processor 420, which may be connected via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input means 430 may be used to receive input numbers, character information, or voice information, and to generate key signal inputs related to user settings and function control of the electronic device. The output device 440 may include a display screen, speakers, or other electronic equipment.
The electronic equipment provided by the embodiment of the application can achieve the technical effects that dynamic parameter assembly based on the text file automatically executes HTTP, and the problem that a large amount of data are acquired by relying on an HTTP interface is effectively solved.
EXAMPLE five
An embodiment of the present invention further provides a storage medium containing computer-executable instructions, where the computer-executable instructions are executed by a computer processor to perform a data acquisition method, and the method includes:
determining a text file comprising at least two data acquisition request parameters;
analyzing the text file by adopting a data warehouse client tool, and determining the data column indexes to which the at least two data acquisition request parameters belong;
and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a flash Memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take a variety of forms, including, but not limited to: an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for data acquisition, the method comprising:
determining a text file comprising at least two data acquisition request parameters;
analyzing the text file by adopting a data warehouse client tool, and determining the data column indexes to which the at least two data acquisition request parameters belong;
and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data.
2. The method of claim 1, wherein determining a text file including at least two data retrieval request parameters comprises:
determining a text file according to at least two data acquisition request parameters;
wherein the text file comprises at least two lines of data; the row of data includes a data field and a column separator; the at least two data acquisition request parameters are represented in data fields and separated by a column separator.
3. The method of claim 1, wherein the parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong comprises:
analyzing the text file by adopting a data warehouse client tool to obtain a data structure of the inline data of the text file;
determining data column indexes to which the at least two data acquisition request parameters belong according to the data structure of the row of data; and the data column index represents the positions of the at least two data acquisition request parameters.
4. The method according to claim 1, wherein organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to obtain target data, and processing the obtained target data, comprises:
determining the value of a data field included in the data column index according to the data column index to which the at least two data acquisition request parameters belong;
and organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data, and processing the acquired target data.
5. The method of claim 1, wherein the parsing the text file using a data warehouse client tool to determine the data column index to which the at least two data acquisition request parameters belong further comprises:
when the text file is analyzed, if the analysis is wrong and the analysis error is a predicted error, suspending the data acquisition request;
or if the server side has an analysis error, automatically stopping analyzing the text file and recording the analysis error by adopting a recording log; and when the text file is analyzed again, continuing to initiate a data acquisition request according to the position recorded by the recording log.
6. A data acquisition apparatus, characterized in that the apparatus comprises:
the text file determining module is used for determining a text file comprising at least two data acquisition request parameters;
the data column index determining module is used for analyzing the text file by adopting a data warehouse client tool and determining the data column index to which the at least two data acquisition request parameters belong;
and the data acquisition request module is used for organizing the values of the data fields included in the data column index into request parameters, initiating a request to a server to acquire target data and processing the acquired target data.
7. The apparatus of claim 6, wherein the text file determining module comprises:
determining a text file according to at least two data acquisition request parameters;
wherein the text file comprises at least two lines of data; the row of data includes a data field and a column separator; the at least two data acquisition request parameters are represented in data fields and separated by a column separator.
8. The apparatus of claim 6, wherein the data column index determining module comprises:
analyzing the text file by adopting a data warehouse client tool to obtain a data structure of the inline data of the text file;
determining data column indexes to which the at least two data acquisition request parameters belong according to the data structure of the row of data; and the data column index represents the positions of the at least two data acquisition request parameters.
9. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the data acquisition method of any one of claims 1-5.
10. A storage medium containing computer-executable instructions for performing the data acquisition method of any one of claims 1-5 when executed by a computer processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210491976.8A CN114860658A (en) | 2022-05-07 | 2022-05-07 | Data acquisition method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210491976.8A CN114860658A (en) | 2022-05-07 | 2022-05-07 | Data acquisition method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114860658A true CN114860658A (en) | 2022-08-05 |
Family
ID=82636336
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210491976.8A Pending CN114860658A (en) | 2022-05-07 | 2022-05-07 | Data acquisition method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114860658A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116384341A (en) * | 2022-12-16 | 2023-07-04 | 西安航天动力试验技术研究所 | Engine test data processing method, storage medium and equipment |
-
2022
- 2022-05-07 CN CN202210491976.8A patent/CN114860658A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116384341A (en) * | 2022-12-16 | 2023-07-04 | 西安航天动力试验技术研究所 | Engine test data processing method, storage medium and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109582660B (en) | Data blood margin analysis method, device, equipment, system and readable storage medium | |
| CN108292323B (en) | Database operations using metadata of data sources | |
| CN107133267B (en) | Method and device for querying elastic search cluster, electronic equipment and readable storage medium | |
| US10713247B2 (en) | Executing queries for structured data and not-structured data | |
| US9959310B2 (en) | Accessing single entities in OData entity sets | |
| US11586585B2 (en) | Method and system for historical call lookup in distributed file systems | |
| US20200349151A1 (en) | Methods, systems, and computer readable mediums for performing an aggregated free-form query | |
| CN106294695A (en) | A kind of implementation method towards the biggest data search engine | |
| CN106687955B (en) | Simplifying invocation of an import procedure to transfer data from a data source to a data target | |
| CN111221791A (en) | Method for importing multi-source heterogeneous data into data lake | |
| US20180349455A1 (en) | Methods, systems, and computer readable mediums for performing a free-form query | |
| US11494395B2 (en) | Creating dashboards for viewing data in a data storage system based on natural language requests | |
| CN110704476A (en) | Data processing method, device, equipment and storage medium | |
| US10901811B2 (en) | Creating alerts associated with a data storage system based on natural language requests | |
| US11263542B2 (en) | Technologies for auto discover and connect to a rest interface | |
| US10776368B1 (en) | Deriving cardinality values from approximate quantile summaries | |
| CN111221851A (en) | Lucene-based mass data query and storage method and device | |
| CN111221785A (en) | Semantic data lake construction method of multi-source heterogeneous data | |
| Kuderu et al. | Relational database to NoSQL conversion by schema migration and mapping | |
| CN114860658A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN114969441A (en) | Knowledge mining engine system based on graph database | |
| CN108959294B (en) | Method and device for accessing search engine | |
| CN110019077A (en) | Log inquiring method, device, equipment and computer readable storage medium | |
| US11200230B2 (en) | Cost-based optimization for document-oriented database queries | |
| CN104331517A (en) | Retrieval method and retrieval device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |