CN114116693A

CN114116693A - Data acquisition method, system, electronic device and medium

Info

Publication number: CN114116693A
Application number: CN202111360703.1A
Authority: CN
Inventors: 张昌达; 黄书珽; 刘力
Original assignee: Ctrip Travel Information Service Shanghai Co Ltd
Current assignee: Ctrip Travel Information Service Shanghai Co Ltd
Priority date: 2021-11-17
Filing date: 2021-11-17
Publication date: 2022-03-01

Abstract

The invention provides a data acquisition method, a system, electronic equipment and a medium, wherein the method comprises the steps of S1, dividing service data in a production environment into a plurality of data blocks; s2, removing duplication of the service data in each data block according to a preset duplication removing rule to obtain duplication removing data; the deduplication rule is data with the same characteristic reserved; s3, acquiring an acquisition rule, and storing the acquired data in a target database; the collected data is data matched with the collection rule in the de-duplication data; s4, under the condition of receiving the query request, acquiring the target data corresponding to the query request from the target database, aggregating the target data, and providing the aggregated target data to the query party of the query request. According to the invention, data are screened and classified, and the screening is passed, so that the data types included in the result data are various, and the data types required by the test are met.

Description

Data acquisition method, system, electronic device and medium

Technical Field

The present invention relates to the field of data processing, and in particular, to a data acquisition method, system, electronic device, and medium.

Background

In the data test, all the service data is collected from the production environment for test, or part of the service data is collected for test. Because the data volume of the business data in the production environment is very huge, whether all the business data can be successfully collected depends on the performance of the hardware equipment, and even if the performance of the hardware equipment supports the collection of all the business data, the data collection efficiency is very low. And collected part of service data cannot contain all data types, and cannot meet the requirement of comprehensive testing.

Disclosure of Invention

The technical problem to be solved by the invention is to provide a data acquisition method, a system, an electronic device and a medium, aiming at overcoming the defects that in the prior art, a mode of acquiring service data for testing either needs hardware support with good performance or cannot meet the requirement of comprehensive testing.

The invention solves the technical problems through the following technical scheme:

a method of data acquisition, the method comprising the steps of:

s1, dividing the service data in the production environment into a plurality of data blocks;

s2, removing duplication of the service data in each data block according to a preset duplication removing rule to obtain duplication removing data; the deduplication rule is data with the same characteristic reserved;

s3, acquiring the acquired data and storing the acquired data in a target database;

the collected data is data matched with a collection rule in the de-duplication data;

s4, under the condition of receiving the query request, acquiring the target data corresponding to the query request from the target database, aggregating the target data, and providing the aggregated target data to the query party of the query request.

Preferably, step S1 includes:

s11, classifying the service data according to the time of the service data or the size of the service data to obtain the data blocks.

Preferably, step S1 includes:

and processing the data blocks through asynchronous multithreading to convert the format of the service data in each data block into a target format.

Preferably, step S2 is followed by:

and S5, marking the duplicate removal data with corresponding labels according to field meanings, and displaying the labels, wherein the labels correspond to the query keywords carried by the query request.

Preferably, the target data is obtained from the production environment in case the target data cannot be obtained from the target database.

As a second aspect of the present invention, the present invention provides a data acquisition system, comprising a classification module, a deduplication module, an acquisition module, and an aggregation module;

the classification module is used for dividing the service data in the production environment into a plurality of data blocks;

the duplication eliminating module is used for eliminating duplication of the service data in each data block according to a preset duplication eliminating rule to obtain duplication eliminating data; the deduplication rules are data that retain the same characteristics,

the acquisition module is used for acquiring acquired data and storing the acquired data in a target database; the collected data is the data matched with the collection rule in the de-duplication data,

the aggregation module is used for acquiring target data corresponding to the query request from the target database under the condition of receiving the query request, aggregating the target data and providing the aggregated target data to a query party of the query request.

Preferably, the classification module comprises: and the classification unit is used for classifying the service data according to the time of generating the service data or the size of the service data to obtain the data blocks.

Preferably, the data acquisition system further comprises: and the format processing module is used for processing the data blocks through asynchronous multithreading so as to convert the format of the service data in each data block into a target format.

Preferably, the data acquisition system further comprises: and the label module is used for marking the duplicate removal data with a corresponding label according to field meaning and displaying the label, wherein the label corresponds to the query keyword carried by the query request.

Preferably, the data acquisition system further comprises: the target data judging module is used for judging whether the target data can be acquired from a received target database;

and under the condition that the target data cannot be acquired from the target database, acquiring the target data from the production environment.

The invention also provides an electronic device, which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the computer program to realize the data acquisition method.

The invention also provides a computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, carries out the data acquisition method of the invention.

The positive progress effects of the invention are as follows:

and removing the duplicate of the data according to the duplicate removal rule, storing the data of which the duplicate removal data is matched with the acquisition rule in a target database, finally acquiring the target data of the query request, and aggregating the target data to obtain the data conforming to the duplicate removal rule.

Drawings

Fig. 1 is a schematic flow chart of a data acquisition method according to embodiment 1 of the present invention.

Fig. 2 is a detailed flowchart of the data acquisition method according to embodiment 1 of the present invention.

Fig. 3 is a schematic structural diagram of a data acquisition system according to embodiment 2 of the present invention.

Fig. 4 is a schematic structural diagram of an electronic device according to embodiment 3 of the present invention.

Detailed Description

The invention is further illustrated by the following examples, which are not intended to limit the scope of the invention.

Example 1

Referring to fig. 1, the present embodiment provides a data acquisition method. The method comprises the following steps:

s2, removing duplication of the service data in each data block according to a preset duplication removing rule to obtain duplication removing data; the deduplication rule is data retaining the same characteristics;

the term "identical" means that the field values of the fields in the data are identical. The data is taken as travel order data for example, and the fields of the travel order can be divided into a destination, an origin, a vehicle used and the like; taking the data as the hotel order as an example, the fields of the hotel order data can be divided into hotel star level, room type, price interval and the like. When the field values of the fields are the same, i.e., the characteristics are the same, the travel order data is the same for the destination, or the origin, or the vehicle; for hotel order data, the hotel star level, the room type or the price interval are equal; the deduplication rule is to retain data that meets a characteristic.

Specifically, when there are several orders, the destination of the order is 10 orders in Shanghai, and the deduplication rule is to filter the orders not destined for Shanghai and filter the 10 orders.

The deduplication rules may be filtered according to the same characteristics of data that need to be retained.

S3, acquiring an acquisition rule, and storing the acquired data in a target database;

the collected data is data matched with the collection rule in the duplicate removal data;

the collection rule is at least one of the characteristics in the data, and can be artificially preset;

and S4, under the condition of receiving the query request, acquiring target data corresponding to the query request from the target database, aggregating the target data, and providing the aggregated target data to the query party of the query request.

The target data is data which is subjected to duplicate removal processing on the business data in the production environment and is matched with the collection rule and the keywords carried by the query request, when the data needs to be queried, the types of the business data needed in the production environment can be contained as much as possible, the data quantity is small, and comprehensive testing, analysis and the like on the business functions can be realized based on the target data.

The aggregated target data can be applied to data transmission to avoid unnecessary data, and the utilization rate of the data is improved.

In step S2, the business data, taking the order data as an example, divides all orders into several data blocks, the data blocks include several order data,

removing duplication of the data block according to duplication removal rules to obtain duplication removal data, wherein the duplication removal data are data with the same characteristics;

for example, the data block includes order data a, order data B, and other order data. Wherein, only the destination of the order data A and the order data B is Shanghai;

when the deduplication rule is that the destination (field) is shanghai (field value), all order data is deduplicated, and finally only order a and order B are reserved.

The collection rule is at least one of characteristics in the deduplication rule, and when the collection rule is the destination Shanghai, data of the destination Shanghai are selected and extracted.

When the service data is order data, the characteristic may be a departure place, a destination, a vehicle, a shift of the vehicle, a departure time, an arrival time, and the like of the order.

In the business data, the business data is not limited to orders, but can also be hotel data, wherein the characteristics in the hotel data comprise the house type, the business district, the star level and the like of the hotel;

the business data can also be scenic spot order message data, the characteristics of which comprise scenic spot levels, peripheral public transportation information and the like, and the characteristics can be set manually, for example, the characteristics can be set according to the type required in required data testing.

The duplicate removal data is reserved with one piece of data with the same property, repeated data with the same property is removed, and when the test is needed, the data is reflowed, only the data which is subjected to the past duplication is needed to be obtained, the data which contains all the data with the same property is not needed to be read, namely, the reread service data is not needed to be obtained, the data quantity is small, and the requirement of the test type is met.

In step S3, the collected data is data matched with the collection rule in the deduplication data;

the information of the fields in the collected data is shown in the following table, and in this embodiment, a deduplication rule can be formulated according to the meaning of the fields, and a collection rule can be formulated according to the meaning of the fields.

The corresponding meanings of the fields are shown in the following table.

Wherein, varchar, int, timestamp, tinyint in the type are field types, the length is the length of the character string byte, the default value is the initial value of each field, and the default value is described as the Chinese meaning of the field name.

Referring to fig. 2, specifically, step S1 includes: and S11, classifying the service data blocks according to the service data generation time or the size of the service data to obtain a plurality of data blocks.

Specifically, step S1 includes: the data blocks are processed through asynchronous multithreading to convert the format of the data in each data block into a target format.

In one embodiment, the format of the data is various, the target format can be set according to the actual requirement, and the target format is not a specific format.

The format of each data block is converted into the target format, and the formats of the data blocks are subjected to unified data format, so that the data blocks can be quickly processed during processing.

When the data in the data block is converted into the target format, the data in the data block can avoid the conversion of the data format, so that the burden of a computer is reduced.

Specifically, step S4 is followed by: and S5, marking the duplicate removal data with corresponding labels according to field meanings, and displaying the labels, wherein the labels correspond to the query keywords carried by the query request, and the query keywords are determined by the user according to the labels displayed by the user.

In one embodiment, the field meaning may be a destination, origin, vehicle used, hotel star, hotel room type, and the like. Taking the field meaning "destination is shanghai" as an example, the query keyword may be "destination" or "shanghai". The tags are not in a particular format, so long as they are easy for the user to understand and correspond to the meaning of the desired keyword.

The labels and fields may be as shown in the following table:

wherein only id is a primary key, and the rest are not primary keys; only "tags" and "fieldValue" are allowed to be empty, and the rest are not allowed to be empty; only "change _ createstame" and "change _ lasttime" have default values, and the others have no default values; only id is self-increment, the rest are not self-increment.

In one embodiment, if the target data cannot be obtained from the target database, no data in the target database satisfies the query request. For example, when the target database has order data for Shanghai, Hangzhou and Beijing, but now needs to query the order data for Nanjing, only Shanghai is in the target database. Order data of Hangzhou and Beijing, which is the order data of Nanjing in the destination, needs to be acquired from the source of the target data and the production environment. And even if the target database does not meet the test environment, the data meeting the test requirements in the test environment can be ensured.

Example 2

Referring to fig. 3, embodiment 2 provides a data acquisition system, which includes a classification module 201, a deduplication module 202, an acquisition module 203, and an aggregation module 204;

the classification module 201 is configured to divide the service data in the production environment into a plurality of data blocks;

the deduplication module 202 is configured to perform deduplication on the service data in each data block according to a preset deduplication rule to obtain deduplication data; the deduplication rules are data that retain the same characteristics,

the acquisition module 203 is used for acquiring acquired data and storing the acquired data in a target database; the collected data is the data matched with the collection rule in the de-duplication data,

the aggregation module 204 is configured to, in a case that a query request is received, obtain target data corresponding to the query request from the target database, aggregate the target data, and provide the aggregated target data to a querying party of the query request.

Specifically, the classification module 201 includes: a classifying unit 205, configured to classify the service data according to a time of generating the service data or a size of the service data, so as to obtain the data blocks.

Specifically, the data acquisition system further comprises: a format processing module 206, wherein the format processing module 206 is configured to process the data blocks through asynchronous multithreading to convert the format of the service data in each data block into a target format.

Specifically, the data acquisition system further comprises: a label module 207, where the label module 207 is configured to print a corresponding label on the duplicate removal data according to a field meaning, and display the label, where the label corresponds to the query keyword carried by the query request.

Specifically, the data acquisition system further includes:

a target data determining module 208, where the target data determining module 208 is configured to determine whether the target data can be obtained from a target database;

Example 3

Fig. 4 is a schematic structural diagram of an electronic device provided in this embodiment. The electronic device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, and the processor executes the program to realize the data acquisition and processing method of the embodiment 1. The electronic device 30 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.

As shown in fig. 4, the electronic device 30 may be embodied in the form of a general purpose computing device, which may be, for example, a server device. The components of the electronic device 30 may include, but are not limited to: the at least one processor 31, the at least one memory 32, and a bus 33 connecting the various system components (including the memory 32 and the processor 31).

The bus 33 includes a data bus, an address bus, and a control bus.

The memory 32 may include volatile memory, such as Random Access Memory (RAM)321 and/or cache memory 322, and may further include Read Only Memory (ROM) 323.

Memory 32 may also include a program/utility 325 having a set (at least one) of program modules 324, such program modules 324 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

The processor 31 executes various functional applications and data processing, such as the data acquisition method of embodiment 1 of the present invention, by executing the computer program stored in the memory 32.

The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). Such communication may be through input/output (I/O) interfaces 35. Also, model-generating device 30 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via network adapter 36. As shown, network adapter 36 communicates with the other modules of model-generating device 30 via bus 33. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the model-generating device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

It should be noted that although in the above detailed description several units/modules or sub-units/modules of the electronic device are mentioned, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module according to embodiments of the invention. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.

Example 4

The present embodiment provides a computer-readable storage medium on which a computer program is stored, the program implementing the data acquisition method of embodiment 1 when executed by a processor.

More specific examples, among others, that the readable storage medium may employ may include, but are not limited to: a portable disk, a hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.

In a possible implementation manner, the present invention can also be implemented in the form of a program product, which includes program code for causing a terminal device to execute the data acquisition method implementing embodiment 1 when the program product runs on the terminal device.

Where program code for carrying out the invention is written in any combination of one or more programming languages, the program code may be executed entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device and partly on a remote device or entirely on the remote device.

While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.

Claims

1. A data acquisition method, characterized in that the data acquisition method comprises the steps of:

2. The data acquisition method as set forth in claim 1, wherein the step S1 includes:

3. The data acquisition method as set forth in claim 1, wherein the step S1 includes:

4. The data acquisition method as claimed in claim 1, wherein step S2 is followed by further comprising:

5. The data collection method of claim 1, wherein the target data is obtained from the production environment in the event that the target data is not available from the target database.

6. A data acquisition system is characterized by comprising a classification module, a duplication elimination module, an acquisition module and an aggregation module;

7. The data acquisition system of claim 6 wherein the classification module comprises: and the classification unit is used for classifying the service data according to the time of generating the service data or the size of the service data to obtain the data blocks.

8. The data acquisition system of claim 6, further comprising:

and the format processing module is used for processing the data blocks through asynchronous multithreading so as to convert the format of the service data in each data block into a target format.

9. The data acquisition system of claim 6, further comprising:

and the label module is used for marking the duplicate removal data with a corresponding label according to field meaning and displaying the label, wherein the label corresponds to the query keyword carried by the query request.

10. The data acquisition system of claim 6, further comprising:

the target data judging module is used for judging whether the target data can be acquired from a received target database;

11. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data acquisition method of any one of claims 1 to 5 when executing the computer program.

12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data acquisition method of any one of claims 1 to 5.