CN115599869B

CN115599869B - Data acquisition method and device, electronic equipment and storage medium

Info

Publication number: CN115599869B
Application number: CN202211597947.6A
Authority: CN
Inventors: 康海峰
Original assignee: Lianren Healthcare Big Data Technology Co Ltd
Current assignee: Lianren Healthcare Big Data Technology Co Ltd
Priority date: 2022-12-14
Filing date: 2022-12-14
Publication date: 2023-03-14
Anticipated expiration: 2042-12-14
Also published as: CN115599869A

Abstract

The invention discloses a data acquisition method, a data acquisition device, electronic equipment and a storage medium. The method comprises the following steps: acquiring original data uploaded by a client, and storing the original data uploaded by the client to a first backup data table; performing first data quality control on the original data uploaded by the client to obtain first quality control processing data; transmitting the first quality control processing data to a message queue, calling data from the message queue, and performing second data quality control on the called data to obtain second quality control processing data; and storing the second quality control processing data into a second backup data table, and storing the second quality control processing data into a target database. According to the technical scheme, the first backup data table and the second backup data table are added, so that data in the transmission and processing processes of the data are backed up, and the data loss is prevented.

Description

Data acquisition method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data acquisition method and apparatus, an electronic device, and a storage medium.

Background

Under the condition of large-batch data acquisition, the accumulation of consumption transaction processing of large amount of data causes that the performance is slowed down after a queue is blocked, and the transaction can be automatically cleaned after being unprocessed for a long time, so that the phenomenon of data packet loss occurs.

The existing data acquisition processing scheme is shown in fig. 1, a database adopts a split-database design, the database includes a PG (postgresql) database and an oracle database, the two transactions of the PG database and the oracle database are asynchronous, and the two transactions cannot be in pace, so that data quality control and storage are realized when the transaction processing of the PG database is finished, but the data cannot fall into the oracle database, a large amount of memory space is consumed through a redis queue processing mechanism, memory overflow and server crash can be caused under the condition that the transaction is not processed timely, and data loss occurs.

In the process of implementing the present invention, the inventor finds that at least the following technical problems exist in the prior art: the existing data acquisition scheme has the problem of data loss.

Disclosure of Invention

The invention provides a data acquisition method, a data acquisition device, electronic equipment and a storage medium, wherein a backup data table is added to prevent data loss.

According to an aspect of the present invention, there is provided a data acquisition method, including:

acquiring original data uploaded by a client, and storing the original data uploaded by the client to a first backup data table;

performing first data quality control on the original data uploaded by the client to obtain first quality control processing data;

transmitting the first quality control processing data to a message queue, calling data from the message queue, and performing second data quality control on the called data to obtain second quality control processing data;

and storing the second quality control processing data into a second backup data table, and storing the second quality control processing data into a target database.

According to another aspect of the present invention, there is provided a data acquisition apparatus comprising:

the first data backup module is used for acquiring original data uploaded by a client and storing the original data uploaded by the client to a first backup data table;

the first data quality control module is used for performing first data quality control on the original data uploaded by the client to obtain first quality control processing data;

the second data quality control module is used for transmitting the first quality control processing data to a message queue, calling the data from the message queue, and performing second data quality control on the called data to obtain second quality control processing data;

and the second data backup module is used for storing the second quality control processing data to a second backup data table and storing the second quality control processing data into a target database.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data acquisition method of any of the embodiments of the invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the data acquisition method according to any one of the embodiments of the present invention when the computer instructions are executed.

According to the technical scheme of the embodiment of the invention, original data uploaded by a client side are obtained and stored in a first backup data table, so that the original data are backed up, the data are ensured to be traceable, further, first data quality control is carried out on the original data uploaded by the client side, first quality control processing data are obtained, the first quality control processing data are transmitted to a message queue, the data are called from the message queue, and second data quality control is carried out on the called data, so that second quality control processing data are obtained; and storing the second quality control processing data into a second backup data table to backup the data after quality control processing, thereby preventing the data from being lost and successfully storing the data into the target database.

It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is an architecture diagram of a data acquisition process of the prior art;

fig. 2 is a flowchart of a data acquisition method according to an embodiment of the present invention;

FIG. 3 is a flowchart of a data acquisition method according to a second embodiment of the present invention;

FIG. 4 is an architecture diagram of a data acquisition process according to a second embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a data acquisition device according to a third embodiment of the present invention;

fig. 6 is a schematic structural diagram of an electronic device implementing the data acquisition method according to the embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 2 is a flowchart of a data acquisition method according to an embodiment of the present invention, where the present embodiment is applicable to a situation of large-batch data acquisition, the method may be executed by a data acquisition device, the data acquisition device may be implemented in a form of hardware and/or software, and the data acquisition device may be configured in a server. As shown in fig. 2, the method includes:

s110, acquiring original data uploaded by a client, and storing the original data uploaded by the client to a first backup data table.

In this embodiment, the client is in communication connection with the server, the client can upload data to the server, and the server receives the original data uploaded by the client. Raw data refers to unprocessed or reduced data, e.g., raw data can be a variety of data that is stored for use by a client. The first backup data table is used for storing original data uploaded by the client.

For example, in a medical scenario, the client may be a hospital client, and the server receives original data uploaded by the hospital client, where the original data may include but is not limited to an electronic medical record, a medical order, an examination, and the like, and stores the original data in the first backup data table, so that the original data is backed up, and the data is guaranteed to be traceable. The first backup data table may include, but is not limited to, a table main key, a hospital code, data uploading time, a message unique code, an uploading data type, a message, and the like.

And S120, performing first data quality control on the original data uploaded by the client to obtain first quality control processing data.

In this embodiment, the data quality control refers to a process of screening original data according to a preset quality control standard. The preset quality control standard can be obtained according to the settlement list filling specification issued by the state. Illustratively, medical costs require a digital type and cannot be in units; alternatively, the disease code, must conform to ICD-10; still alternatively, the number of disease names cannot exceed 50.

It should be emphasized that the first quality control processing data is the data which passes the quality control, and the data which does not pass the quality control does not participate in the subsequent processing.

S130, transmitting the first quality control processing data to a message queue, calling the data from the message queue, and performing second data quality control on the called data to obtain second quality control processing data.

It should be noted that the first data quality control and the second data quality control may be data processing procedures with two different quality control standards. Illustratively, the first data quality control may be a relatively coarse quality control, and the second data quality control may be more stringent and finer in quality control criteria than the first data quality control, and may result in compliance data that can be binned.

In this embodiment, the message queue refers to a container for storing messages in the transmission process of the messages. Illustratively, the message queue may be a RocketMQ queue.

And S140, storing the second quality control processing data into a second backup data table, and storing the second quality control processing data into a target database.

In this embodiment, the second backup data table is a backup table for storing the second quality control processing data. The second quality control processing data is stored in the second backup data table, so that the second quality control processing data can be backed up, and the data loss is avoided. The second backup data table may include, but is not limited to, a table main key, a hospital code, a data upload time, a message unique code, an upload data type, a message, and the like. The target database may be a single database, for example the target database may be an oracle database.

In some optional embodiments, performing first data quality control on the original data uploaded by the client to obtain first quality control processing data includes: if the original data uploaded by the client side passes the first data quality control, writing quality control passing information into a warehousing record table, and taking the original data after the quality control as first quality control processing data, wherein the quality control passing information comprises a data identifier; correspondingly, the method further comprises the following steps: and calling a warehousing record table based on the data identification, and updating the warehousing record table to the target database.

The warehousing record table is used for recording quality control passing information, and the quality control passing information may include but is not limited to data identification, messages, abnormal information, warehousing state information and the like. The data identification has uniqueness.

Illustratively, the original data uploaded by the client is subjected to quality control through Remote Procedure Call (RPC) quality control service, and if the quality control passes, quality control pass information is written into the storage record table to implement data recording.

In some optional embodiments, after performing the first data quality control on the raw data uploaded by the client, the method further includes: and writing the quality control result of the first data quality control into a quality control result table, and feeding back the quality control result to the client, wherein the quality control result of the first data quality control comprises a quality control passing result and a quality control failing result.

And the quality control result table is used for recording the quality control result of the first data quality control.

Illustratively, taking a medical scene as an example, if quality control passes, recording a quality control passing result in a quality control result table, and feeding back the quality control passing result to a corresponding hospital client, and if the quality control fails, recording a quality control failing result in the quality control result table, and feeding back the quality control failing result to the corresponding hospital client, so as to record and feed back the quality control result, and enable the hospital client to know the data acquisition condition.

According to the technical scheme of the embodiment of the invention, original data uploaded by a client is obtained and stored in a first backup data table, so that the original data is backed up, the data is ensured to be traceable, further, first data quality control is carried out on the original data uploaded by the client to obtain first quality control processing data, the first quality control processing data is transmitted to a message queue, the data is called from the message queue, and second data quality control is carried out on the called data to obtain second quality control processing data; and storing the second quality control processing data into a second backup data table to backup the data after quality control processing, thereby preventing the data from being lost and successfully storing the data into the target database.

Example two

Fig. 3 is a flowchart of a data acquisition method provided in a second embodiment of the present invention, and the method in this embodiment may be combined with each alternative in the data acquisition method provided in the foregoing embodiment. The data acquisition method provided by the embodiment is further optimized. Optionally, after storing the original data uploaded by the client in the first backup data table, the method further includes: and if the first quality control processing data is lost in the process of uploading the first quality control processing data to a message queue, acquiring the original data uploaded by the client from the first backup data table, and performing data quality control on the original data uploaded by the client. And after storing the second quality control processing data to a second backup data table, the method further comprises: if the second quality control processing data cannot be consumed, acquiring the second quality control processing data from the second backup data table, and performing consumption retry processing on the second quality control processing data in the second backup data table; if the consumption retry processing fails, exception information is generated.

As shown in fig. 3, the method includes:

s210, acquiring original data uploaded by a client, and storing the original data uploaded by the client to a first backup data table.

S220, performing first data quality control on the original data uploaded by the client to obtain first quality control processing data.

And S230, transmitting the first quality control processing data to a message queue, calling the data from the message queue, and performing second data quality control on the called data to obtain second quality control processing data.

And S240, storing the second quality control processing data into a second backup data table, and storing the second quality control processing data into a target database.

And S250, if the first quality control processing data is lost in the process of uploading to the message queue, acquiring the original data uploaded by the client from the first backup data table, and performing data quality control on the original data uploaded by the client.

It can be understood that, in the embodiment, by adding the first backup data table to perform backup of the original data, the backed-up original data can be called to perform subsequent data processing when a message is lost during data transmission, so that the situation of data loss can be coped with, and data acquisition safety is ensured.

S260, if the second quality control processing data cannot be consumed, acquiring the second quality control processing data from the second backup data table, and performing consumption retry processing on the second quality control processing data in the second backup data table; if the consumption retry processing fails, exception information is generated.

The consumption retry processing refers to an operation process of consuming the data which fails to be consumed or cannot be consumed again. The retry number of the consumption retry process may be set according to the data acquisition requirement, and is not limited herein.

It can be understood that, in this embodiment, the second backup data table is added to perform backup of the quality control processing data, and the backed-up quality control processing data can be called to perform consumption retry processing under the condition that the data cannot be consumed or the consumption fails, so that the condition of data loss can be coped with, and the data acquisition safety is ensured.

In some optional embodiments, storing the second quality control processing data to a second backup data table includes: and storing the second quality control processing data to a second backup data table in a CLOB form.

It can be understood that the CLOB form can directly store the words, which facilitates data management.

In some optional embodiments, after storing the second quality control processing data in the form of CLOB to the second backup data table, the method further comprises: and classifying the second quality control processing data in the CLOB form to obtain consumption success data and consumption failure data.

For example, after the data passing the quality control is stored in the form of CLOB, the data may be classified, and the classification result may include consumption success data and consumption failure data, so that a label of consumption success may be printed on the consumption success data, and a label of consumption failure may be printed on the consumption failure data, so as to perform problem troubleshooting in the following.

For example, fig. 4 is an architecture diagram of a data acquisition process according to an embodiment of the present invention. And (4) performing authority verification through a thread scheduling manager (dispatcher project), and storing the total amount of original data uploaded by the hospital. And further, calling a quality control project to control the data quality, and if the quality control passes, writing quality control passing information into a warehousing record table and writing a RockettMQ message. And further monitoring the RocktMQ queue, calling a warehousing record table according to the data identification and updating the warehousing record table to an oracle database. Wherein, the data identification can be a global unique ID. In addition, the architecture in fig. 4 eliminates the redis queue, thereby reducing the use of memory, and increasing the message carrying capacity by consuming the disk space of the server for storage.

According to the technical scheme of the embodiment of the invention, the first backup data table is added for backing up the original data, the backed-up original data can be called for subsequent data processing under the condition that the message is lost in data transmission, the second backup data table is added for backing up the quality control processing data, and the backed-up quality control processing data can be called for retry consumption processing under the condition that the data cannot be consumed or the consumption fails, so that the condition of data loss can be coped with, and the data acquisition safety is ensured.

EXAMPLE III

Fig. 5 is a schematic structural diagram of a data acquisition device according to a third embodiment of the present invention. As shown in fig. 5, the apparatus includes:

the first data backup module 310 is configured to obtain original data uploaded by a client, and store the original data uploaded by the client in a first backup data table;

the first data quality control module 320 is configured to perform first data quality control on the original data uploaded by the client to obtain first quality control processing data;

the second data quality control module 330 is configured to transmit the first quality control processing data to a message queue, call data from the message queue, and perform second data quality control on the called data to obtain second quality control processing data;

and the second data backup module 340 is configured to store the second quality control processing data in a second backup data table, and store the second quality control processing data in a target database.

In some optional embodiments, the first data quality control module 320 is specifically configured to:

if the original data uploaded by the client side passes the first data quality control, writing quality control passing information into a warehousing record table, and taking the original data after the quality control as first quality control processing data, wherein the quality control passing information comprises a data identifier;

correspondingly, the device further comprises:

and the warehousing record table updating module is used for calling the warehousing record table based on the data identification and updating the warehousing record table to the target database.

In some optional embodiments, the apparatus further comprises:

and the quality control result feedback module is used for writing the quality control result of the first data quality control into a quality control result table and feeding the quality control result back to the client, wherein the quality control result of the first data quality control comprises a quality control passing result and a quality control failing result.

In some optional embodiments, the apparatus further comprises:

and the message loss processing module is used for acquiring the original data uploaded by the client from the first backup data table and performing data quality control on the original data uploaded by the client if the first quality control processing data is lost in the process of uploading the first quality control processing data to the message queue.

In some optional embodiments, the second data backup module 340 includes:

and the CLOB form backup unit is used for storing the second quality control processing data to a second backup data table in a CLOB form.

In some optional embodiments, the second data backup module 340 is further configured to:

and classifying the second quality control processing data in the CLOB form to obtain consumption success data and consumption failure data.

In some optional embodiments, the apparatus further comprises:

the message unconsumed processing module is used for acquiring second quality control processing data from the second backup data table if the second quality control processing data cannot be consumed, and carrying out consumption retry processing on the second quality control processing data in the second backup data table;

and the abnormal information generation module is used for generating abnormal information if the consumption retry processing fails.

The data acquisition device provided by the embodiment of the invention can execute the data acquisition method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

Example four

FIG. 6 illustrates a schematic structural diagram of an electronic device 10 that may be used to implement an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

Processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as a data acquisition method, including:

In some embodiments, the data acquisition method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data acquisition method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data acquisition method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of data acquisition, comprising:

storing the second quality control processing data to a second backup data table, and storing the second quality control processing data into a target database;

the method for performing first data quality control on the original data uploaded by the client to obtain first quality control processing data comprises the following steps:

correspondingly, the method further comprises the following steps:

and calling a warehousing record table based on the data identification, and updating the warehousing record table to the target database.

2. The method of claim 1, wherein after the performing a first data quality control on the raw data uploaded by the client, the method further comprises:

and writing the quality control result of the first data quality control into a quality control result table, and feeding the quality control result back to the client, wherein the quality control result of the first data quality control comprises a quality control passing result and a quality control failing result.

3. The method of claim 1, wherein after storing the original data uploaded by the client to the first backup data table, the method further comprises:

and if the first quality control processing data is lost in the process of uploading the first quality control processing data to a message queue, acquiring the original data uploaded by the client from the first backup data table, and performing data quality control on the original data uploaded by the client.

4. The method of claim 1, wherein storing the second quality control process data to a second backup data table comprises:

and storing the second quality control processing data to a second backup data table in a CLOB form.

5. The method of claim 4, wherein after storing the second quality control process data in a CLOB format to a second backup data table, the method further comprises:

6. The method of claim 1, wherein after storing the second quality control process data to a second backup data table, the method further comprises:

if the second quality control processing data cannot be consumed, acquiring the second quality control processing data from the second backup data table, and performing consumption retry processing on the second quality control processing data in the second backup data table;

if the consumption retry processing fails, exception information is generated.

7. A data acquisition device, comprising:

the second data backup module is used for storing the second quality control processing data into a second backup data table and storing the second quality control processing data into a target database;

the first data quality control module is specifically used for:

if the original data uploaded by the client side pass the first data quality control, writing quality control passing information into a warehousing record table, and taking the original data after the quality control as first quality control processing data, wherein the quality control passing information comprises a data identifier;

correspondingly, the device further comprises:

8. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data acquisition method of any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions for causing a processor to perform the data acquisition method of any one of claims 1-6 when executed.