CN114547011A - Data extraction method and device, electronic equipment and storage medium - Google Patents
Data extraction method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114547011A CN114547011A CN202210161359.1A CN202210161359A CN114547011A CN 114547011 A CN114547011 A CN 114547011A CN 202210161359 A CN202210161359 A CN 202210161359A CN 114547011 A CN114547011 A CN 114547011A
- Authority
- CN
- China
- Prior art keywords
- data
- interface
- field
- sequence
- updated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Abstract
The invention relates to the data processing technology and discloses a data extraction method, which comprises the steps of sequencing all data interfaces according to the priority of a data source to obtain an interface sequence; extracting data from a data source corresponding to a first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data; judging whether a blank field exists in the first interface data; when a blank field exists in the first interface data, extracting data from other interfaces to perform supplementary replacement on the blank field in the first interface data until the first interface data does not have the blank field or all the interfaces are completely extracted, obtaining updated first interface data, and sending the data to a preset terminal. The invention also relates to a blockchain technique, the first interface data may be stored in a blockchain link point. The invention also provides a data extraction device, equipment and a medium. The invention can improve the efficiency of data extraction.
Description
Technical Field
The present invention relates to data processing technologies, and in particular, to a data extraction method and apparatus, an electronic device, and a storage medium.
Background
With the advent of the big data era, the current data application platform often interfaces with the multiple data sources, and because different data sources are different, how to extract data from different data sources becomes a problem that needs to be researched.
However, the current data extraction method needs to extract data from all data sources for integration, which takes much time, resulting in low data extraction efficiency.
Disclosure of Invention
The invention provides a data extraction method, a data extraction device, electronic equipment and a storage medium, and mainly aims to improve the data extraction efficiency.
Acquiring data interfaces of different data sources and corresponding data source priorities, and sequencing all the data interfaces according to the data source priorities to obtain interface sequences;
extracting data from a data source corresponding to a first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data;
judging whether a blank field exists in the first interface data;
when no blank field exists in the first interface data, the first interface data are sent to preset terminal equipment of a data extraction person;
when a blank field exists in the first interface data, updating the interface number of the interface sequence, and judging whether the interface number in the updated interface sequence is greater than zero or not;
when the number of the interfaces in the updated interface sequence is larger than zero, extracting data from a data source corresponding to a first data interface in the updated interface sequence, and mapping the data to a preset field to obtain second interface data;
carrying out data replacement of the same field on a blank field in the first interface data by using the second interface data to obtain updated first interface data, and returning to the step of judging whether the blank field exists in the first interface data;
and when the number of the interfaces in the updated interface sequence is not more than zero, sending the first interface data to the terminal equipment.
Optionally, the extracting data from the data source corresponding to the first data interface in the interface sequence and mapping the data to a preset field to obtain first interface data includes:
acquiring configuration information and a field mapping rule of a first data interface in the interface sequence;
performing data extraction on the data source corresponding to the first data interface by using the configuration information to obtain initial extraction data;
and mapping the initial extraction data to a preset field to obtain the first interface data.
Optionally, the determining whether a blank field exists in the first interface data includes:
inquiring whether a value corresponding to each field in the first interface data is a null value;
when the value corresponding to the field in the first interface data is null, determining the field as a blank field.
Optionally, the performing, by using the second interface data, data replacement of the same field for a blank field in the first interface data to obtain updated first interface data includes:
determining the blank field as a field to be replaced;
extracting a value corresponding to the field to be replaced in the second interface data to obtain a numerical value to be replaced;
and replacing the value of the same field in the first interface data with the value to be replaced to obtain updated first interface data.
Optionally, the sending the first interface data to the terminal device includes:
packaging the first interface data into a file with a preset file format to obtain a target file;
and sending the target file to the terminal equipment.
Optionally, the updating the number of interfaces of the interface sequence includes:
and deleting the first data interface in the interface sequence to obtain an updated interface sequence.
Optionally, the performing, by using the configuration information, data extraction on the data source to which the first data interface belongs to obtain initial extraction data includes:
analyzing the input/output parameters of the first data interface from the configuration information;
determining a data source corresponding to the configuration information as a target data source;
converting the input/output parameters into a data source executable language of the target data source;
and sending the data source executable language to the target data source, and receiving data returned by the target data source according to the data source executable language to obtain the initial extracted data.
In order to solve the above problem, the present invention also provides a data extraction apparatus, including:
the data extraction module is used for acquiring data interfaces of different data sources and corresponding data source priorities, and sequencing all the data interfaces according to the data source priorities to obtain an interface sequence; extracting data from a data source corresponding to a first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data;
a blank field judging module, configured to judge whether a blank field exists in the first interface data; when no blank field exists in the first interface data, the first interface data are sent to preset terminal equipment of a data extraction person; when a blank field exists in the first interface data, updating the interface number of the interface sequence, and judging whether the interface number in the updated interface sequence is greater than zero or not;
the interface number judging module is used for extracting data from a data source corresponding to a first data interface in the updated interface sequence and mapping the data to a preset field to obtain second interface data when the number of interfaces in the updated interface sequence is greater than zero; carrying out data replacement of the same field on a blank field in the first interface data by using the second interface data to obtain updated first interface data, and returning to the step of judging whether the blank field exists in the first interface data; and when the number of the interfaces in the updated interface sequence is not more than zero, sending the first interface data to the terminal equipment.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one computer program; and
and the processor executes the computer program stored in the memory to realize the data extraction method.
In order to solve the above problem, the present invention further provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the data extraction method described above.
The embodiment of the invention judges whether a blank field exists in the first interface data; when a blank field exists in the first interface data, updating the interface number of the interface sequence, and judging whether the interface number in the updated interface sequence is greater than zero or not; when the number of the interfaces in the updated interface sequence is larger than zero, extracting data from a data source corresponding to a first data interface in the updated interface sequence, and mapping the data to a preset field to obtain second interface data; and performing data replacement of the same field on the blank field in the first interface data by using the second interface data to obtain updated first interface data, extracting data from a data source with a high priority in a limited way, and extracting missing data from other data sources for supplement if the extracted data is incomplete, so that the data does not need to be extracted from all the data sources for integration, and the data is extracted as required, thereby improving the efficiency of data extraction. Therefore, the data extraction method, the data extraction device, the electronic equipment and the readable storage medium provided by the embodiment of the invention improve the data extraction efficiency.
Drawings
Fig. 1 is a schematic flow chart of a data extraction method according to an embodiment of the present invention;
fig. 2 is a schematic block diagram of a data extraction apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an internal structure of an electronic device implementing a data extraction method according to an embodiment of the present invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the invention provides a data extraction method. The execution subject of the data extraction method includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiments of the present application. In other words, the data extraction method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a block chain platform. The server includes but is not limited to: the cloud server can be an independent server, or can be a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, Network service, cloud communication, middleware service, domain name service, security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.
Referring to fig. 1, a schematic flow chart of a data extraction method according to an embodiment of the present invention is shown, where in the embodiment of the present invention, the data extraction method includes:
s1, acquiring data interfaces of different data sources and corresponding data source priorities, and sequencing all the data interfaces according to the data source priorities to obtain an interface sequence.
In detail, in the embodiment of the present invention, the data interface is an interface that extracts data from the data source, and the data source priority is an index that evaluates reliability of data of different data sources, and optionally, the data source priority may be represented by a numerical value or a priority level.
Further, in the embodiment of the present invention, all the data interfaces are sorted according to the priority of the data source, so as to obtain the interface sequence.
Optionally, in the interface sequence in the embodiment of the present invention, the interface corresponding to the data source with the higher priority is ranked in front, and the interface corresponding to the data source with the lower priority is ranked behind, for example: the data source priority is represented by a numerical value, and two data interfaces are shared: the data interface A and the data interface B are respectively connected, the priority of a data source corresponding to the data interface A is 0.8, the priority of a data source corresponding to the data interface B is 0.7, and then the interface sequence is [ data interface A, data interface B ].
And S2, extracting data from the data source corresponding to the first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data.
In detail, in the embodiment of the present invention, extracting data from a data source corresponding to a first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data, includes:
step A: acquiring configuration information and a field mapping rule of a first data interface in the interface sequence;
in the embodiment of the invention, the configuration information is preset interface configuration information and can be adjusted according to actual requirements.
Optionally, in the embodiment of the present invention, the interface configuration information includes information such as a data source identifier and an interface input/output parameter, so that the data source identifier may be parsed from the configuration information, and the input/output parameter corresponding to the data interface may be parsed from the configuration information.
And B: performing data extraction on the data source corresponding to the first data interface by using the configuration information to obtain initial extraction data;
in the embodiment of the present invention, the data source corresponding to the first interface is a data source to which the first interface belongs.
Specifically, in the embodiment of the present invention, performing data extraction on the data source corresponding to the first data interface by using the configuration information to obtain initial extraction data, includes:
step a: analyzing the input/output parameters of the first data interface from the configuration information;
step b: determining a data source corresponding to the configuration information as a target data source, and converting the input/output parameters into a data source executable language of the target data source;
in this embodiment of the present invention, a data source to which a data interface corresponding to the configuration information belongs is a target data source.
In another embodiment of the present invention, a data source identifier may also be parsed from the configuration information, and a data source corresponding to the data source identifier is determined as a target data source, where the data source identifier may be a data source id, a data source name, and the like, and the corresponding data source is identified by the data source identifier, which is not limited in this embodiment of the present invention.
Further, in another embodiment of the present invention, a preset data source adapter may be further utilized to convert the input/output parameters into a data source executable language of the target data source, where for the data source executable language, the data source executable language corresponds to a data source one to one, for example: if the type of the data source is Mysql, the executable Language of the data source may be SQL (Structured Query Language), which is not limited in the embodiment of the present invention. In the embodiment of the invention, for different types of data sources, corresponding pre-configured data source adapters exist, which means that for each type of data source, one data source adapter exists corresponding to the data source adapter.
Step c: and sending the data source executable language to the target data source, and receiving data returned by the target data source according to the data source executable language to obtain the initial extracted data.
Step C: and mapping the initial extraction data to a preset field to obtain the first interface data.
For example: the initial extracted data is 'name: Zhang III', address: beijing, the field mapping rule is that the field of 'name' is mapped into a preset field of 'name', the field of 'address' is mapped into a preset field of 'address', and then the first interface data is 'name: Zhang III', address: beijing ″.
In the embodiment of the invention, the returned data is mapped into different preset fields by utilizing the field mapping rule, so that the data extracted from the data source is ensured to have fixed fields and data formats.
In another embodiment of the present invention, the first interface data may be stored in a block link point, and the data access efficiency is improved by using the characteristic of high throughput of the block link point.
And S3, judging whether a blank field exists in the first interface data.
In the embodiment of the invention, whether a blank field exists in the first interface data is judged, so that whether the data extracted from the data source is complete or not is judged.
In detail, the determining whether a blank field exists in the first interface data in the embodiment of the present invention includes:
inquiring whether a value corresponding to each field in the first interface data is a null value;
when the value corresponding to the field in the first interface data is null, determining the field as a blank field.
For example: the first interface data comprises two fields which are a name field and an address field respectively, wherein the value corresponding to the name field is Zhang III, the value corresponding to the address field is null, and the address field is a blank field.
And S4, when no blank field exists in the first interface data, sending the first interface data to preset terminal equipment of a data extraction person.
In the embodiment of the present invention, when no blank field exists in the first interface data, it indicates that the data corresponding to all fields of the first interface data is successfully extracted, that is, the first interface data includes all data that needs to be extracted.
Further, the embodiment of the invention sends the first interface data to a preset terminal device of a data extraction person. The terminal device includes: intelligent terminals such as mobile phones, computers and tablets.
And S5, when a blank field exists in the first interface data, updating the interface number of the interface sequence, and judging whether the interface number in the updated interface sequence is larger than zero.
In the embodiment of the present invention, if there is a blank field in the first interface data, it indicates that the data in the data source is incomplete, and all the required data cannot be extracted.
Specifically, in the embodiment of the present invention, updating the number of interfaces of the interface sequence includes: and deleting the first data interface in the interface sequence to obtain an updated interface sequence.
In another embodiment of the present invention, the updating the number of interfaces of the interface sequence includes: and selecting all the data interfaces in the second and the following interface sequences to obtain the updated interface sequence.
Further, in order to determine whether the interface sequence can also satisfy the next update, the embodiment of the present invention needs to determine whether the number of interfaces in the updated interface sequence is greater than zero.
And S6, when the number of the interfaces in the updated interface sequence is larger than zero, extracting data from a data source corresponding to the first data interface in the updated interface sequence, and mapping the data to a preset field to obtain second interface data.
In the embodiment of the present invention, a method for extracting data mapping from the data source corresponding to the first data interface in the updated interface sequence to the preset field is similar to the method described in step S3, and is not described herein again.
S7, carrying out data replacement of the same field on the blank field in the first interface data by using the second interface data to obtain updated first interface data, and returning to S3.
Specifically, in the embodiment of the present invention, performing data replacement on a blank field in the first interface data by using the second interface data to obtain updated first interface data, where the data replacement includes:
determining the blank field as a field to be replaced;
extracting a value corresponding to the field to be replaced in the second interface data to obtain a numerical value to be replaced;
and replacing the value of the same field in the first interface data with the value to be replaced to obtain updated first interface data.
For example: and if the blank field is an address field and the address field in the second interface data is Beijing, replacing the null value corresponding to the address field in the first interface data by the Beijing to obtain the updated first interface data.
And S8, when the number of the interfaces in the updated interface sequence is not more than zero, sending the first interface data to the terminal equipment.
In another embodiment of the present invention, for convenience of query, before the first interface data is sent to the terminal device, the first interface data may be further packaged into a file in a preset file format to obtain an object file, and the object file is sent to the terminal device.
Optionally, the file format in the embodiment of the present invention includes but is not limited to: txt, xlsx, etc.
Fig. 2 is a functional block diagram of the data extraction device according to the present invention.
The data extraction apparatus 100 of the present invention may be installed in an electronic device. According to the implemented functions, the data extraction device may include a data extraction module 101, a blank field judgment module 102, and an interface number judgment module 103, which may also be referred to as a unit, and refers to a series of computer program segments that can be executed by a processor of an electronic device and can perform fixed functions, and are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the data extraction module 101 is configured to obtain data interfaces of different data sources and corresponding data source priorities, and sort all the data interfaces according to the data source priorities to obtain an interface sequence; extracting data from a data source corresponding to a first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data;
the blank field determining module 102 is configured to determine whether a blank field exists in the first interface data; when no blank field exists in the first interface data, the first interface data are sent to preset terminal equipment of a data extraction worker; when a blank field exists in the first interface data, updating the interface number of the interface sequence, and judging whether the interface number in the updated interface sequence is greater than zero or not;
the interface number judging module 103 is configured to, when the number of interfaces in the updated interface sequence is greater than zero, extract data from a data source corresponding to a first data interface in the updated interface sequence and map the data to a preset field to obtain second interface data; carrying out data replacement of the same field on a blank field in the first interface data by using the second interface data to obtain updated first interface data, and returning to the step of judging whether the blank field exists in the first interface data; and when the number of the interfaces in the updated interface sequence is not more than zero, sending the first interface data to the terminal equipment.
In detail, in the embodiment of the present invention, when the modules in the data extraction device 100 are used, the same technical means as the data extraction method described in fig. 1 are used, and the same technical effect can be produced, which is not described herein again.
Fig. 3 is a schematic structural diagram of an electronic device implementing the data extraction method according to the present invention.
The electronic device may comprise a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may further comprise a computer program, such as a data extraction program, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device, for example a removable hard disk of the electronic device. The memory 11 may also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 may be used not only to store application software installed in the electronic device and various types of data, such as codes of a data extraction program, but also to temporarily store data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device by operating or executing programs or modules (e.g., data extraction programs, etc.) stored in the memory 11 and calling data stored in the memory 11.
The communication bus 12 may be a PerIPheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The communication bus 12 is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
Fig. 3 shows only an electronic device having components, and those skilled in the art will appreciate that the structure shown in fig. 3 does not constitute a limitation of the electronic device, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are realized through the power management device. The power source may also include any component of one or more dc or ac power sources, recharging devices, power failure classification circuits, power converters or inverters, power status indicators, and the like. The electronic device may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Optionally, the communication interface 13 may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.
Optionally, the communication interface 13 may further include a user interface, which may be a Display (Display), an input unit (such as a Keyboard (Keyboard)), and optionally, a standard wired interface, or a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The data extraction program stored in the memory 11 of the electronic device is a combination of a plurality of computer programs, and when running in the processor 10, can realize:
acquiring data interfaces of different data sources and corresponding data source priorities, and sequencing all the data interfaces according to the data source priorities to obtain interface sequences;
extracting data from a data source corresponding to a first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data;
judging whether a blank field exists in the first interface data;
when no blank field exists in the first interface data, the first interface data are sent to preset terminal equipment of a data extraction person;
when a blank field exists in the first interface data, updating the interface number of the interface sequence, and judging whether the interface number in the updated interface sequence is greater than zero or not;
when the number of the interfaces in the updated interface sequence is larger than zero, extracting data from a data source corresponding to a first data interface in the updated interface sequence, and mapping the data to a preset field to obtain second interface data;
carrying out data replacement of the same field on a blank field in the first interface data by using the second interface data to obtain updated first interface data, and returning to the step of judging whether the blank field exists in the first interface data;
and when the number of the interfaces in the updated interface sequence is not more than zero, sending the first interface data to the terminal equipment.
Specifically, the processor 10 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1 for a specific implementation method of the computer program, which is not described herein again.
Further, the electronic device integrated module/unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a computer readable storage medium. The computer readable medium may be non-volatile or volatile. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
Embodiments of the present invention may also provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program may implement:
acquiring data interfaces of different data sources and corresponding data source priorities, and sequencing all the data interfaces according to the data source priorities to obtain interface sequences;
extracting data from a data source corresponding to a first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data;
judging whether a blank field exists in the first interface data;
when no blank field exists in the first interface data, the first interface data are sent to preset terminal equipment of a data extraction person;
when a blank field exists in the first interface data, updating the interface number of the interface sequence, and judging whether the interface number in the updated interface sequence is greater than zero or not;
when the number of the interfaces in the updated interface sequence is larger than zero, extracting data from a data source corresponding to a first data interface in the updated interface sequence, and mapping the data to a preset field to obtain second interface data;
carrying out data replacement of the same field on a blank field in the first interface data by using the second interface data to obtain updated first interface data, and returning to the step of judging whether the blank field exists in the first interface data;
and when the number of the interfaces in the updated interface sequence is not more than zero, sending the first interface data to the terminal equipment.
Further, the computer usable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not to denote any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method of data extraction, the method comprising:
acquiring data interfaces of different data sources and corresponding data source priorities, and sequencing all the data interfaces according to the data source priorities to obtain interface sequences;
extracting data from a data source corresponding to a first data interface in the interface sequence and mapping the data to a preset field to obtain first interface data;
judging whether a blank field exists in the first interface data;
when no blank field exists in the first interface data, the first interface data are sent to preset terminal equipment of a data extraction person;
when a blank field exists in the first interface data, updating the number of the interfaces of the interface sequence, and judging whether the number of the interfaces in the updated interface sequence is larger than zero or not;
when the number of the interfaces in the updated interface sequence is larger than zero, extracting data from a data source corresponding to a first data interface in the updated interface sequence, and mapping the data to a preset field to obtain second interface data;
carrying out data replacement of the same field on a blank field in the first interface data by using the second interface data to obtain updated first interface data, and returning to the step of judging whether the blank field exists in the first interface data;
and when the number of the interfaces in the updated interface sequence is not more than zero, sending the first interface data to the terminal equipment.
2. The data extraction method according to claim 1, wherein the extracting data from the data source corresponding to the first data interface in the interface sequence, which is mapped to a preset field, to obtain the first interface data, includes:
acquiring configuration information and a field mapping rule of a first data interface in the interface sequence;
performing data extraction on the data source corresponding to the first data interface by using the configuration information to obtain initial extraction data;
and mapping the initial extraction data to a preset field to obtain the first interface data.
3. The data extraction method of claim 1, wherein the determining whether a blank field exists in the first interface data comprises:
inquiring whether a value corresponding to each field in the first interface data is a null value;
when the value corresponding to the field in the first interface data is null, determining the field as a blank field.
4. The data extraction method according to claim 1, wherein the performing, by using the second interface data, data replacement of the same field for a blank field in the first interface data to obtain updated first interface data includes:
determining the blank field as a field to be replaced;
extracting a value corresponding to the field to be replaced in the second interface data to obtain a numerical value to be replaced;
and replacing the value of the same field in the first interface data with the value to be replaced to obtain updated first interface data.
5. The data extraction method of claim 1, wherein the sending the first interface data to the terminal device comprises:
packaging the first interface data into a file with a preset file format to obtain a target file;
and sending the target file to the terminal equipment.
6. The data extraction method according to any one of claims 1 to 5, wherein the updating the interface number of the interface sequence comprises:
and deleting the first data interface in the interface sequence to obtain an updated interface sequence.
7. The data extraction method according to claim 2, wherein the extracting data from the data source to which the first data interface belongs using the configuration information to obtain initial extraction data comprises:
analyzing the input/output parameters of the first data interface from the configuration information;
determining a data source corresponding to the configuration information as a target data source;
converting the input/output parameters into a data source executable language of the target data source;
and sending the data source executable language to the target data source, and receiving data returned by the target data source according to the data source executable language to obtain the initial extracted data.
8. A data extraction apparatus, comprising:
the data extraction module is used for acquiring data interfaces of different data sources and corresponding data source priorities, and sequencing all the data interfaces according to the data source priorities to obtain an interface sequence; extracting data from a data source corresponding to a first data interface in the interface sequence, and mapping the data to a preset field to obtain first interface data;
a blank field judging module, configured to judge whether a blank field exists in the first interface data; when no blank field exists in the first interface data, the first interface data are sent to preset terminal equipment of a data extraction person; when a blank field exists in the first interface data, updating the interface number of the interface sequence, and judging whether the interface number in the updated interface sequence is greater than zero or not;
the interface number judging module is used for extracting data from a data source corresponding to a first data interface in the updated interface sequence and mapping the data to a preset field to obtain second interface data when the number of interfaces in the updated interface sequence is greater than zero; carrying out data replacement of the same field on a blank field in the first interface data by using the second interface data to obtain updated first interface data, and returning to the step of judging whether the blank field exists in the first interface data; and when the number of the interfaces in the updated interface sequence is not more than zero, sending the first interface data to the terminal equipment.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor;
wherein the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data extraction method of any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements a data extraction method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210161359.1A CN114547011A (en) | 2022-02-22 | 2022-02-22 | Data extraction method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210161359.1A CN114547011A (en) | 2022-02-22 | 2022-02-22 | Data extraction method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114547011A true CN114547011A (en) | 2022-05-27 |
Family
ID=81677602
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210161359.1A Pending CN114547011A (en) | 2022-02-22 | 2022-02-22 | Data extraction method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114547011A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116630034A (en) * | 2023-07-21 | 2023-08-22 | 杭银消费金融股份有限公司 | Wind control data processing system and method |
-
2022
- 2022-02-22 CN CN202210161359.1A patent/CN114547011A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116630034A (en) * | 2023-07-21 | 2023-08-22 | 杭银消费金融股份有限公司 | Wind control data processing system and method |
| CN116630034B (en) * | 2023-07-21 | 2023-11-07 | 杭银消费金融股份有限公司 | Wind control data processing system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114979120B (en) | Data uploading method, device, equipment and storage medium | |
| CN112115145A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN112418798A (en) | Information auditing method and device, electronic equipment and storage medium | |
| CN113806434B (en) | Big data processing method, device, equipment and medium | |
| CN113890712A (en) | Data transmission method and device, electronic equipment and readable storage medium | |
| CN114398194A (en) | Data collection method and device, electronic equipment and readable storage medium | |
| CN114386509A (en) | Data fusion method and device, electronic equipment and storage medium | |
| CN112949278A (en) | Data checking method and device, electronic equipment and readable storage medium | |
| CN113868528A (en) | Information recommendation method and device, electronic equipment and readable storage medium | |
| CN114844844A (en) | Delay message processing method, device, equipment and storage medium | |
| CN114491047A (en) | Multi-label text classification method and device, electronic equipment and storage medium | |
| CN114547011A (en) | Data extraction method and device, electronic equipment and storage medium | |
| CN113468175A (en) | Data compression method and device, electronic equipment and storage medium | |
| CN112580079A (en) | Authority configuration method and device, electronic equipment and readable storage medium | |
| CN114840388A (en) | Data monitoring method and device, electronic equipment and storage medium | |
| CN113342867A (en) | Data distribution and management method and device, electronic equipment and readable storage medium | |
| CN113254446A (en) | Data fusion method and device, electronic equipment and medium | |
| CN113421009B (en) | Accessory approval method and device, electronic equipment and readable storage medium | |
| CN115174698B (en) | Market data decoding method, device, equipment and medium based on table entry index | |
| CN115002100B (en) | File transmission method and device, electronic equipment and storage medium | |
| CN113704616B (en) | Information pushing method and device, electronic equipment and readable storage medium | |
| CN113672565B (en) | File marking method and device, electronic equipment and storage medium | |
| CN115033605A (en) | Data query method and device, electronic equipment and storage medium | |
| CN115225458A (en) | Alarm notification method, alarm notification device, electronic equipment and storage medium | |
| CN115174555A (en) | File transmission method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |