CN110750569A

CN110750569A - Data extraction method, device, equipment and storage medium

Info

Publication number: CN110750569A
Application number: CN201910989413.XA
Authority: CN
Inventors: 高堆峰; 刘世平; 高琪; 王涛
Original assignee: Beijing Ruian Technology Co Ltd
Current assignee: Beijing Ruian Technology Co Ltd
Priority date: 2019-10-17
Filing date: 2019-10-17
Publication date: 2020-02-04

Abstract

The embodiment of the invention discloses a data extraction method, a data extraction device, data extraction equipment and a storage medium. The method comprises the following steps: determining target attribute information from at least two pieces of existing attribute information in response to an operation request of a user; extracting first target data from an original database according to the target attribute information; extracting common attribute information of the first target data; wherein the public attribute information is different from the existing attribute information; and extracting second target data from the original database according to the public attribute information, and determining an operation result according to the first target data and the second target data. So as to enable the user to quickly obtain the required data information.

Description

Data extraction method, device, equipment and storage medium

Technical Field

Embodiments of the present invention relate to data processing technologies, and in particular, to a data extraction method, apparatus, device, and storage medium.

Background

With the rapid development of computer technology, a large amount of data is received by users, however, the users receive much useless information.

The current searching mode can only carry out data screening according to the searching of the user, and the data screened by the mode has a large amount of useless information. Therefore, a method is needed to enable a user to quickly obtain the required data information.

Disclosure of Invention

The invention provides a data extraction method, a data extraction device, data extraction equipment and a storage medium, which are used for enabling a user to quickly obtain required data information.

In a first aspect, an embodiment of the present invention provides a data extraction method, including:

determining target attribute information from at least two pieces of existing attribute information in response to an operation request of a user;

extracting first target data from an original database according to the target attribute information;

extracting common attribute information of the first target data; wherein the public attribute information is different from the existing attribute information;

and extracting second target data from the original database according to the public attribute information, and determining an operation result according to the first target data and the second target data.

In a second aspect, an embodiment of the present invention further provides a data extraction apparatus, including:

the target attribute information determining module is used for responding to an operation request of a user and determining target attribute information from at least two pieces of existing attribute information;

the first target data extraction module is used for extracting first target data from an original database according to the target attribute information;

the public attribute information extraction module is used for extracting public attribute information of the first target data; wherein the public attribute information is different from the existing attribute information;

and the operation result determining module is used for extracting second target data from the original database according to the public attribute information and determining an operation result according to the first target data and the second target data.

In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the data extraction method according to any one of the embodiments when executing the program.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data extraction method according to any one of the embodiments.

The method comprises the steps of determining target attribute information from at least two existing attribute information by responding to an operation request of a user; extracting first target data from an original database according to the target attribute information; extracting common attribute information of the first target data; wherein the public attribute information is different from the existing attribute information; and extracting second target data from the original database according to the public attribute information, and determining an operation result according to the first target data and the second target data so as to enable a user to quickly obtain required data information.

Drawings

Fig. 1 is a schematic flow chart of a data extraction method provided in a first embodiment of the present invention;

fig. 2 is a schematic flow chart of a data extraction method provided in the second embodiment of the present invention;

FIG. 3 is a diagram of a data extraction model platform according to a second embodiment of the present invention;

fig. 4 is a schematic structural diagram of a data extraction apparatus provided in a third embodiment of the present invention;

fig. 5 is a schematic structural diagram of an apparatus provided in the fourth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

Example one

Fig. 1 is a schematic flow chart of a data extraction method according to an embodiment of the present invention, where the present embodiment is applicable to a case where required data is extracted from multiple data, and the method may be executed by a data extraction device, where the data extraction device may be implemented in a software and/or hardware manner, and may be integrated in an electronic device, and specifically includes the following steps:

step 110, in response to the operation request of the user, determining target attribute information from at least two existing attribute information.

In this embodiment, the operation request refers to a data request message sent by a user, and is used for screening data. The existing attribute information is a condition for data screening and is configured in the original database. The screening conditions can be displayed through a visual interface, and a user can select the existing attribute information through the visual interface.

The existing attribute information includes, but is not limited to, the APP category, the user usage record, or the search keyword of the user in the page. Illustratively, the existing attribute information may be a shopping record of a shopping APP or an access location of a map APP.

Further, the target attribute information is attribute information selected by the user from the existing attribute information. Illustratively, the target attribute information that the user can select from the visualized interface is a shopping record on the web.

And 120, extracting first target data from an original database according to the target attribute information.

In this embodiment, the original database has a plurality of data information, wherein the original database is a distributed database and is used for concurrently processing the operation request. When the first target data is extracted from the original database, data extraction can be performed by nodes distributed in the original database.

Further, the original database may be a Hadoop (distributed File System) in a distributed database. Hadoop is a distributed data and computation framework, a large number of semi-structured data sets can be stored, and data can be randomly stored, so that data loss cannot be caused due to failure of one disk. Therefore, a large amount of data is stored in Hadoop, and the integrity of the data can be guaranteed. Meanwhile, Hadoop can perform distributed calculation, namely data can be processed quickly across multiple machines, and therefore the data extraction speed can be improved. Hadoop has high fault tolerance, can provide high throughput to access data of an application program, and is suitable for the application program with a super large data set. The most core design of the Hadoop framework is that Hadoop provides storage for a large amount of data.

For example, if the target attribute information is a shopping record on the internet, data search is performed in the original database according to the shopping record on the internet, and data search is performed according to a node in the original database.

Step 130, extracting public attribute information of the first target data; wherein the common attribute information is different from the existing attribute information.

In this embodiment, the common attribute information refers to a common feature in the first target data, and exemplarily, the first target data is screened out by using the type of the APP by the user, where the type of the APP is the existing attribute information, and a large amount of data of the exercise device exists in the screened out first target data, so that the user can know that the exercise attribute is the common attribute information of the first target data. Wherein the motion attribute information is different from attribute information of a type using the APP. Therefore, the hidden data characteristics in the first target data can be obtained through data analysis, and the dimensionality of the existing attribute information in the original database can be further improved.

Step 140, extracting second target data from the original database according to the public attribute information, and determining an operation result according to the first target data and the second target data.

In this embodiment, the second target data is data that is screened by the user through the common attribute information, and for example, data that is screened by the user through the exercise device about exercise is the second target data. Further, the first target data and the second target data are both sent to the user as operation results. Wherein the second target data is capable of being used to analyze the user representation.

The embodiment of the invention determines the target attribute information from at least two existing attribute information by responding to the operation request of a user; extracting first target data from an original database according to the target attribute information; extracting common attribute information of the first target data; wherein the public attribute information is different from the existing attribute information; and extracting second target data from the original database according to the public attribute information, and determining an operation result according to the first target data and the second target data so as to enable a user to quickly obtain required data information.

Example two

Fig. 2 is a schematic flowchart of a data extraction method according to a second embodiment of the present invention, where this embodiment is applicable to a case where required data is extracted from many data, and the method may be executed by a data extraction apparatus, where the apparatus may be implemented in a software and/or hardware manner, and may be integrated in an electronic device, and an architecture diagram of a specific implementation process may be combined with the schematic diagram of a data extraction model platform shown in fig. 3. Further, a data extraction method specifically includes the following steps:

step 210, in response to an operation request of a user, determining target attribute information from at least two existing attribute information.

Step 220, extracting first target data from the original database according to the target attribute information.

Specifically, the operation request of the user is issued to the original database through the Web, and the first target data is extracted by the distributed task extraction program in the original database.

Step 230, extracting the public attribute information of the first target data; wherein the common attribute information is different from the existing attribute information.

Step 240, adding the public attribute information to an original database to update the existing attribute information.

In this embodiment, the public attribute information is added to the original database, so that the dimension of the existing attribute information in the original database is increased, and when the user performs data extraction next time, the dimension of the existing attribute information presented in the visual interface is increased by the public attribute information added last time. For example, there is no motion attribute information in the existing attribute information, and when the data is searched for the first time, the common attribute information is motion attribute information, and the user will include the motion attribute information in the existing attribute information at the next data extraction time. In particular, reference may be made to a schematic diagram of a data extraction model platform shown in fig. 3.

And step 250, extracting second target data from the original database according to the public attribute information, and determining an operation result according to the first target data and the second target data.

Step 260, storing the second target data in a target database for responding to a new operation request.

In this embodiment, the target database is used to store the second target data extracted in the past, and when the user sends a request for querying the user portrait, the target database is directly accessed. And obtaining a data image of the user through the second target data extracted from the target database so as to be convenient for pushing the relevant data information of the user in the following. After the second target data is stored in the target database, when the user sends a query request for the second target data, the user can directly access the target database without accessing the data from the original database again, and the required data information can be accessed quickly and accurately.

The embodiment of the invention determines the target attribute information from at least two existing attribute information by responding to the operation request of a user; extracting first target data from an original database according to the target attribute information; extracting common attribute information of the first target data; adding the public attribute information to an original database to update the existing attribute information; and extracting second target data from the original database according to the public attribute information, determining an operation result according to the first target data and the second target data, and storing the second target data into a target database for responding to a new operation request. The method can update the existing attribute information, so that the subsequent data screening can be more accurate.

EXAMPLE III

Fig. 4 is a schematic structural diagram of a data extraction apparatus according to a third embodiment of the present invention. The data extraction device provided by the embodiment of the invention can execute the data extraction method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. As shown in fig. 4, the apparatus includes:

a target attribute information determination module 401, configured to determine target attribute information from at least two existing attribute information in response to an operation request of a user;

a first target data extraction module 402, configured to extract first target data from an original database according to the target attribute information;

a common attribute information extraction module 403, configured to extract common attribute information of the first target data; wherein the public attribute information is different from the existing attribute information;

an operation result determining module 404, configured to extract second target data from the original database according to the common attribute information, and determine an operation result according to the first target data and the second target data.

The device further comprises:

and the public attribute information adding module is used for adding the public attribute information into an original database so as to update the existing attribute information.

The device further comprises:

and the second target data storage module is used for storing the second target data into a target database and responding to a new operation request.

The original database is a distributed database for concurrently processing the operation request.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the above-described apparatus may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

Example four

Fig. 5 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention, and fig. 5 is a schematic structural diagram of an exemplary apparatus suitable for implementing the embodiment of the present invention. The device 12 shown in fig. 5 is only an example and should not bring any limitations to the functionality and scope of use of the embodiments of the present invention.

As shown in FIG. 5, device 12 is in the form of a general purpose computing device. The components of device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by device 12 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of embodiments described herein.

Device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with device 12, and/or with any devices (e.g., network card, modem, etc.) that enable device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, the device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via the network adapter 20. As shown in FIG. 5, the network adapter 20 communicates with the other modules of the device 12 via the bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, to implement a data extraction method provided by an embodiment of the present invention, including:

EXAMPLE five

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program (or referred to as a computer-executable instruction) is stored, where the computer program, when executed by a processor, can implement a data extraction method according to any of the embodiments described above, where the computer program includes:

Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method of data extraction, comprising:

2. The method according to claim 1, wherein after extracting the common attribute information of the first target data, further comprising:

and adding the public attribute information into an original database to update the existing attribute information.

3. The method of claim 1, after retrieving second target data from the primary database, further comprising:

and storing the second target data into a target database for responding to a new operation request.

4. The method of claim 1, wherein the primary database is a distributed database for concurrent processing of the operation requests.

5. A data extraction apparatus, comprising:

6. The apparatus of claim 5, further comprising:

7. The apparatus of claim 5, further comprising:

8. The apparatus of claim 5, wherein the primary database is a distributed database for concurrent processing of the operation requests.

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data extraction method according to any one of claims 1-4 when executing the program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data extraction method according to any one of claims 1 to 4.