CN111949850B - Multi-source data acquisition method, device, equipment and storage medium - Google Patents
Multi-source data acquisition method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111949850B CN111949850B CN202010819608.2A CN202010819608A CN111949850B CN 111949850 B CN111949850 B CN 111949850B CN 202010819608 A CN202010819608 A CN 202010819608A CN 111949850 B CN111949850 B CN 111949850B
- Authority
- CN
- China
- Prior art keywords
- data
- acquired
- access
- memory
- acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000002159 abnormal effect Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000013500 data storage Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000003287 optical effect Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Abstract
The embodiment of the invention discloses a method, a device, equipment and a storage medium for acquiring multi-source data. Comprising the following steps: determining an access strategy according to the data source type of the data to be acquired; converting the structured data in the data to be acquired into a set format according to the access strategy, and transmitting the data in the set format to a set acquisition queue; and uploading unstructured data in the data to be acquired to a setting memory according to the access strategy. According to the multi-source data acquisition method disclosed by the embodiment of the invention, corresponding access strategies are determined for different data sources, structured data in the data to be acquired are converted into a set format based on the determined access strategies and then are sent to the set acquisition queue, unstructured data in the data to be acquired are uploaded to the set memory, the acquisition of multi-source heterogeneous data is realized, and the data acquisition efficiency is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of data acquisition, in particular to a method, a device, equipment and a storage medium for acquiring multi-source data.
Background
Along with the rapid development of big data, the rapid development of the Internet of things and the Internet brings the information industry to a new climax, and data acquisition is used as a core technology of the information industry, and aiming at multi-source heterogeneous data acquisition, the data acquisition has become a core technical problem of the Internet of things and an Internet system.
The data acquisition system of the current market open source is: flume, datax, logstash, etc., which are not well supported for collecting some common structured data, but for some unstructured data as well as structured data in a special format. In the big data age, the timeliness of the information is limited, and the method is particularly important to collect multi-source heterogeneous data in a short time.
Disclosure of Invention
The embodiment of the invention provides a method, a device, equipment and a storage medium for acquiring multi-source data, which can realize the acquisition of multi-source heterogeneous data and improve the data acquisition efficiency.
In a first aspect, an embodiment of the present invention provides a method for collecting multi-source data, including:
determining an access strategy according to the data source type of the data to be acquired;
converting the structured data in the data to be acquired into a set format according to the access strategy, and transmitting the data in the set format to a set acquisition queue;
and uploading unstructured data in the data to be acquired to a setting memory according to the access strategy.
Further, before determining the access policy according to the data source type of the data to be collected, the method further comprises:
configuring task parameters according to each data source type, and determining field information to be acquired;
generating access strategies corresponding to the data sources respectively according to the task parameters and the field information;
and uploading the access strategy to an acquisition system.
Further, converting the structured data in the data to be collected into a set format according to the access policy, including:
acquiring keywords in the structured data and values corresponding to the keywords according to the field information;
and generating data in a set format according to the keywords and the numerical values.
Further, uploading unstructured data in the data to be acquired to a setting memory according to the access policy includes:
and uploading the unstructured data in the data to be acquired to a setting memory when the amount of the unstructured data reaches a first setting value and/or the acquired time length exceeds a second setting value.
Further, before converting the structured data in the data to be collected into a set format according to the access policy, the method further includes:
determining the type of the data to be acquired according to the access strategy; the types include structured data and unstructured data.
Further, the method further comprises the following steps:
counting the number of the collected structured data and unstructured data to obtain a collection amount;
counting abnormal data quantity;
counting data output quantity; the data output quantity comprises the data quantity sent to the set acquisition queue and the data quantity uploaded to the set memory;
and displaying the acquired quantity, the abnormal data quantity and the data output quantity in real time.
Further, the set collection queue is a Kafka queue; the setting memory is HDFS.
In a second aspect, an embodiment of the present invention further provides a device for collecting multi-source data, including:
the access strategy determining module is used for determining an access strategy according to the data source type of the data to be acquired;
the format conversion module is used for converting the structured data in the data to be acquired into a set format according to the access strategy and sending the data in the set format to a set acquisition queue;
and the data storage module is used for uploading unstructured data in the data to be acquired to a setting memory according to the access strategy.
In a third aspect, an embodiment of the present invention further provides a computer apparatus, the apparatus including: the multi-source data acquisition method comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the multi-source data acquisition method according to the embodiment of the invention when executing the program.
In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where a computer program is stored, where the program when executed by a processing device implements a method for collecting multi-source data according to an embodiment of the present invention.
The embodiment of the invention provides a method, a device, equipment and a storage medium for collecting multi-source data, which are characterized in that firstly, an access strategy is determined according to the data source type of the data to be collected, then structured data in the data to be collected are converted into a set format according to the access strategy, the data in the set format are sent to a set collection queue, and finally unstructured data in the data to be collected are uploaded to a set memory according to the access strategy. According to the multi-source data acquisition method disclosed by the embodiment of the invention, corresponding access strategies are determined for different data sources, structured data in the data to be acquired are converted into a set format based on the determined access strategies and then are sent to the set acquisition queue, unstructured data in the data to be acquired are uploaded to the set memory, the acquisition of multi-source heterogeneous data is realized, and the data acquisition efficiency is improved.
Drawings
FIG. 1 is a flow chart of a method for collecting multi-source data according to a first embodiment of the present invention;
fig. 2 is a schematic structural diagram of a multi-source data acquisition device according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a computer device in a third embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
Example 1
Fig. 1 is a flowchart of a method for collecting multi-source data according to a first embodiment of the present invention, where the method may be applied to a case of collecting data of different data sources, and the method may be performed by a multi-source data collecting device, where the device may be composed of hardware and/or software and may be generally integrated in a device having a multi-source data collecting function, where the device may be an electronic device such as a server or a server cluster. As shown in fig. 1, the method specifically includes the following steps:
step 110, determining an access strategy according to the data source type of the data to be collected.
The types of data sources may include data block types, FPT types, kafka types, and the like. The access policy may be determined according to the class of the data source, and includes task parameters to be configured, field information to be collected, and the like. For example: if the type of the data source is the database type, parameters to be configured include URL, user name, password, etc. In this embodiment, the data source types are in one-to-one correspondence with the access policies.
Optionally, before determining the access policy according to the data source type of the data to be collected, the method further includes the following steps: configuring task parameters according to each data source type, and determining field information to be acquired; generating access strategies corresponding to the data sources respectively according to the task parameters and the field information; and uploading the access strategy to the acquisition system.
The process of uploading the access policy into the acquisition system may be uploading the access policy through a web page of the acquisition system.
Specifically, when data is collected, the type of the data source is firstly judged, and then a corresponding access strategy is selected according to the type of the data source so as to collect the data according to the access strategy.
And 120, converting the structured data in the data to be acquired into a set format according to the access strategy, and sending the data in the set format to a set acquisition queue.
Wherein, the set acquisition queue may be a Kafka queue. The access policy includes a policy for judging the data type. In this embodiment, before the structured data in the data to be collected is converted into the set format according to the access policy, the method further includes a step of determining whether the data to be collected is structured data or unstructured data according to the access policy.
Specifically, the method for converting the structured data in the data to be collected into the set format according to the access policy may be: acquiring a keyword in the structured data and a numerical value corresponding to the keyword according to the field information; and generating data in a set format according to the keywords and the numerical values.
Wherein, the set format may be a k-value format. In this embodiment, the key is obtained by the access policy, and then the extracted data value is value, so as to form the k-v format data. And sending the structured data to a Kafka queue, so that subsequent processing and extraction analysis are facilitated.
And 130, uploading unstructured data in the data to be acquired to a setting memory according to an access strategy.
Wherein, the setting memory may be HDFS. Specifically, the method for uploading unstructured data in the data to be collected to the setting memory according to the access policy may be: and uploading the unstructured data in the data to be acquired to a setting memory when the amount of the unstructured data reaches a first setting value and/or the acquired time length exceeds a second setting value.
Specifically, the Mapfile of Hadoop is combined and then uploaded to the HDFS.
Optionally, the system also has a data access statistics function, a task state query function and the like. The method further comprises the steps of: counting the number of the collected structured data and unstructured data to obtain a collection amount; counting abnormal data quantity; counting data output quantity; the data output quantity comprises the data quantity sent to the set collection queue and the data quantity uploaded to the set memory; and displaying the acquired quantity, the abnormal data quantity and the data output quantity in real time.
According to the technical scheme, firstly, an access strategy is determined according to the data source type of data to be collected, then structured data in the data to be collected are converted into a set format according to the access strategy, the data in the set format are sent to a set collection queue, and finally unstructured data in the data to be collected are uploaded to a set memory according to the access strategy. According to the multi-source data acquisition method disclosed by the embodiment of the invention, corresponding access strategies are determined for different data sources, structured data in the data to be acquired are converted into a set format based on the determined access strategies and then are sent to the set acquisition queue, unstructured data in the data to be acquired are uploaded to the set memory, the acquisition of multi-source heterogeneous data is realized, and the data acquisition efficiency is improved.
Example two
Fig. 2 is a schematic structural diagram of a multi-source data acquisition device according to a second embodiment of the present invention. As shown in fig. 2, the apparatus includes: an access policy determination module 210, a format conversion module 220 and a data storage module 230.
An access policy determining module 210, configured to determine an access policy according to a data source type of data to be collected;
the format conversion module 220 is configured to convert the structured data in the data to be collected into a set format according to the access policy, and send the data in the set format to the set collection queue;
the data storage module 230 is configured to upload unstructured data in the data to be collected to the setting memory according to the access policy.
Optionally, the method further comprises: an access policy generation module, configured to:
configuring task parameters according to each data source type, and determining field information to be acquired;
generating access strategies corresponding to the data sources respectively according to the task parameters and the field information;
and uploading the access strategy to the acquisition system.
Optionally, the format conversion module 220 is further configured to:
acquiring a keyword in the structured data and a numerical value corresponding to the keyword according to the field information;
and generating data in a set format according to the keywords and the numerical values.
Optionally, the data storage module 230 is further configured to:
and uploading the unstructured data in the data to be acquired to a setting memory when the amount of the unstructured data reaches a first setting value and/or the acquired time length exceeds a second setting value.
Optionally, the method further comprises: a data type determining module for:
determining the type of data to be acquired according to an access strategy; types include structured data and unstructured data.
Optionally, the method further comprises: a statistics module for:
counting the number of the collected structured data and unstructured data to obtain a collection amount;
counting abnormal data quantity;
counting data output quantity; the data output quantity comprises the data quantity sent to the set collection queue and the data quantity uploaded to the set memory;
and displaying the acquired quantity, the abnormal data quantity and the data output quantity in real time.
Optionally, setting the acquisition queue as a Kafka queue; the memory is set to HDFS.
The device can execute the method provided by all the embodiments of the invention, and has the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in this embodiment can be found in the methods provided in all the foregoing embodiments of the invention.
Example III
Fig. 3 is a schematic structural diagram of a computer device according to a third embodiment of the present invention. FIG. 3 illustrates a block diagram of a computer device 312 suitable for use in implementing embodiments of the present invention. The computer device 312 shown in fig. 3 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention. Device 312 is a computing device that is typically the acquisition function of multi-source data.
As shown in FIG. 3, computer device 312 is in the form of a general purpose computing device. Components of computer device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 that connects the different system components (including the storage device 328 and the processor 316).
Bus 318 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include industry standard architecture (Industry Standard Architecture, ISA) bus, micro channel architecture (Micro Channel Architecture, MCA) bus, enhanced ISA bus, video electronics standards association (Video Electronics Standards Association, VESA) local bus, and peripheral component interconnect (Peripheral Component Interconnect, PCI) bus.
Computer device 312 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer device 312 and includes both volatile and nonvolatile media, removable and non-removable media.
The storage 328 may include computer system-readable media in the form of volatile memory, such as random access memory (Random Access Memory, RAM) 330 and/or cache memory 332. The computer device 312 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 334 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 3, commonly referred to as a "hard disk drive"). Although not shown in fig. 3, a disk drive for reading from and writing to a removable nonvolatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from and writing to a removable nonvolatile optical disk (e.g., a Compact Disc-Read Only Memory (CD-ROM), digital versatile Disc (Digital Video Disc-Read Only Memory, DVD-ROM), or other optical media), may be provided. In such cases, each drive may be coupled to bus 318 through one or more data medium interfaces. Storage 328 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of embodiments of the invention.
Programs 336 having a set (at least one) of program modules 326 may be stored, for example, in storage 328, such program modules 326 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 326 generally perform the functions and/or methods in the described embodiments of the invention.
The computer device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), one or more devices that enable a user to interact with the computer device 312, and/or any devices (e.g., network card, modem, etc.) that enable the computer device 312 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 322. Moreover, the computer device 312 may also communicate with one or more networks such as a local area network (Local Area Network, LAN), a wide area network Wide Area Network, a WAN) and/or a public network such as the internet via the network adapter 320. As shown, network adapter 320 communicates with other modules of computer device 312 via bus 318. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with computer device 312, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk array (Redundant Arrays of Independent Disks, RAID) systems, tape drives, data backup storage systems, and the like.
The processor 316 executes programs stored in the storage 328 to perform various functional applications and data processing, such as implementing the multi-source data collection method provided by the above-described embodiments of the present invention.
Example IV
The embodiment of the invention provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and when the program is executed by a processing device, the method for counting database data in the embodiment of the invention is realized. The computer readable medium of the present invention described above may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: determining an access strategy according to the data source type of the data to be acquired; converting the structured data in the data to be acquired into a set format according to the access strategy, and transmitting the data in the set format to a set acquisition queue; and uploading unstructured data in the data to be acquired to a setting memory according to the access strategy.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (9)
1. A method for collecting multi-source data, comprising:
determining an access strategy according to the data source type of the data to be acquired;
converting the structured data in the data to be acquired into a set format according to the access strategy, and transmitting the data in the set format to a set acquisition queue;
uploading unstructured data in the data to be acquired to a setting memory according to the access strategy;
counting the number of the collected structured data and unstructured data to obtain collection quantity;
counting abnormal data quantity;
counting data output quantity; the data output quantity comprises the data quantity sent to the set acquisition queue and the data quantity uploaded to the set memory;
and displaying the acquired quantity, the abnormal data quantity and the data output quantity in real time.
2. The method of claim 1, further comprising, prior to determining the access policy based on the data source type of the data to be collected:
configuring task parameters according to each data source type, and determining field information to be acquired;
generating access strategies corresponding to the data sources respectively according to the task parameters and the field information;
and uploading the access strategy to an acquisition system.
3. The method of claim 2, wherein converting structured data in the data to be collected into a set format according to the access policy comprises:
acquiring keywords in the structured data and values corresponding to the keywords according to the field information;
and generating data in a set format according to the keywords and the numerical values.
4. The method of claim 1, wherein uploading unstructured data in the data to be collected into a settings store according to the access policy comprises:
and uploading the unstructured data in the data to be acquired to a setting memory when the amount of the unstructured data reaches a first setting value and/or the acquired time length exceeds a second setting value.
5. The method of claim 1, further comprising, prior to converting structured data in the data to be collected into a set format according to the access policy:
determining the type of the data to be acquired according to the access strategy; the types include structured data and unstructured data.
6. The method of claim 1, wherein the set acquisition queue is a Kafka queue; the setting memory is HDFS.
7. A multi-source data acquisition device, comprising:
the access strategy determining module is used for determining an access strategy according to the data source type of the data to be acquired;
the format conversion module is used for converting the structured data in the data to be acquired into a set format according to the access strategy and sending the data in the set format to a set acquisition queue;
the data storage module is used for uploading unstructured data in the data to be acquired to a setting memory according to the access strategy;
the statistics module is used for counting the number of the collected structured data and unstructured data to obtain collection quantity;
counting abnormal data quantity;
counting data output quantity; the data output quantity comprises the data quantity sent to a set acquisition queue and the data quantity uploaded to the set memory;
and displaying the acquired quantity, the abnormal data quantity and the data output quantity in real time.
8. A computer device, the device comprising: comprising a memory, a processor and a computer program stored on the memory and executable on the processor, said processor implementing the method for collecting multi-source data according to any of claims 1-6 when said program is executed.
9. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processing device, implements a method of acquisition of multisource data according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010819608.2A CN111949850B (en) | 2020-08-14 | 2020-08-14 | Multi-source data acquisition method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010819608.2A CN111949850B (en) | 2020-08-14 | 2020-08-14 | Multi-source data acquisition method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111949850A CN111949850A (en) | 2020-11-17 |
| CN111949850B true CN111949850B (en) | 2024-03-22 |
Family
ID=73342345
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010819608.2A Active CN111949850B (en) | 2020-08-14 | 2020-08-14 | Multi-source data acquisition method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111949850B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112433998B (en) * | 2020-11-20 | 2022-01-21 | 广东电网有限责任公司佛山供电局 | Multisource heterogeneous data acquisition and convergence system and method based on power system |
| CN112650757A (en) * | 2020-12-30 | 2021-04-13 | 创业慧康科技股份有限公司 | Method, device, equipment and medium for storing structured and unstructured data |
| CN114791840B (en) * | 2021-01-26 | 2023-09-22 | 武汉斗鱼网络科技有限公司 | Data assembling method, device, electronic equipment and medium |
| CN115391413A (en) * | 2022-10-26 | 2022-11-25 | 中国电子科技集团公司第十五研究所 | Multi-source data access aggregation method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107590749A (en) * | 2017-09-07 | 2018-01-16 | 北京国电通网络技术有限公司 | A kind of processing method and system with electricity consumption data |
| CN108121508A (en) * | 2017-12-15 | 2018-06-05 | 华中师范大学 | Multi-source heterogeneous data collecting system and processing method based on education big data |
| CN108491499A (en) * | 2018-03-20 | 2018-09-04 | 苏州思必驰信息科技有限公司 | Collecting method, data acquisition platform, client and service server |
| CN111078765A (en) * | 2019-11-13 | 2020-04-28 | 北京中盾安全技术开发公司 | View base system based on Hadoop system architecture and construction method thereof |
-
2020
- 2020-08-14 CN CN202010819608.2A patent/CN111949850B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107590749A (en) * | 2017-09-07 | 2018-01-16 | 北京国电通网络技术有限公司 | A kind of processing method and system with electricity consumption data |
| CN108121508A (en) * | 2017-12-15 | 2018-06-05 | 华中师范大学 | Multi-source heterogeneous data collecting system and processing method based on education big data |
| CN108491499A (en) * | 2018-03-20 | 2018-09-04 | 苏州思必驰信息科技有限公司 | Collecting method, data acquisition platform, client and service server |
| CN111078765A (en) * | 2019-11-13 | 2020-04-28 | 北京中盾安全技术开发公司 | View base system based on Hadoop system architecture and construction method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111949850A (en) | 2020-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111949850B (en) | Multi-source data acquisition method, device, equipment and storage medium | |
| CN110147398B (en) | Data processing method, device, medium and electronic equipment | |
| CN110362544B (en) | Log processing system, log processing method, terminal and storage medium | |
| CN110008045B (en) | Method, device and equipment for aggregating microservices and storage medium | |
| CN107370806B (en) | HTTP status code monitoring method, device, storage medium and electronic equipment | |
| CN110457256A (en) | Date storage method, device, computer equipment and storage medium | |
| US11188443B2 (en) | Method, apparatus and system for processing log data | |
| WO2014173151A1 (en) | Method, device and terminal for data processing | |
| CN113485962B (en) | Log file storage method, device, equipment and storage medium | |
| CN111258978B (en) | Data storage method | |
| CN111985831A (en) | Scheduling method and device of cloud computing resources, computer equipment and storage medium | |
| CN115297183B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112907942A (en) | Vehicle scheduling method, device, equipment and medium based on edge calculation | |
| CN117131281A (en) | Public opinion event processing method, apparatus, electronic device and computer readable medium | |
| CN111611479B (en) | Data processing method and related device for network resource recommendation | |
| CN111930385A (en) | Data acquisition method, device, equipment and storage medium | |
| CN111666449B (en) | Video retrieval method, apparatus, electronic device, and computer-readable medium | |
| CN115756448A (en) | Method, device, equipment and medium for acquiring vehicle cloud data acquisition system architecture | |
| CN115309802A (en) | User distribution thermodynamic diagram acquisition method and device, electronic equipment and storage medium | |
| CN110955709B (en) | Data processing method and device and electronic equipment | |
| CN114756301A (en) | Log processing method, device and system | |
| CN113919446A (en) | Method and device for model training and similarity determination of multimedia resources | |
| KR101878291B1 (en) | Big data management system and management method thereof | |
| CN112035256A (en) | Resource allocation method, device, electronic equipment and medium | |
| CN114201493B (en) | Data access method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |