CN111104233B - Data acquisition method, device, computer equipment and computer readable storage medium - Google Patents
Data acquisition method, device, computer equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111104233B CN111104233B CN201911146802.2A CN201911146802A CN111104233B CN 111104233 B CN111104233 B CN 111104233B CN 201911146802 A CN201911146802 A CN 201911146802A CN 111104233 B CN111104233 B CN 111104233B
- Authority
- CN
- China
- Prior art keywords
- data
- target system
- memory
- preset
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/545—Interprogram communication where tasks reside in different layers, e.g. user- and kernel-space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1847—File system types specifically adapted to static storage, e.g. adapted to flash memory or SSD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
- Hardware Redundancy (AREA)
Abstract
The embodiment of the invention provides a data acquisition method, a data acquisition device, computer equipment and a computer readable storage medium. The method comprises the steps of storing data in a memory by acquiring the data of an external data source, judging whether a target system corresponding to a data receiver meets a preset survival condition or not, if so, forwarding the data to the target system, and if not, forwarding the data to a preset memory for temporary storage, and when a subsequent target system is available, forwarding the data to realize data acquisition. By temporarily storing the data transferred to the preset memory when the target system is judged to be unavailable and transferring the data to the target system when the subsequent target system is available, the high availability can be realized under the condition of ensuring high-performance data acquisition, and particularly, the high performance, the high availability and the stability of the whole data acquisition link are improved for the acquisition of mass data such as big data.
Description
Technical Field
The present invention relates to the field of data acquisition technologies, and in particular, to a data acquisition method, apparatus, computer device, and computer readable storage medium.
Background
In a traditional data acquisition scheme, for example, in particular to a mass data acquisition scheme such as big data, a core component comprises a real-time acquisition device, the real-time acquisition device comprises a data reading component and a data sending component, and the data sending component is responsible for sending data to a target system, for example, a file system, a NoSql database, a message queue and other target systems corresponding to data receivers. Once the target system is unavailable, the data transmission component may not find the transmission target, resulting in the real-time collector being unavailable, which may result in data extrusion, loss, and unavailability of the entire collection link. Thus, in the conventional art, a real-time collector of a data collection link cannot achieve a truly high availability.
Disclosure of Invention
The embodiment of the invention provides a data acquisition method, a data acquisition device, computer equipment and a computer readable storage medium, which can solve the problem of low high availability of a real-time acquisition device in the traditional technology.
In a first aspect, an embodiment of the present invention provides a data acquisition method, including: acquiring data of an external data source and storing the data into a memory; judging whether a target system corresponding to a data receiver meets a preset survival condition or not, wherein the target system is used for receiving preset components of the acquired data contained in a data acquisition system, and the target system survival means that the target system is in a state of normally receiving the data; if the target system meets the preset survival condition, reading the data from the memory, and forwarding the data to the target system to realize the data acquisition; and if the target system does not meet the preset survival condition, reading the data from the memory, forwarding the data to a preset memory for temporary storage, and forwarding the data to the target system when the target system is available subsequently so as to realize data acquisition.
In a second aspect, an embodiment of the present invention further provides a data acquisition device, including: the acquisition unit is used for acquiring data of an external data source and storing the data into the memory; the first judging unit is used for judging whether a target system corresponding to the data receiving party meets a preset survival condition or not, wherein the target system is used for receiving preset components of the acquired data contained in the data acquisition system, and the target system survival refers to the state that the target system is in a normal data receiving state; the first forwarding unit is used for reading the data from the memory and forwarding the data to the target system if the target system meets the preset survival condition so as to realize the data acquisition; and the temporary storage unit is used for reading the data from the memory and forwarding the data to a preset storage for temporary storage if the target system does not meet the preset survival condition, and forwarding the data to the target system when the target system is available subsequently so as to realize the data acquisition.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the data collection method when executing the computer program.
In a fourth aspect, embodiments of the present invention further provide a computer readable storage medium storing a computer program, which when executed by a processor causes the processor to perform the data acquisition method.
The embodiment of the invention provides a data acquisition method, a data acquisition device, computer equipment and a computer readable storage medium. When the embodiment of the invention realizes data acquisition, the data of an external data source are acquired and stored in a memory, whether a target system corresponding to a data receiver meets a preset survival condition is judged, if the target system meets the preset survival condition, the data is read from the memory and forwarded to the target system, so that the data acquisition is realized, if the target system does not meet the preset survival condition, the data is read from the memory and forwarded to a preset memory for temporary storage, and when the target system is available, the data is forwarded to the target system, so that the data acquisition is realized. When the target system is judged to be unavailable, the data real-time collector forwards the data to the preset memory for temporary storage, and when the subsequent target system is available, the data is forwarded to the target system, so that the problem that the real-time collector cannot find a sending target to cause data extrusion, loss and unavailability of the whole collection link due to the unavailability of the target system is avoided.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an application environment architecture of a data acquisition method according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 4 is a schematic block diagram of a data acquisition device according to an embodiment of the present invention; and
fig. 5 is a schematic block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be understood that the terms "comprises" and "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic flowchart of a data acquisition method according to an embodiment of the present invention, and fig. 2 is a schematic view of an application environment architecture of the data acquisition method according to the embodiment of the present invention. As shown in fig. 1, the method includes the following steps S101-S104:
s101, acquiring data of an external data source and storing the data into a memory.
In particular, in data acquisition, there are two general ways to acquire data of an external data source, one way is to actively go to the external data source to read, and the other way is to receive the data sent by the external data source, i.e. to read the data of the external data source or receive the data sent by the external data source. In the data collection, the core component includes a data real-time collector, as shown in fig. 2, taking data of an external data source as an example, in the embodiment of the present invention, the data real-time collector includes a data reading component (i.e. Reader component), an intermediate Storage component (i.e. Storage component) and a sending component (i.e. Sender component), besides the data real-time collector, the data collection system further includes a preset memory (e.g. disk file), where the Reader component is responsible for reading data, and a manner of reading data may be reading data in a file, reading data in a message queue, etc., and writing the data into the Storage component, and the Storage adopts a memory queue to store the data. The data real-time collector reads data from an external data source according to a certain period, and the data is stored in the memory queue after being read, namely the data of the external data source is read and stored in the memory.
S102, judging whether a target system corresponding to a data receiver meets a preset survival condition, wherein the target system is used for receiving preset components of the acquired data in the data acquisition system, and the target system survival means that the target system is in a state of normally receiving the data.
Specifically, the data real-time collector confirms a target system corresponding to the data receiving party, for example, a downstream system in the data collecting system, through the sending component (i.e., the Sender component) so as to judge whether the target system meets a preset survival condition, thereby judging whether the target system survives, namely, judging whether the target system is in a normal connection state so as to be capable of normally receiving data. The target system survival means that the target system is in a state of normally receiving data, wherein the target system is in a state of receiving preset components of the collected data in the data collection system, for example, the target system comprises a file system, a NoSql database, a message queue and other data receivers.
And S103, if the target system meets the preset survival condition, reading the data from the memory, and forwarding the data to the target system to realize the data acquisition.
Specifically, if the data real-time collector judges that the target system meets the preset survival condition, namely that the target system can normally receive data, the data is read from the memory, and the data is forwarded to the target system, so that the data collection is realized. With continued reference to fig. 2, as shown in fig. 2, external data read by the data real-time collector is forwarded to the target system through a path 1 correspondingly described by a black line, so as to realize data collection, especially for collection of massive data such as big data, and the like, because the data is stored in a memory, high performance of the data real-time collector during data collection processing can be realized.
And S104, if the target system does not meet the preset survival condition, reading the data from the memory, forwarding the data to a preset storage for temporary storage, and forwarding the data to the target system when the target system is available subsequently so as to realize data acquisition.
Specifically, in order to ensure high performance of the real-time data collector, the Storage component stores data in a memory queue, if the Sender component does not timely send the data, the memory can be exploded due to excessive data, so that the memory overflows, once the real-time data collector finds that a target system is unavailable, the Sender component writes the data acquired from the memory into a preset memory, such as a disk file, and switches the current state of the Sender component in the real-time data collector to the condition that the target system is unavailable, namely if the real-time data collector judges that the target system does not meet the preset survival condition, the target system cannot normally receive the data, the data is read from the memory, and is forwarded to the preset memory for temporary Storage, and when a subsequent target system is available, the data is forwarded to the target system to realize data collection. With continued reference to fig. 2, as shown in fig. 2, the data real-time collector determines that the target system does not meet the preset survival condition, and forwards the read external data to the data sending component through a path 1 described by a black line, and then forwards the read external data to a preset memory (for example, a disk file) for temporary storage through a path 2 correspondingly described by a dotted line, and when the subsequent target system is available, forwards the data to the target system from the preset memory through a path 3 (1) correspondingly described by a dotted line, so as to realize data collection, thereby realizing high availability of the data real-time collector.
In a general case, in a real-time large data acquisition scheme, a real-time acquisition device cannot ensure zero loss of data on the premise of ensuring high performance, acquired data can be selected to be cached in a memory or to fall to a disk, if the data is cached in the memory, high performance of data acquisition and transmission can be ensured, but if the situation that a target system is unavailable is encountered, the acquisition device is wrong, so that the data is lost, and if the data is selected to fall to the local disk, the data is not lost when a program is wrong or a server is down, but if the data is landed to the disk, the performance is reduced. In the embodiment of the invention, the data is cached in the memory and combined with the preset memory corresponding to the magnetic disk, so that the high performance of data acquisition and transmission is ensured by caching the data in the memory when the target system can normally receive the data, and the high availability of data acquisition and transmission can be ensured by caching the data in the preset memory for temporary storage when the target system cannot normally receive the data, and the loss of the data is avoided, thereby realizing the high performance and high availability of the data acquisition and transmission.
When the embodiment of the invention realizes data acquisition, the data of an external data source are acquired and stored in a memory, whether a target system corresponding to a data receiver meets a preset survival condition is judged, if the target system meets the preset survival condition, the data is read from the memory and forwarded to the target system, so that the data acquisition is realized, if the target system does not meet the preset survival condition, the data is read from the memory and forwarded to a preset memory for temporary storage, and when the target system is available, the data is forwarded to the target system, so that the data acquisition is realized. When the target system is judged to be unavailable, the data real-time collector forwards the data to the preset memory for temporary storage, and when the subsequent target system is available, the data is forwarded to the target system, so that the problem that the real-time collector cannot find a sending target to cause data extrusion, loss and unavailability of the whole collection link due to the unavailability of the target system is avoided.
In one embodiment, the step of determining whether the target system corresponding to the data receiver meets the preset survival condition includes:
according to a preset period, confirming a target system corresponding to a data receiver in a heartbeat mode to judge whether the target system survives;
if the connection with the target system is judged to be abnormal according to the heartbeat feedback, judging that the target system does not survive;
and if the target system is judged to be not survived and the number of times of confirmation with the target system exceeds the preset number of times, judging that the target system does not meet the preset survival condition.
The heartbeat mechanism is a mechanism for sending a self-defined structure body (heartbeat packet) at fixed time to enable the opposite party to know that the opposite party is still alive so as to ensure the validity of the connection, for example, the heartbeat mechanism in the TCP protocol can send an empty data packet (keep alive probe) with an ACK mark to the opposite party, and after receiving the ACK packet, if the connection is all normal, the opposite party shall reply an ACK; if the connection is incorrect (e.g., the partner restarted, the connection state is lost), a RST should be returned; if the other party does not reply, the server resends the ACK at intervals, and if a plurality of continuous packets are ignored, the connection is disconnected.
Specifically, the data sending component in the data real-time collector confirms with a target system corresponding to a data receiver in a heartbeat manner according to a preset period to judge whether the target system survives, if the target system is judged to be connected with the target system normally according to heartbeat feedback, the target system is judged to survive, the data can be continuously sent to the target system so that the target system normally receives the data, if the target system is judged to be not survived according to heartbeat feedback, the target system can not normally receive the data, and further, if the target system is judged to be not survived and the number of times of confirmation with the target system exceeds a preset number of times, the target system is judged to not meet the preset survival condition, and the connection is disconnected, so that the data can not be sent to the target system. With continued reference to fig. 2, the sender component periodically performs heartbeat confirmation with the target system to determine whether the target system is alive, retries after failing to send data, and fails to send the data to the target system after n retries, and determines that the target system is unavailable. And through repeated confirmation, the state of the target system is determined, and the accuracy of judging the state of the target system is improved.
Referring to fig. 3, fig. 3 is another flow chart of a data acquisition method according to an embodiment of the invention. Step S301 to step S304 in fig. 3 correspond to step S101 to step S104 in fig. 1, respectively, and are not described herein. As shown in fig. 3, in this embodiment, after the step of reading the data from the memory and forwarding the data to a preset memory for temporary storage, the method further includes:
s305, continuously judging whether the target system meets the preset survival condition according to the preset period;
s306, if the target system meets the preset survival condition, reading the data from the preset memory, sending the data to the target system, and starting to continue to append the data contained in the memory to the preset memory.
Specifically, the data sending component in the data real-time collector continues to determine whether the target system meets the preset survival condition according to the preset period, if the target system meets the preset survival condition, reads the data from the preset memory (such as a magnetic disk) and sends the data to the target system, starts to continue to add the data contained in the memory to the preset memory, and if the target system does not meet the preset survival condition, returns to execute the step of continuing to determine whether the target system meets the preset survival condition according to the preset period, so as to monitor the state of the target system continuously. With continued reference to fig. 2, the Sender component will continue to periodically send a heartbeat to the target system, attempt to connect, and if the target system is found to be available, switch the current state of the Sender component to the target system being available, the Sender component will switch from reading data from the memory to reading data in the disk file from the disk via path 2 to storing the data in the preset memory, and send the data in the disk file to the target system via path 3 (1). At this time, the Sender component reads data from the disk file and sends the read data to the target system, but does not read the data from the memory (i.e. Storage component), and meanwhile, the target system is still unavailable, and the step of continuously judging whether the target system meets the preset survival condition according to the preset period is performed to continuously monitor the state of the target system until the target system meets the preset survival condition or the data acquisition system finishes serving, in order to avoid that the memory can be exploded due to excessive data, and the memory overflows, an additional component (i.e. replier) is started to read the data in the memory queue from the memory and is added to the disk through a path 3 (2), so that the data acquisition is realized through the data acquisition processes correspondingly described by the paths 3 (2) and 3 (1), and the target system is still unavailable if the target system still does not meet the preset survival condition.
Further, the method further comprises: the available memory space of a preset size is maintained.
The step of maintaining the available memory space of the preset size comprises: and keeping the preset proportion of the data in the memory in the available memory.
Specifically, when the replier component is started to read the data in the memory queue and append the data to the preset memory, a certain free memory is reserved in the memory, so that data loss caused by memory overflow is avoided, the safety of the data is ensured, for example, the data in the memory is kept in 90% of the available memory, and backlogged temporary data is sent on the premise of ensuring the data order.
Further, the step of transmitting the data to the target system includes:
and sending the data to the target system in a zero copy mode.
The Zero copy (Zero copy) technique refers to that when a computer executes an operation, the CPU does not need to copy data from a memory to another specific area. This technique is typically used to save CPU cycles and memory bandwidth when transmitting files over a network.
Specifically, referring to fig. 2, if the Sender component in the real-time data collector determines that the target system is available, the current state of the Sender component is switched to the target system for use, and the Sender component is switched to read the data in the disk file and sends the data to the target system with high performance through the zero copy technology, so that the speed of adding the data in the memory to the preset memory is lower than the speed of reading the data in the preset memory by the data sending component, the efficiency of sending the data in the disk to the target system by the Sender component is improved, and the backlog data in the disk file can be forwarded as soon as possible.
In one embodiment, after the step of reading the data from the preset memory and sending the data to the target system, and starting to continue to append the data contained in the memory to the preset memory, the method further includes:
s307, judging whether the data contained in the preset memory is sent to the target system or not;
and S308, switching the reading of the data from the preset memory to the reading of the data from the memory if the data contained in the preset memory is sent to the target system, and forwarding the data read from the memory to the target system.
Specifically, the sending component in the real-time collector judges whether the data contained in the preset memory is sent to the target system or not, if the data contained in the preset memory is sent to the target system, the data read from the preset memory is switched to the data read from the memory, the data read from the memory is forwarded to the target system, and particularly, for collecting massive data such as big data, high-performance data transmission is realized, if the data stored in the preset memory is not sent to the target system, the step of judging whether the data stored in the preset memory is sent to the target system or not is executed, so that the sending state of the data stored in the preset memory is continuously monitored. With continued reference to fig. 2, for example, the replier component is started to read data in the memory queue and append the data to the disk, and the data in the memory is kept in 90% of the available memory, so that backlog data is efficiently transmitted on the premise of ensuring the data order, and once the data in the disk file is read and transmitted, the Sender component is switched from reading the disk file to reading the memory data, so that the data is read from the memory continuously through the path 1, the high performance of the real-time data collector is realized, and the high performance transmission of the data is realized.
It should be noted that, in the data acquisition method described in each of the foregoing embodiments, the technical features included in the different embodiments may be recombined as needed to obtain a combined implementation, which is within the scope of protection claimed in the present invention.
Referring to fig. 4, fig. 4 is a schematic block diagram of a data acquisition device according to an embodiment of the present invention. Corresponding to the data acquisition method, the embodiment of the invention also provides a data acquisition device. As shown in fig. 4, the data acquisition device includes a unit for performing the above-described data acquisition method, and the device may be configured in a computer apparatus such as a server. Specifically, referring to fig. 4, the data acquisition device 400 includes an acquisition unit 401, a first determination unit 402, a first forwarding unit 403, and a temporary storage unit 404.
The acquiring element 401 is configured to acquire data of an external data source, and store the data into the memory;
a first judging unit 402, configured to judge whether a target system corresponding to a data receiving party meets a preset survival condition, where the target system is a preset component part included in a data acquisition system and used for receiving acquired data, and the target system survival refers to a state that the target system is in a normal data receiving state;
A first forwarding unit 403, configured to read the data from the memory and forward the data to the target system to implement the data acquisition if the target system meets the preset survival condition;
and the temporary storage unit 403 is configured to read the data from the memory and forward the data to a preset storage for temporary storage if the target system does not meet the preset survival condition, and forward the data to the target system when the target system is available subsequently, so as to realize the data acquisition.
In one embodiment, the first determining unit 402 includes:
the judging subunit is used for confirming the target system corresponding to the data receiver in a heartbeat mode according to a preset period so as to judge whether the target system survives;
the first judging subunit is used for judging that the target system does not survive if the connection with the target system is judged to be abnormal according to heartbeat feedback;
and the second judging subunit is used for judging that the target system does not meet the preset survival condition if the target system is judged not to survive and the number of times of confirmation with the target system exceeds the preset number of times.
In one embodiment, the data acquisition device 400 further comprises:
the second judging unit is used for continuously judging whether the target system meets the preset survival condition according to the preset period;
and the sending unit is used for reading the data from the preset memory and sending the data to the target system if the target system meets the preset survival condition, and starting to continue to append the data contained in the memory to the preset memory.
In one embodiment, the data acquisition device 400 further comprises:
a third judging unit, configured to judge whether the data contained in the preset memory is sent to the target system;
and the switching unit is used for switching the reading of the data from the preset memory to the reading of the data from the memory and forwarding the data read from the memory to the target system if the data contained in the preset memory are sent to the target system.
In one embodiment, the first forwarding unit 403 is specifically configured to send the data to the target system in a zero-copy manner.
It should be noted that, as those skilled in the art can clearly understand the specific implementation process of the data acquisition device and each unit, reference may be made to the corresponding description in the foregoing method embodiments, and for convenience and brevity of description, details are not repeated herein.
Meanwhile, the dividing and connecting modes of the units in the data acquisition device are only used for illustration, and in other embodiments, the data acquisition device can be divided into different units according to the needs, and different connecting sequences and modes can be adopted for the units in the data acquisition device so as to complete all or part of functions of the data acquisition device.
The data acquisition device described above may be implemented in the form of a computer program that is executable on a computer device as shown in fig. 5.
Referring to fig. 5, fig. 5 is a schematic block diagram of a computer device according to an embodiment of the present invention. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part of another device.
With reference to FIG. 5, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, causes the processor 502 to perform a data acquisition method as described above.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the execution of a computer program 5032 in the non-volatile storage medium 503, which computer program 5032, when executed by the processor 502, causes the processor 502 to perform a data acquisition method as described above.
The network interface 505 is used for network communication with other devices. It will be appreciated by those skilled in the art that the architecture shown in fig. 5 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting of the computer device 500 to which the present inventive arrangements may be implemented, as a particular computer device 500 may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components. For example, in some embodiments, the computer device may include only a memory and a processor, and in such embodiments, the structure and function of the memory and the processor are consistent with the embodiment shown in fig. 5, and will not be described again.
Wherein the processor 502 is configured to execute a computer program 5032 stored in a memory to implement the steps of: acquiring data of an external data source and storing the data into a memory; judging whether a target system corresponding to a data receiver meets a preset survival condition or not, wherein the target system is used for receiving preset components of the acquired data contained in a data acquisition system, and the target system survival means that the target system is in a state of normally receiving the data; if the target system meets the preset survival condition, reading the data from the memory, and forwarding the data to the target system to realize the data acquisition; and if the target system does not meet the preset survival condition, reading the data from the memory, forwarding the data to a preset memory for temporary storage, and forwarding the data to the target system when the target system is available subsequently so as to realize data acquisition.
In an embodiment, when the step of determining whether the target system corresponding to the data receiver meets the preset survival condition is implemented by the processor 502, the following steps are specifically implemented:
According to a preset period, confirming a target system corresponding to a data receiver in a heartbeat mode to judge whether the target system survives;
if the connection with the target system is judged to be abnormal according to the heartbeat feedback, judging that the target system does not survive;
and if the target system is judged to be not survived and the number of times of confirmation with the target system exceeds the preset number of times, judging that the target system does not meet the preset survival condition.
In one embodiment, after implementing the step of reading the data from the memory and forwarding the data to a preset memory for temporary storage, the processor 502 further implements the following steps:
continuously judging whether the target system meets the preset survival condition according to the preset period;
and if the target system meets the preset survival condition, reading the data from the preset memory, sending the data to the target system, and starting to continue to append the data contained in the memory to the preset memory.
In one embodiment, after implementing the step of reading the data from the preset memory and sending the data to the target system, and starting the step of continuously adding the data contained in the memory to the preset memory, the processor 502 further implements the following steps:
Judging whether the data contained in the preset memory is sent to the target system or not;
and if the data contained in the preset memory is sent to the target system, switching the reading of the data from the preset memory to the reading of the data from the memory, and forwarding the data read from the memory to the target system.
In one embodiment, when implementing the step of sending the data to the target system, the processor 502 specifically implements the following steps:
and sending the data to the target system in a zero copy mode.
It should be appreciated that in embodiments of the present invention, the processor 502 may be a Central processing unit (Central ProcessingUnit, CPU), and the processor 502 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf Programmable gate arrays (FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It will be appreciated by those skilled in the art that all or part of the flow of the method of the above embodiments may be implemented by a computer program, which may be stored on a computer readable storage medium. The computer program is executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a computer-readable storage medium. The computer readable storage medium may be a non-volatile computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of:
a computer program product which, when run on a computer, causes the computer to perform the steps of the data acquisition method described in the above embodiments.
The computer readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or a memory of the device. The computer readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the device. Further, the computer readable storage medium may also include both internal storage units and external storage devices of the device.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the apparatus, device and unit described above may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
The storage medium is a physical, non-transitory storage medium, and may be, for example, a U-disk, a removable hard disk, a Read-only memory (ROM), a magnetic disk, or an optical disk.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be combined, divided and deleted according to actual needs. In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The integrated unit may be stored in a storage medium if implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing an electronic device (which may be a personal computer, a terminal, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (8)
1. A method of data acquisition, comprising:
acquiring data of an external data source and storing the data into a memory;
judging whether a target system corresponding to a data receiver meets a preset survival condition or not, wherein the target system is used for receiving preset components of the acquired data contained in a data acquisition system, and the target system survival means that the target system is in a state of normally receiving the data;
if the target system meets the preset survival condition, reading the data from the memory, and forwarding the data to the target system to realize the data acquisition;
if the target system does not meet the preset survival condition, reading the data from the memory, forwarding the data to a preset storage for temporary storage, and forwarding the data to the target system when the target system is available subsequently so as to realize the data acquisition;
The step of judging whether the target system corresponding to the data receiver meets the preset survival condition comprises the following steps:
according to a preset period, confirming a target system corresponding to a data receiver in a heartbeat mode to judge whether the target system survives;
if the connection with the target system is judged to be abnormal according to the heartbeat feedback, judging that the target system does not survive;
if the target system is judged to be not survived and the number of times of confirmation with the target system exceeds the preset number of times, judging that the target system does not meet the preset survival condition;
the step of confirming the target system corresponding to the data receiver in the heartbeat mode to judge whether the target system survives, comprises the following steps:
and sending a custom data packet to the target system and receiving heartbeat feedback of the target system.
2. The method for collecting data according to claim 1, wherein after the step of reading the data from the memory and forwarding the data to a predetermined memory for temporary storage, the method further comprises:
continuously judging whether the target system meets the preset survival condition according to the preset period;
And if the target system meets the preset survival condition, reading the data from the preset memory, sending the data to the target system, and starting to continue to append the data contained in the memory to the preset memory.
3. The method of claim 2, wherein after the step of reading the data from the preset memory and sending the data to the target system, and starting the step of continuing to append the data contained in the memory to the preset memory, the method further comprises:
judging whether the data contained in the preset memory is sent to the target system or not;
and if the data contained in the preset memory is sent to the target system, switching the reading of the data from the preset memory to the reading of the data from the memory, and forwarding the data read from the memory to the target system.
4. The data acquisition method of claim 2, wherein the step of transmitting the data to the target system comprises:
and sending the data to the target system in a zero copy mode.
5. A data acquisition device, comprising:
the acquisition unit is used for acquiring data of an external data source and storing the data into the memory;
the first judging unit is used for judging whether a target system corresponding to the data receiving party meets a preset survival condition or not, wherein the target system is used for receiving preset components of the acquired data contained in the data acquisition system, and the target system survival refers to the state that the target system is in a normal data receiving state;
the first forwarding unit is used for reading the data from the memory and forwarding the data to the target system if the target system meets the preset survival condition so as to realize the data acquisition;
the temporary storage unit is used for reading the data from the memory and forwarding the data to a preset storage for temporary storage if the target system does not meet the preset survival condition, and forwarding the data to the target system when the target system is available subsequently so as to realize the data acquisition;
the first judgment unit includes:
the judging subunit is used for confirming the target system corresponding to the data receiver in a heartbeat mode according to a preset period so as to judge whether the target system survives;
The first judging subunit is used for judging that the target system does not survive if the connection with the target system is judged to be abnormal according to heartbeat feedback;
a second determination subunit, configured to determine that the target system does not meet the preset survival condition if it is determined that the target system does not survive and the number of times of confirmation with the target system exceeds a preset number of times;
the step of confirming the target system corresponding to the data receiver in the heartbeat mode to judge whether the target system survives, comprises the following steps:
and sending a custom data packet to the target system and receiving heartbeat feedback of the target system.
6. The data acquisition device of claim 5, further comprising:
the second judging unit is used for continuously judging whether the target system meets the preset survival condition according to the preset period;
and the sending unit is used for reading the data from the preset memory and sending the data to the target system if the target system meets the preset survival condition, and starting to continue to append the data contained in the memory to the preset memory.
7. A computer device comprising a memory and a processor coupled to the memory; the memory is used for storing a computer program; the processor is configured to execute a computer program stored in the memory to perform the steps of the data acquisition method according to any one of claims 1-4.
8. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, causes the processor to perform the steps of the data acquisition method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911146802.2A CN111104233B (en) | 2019-11-21 | 2019-11-21 | Data acquisition method, device, computer equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911146802.2A CN111104233B (en) | 2019-11-21 | 2019-11-21 | Data acquisition method, device, computer equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111104233A CN111104233A (en) | 2020-05-05 |
| CN111104233B true CN111104233B (en) | 2023-09-26 |
Family
ID=70421674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911146802.2A Active CN111104233B (en) | 2019-11-21 | 2019-11-21 | Data acquisition method, device, computer equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111104233B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113783939B (en) * | 2021-08-20 | 2024-09-03 | 奇安信科技集团股份有限公司 | File transmission method and device, electronic equipment and storage medium |
| CN116541448B (en) * | 2023-05-10 | 2023-12-08 | 汉友科技有限公司 | Data integration processing method and device based on SaaS |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107797825A (en) * | 2017-10-24 | 2018-03-13 | 努比亚技术有限公司 | A kind of data transmission method for uplink, terminal and computer-readable recording medium |
| CN109739921A (en) * | 2019-01-07 | 2019-05-10 | 北京云基数技术有限公司 | A kind of big data acquisition system and method |
| CN209373598U (en) * | 2019-02-21 | 2019-09-10 | 深圳市甲易科技有限公司 | Real time data and external data association process system |
| CN110362450A (en) * | 2019-07-16 | 2019-10-22 | 深圳市网心科技有限公司 | A kind of log data acquisition method, device and computer readable storage medium |
-
2019
- 2019-11-21 CN CN201911146802.2A patent/CN111104233B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107797825A (en) * | 2017-10-24 | 2018-03-13 | 努比亚技术有限公司 | A kind of data transmission method for uplink, terminal and computer-readable recording medium |
| CN109739921A (en) * | 2019-01-07 | 2019-05-10 | 北京云基数技术有限公司 | A kind of big data acquisition system and method |
| CN209373598U (en) * | 2019-02-21 | 2019-09-10 | 深圳市甲易科技有限公司 | Real time data and external data association process system |
| CN110362450A (en) * | 2019-07-16 | 2019-10-22 | 深圳市网心科技有限公司 | A kind of log data acquisition method, device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111104233A (en) | 2020-05-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101453182B1 (en) | Interruption, at least in part, of frame transmission | |
| US9866351B2 (en) | Communication method and communication apparatus | |
| CN111104233B (en) | Data acquisition method, device, computer equipment and computer readable storage medium | |
| JP2020537405A5 (en) | ||
| CN109587822B (en) | Information transmission control method, information reception control device, and storage medium | |
| JP2017521016A (en) | Data packet transmitting apparatus, data packet transmitting method, data processing apparatus and data processing method | |
| KR20150131327A (en) | Network transmission adjustment based on application-provided transmission metadata | |
| CN113626235A (en) | Fault tolerance processing method and device and server | |
| JP4767336B2 (en) | Mail server system and congestion control method | |
| CN110445666B (en) | Network quality detection method and device and server | |
| CN103139098B (en) | Data transfer control method, device and system and fibre channel media | |
| CN111211933A (en) | Method and device for determining quality of transmission link | |
| CN115514698A (en) | Protocol calculation method, switch, cross-device link aggregation system and storage medium | |
| CN116303173B (en) | Method, device and system for reducing RDMA engine on-chip cache and chip | |
| CN117579611A (en) | Data transmission method and device, storage medium and electronic device | |
| CN112039835B (en) | Adaptive polling time estimation method, device, equipment and storage medium | |
| CN111970497A (en) | Video stream processing method and device, SDN controller and storage medium | |
| CN111447253A (en) | Buffer management method, system, device and storage medium | |
| CN111641565A (en) | Method, device and system for transmitting Ethernet message | |
| CN114125064A (en) | Method and device for ensuring equipment positioning reliability, computer equipment and storage medium | |
| CN112866187B (en) | Path switching method and path switching device | |
| CN109905447B (en) | Information transmission method and device | |
| JP7039959B2 (en) | Communication systems, relay devices, and nodes | |
| CN111480319B (en) | Throughput testing | |
| CN115955271B (en) | Satellite data transmission method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |