CN112417022A - Method, device, terminal and storage medium for acquiring data - Google Patents
Method, device, terminal and storage medium for acquiring data Download PDFInfo
- Publication number
- CN112417022A CN112417022A CN202011362965.7A CN202011362965A CN112417022A CN 112417022 A CN112417022 A CN 112417022A CN 202011362965 A CN202011362965 A CN 202011362965A CN 112417022 A CN112417022 A CN 112417022A
- Authority
- CN
- China
- Prior art keywords
- data
- data packet
- target
- packet
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000012544 monitoring process Methods 0.000 claims abstract description 22
- 238000004590 computer program Methods 0.000 claims description 21
- 238000012545 processing Methods 0.000 abstract description 12
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 4
- 238000001914 filtration Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a method, a device, a terminal and a storage medium for acquiring data, relates to the technical field of data processing, and can effectively improve the efficiency of data acquisition. The method for acquiring data comprises the following steps: if monitoring that at least one interface acquires first data, acquiring at least one first data packet corresponding to a target interface from the first data; and obtaining target transaction data according to the at least one first data packet.
Description
Technical Field
The present application belongs to the technical field of data processing, and in particular, to a method, an apparatus, a terminal, and a storage medium for acquiring data.
Background
With the development of information technology, data is becoming an indispensable part in the information field. Generally, a business system such as a transaction system is used to acquire necessary data so as to update the current data in a target system based on the data. In order to acquire data, a database in which related service data is stored in advance is polled uninterruptedly to acquire the data, and corresponding data processing is completed based on the acquired data.
However, when data is obtained by polling the interface library, the database cannot be accessed frequently in order to reduce the pressure of the database, but when the time interval for accessing the interface library is too long, the data cannot be returned to the target system in time so as to complete the corresponding processing of the data, thereby greatly reducing the user experience on the service system. Therefore, when the data is acquired by using the current data acquisition mode, the efficiency is not high.
Disclosure of Invention
The embodiment of the application provides a method, a device, a terminal and a storage medium for acquiring data, and aims to solve the problem of low efficiency when data acquisition is performed by using the current data acquisition mode.
In a first aspect, an embodiment of the present application provides a method for acquiring data, including:
if monitoring that at least one interface acquires first data, acquiring at least one first data packet corresponding to a target interface from the first data;
and obtaining target transaction data according to the at least one first data packet.
By adopting the method for acquiring data, whether the interface acquires the first data or not is monitored in real time, and if the first data is acquired by at least one interface, at least one first data packet corresponding to the target interface is acquired from the first data, so that the condition that the designated object sends data to the interface can be conveniently known. Furthermore, target transaction data are obtained according to the at least one first data packet, so that data can be obtained in time, the data obtaining efficiency is improved, and the requirements of a data obtaining scene can be better met.
Optionally, the interface is identified by a port number and/or an IP address;
if it is monitored that at least one interface acquires first data, acquiring at least one first data packet corresponding to a target interface from the first data, including:
and if the monitoring shows that the at least one interface acquires the first data, determining the at least one first data packet corresponding to the target interface from the first data according to the port number and/or the IP address of the target interface.
Optionally, the obtaining target transaction data according to the at least one first data packet includes:
determining at least one target data packet from the at least one first data packet, wherein the target data packet comprises the target transaction data;
and analyzing the at least one target data packet to obtain the target transaction data.
Optionally, the determining at least one target data packet from the at least one first data packet includes:
for each first data packet, obtaining a first data length of the first data packet, where the first data length is a length of data about the first data packet sent by a specified object to the target interface;
if the first data length is equal to the length of the first data packet, acquiring a second data length of the first data packet, wherein the second data length is the data length of a packet body in the first data packet;
and if the second data length is equal to the data length corresponding to the packet body in the first data packet, taking the first data packet as a target data packet.
Optionally, after determining that the first data length is equal to the length of the first data packet and obtaining the second data length, the method further includes:
if the second data length is larger than the acquired data length corresponding to the packet body in the first data packet, and the data length of the packet body of the first data packet is larger than a preset data length, determining that the first data packet lacks part of packet data;
if monitoring that at least one interface acquires second data, acquiring at least one second data packet corresponding to the target interface from the second data;
acquiring a remaining data packet corresponding to the stored first data packet from the at least one second data packet, wherein the remaining data packet is a data packet corresponding to part of body data lacking in body data of the first data packet;
merging the packet bodies of the rest data packets with the stored packet body of the first data packet to form a new data packet;
and taking the new data packet as the target data packet.
Optionally, the analyzing the at least one target data packet to obtain the target transaction data includes:
for each target data packet, acquiring the file type of the target data packet;
if the file type accords with a preset file type, acquiring a first identification parameter in transaction data included in the target data packet;
and if the first identification parameter is consistent with a preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
Optionally, if the first identifier parameter is consistent with a preset first identifier parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet, including:
if the first identification parameter of the target data packet is consistent with the preset first identification parameter, acquiring a second identification parameter in transaction data included in the target data packet;
and if the second identification parameter is consistent with a preset second identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
In a second aspect, an embodiment of the present application provides an apparatus for acquiring data, including:
the first obtaining module is used for obtaining at least one first data packet corresponding to a target interface from the first data if the monitoring that at least one interface obtains the first data is carried out;
and the second acquisition module is used for acquiring target transaction data according to the at least one first data packet.
In a third aspect, an embodiment of the present application provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the method when executing the computer program.
In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the method.
In a fifth aspect, embodiments of the present application provide a computer program product, which, when run on a terminal, causes the terminal to perform the method of any one of the first aspect.
It is understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description of the first aspect, and are not described herein again.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a schematic flow chart diagram of a method for acquiring data according to an embodiment of the present application;
FIG. 2 is a diagram illustrating an application scenario of a method for acquiring data according to an embodiment of the present application;
fig. 3 is a flowchart illustrating a specific implementation of step S12 of the method for acquiring data according to an embodiment of the present application;
fig. 4 is a flowchart illustrating a specific implementation of step S21 of the method for acquiring data according to an embodiment of the present application;
FIG. 5 is a schematic flow chart diagram of a method for acquiring data according to another embodiment of the present application;
fig. 6 is a flowchart illustrating a specific implementation of step S22 of the method for acquiring data according to another embodiment of the present application;
FIG. 7 is a schematic structural diagram of an apparatus for acquiring data according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a terminal according to another embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
As used in this specification and the appended claims, the term "if" may be interpreted in context to mean "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
In order to explain the technical solution described in the present application, the following description will be given by way of specific examples.
Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of a method for acquiring data according to an embodiment of the present disclosure. In this embodiment, the method for acquiring data is used to acquire target transaction data in a process that a service system needs to acquire data for processing, and an execution subject of the method is a terminal. The terminal includes but is not limited to a mobile terminal such as a smart phone, a tablet computer, a wearable device, and the like, and may also be a desktop computer, a robot, a server, and the like.
The method for acquiring data as shown in fig. 1 comprises the following steps:
s11: and if monitoring that at least one interface acquires first data, acquiring at least one first data packet corresponding to a target interface from the first data.
In step S11, the interface is a data specification for implementing data acquisition and transmission. It will be appreciated that data may be retrieved based on the interface and stored to a designated database, or sent out through the interface.
The target interface is used to describe the interface that receives the target transaction data.
The first data is data which is sent to the data source through the interfaces based on the data requirements corresponding to the interfaces and is returned by the data source. It will be appreciated that the data acquisition requirements of the respective interfaces may be described by the first data.
The first data packet is used for describing data requirements corresponding to the target interface. It is understood that, in the data packet, transaction data describing the data acquisition requirement corresponding to the target interface is recorded. In some embodiments, the first packet is a packet corresponding to a Tabular Data Stream protocol (TDS protocol).
It should be noted that, in order to determine whether the target interface acquires the first data, so as to acquire the at least one first data packet corresponding to the target interface from the first data corresponding to the target interface, before monitoring the at least one interface to acquire the first data, an environment in the terminal for monitoring the at least one interface to acquire the first data is configured based on the code script, and then the running of the code script is controlled to monitor the first data acquired by the at least one interface. The code script configures an Internet Protocol Address (IP Address), a name, and an interface of the network device for acquiring the data packet. It is understood that, when controlling the code script to run, the first data returned by each data source may be captured based on the code script, and it is further confirmed that each interface is to acquire the first data, so that at least one first data packet corresponding to the target interface is acquired from the first data to be acquired by the target interface.
For example, before monitoring at least one interface to obtain first data, an environment for monitoring the at least one interface to obtain the first data in the terminal is configured based on the code script, that is, a network device interface corresponding to the terminal is obtained based on the code script, and a working mode of the network device interface is configured as a target mode, for example, the network device interface is configured as a promiscuous mode, so that in the working mode, the network device interface may capture all data in a communication network to which the terminal is connected, that is, first data transmitted back to the at least one interface by the data source may be monitored.
In this embodiment, when it is monitored that at least one interface acquires first data, at least one first data packet corresponding to a target interface is acquired from the first data, and the first data can be used for describing a data acquisition requirement of the target interface, so that the at least one first data packet corresponding to the target interface is acquired from the first data, and the first data packet describing a data requirement of a target system can be better acquired. In addition, in this embodiment, when monitoring at least one interface to obtain first data, the first data packet corresponding to the target interface is directly obtained from the first data corresponding to the interface, so that a corresponding data obtaining requirement is met, and thus, a database corresponding to the interface does not need to be accessed, and the first data packet corresponding to the data obtaining requirement is obtained, which can effectively reduce the pressure of the database, and the data meeting the target system can be obtained without accessing the database, so that the data obtaining efficiency is higher.
When at least one first data packet corresponding to the target interface is obtained from the first data, the following two scenarios may be included, but not limited to.
Scene 1: when a data acquisition request of a target system is received, monitoring whether at least one interface acquires first data or not, and if the monitoring that at least one interface acquires the first data, acquiring at least one first data packet corresponding to the target interface from the first data.
For example, when a user performs a transaction operation in a transaction system, the transaction system needs to acquire transaction data corresponding to the transaction operation of the user from a transaction program corresponding to a transaction exchange. Therefore, in order to efficiently acquire the transaction data, the first data acquired by at least one interface is monitored, so that at least one first data packet corresponding to the target interface is acquired from the first data.
Scene 2: in order to update data in the trading system, data in the trading market needs to be periodically acquired to a trading program corresponding to a trading exchange. Therefore, in order to efficiently acquire the transaction data, the first data acquired by at least one interface is monitored, so that at least one first data packet corresponding to the target interface is acquired from the first data.
It should be understood that, in practical applications, the at least one first data packet corresponding to the target interface may be obtained from the first data based on the target system, or the at least one first data packet corresponding to the target interface may be obtained from the first data by a terminal connected to the target system, and then returned to the target system.
In an embodiment, at least one interface is cyclically monitored to acquire first data, and when the at least one interface is monitored to acquire the first data, at least one first data packet corresponding to a target interface is acquired from the first data.
In this embodiment, when cyclically monitoring at least one interface to acquire first data, that is, when at least one first data packet corresponding to a target interface is acquired from the first data, a new round of monitoring is started, that is, monitoring at least one interface to acquire the first data is restarted.
As a possible implementation manner of this embodiment, the interface is identified by a port number and/or an IP address;
if it is monitored that at least one interface acquires first data, acquiring at least one first data packet corresponding to a target interface from the first data, including:
and if the monitoring shows that the at least one interface acquires the first data, determining the at least one first data packet corresponding to the target interface from the first data according to the port number and/or the IP address of the target interface.
In the present embodiment, the port number is an identifier configured to distinguish between the respective interfaces, and the identifier includes a number. Through the port number and the definition of the preset port number, the service content provided by the interface can be known. The IP address is a logical address that the terminal corresponding to the interface has on the internet.
In this embodiment, each interface has a unique port number and/or IP address, and in order to accurately transmit data to each interface, the data packet includes the port number and/or IP address of the interface. Therefore, in order to determine the at least one first data packet corresponding to the target interface from the first data corresponding to the target interface, the port number and/or the IP address of the target interface may be matched with the port number and/or the IP address included in the data packet, and if the matching is successful, the data packet is a data packet to be received by the target interface, and the data packet may include transaction data required by the data of the target system.
Specifically, a port number and/or an IP address of each data packet in the first data is obtained, the port number and/or the IP address of each data packet is matched with a port number and/or an IP address of the target interface, and the first data packet corresponding to the target interface is obtained according to a matching result. It can be understood that, since the first data corresponding to the target interface is stored in the database corresponding to the target interface through the target interface, the first data packet corresponding to the target interface is obtained as a copied version.
For example, as shown in fig. 2, when the specified object is an inventory program of the exchange, the inventory program needs to write back transaction data to the interface library based on the port number and the IP address of the interface, so as to store the transaction data in the database through the interface in the interface library. Meanwhile, the terminal is configured with a code script used for acquiring the transaction data packet, so that the target transaction data in the acquired transaction data packet is returned to the target system through the code script, and the target system completes related data processing according to the target transaction data. In order to acquire the transaction data packet written back to the interface library by the disc reporting program, the network interface can be configured into a preset mode through the configured code script, so that the terminal can capture all data in the communication network based on the network interface, and in order to accurately acquire the transaction data packet written back to the interface library by the disc reporting program, the code script is configured with a corresponding data packet filtering expression, so that the transaction data packet written back to the interface library by the non-disc reporting program is filtered by the filtering expression. For example, the filtering expression configured in the code script is "tcp & & dst port number & & dst host interface IP address and len > 60", based on the filtering expression, a data packet of a non-report program written back to the interface library can be filtered, and only a transaction data packet of a report program written back to the interface library is reserved, that is, when at least one interface is monitored to obtain network data, the at least one transaction data packet corresponding to a target interface in the interface library can be obtained from the network data according to the port number and the IP address of the target interface, so as to obtain the transaction data through the code script configured in the terminal before the report program does not write the transaction data into the interface library, that is, the transaction data is directly obtained by bypassing the time consuming operation of the writing interface library, so as to greatly improve the efficiency of the target system for obtaining the transaction data, and meanwhile, the target system does not need to frequently poll the interface library for querying the transaction data, the pressure of the database corresponding to the interface library can be effectively reduced.
S12: and obtaining target transaction data according to the at least one data packet.
In step S12, the target transaction data is transaction data that satisfies the data acquisition requirement. For example, the target trading data is trading data for updating a target trading unit in a trading system.
In this embodiment, since the obtained at least one data packet is a data packet that meets the data obtaining requirement corresponding to the target interface, the at least one data packet may be analyzed according to a preset policy after obtaining the at least one first data packet corresponding to the target interface from the first data, so as to obtain the target transaction data. The preset strategy is used for describing a logic process for analyzing and processing at least one data packet to obtain target transaction data.
In one embodiment, a transaction call package is formed based on the target transaction data, and the transaction call package is returned to the target system, so that the target system performs corresponding data processing according to the transaction call package.
As shown in fig. 3, as a possible implementation manner of this embodiment, the obtaining target transaction data according to the at least one first data packet includes:
s21: and determining at least one target data packet from the at least one first data packet, wherein the target data packet comprises the target transaction data.
S22: and analyzing the at least one target data packet to obtain the target transaction data.
In this embodiment, the target data packet records a data packet that satisfies the target data acquisition requirement.
The method includes the steps of determining at least one target data packet from the at least one first data packet, namely determining the data packet recorded with target transaction data from the at least one first data packet describing various data acquisition requirements corresponding to a target interface, so as to directly acquire the target data packet and return the target data packet to a target system, or a terminal connected with the target system, so as to return the target data packet to the target system through the terminal, and finally completing data processing through the target system according to the target transaction data recorded in the target data packet, such as completing update of transaction data or displaying data.
As shown in fig. 4, as a possible implementation manner of this embodiment, the determining at least one target data packet from the at least one first data packet includes:
s31: for each first data packet, obtaining a first data length of the first data packet, where the first data length is a length of data about the first data packet sent by a specified object to the target interface.
S32: if the first data length is equal to the length of the first data packet, acquiring a second data length of the first data packet, wherein the second data length is the data length of a packet body in the first data packet.
S33: and if the second data length is equal to the data length corresponding to the packet body in the first data packet, taking the first data packet as a target data packet.
In this embodiment, the designated object is a terminal that directly returns the first data to the interface or an intermediate terminal that forwards the first data sent by the data source. The designated object and the terminal acquiring the first data packet are different terminals.
It is understood that the data packet has a specific data format, and the identification field in the data packet is predefined as a body in which transaction data describing the data acquisition requirement is recorded.
Illustratively, the first data packet is a data packet of a TDS protocol, and the data field included in the data format of the data packet includes a data packet header and a data field, where the data field is used as a packet body.
In order to determine whether the obtained data corresponding to the target interface is complete, when at least one target data packet is determined from at least one first data packet, a first data length of the first data packet is obtained for each first data packet, so that the length of the data, which is sent by a specified object to the target interface and is related to the first data packet, is known through the first data length. And if the first data length is equal to the length of the first data packet, the fact that a complete data packet is acquired is indicated. And further acquiring a second data length used for describing the data length of the packet body in the first data packet, so as to determine whether the acquired packet body in the first data packet is complete or not through the second data length, and when the second data length is equal to the data length corresponding to the packet body in the first data packet, indicating that the acquired packet body of the first data packet is complete, the first data packet can be used as a target data packet.
In an embodiment, the data packet has a specific data format, the first data length is recorded in a specific field of the data format, and the first data length of the data packet is extracted according to a preset data length obtaining strategy.
The preset data length obtaining strategy is used for describing a logic process for obtaining the first data length. For example, the data format of the data packet is identified, and when the data format is consistent with the data format recorded in the preset data length acquisition policy, the data recorded in the preset field is extracted, and the data is used as the first data length.
As shown in fig. 5, as a possible implementation manner of this embodiment, after determining that the first data length is equal to the length of the first data packet and acquiring the second data length, the method further includes:
s41: and if the second data length is larger than the acquired data length corresponding to the packet body in the first data packet, and the data length of the packet body of the first data packet is larger than a preset data length, determining that the first data packet lacks part of packet data.
S42: and if the monitoring shows that at least one interface acquires the second data, acquiring at least one second data packet corresponding to the target interface from the second data.
S43: and acquiring a remaining data packet corresponding to the stored first data packet from the at least one second data packet, wherein the remaining data packet is a data packet corresponding to part of the body data lacking in the body data of the first data packet.
S44: and merging the packet bodies of the rest data packets with the stored packet body of the first data packet to form a new data packet.
S45: and taking the new data packet as the target data packet.
In this embodiment, the first data is data that is sent to the data source based on the data requirement corresponding to each interface and is returned by the data source. It will be appreciated that the data requirements of the respective interfaces may be described by the first data.
The second data packet is used for describing the data requirement corresponding to the target interface. It is understood that, in the data packet, transaction data describing the data acquisition requirement corresponding to the target interface is recorded.
And the preset data length is used for describing the minimum packet body length meeting the data retention requirement. For example, the preset data length is limited to be greater than 0, otherwise, when the data length of the packet body is equal to 0, the first data packet corresponding to the packet body is directly discarded.
The remaining data packet is a data packet corresponding to part of the body data lacking in the body data of the first data packet.
It can be understood that, because the second data length is the data length of the packet body in the first data packet, when the second data length is greater than the data length corresponding to the acquired packet body in the first data packet, that is, when the second data length is the data length corresponding to the acquired packet body in the first data packet, that is, when the second data length indicates the first data packet acquired from the first data corresponding to the target interface, the packet body in which one piece of information is completely recorded is not acquired, that is, the first data packet lacks part of packet data. Therefore, it is necessary to continue to monitor the second data acquired by the interface, so as to acquire at least one second data packet corresponding to the target interface from the second data, and acquire the remaining data packets corresponding to the stored first data packet from the at least one second data packet. Further, merging the packet bodies of the remaining data packets with the stored packet body of the first data packet to form a new data packet; and taking the new data packet as the target data packet.
In this embodiment, since it is not known when the data packet corresponding to the part of the body data missing from the body data of the first data packet is obtained, the first data packet may be stored in the preset storage space, so that when the remaining data packets are obtained, the first data packet is obtained from the preset storage space, and the bodies of the remaining data packets and the stored body of the first data packet are merged to form a new data packet.
As shown in fig. 6, as a possible implementation manner of this embodiment, the analyzing the at least one target data packet to obtain the target transaction data includes:
s51: for each target data packet, acquiring the file type of the target data packet;
s52: if the file type accords with a preset file type, acquiring a first identification parameter in transaction data included in the target data packet;
s53: and if the first identification parameter is consistent with a preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
In this embodiment, the file type is used to describe the manner in which information is encoded for use in storing the information.
The first identification parameter is a distinguishing parameter describing whether the data packet satisfies the target data acquisition request. For example, the first identification parameter is "INSERT INTO table name".
The transaction data is used for describing data containing identification parameters and transaction parameters, and whether the data packet is a target data packet meeting certain data acquisition requirements can be determined based on the identification parameters in the transaction data.
The transaction parameters are used to describe data acquisition requirements. It can be understood that the target transaction parameter in the target data packet can be obtained based on the transaction parameter, and the target transaction parameter is a portion that does not include the identification parameter in the transaction parameter because the target transaction parameter is data that only describes transaction changes.
It can be understood that the preset first identification parameter is data identifying a specific data acquisition request, so that the first identification parameter is matched with the preset first identification parameter, if the matching result indicates that the matching is consistent, the target data packet is a data packet corresponding to the specific data acquisition request, and the transaction parameter in the target data packet may meet the corresponding data acquisition request, so that the target transaction data in the target data packet can be obtained according to the transaction parameter in the target data packet, that is, the target transaction data is extracted from the transaction parameter of the target data packet.
As a possible implementation manner of this embodiment, if the first identifier parameter is consistent with a preset first identifier parameter, obtaining the target transaction data in the target data packet according to the transaction parameter in the target data packet includes:
if the first identification parameter of the target data packet is consistent with the preset first identification parameter, acquiring a second identification parameter in transaction data included in the target data packet;
and if the second identification parameter is consistent with a preset second identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
In this embodiment, the second identification parameter is used to describe a parameter describing whether the data packet satisfies the target data obtaining request. For example, whether the transaction data in the target data packet is the data corresponding to the target transaction unit is determined by the second identification parameter. It can be understood that, when the first identification parameter of the target data packet is consistent with the preset first identification parameter, it indicates that the transaction data in the target data packet is the required transaction data, but in order to determine whether the transaction data is the data corresponding to the target transaction unit in the target system, it is also necessary to acquire the second identification parameter in the transaction data included in the target data packet, so as to determine whether the transaction data is the transaction data required by the target transaction unit in the target system through the second identification parameter. If the second identification parameter is consistent with the preset second identification parameter, the target data packet is a data packet meeting the data acquisition requirement, so that the target transaction data in the target data packet can be obtained according to the transaction parameter in the target data packet, and a target system can complete corresponding data processing according to the target transaction data.
By adopting the method for acquiring data provided by the above embodiment of the present application, whether the interface acquires the first data is monitored in real time, and if it is monitored that at least one interface acquires the first data, at least one first data packet corresponding to the target interface is acquired from the first data, so that the situation that the designated object sends data to the interface is known. Furthermore, target transaction data are obtained according to the at least one first data packet, so that data can be obtained in time, the data obtaining efficiency is improved, and the requirements of a data obtaining scene can be better met.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Fig. 7 shows a block diagram of a device for acquiring data according to an embodiment of the present application, which corresponds to the method for acquiring data according to the foregoing embodiment, and only shows portions related to the embodiment of the present application for convenience of description.
Referring to fig. 7, the apparatus 100 includes:
a first obtaining module 101, configured to obtain at least one first data packet corresponding to a target interface from first data if it is monitored that at least one interface obtains the first data;
the second obtaining module 102 is configured to obtain the target transaction data according to the at least one first data packet.
Optionally, the interface is identified by a port number and/or an IP address.
The first obtaining module 101 is further configured to determine, according to the port number and/or the IP address of the target interface, the at least one first data packet corresponding to the target interface from the first data if it is monitored that the at least one interface obtains the first data.
Optionally, the second obtaining module 102 further includes a determining unit and an analyzing unit.
The determining unit is configured to determine at least one target data packet from the at least one first data packet, where the target data packet includes the target transaction data.
And the analysis unit is used for analyzing the at least one target data packet to obtain the target transaction data.
Optionally, the determining unit is further configured to obtain, for each first data packet, a first data length of the first data packet, where the first data length is a length of data of the first data packet that is sent by a specified object to the target interface; if the first data length is equal to the length of the first data packet, acquiring a second data length of the first data packet, wherein the second data length is the data length of a packet body in the first data packet; and if the second data length is equal to the data length corresponding to the packet body in the first data packet, taking the first data packet as a target data packet.
Optionally, the determining unit is further configured to, after determining that the first data length is equal to the length of the first data packet and acquiring the second data length, further include: and if the second data length is larger than the acquired data length corresponding to the packet body in the first data packet, and the data length of the packet body of the first data packet is larger than a preset data length, determining that the first data packet lacks part of packet data.
The first obtaining module 101 is further configured to, if it is monitored that at least one interface obtains second data, obtain at least one second data packet corresponding to the target interface from the second data.
The determining unit is further configured to acquire a remaining data packet corresponding to the stored first data packet from the at least one second data packet, where the remaining data packet is a data packet corresponding to part of body data lacking in body data of the first data packet; merging the packet bodies of the rest data packets with the stored packet body of the first data packet to form a new data packet; and taking the new data packet as the target data packet.
Optionally, the parsing unit is further configured to obtain, for each target data packet, a file type of the target data packet; if the file type accords with a preset file type, acquiring a first identification parameter in transaction data included in the target data packet; and if the first identification parameter is consistent with a preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
Optionally, the parsing unit is further configured to obtain a second identification parameter in the transaction data included in the target data packet if the first identification parameter of the target data packet is consistent with a preset first identification parameter; and if the second identification parameter is consistent with a preset second identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
Fig. 8 is a schematic structural diagram of a terminal according to an embodiment of the present application. As shown in fig. 8, the terminal 8 of this embodiment includes: at least one processor 80 (only one processor is shown in fig. 8), a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, the processor 80 implementing the steps in any of the various method embodiments described above when executing the computer program 82.
The terminal 8 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The terminal may include, but is not limited to, a processor 80, a memory 81. Those skilled in the art will appreciate that fig. 8 is merely an example of the terminal 8 and does not constitute a limitation of the terminal 8, and may include more or less components than those shown, or combine some components, or different components, such as input and output devices, network access devices, etc.
The Processor 80 may be a Central Processing Unit (CPU), and the Processor 80 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 81 may in some embodiments be an internal storage unit of the terminal 8, such as a hard disk or a memory of the terminal 8. The memory 81 may also be an external storage device of the terminal 8 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal 8. Further, the memory 81 may also include both an internal storage unit and an external storage device of the terminal 8. The memory 81 is used for storing an operating system, an application program, a BootLoader (BootLoader), data, and other programs, such as program codes of the computer program. The memory 81 may also be used to temporarily store data that has been output or is to be output.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned method embodiments.
The embodiments of the present application provide a computer program product, which when running on a terminal, enables the terminal to implement the steps in the above method embodiments when executed.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a device/terminal, recording medium, computer Memory, Read-Only Memory (ROM), Random-Access Memory (RAM), electrical carrier wave signals, telecommunications signals, and software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal and method may be implemented in other ways. For example, the above-described apparatus/terminal embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A method for acquiring data, comprising:
if monitoring that at least one interface acquires first data, acquiring at least one first data packet corresponding to a target interface from the first data;
and obtaining target transaction data according to the at least one first data packet.
2. The method of claim 1, wherein the interface is identified by a port number and/or an IP address;
if it is monitored that at least one interface acquires first data, acquiring at least one first data packet corresponding to a target interface from the first data, including:
and if the monitoring shows that the at least one interface acquires the first data, determining the at least one first data packet corresponding to the target interface from the first data according to the port number and/or the IP address of the target interface.
3. The method of claim 1 or 2, wherein obtaining target transaction data based on the at least one first data packet comprises:
determining at least one target data packet from the at least one first data packet, wherein the target data packet comprises the target transaction data;
and analyzing the at least one target data packet to obtain the target transaction data.
4. The method of claim 3, wherein said determining at least one destination packet from said at least one first packet comprises:
for each first data packet, obtaining a first data length of the first data packet, where the first data length is a length of data about the first data packet sent by a specified object to the target interface;
if the first data length is equal to the length of the first data packet, acquiring a second data length of the first data packet, wherein the second data length is the data length of a packet body in the first data packet;
and if the second data length is equal to the data length corresponding to the packet body in the first data packet, taking the first data packet as a target data packet.
5. The method of claim 4, wherein after determining that the first data length is equal to the length of the first data packet and obtaining the second data length, further comprising:
if the second data length is larger than the acquired data length corresponding to the packet body in the first data packet, and the data length of the packet body of the first data packet is larger than a preset data length, determining that the first data packet lacks part of packet data;
if monitoring that at least one interface acquires second data, acquiring at least one second data packet corresponding to the target interface from the second data;
acquiring a remaining data packet corresponding to the stored first data packet from the at least one second data packet, wherein the remaining data packet is a data packet corresponding to part of body data lacking in body data of the first data packet;
merging the packet bodies of the rest data packets with the stored packet body of the first data packet to form a new data packet;
and taking the new data packet as the target data packet.
6. The method of claim 3, wherein said parsing said at least one target data packet to obtain said target transaction data comprises:
for each target data packet, acquiring the file type of the target data packet;
if the file type accords with a preset file type, acquiring a first identification parameter in transaction data included in the target data packet;
and if the first identification parameter is consistent with a preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
7. The method of claim 6, wherein obtaining the target transaction data in the target data packet according to the transaction parameter in the target data packet if the first identification parameter is consistent with a preset first identification parameter comprises:
if the first identification parameter of the target data packet is consistent with the preset first identification parameter, acquiring a second identification parameter in transaction data included in the target data packet;
and if the second identification parameter is consistent with a preset second identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
8. An apparatus for acquiring data, comprising:
the first obtaining module is used for obtaining at least one first data packet corresponding to a target interface from the first data if the monitoring that at least one interface obtains the first data is carried out;
and the second acquisition module is used for acquiring target transaction data according to the at least one first data packet.
9. A terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011362965.7A CN112417022B (en) | 2020-11-27 | Method, device, terminal and storage medium for acquiring data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011362965.7A CN112417022B (en) | 2020-11-27 | Method, device, terminal and storage medium for acquiring data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112417022A true CN112417022A (en) | 2021-02-26 |
| CN112417022B CN112417022B (en) | 2024-06-07 |
Family
ID=
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101136812A (en) * | 2007-09-28 | 2008-03-05 | 北京北方烽火科技有限公司 | Independent WIMAX base station network performance testing method and system |
| US20170034036A1 (en) * | 2015-07-31 | 2017-02-02 | Aetna Inc. | Computing environment connectivity system |
| US20170356808A1 (en) * | 2016-06-08 | 2017-12-14 | Mark Smith | Retrofit Gas Kiln Atmospheric Monitoring System |
| CN107682125A (en) * | 2017-11-13 | 2018-02-09 | 平安科技（深圳）有限公司 | A kind of data capture method, device, computer equipment and storage medium |
| WO2019072103A1 (en) * | 2017-10-11 | 2019-04-18 | 阿里巴巴集团控股有限公司 | Method and apparatus for obtaining target data |
| CN109800259A (en) * | 2018-12-11 | 2019-05-24 | 深圳市金证科技股份有限公司 | Collecting method, device and terminal device |
| US20190220273A1 (en) * | 2018-01-02 | 2019-07-18 | Alibaba Group Holding Limited | Configuring data collection |
| KR102009336B1 (en) * | 2018-04-25 | 2019-08-12 | 주식회사쿠콘 | Apparatus, method and computer program for cloud scrapping using pre-scrapped bigdata |
| CN111813573A (en) * | 2020-06-29 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Communication method of management platform and robot software and related equipment thereof |
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101136812A (en) * | 2007-09-28 | 2008-03-05 | 北京北方烽火科技有限公司 | Independent WIMAX base station network performance testing method and system |
| US20170034036A1 (en) * | 2015-07-31 | 2017-02-02 | Aetna Inc. | Computing environment connectivity system |
| US20170356808A1 (en) * | 2016-06-08 | 2017-12-14 | Mark Smith | Retrofit Gas Kiln Atmospheric Monitoring System |
| WO2019072103A1 (en) * | 2017-10-11 | 2019-04-18 | 阿里巴巴集团控股有限公司 | Method and apparatus for obtaining target data |
| CN109947795A (en) * | 2017-10-11 | 2019-06-28 | 阿里巴巴集团控股有限公司 | The acquisition methods and device of target data |
| CN107682125A (en) * | 2017-11-13 | 2018-02-09 | 平安科技（深圳）有限公司 | A kind of data capture method, device, computer equipment and storage medium |
| WO2019090997A1 (en) * | 2017-11-13 | 2019-05-16 | 平安科技（深圳）有限公司 | Data acquisition method and device, computer device and storage medium |
| US20190220273A1 (en) * | 2018-01-02 | 2019-07-18 | Alibaba Group Holding Limited | Configuring data collection |
| KR102009336B1 (en) * | 2018-04-25 | 2019-08-12 | 주식회사쿠콘 | Apparatus, method and computer program for cloud scrapping using pre-scrapped bigdata |
| CN109800259A (en) * | 2018-12-11 | 2019-05-24 | 深圳市金证科技股份有限公司 | Collecting method, device and terminal device |
| CN111813573A (en) * | 2020-06-29 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Communication method of management platform and robot software and related equipment thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108427705B (en) | Electronic device, distributed system log query method and storage medium | |
| CN109656487B (en) | Data processing method, device, equipment and storage medium | |
| CN112448969A (en) | Link tracking method, device, system, equipment and readable storage medium | |
| CN110851334A (en) | Flow statistical method, electronic device, system and medium | |
| CN112671878B (en) | Block chain information subscription method, device, server and storage medium | |
| CN106790380A (en) | Data reporting method and device | |
| CN112422485B (en) | Communication method and device of transmission control protocol | |
| CN112929720A (en) | Screen projection method and device, electronic equipment and readable storage medium | |
| CN112417022B (en) | Method, device, terminal and storage medium for acquiring data | |
| CN108282374B (en) | Method, system and readable storage medium for configuring NCSI network card | |
| CN112417022A (en) | Method, device, terminal and storage medium for acquiring data | |
| CN111212153A (en) | IP address checking method, device, terminal equipment and storage medium | |
| US9813927B2 (en) | Mobile terminal flow identification method and apparatus | |
| CN115328734A (en) | Cross-service log processing method and device and server | |
| CN110868360B (en) | Flow statistics method, electronic equipment, system and medium | |
| CN114218000A (en) | Log management method and device, electronic equipment and computer readable storage medium | |
| CN114428737A (en) | Communication method, device, equipment and storage medium based on micro service system | |
| CN115795100A (en) | User event processing method and device, electronic equipment and readable storage medium | |
| CN108683716B (en) | business logic learning and protecting method and learning and protecting device based on big data | |
| CN107749892B (en) | Network reading method and device for conference record, intelligent tablet and storage medium | |
| CN112256654A (en) | Document sharing method and device | |
| WO2014173129A1 (en) | Mobile terminal flow identification method and apparatus | |
| CN117667604B (en) | Data monitoring method, device, electronic equipment and storage medium for tracking event | |
| CN115038089B (en) | Multi-terminal data monitoring and collecting method based on information extraction | |
| CN116886445B (en) | Processing method and device of filtering result, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |