CN112788077B - Data acquisition method and device, computer equipment and computer-readable storage medium - Google Patents
Data acquisition method and device, computer equipment and computer-readable storage medium Download PDFInfo
- Publication number
- CN112788077B CN112788077B CN201911082189.2A CN201911082189A CN112788077B CN 112788077 B CN112788077 B CN 112788077B CN 201911082189 A CN201911082189 A CN 201911082189A CN 112788077 B CN112788077 B CN 112788077B
- Authority
- CN
- China
- Prior art keywords
- data
- field
- bytes
- main
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/535—Tracking the activity of the user
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a data acquisition method, a data acquisition device, computer equipment and a computer readable storage medium. The data acquisition method is applied to an application program server and comprises the following steps: receiving a resource request sent by an application program client; acquiring data content to be transmitted corresponding to the application program client according to the resource request; generating a data record according to the data content to be transmitted; and sending the data record to a data receiving server. By the method and the device, the technical problems that the normal use of the client is influenced and the data is easy to lose when the server collects the data in the prior art can be solved.
Description
Technical Field
The present invention relates to the field of data acquisition technologies, and in particular, to a data acquisition method and apparatus, a computer device, and a computer-readable storage medium.
Background
With the development of display screen technology, processing chip technology and internet technology, mobile terminals are almost instantly transformed into key entrances and main innovative platforms of internet services, and the most important hubs of internet resources, mobile network resources and environmental interaction resources. In the prior art, data such as operating behaviors and user information on a mobile terminal are researched to continuously improve the comfort level and the convenience of the use of the mobile terminal, wherein the relevant data are collected and uploaded to a server, and the data become the basis of the research.
The inventor finds that, when the collected related data is transmitted to the server, the client is usually required to report the data to the server, which consumes traffic of the client and occupies resources of the client, possibly affecting normal use of the client. In addition, the client generally has the problems of delayed reporting, unstable weak network environment, flow limitation and the like, data cannot be reported timely and accurately, and data loss is easy to occur.
Disclosure of Invention
The invention aims to provide a data acquisition method, a data acquisition device, computer equipment and a computer readable storage medium, which are used for solving the technical problems that the normal use of a client is influenced and data is easy to lose when a server side acquires data in the prior art.
In one aspect, the present invention provides a data acquisition method for achieving the above objects.
The data acquisition method is applied to an application program server and comprises the following steps: receiving a resource request sent by an application program client; acquiring data content to be transmitted corresponding to the application program client according to the resource request; generating a data record according to the data content to be transmitted; and sending the data record to a data receiving server.
Further, when receiving resource requests sent by a plurality of application program clients, the step of sending the data records to the data receiving server includes: sequentially splicing the data records corresponding to the application program clients; and sending the spliced data records to a data receiving server.
Further, before sequentially splicing the data records corresponding to each application client, the step of sending the data records to the data receiving server further includes: serializing data in the data records; the step of sequentially splicing the data records corresponding to the application program clients comprises the following steps: and sequentially splicing the serialized data records.
Further, the data record includes the main data and the description information of the main data, and the step of generating the data record according to the data content to be transmitted includes: adding the data content to be transmitted to a predefined data object to obtain effective data; determining metadata corresponding to the valid data, wherein the metadata comprises a plurality of fields; generating field information according to the field name and the field value of the field in the metadata; generating main data according to the valid data and the field information; determining description information of the main data; and generating a data record according to the main data and the description information of the main data.
Further, the description information of the main data includes a data identifier, a first identifier representing whether the main data includes metadata, and a length of the main data; the field information comprises the length of the field name, a second identifier for representing whether other field information exists after the current field information, the length of the field value and the field value.
Further, the step of adding the data content to be transmitted to the predefined data object to obtain the valid data comprises: acquiring a public field and adding the public field to a data object to obtain an intermediate object; and adding the private field in the data to be transmitted to the intermediate object to obtain valid data.
Further, the step of generating field information according to the field name and the field value of the field in the metadata includes: setting the length of the field name in the first x1 bytes of the field information; setting a field name in a byte which is behind the first x1 bytes and adjacent to the first x1 bytes; setting a second identification at the front y1 bit of x2 bytes which are adjacent to the field name and behind the field name; setting the length of a field value to the last y2 bits of x2 bytes, wherein the y1 bit and the y2 bit form x2 bytes; the field value is set to be x2 bytes and adjacent to x2 bytes, wherein x1, x2, y1 and y2 are all natural numbers.
Further, the step of generating the body data according to the valid data and the field information includes: sequentially arranging field information corresponding to each field in the metadata; valid data is set after all field information to generate body data.
Furthermore, the description information of the main data also comprises a check code of the length of the main data; the step of determining the description information of the subject data includes: setting a preset character string as a data identifier in the first x3 bytes of the description information; setting a first identification on the first y3 bits of x4 bytes after x3 bytes and adjacent to the x3 bytes; setting the length of the main data to the last y4 bits of the x4 bytes, wherein the y3 bits and the y4 bits form the x4 bytes; setting a check code in the last x5 bytes of the description information, wherein x3, x4, x5, y3 and y4 are all natural numbers; the step of generating the data record according to the main data and the description information of the main data comprises the following steps: after the body data is set to the description information of the body data, a data record is generated.
In another aspect, the present invention provides a data acquisition device for achieving the above objects.
This data acquisition device sets up in application service end, includes: the receiving module is used for receiving a resource request sent by an application program client; the acquisition module is used for acquiring the data content to be transmitted corresponding to the application program client according to the resource request; the processing module is used for generating a data record according to the data content to be transmitted; and the sending module is used for sending the data record to the data receiving server.
To achieve the above object, the present invention also provides a computer device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method when executing the computer program.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above method.
According to the data acquisition method, the data acquisition device, the computer equipment and the computer readable storage medium, the data receiving server acquires terminal-related data through the application program server, wherein the application program server acquires the data content to be transmitted corresponding to the application program client based on the resource request sent by the application program client, and then generates a data record according to the data content and sends the data record to the data receiving server.
Drawings
Fig. 1 is a flowchart of a data acquisition method according to an embodiment of the present invention;
fig. 2 is a block diagram of a data acquisition apparatus according to a second embodiment of the present invention;
fig. 3 is a hardware structure diagram of a computer device according to a third embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
In order to solve the technical problems that the normal use of a client is influenced and data is easily lost when a server acquires data in the prior art, the research direction of technicians in the field is focused on improving the data transmission process, for example, a better data compression method is adopted to increase the data transmission quantity and reduce the interaction between the client and a data receiving server; the more appropriate data transmission logic is adopted to upload data under the condition of good network quality so as to reduce data loss and the like, but the technical idea still has great limitation when the influence of data acquisition on the client is solved.
The inventor of the application carries out the following research on the process of reporting data of a data receiving server by a client in the prior art: in the prior art, when an application client receives an operation of a user, such as a click operation, a browsing operation, and the like, an application server needs to be called synchronously to perform a resource request, and the application client transmits data related to a user operation line to a data receiving server as data content to be transmitted. The inventor finds that when an application program client sends a resource request to an application program server, the application program server can obtain some data uploaded to a data receiving server by the application program client according to the resource request, and then if the application program server directly uploads the data to the data receiving server, the application program client does not need to upload the data, so that requests and data transmission under one or more public network environments can be reduced, and the problem existing in the process of reporting the data to the data receiving server by the client can be solved.
Based on this, the invention provides a data acquisition method, a device, computer equipment and a computer readable storage medium, the data acquisition method is applied to an application program server, the data acquisition method is communicated with a data receiving server through an intranet, the application program server normally responds to the application program client on one hand when receiving a resource request sent by the application program client, on the other hand, the data content to be transmitted corresponding to the application program client is obtained according to the resource request, then a data record is generated according to the data content to be transmitted, and the data record is sent to the data receiving server through the intranet, that is, by adopting the data acquisition method provided by the invention, the application program server uploads the data to be transmitted to the data receiving server, so that the request and data transmission between the application program client and the data receiving server through the public network are reduced, thereby the influence on the application program client when the application program server acquires the data can be reduced, and the intranet and the data receiving server are usually in the same state, the network environment is stable, and the data loss rate is reduced.
Specific embodiments of the data acquisition method, the data acquisition device, the computer device and the computer-readable storage medium provided by the invention will be described in detail below.
Example one
The embodiment of the invention provides a data acquisition method, which is applied to an application program server side and can reduce the requests and data transmission between an application program client side and a data receiving server side through a public network, thereby reducing the influence on the application program client side when the data receiving server side acquires data and reducing the data loss rate. Specifically, fig. 1 is a flowchart of a data acquisition method according to an embodiment of the present invention, and as shown in fig. 1, the data acquisition method according to the embodiment includes steps S101 to S104 as follows.
Step S101: and receiving a resource request sent by the application program client.
The application program client is positioned at one side of the terminal, receives the operation of the user, interacts with the application program server, and displays corresponding information to the user. For example, when the application client receives a click browsing operation of a user, the application client sends a resource request to the application server to request a resource to be displayed, where the resource request carries data parameters that identify the application client and are related to the click browsing operation.
Step S102: and acquiring the data content to be transmitted corresponding to the application program client according to the resource request.
After receiving the resource request, the application program server analyzes the resource request to obtain data parameters in the resource request, on one hand, corresponding resource information is found according to the data parameters to serve as a response of the resource request and is fed back to the application program client, so that the application program client displays resources required to be browsed by a user according to the response; on the other hand, the data content to be transmitted, which is corresponding to the application program client and needs to be uploaded to the data receiving server, is obtained according to the resource request, optionally, the parameter carried in the resource request includes the data content to be transmitted, or the application program server can obtain the data content to be transmitted according to the parameter carried in the resource request.
Step S103: and generating a data record according to the data content to be transmitted.
When the application program server side obtains the data content to be transmitted, some data processing can be further carried out, or the data can be directly packaged to generate a data record.
Step S104: and sending the data record to a data receiving server.
In the step, the application program server side records the packaged data and sends the data to the data receiving server side through the intranet, so that data acquisition of the data receiving server side is realized.
In the data acquisition method provided in this embodiment, the data receiving server acquires data related to the terminal through the application server, where the application server acquires data content to be transmitted corresponding to the application client based on a resource request sent by the application client, and then generates a data record according to the data content and sends the data record to the data receiving server, so that for the part of data, the application client does not need to be uploaded to the data receiving server on the terminal, and requests and data transmissions between the application client and the data receiving server through a public network are reduced, so that the influence on the application client when the application server acquires data can be reduced.
Optionally, in an embodiment, the step of obtaining, according to the resource request, the data content to be transmitted corresponding to the application client includes: analyzing the resource request to obtain data content carried in the resource request; and/or querying data content corresponding to the resource request at the application program server side.
Specifically, the resource request carries the data content to be transmitted, and in this embodiment, the data content to be transmitted can be obtained by analyzing the resource request; and/or the resource request carries an identification parameter of the application program server, and a database accessible by the application program server stores a corresponding relation between the identification parameter and the data content to be transmitted, so that the application program server can obtain the identification parameter in the resource request and then can inquire the corresponding data content to be transmitted in the database according to the identification parameter. For example, taking an example that a user clicks a video of a certain video player at a mobile phone client, the user clicks the video once to watch, the player client needs to send an Http resource request to the player server once to obtain a corresponding video or a bullet screen resource, and the like, after the player server receives the resource request, the player server parses the resource request to obtain information corresponding to the player client, such as a model, a version of the player client, and obtained resource information, and the like, and in addition, the player server can query an additional terminal, such as a video directory in a user favorite of the player client or a most popular video, and the like, and generates a data record by the player server according to the parsed information of the player client and the information obtained in the user favorite, and sends the data record to the data receiving server.
By adopting the data acquisition method provided by the embodiment, the application program server analyzes the resource request to obtain the data content carried in the resource request and uploads the data content to the data receiving server. Compared with the prior art, the data content corresponding to the resource request is inquired by the application program server and uploaded to the data receiving server, and the data which cannot be acquired by the application program client can be uploaded to the data receiving server, so that the data acquired by the data receiving server is more comprehensive.
Optionally, in an embodiment, when receiving a resource request sent by multiple application clients, the step of sending the data record to the data receiving server includes: sequentially splicing the data records corresponding to the application program clients; and sending the spliced data records to a data receiving server.
Specifically, the application server generally interacts with multiple application clients and receives resource requests sent by the multiple application clients, so in this embodiment, data records corresponding to the application clients are sequentially spliced, data is aggregated, and the aggregated multiple data records are sent to the data receiving server.
By adopting the data acquisition method provided by the embodiment, the data records corresponding to different application program clients are aggregated and uploaded to the data receiving server, so that data interaction between the application program server and the data receiving server can be reduced.
Optionally, in an embodiment, before sequentially splicing the data records corresponding to each application client, the step of sending the data records to the data receiving server further includes: serializing data in the data records; the step of sequentially splicing the data records corresponding to the application program clients comprises the following steps: and sequentially splicing the serialized data records.
Specifically, before sending the spliced data records, the application program server serializes the data in the data records to form binary numbers, and then splices the data records in the binary format.
By adopting the data acquisition method provided by the embodiment, the plurality of data records are transmitted in a binary format, and compared with the method of directly transmitting the plaintext, the content in the data records is not easy to expose, and the safety of the transmitted data content is improved; after the data is binarized, the transmission data volume can be reduced, the transmission efficiency is improved, and on the premise of the same transmission data volume, due to the fact that the data records are serialized, more data records can be spliced in one-time transmission, and therefore the data aggregation capacity can be improved.
Optionally, in an embodiment, the data record includes the subject data and description information of the subject data; the step of generating a data record according to the data content to be transmitted comprises: adding the data content to be transmitted to a predefined data object to obtain effective data; determining metadata corresponding to the valid data, wherein the metadata comprises a plurality of fields; generating field information according to the field name and the field value of the field in the metadata; generating main data according to the valid data and the field information; determining description information of the main data; and generating a data record according to the main data and the description information of the main data.
Specifically, the description information of the body data is used to describe the characteristics of the body data, including the length of the data, the name of the data, and the like. When the application program server generates the data records according to the data contents to be transmitted, one data object can be predefined, and the data contents to be transmitted are added to the data object to obtain effective data, so that the effective data of different data records can have a uniform data format, and the contents and forms of the effective data are standardized. Optionally, the data object is defined based on protobuf, and the data model is unified. The valid data is payload in the data record, and the valid data may correspond to metadata, where the metadata is information describing a data property (property) and is used to support functions such as indicating a storage location, history data, resource search, file record, and the like. When determining the metadata corresponding to the valid data, according to different requirements of the data contents, some data contents have metadata, other data contents do not have metadata, and the metadata may include a plurality of fields, and the field name of each field and the value of each field form a key-value pair. For data content with metadata, generating field information of each field according to key value pairs of the fields in the metadata, wherein the main data comprises valid data and all the field information; for data content without metadata, the body data includes only valid data. And finally, determining the description information of the main data according to the generated main data to generate a data record.
By adopting the data acquisition method provided by the embodiment, based on the predefined data object, the application program server side generates the effective data by adopting the uniform data format for the data to be transmitted, so that the effective data with the uniform data format can be obtained after the data receiving server side receives the data record and analyzes the effective data, and the data receiving server side can conveniently process the data.
Optionally, in an embodiment, the description information of the body data includes a data identifier, a first identifier that characterizes whether the body data includes metadata, and a length of the body data, and the field information includes a length of a field name, a second identifier that characterizes whether there is other field information after the current field information, a length of the field value, and a field value.
Specifically, the description information of the body data includes a data identification, a first identification characterizing whether the body data includes metadata, and a length of the body data. The data identification is an identifier of the data record and can be used for distinguishing the data record from other data in the invention, after a specific character string is agreed by two ends of the application program server and the data receiving server as the identifier, the application program server takes the specific character string as the data identification, and the data receiving server can identify the received data record when receiving the data record with the data identification as the specific character string. The first identifier characterizes whether the main data includes metadata or not by different characters, for example, when the first identifier is a character, the main data includes metadata, and when the first identifier is a character B, the main data does not include metadata.
When the first identification represents that the main data comprises metadata, the main data comprises effective data and field information corresponding to fields in the metadata, and the field information comprises the length of a field name, the field name, a second identification for representing whether other field information exists after the current field information, the length of a field value and the field value. The second identifier represents whether other field information exists after the current field information is represented by different characters, for example, when the second identifier is a C character, other field information exists after the current field information is represented, and when the second identifier is a D character, other field information does not exist after the current field information is represented.
The application program server side represents the data record through the data identification, represents whether the main data comprises the metadata or not through the first identification, sets the first identification as a value representing that the main data does not comprise the metadata when the main data does not comprise the metadata, and obtains the length of the main data according to the length of the effective data; when the main data includes metadata, setting a first identifier as a value characterizing that the main data includes the metadata, and obtaining the length of the main data according to the sum of the length of the valid data and the length of the field information. The metadata of the valid data comprises key value pairs of a plurality of fields, the key value pairs of the fields comprise field names and field values, field information is constructed through the key value pairs of the fields, when each field information is constructed, the length of the field names and the length of the field values are calculated, whether other field information exists after the current field information is represented through a second identifier, if other field information exists after the current field information, the second identifier is set to be the value when other field information exists after the current field information is represented, if other field information does not exist after the current field information, the second identifier is set to be the value when other field information does not exist after the current field information is represented, and the length of the field information in the data records can be calculated after the field information of all the fields is constructed.
The data receiving server side determines the data record through the data identification, then further determines whether the main data comprises metadata through the first identification, and when the main data does not comprise the metadata, the main data is obtained according to the length of the main data, namely effective data is obtained; when the main data comprises metadata, obtaining key value pairs of fields, namely field names and field values, through field information, wherein the key value pairs of all the fields are metadata of effective data, when each field information is analyzed, whether other field information exists after the current field information is determined through a second identifier, if other field information exists after the current field information, the field information is continuously analyzed according to the structure of the field information, if other field information does not exist after the current field information, the length of the effective data can be obtained according to the length of the analyzed field information and the length of the main data, and the data of the length of the effective data is obtained after the current field information, so that the effective data can be obtained.
By adopting the data acquisition method provided by the embodiment, the application program server processes the data to be transmitted to form the data records with the specific data structure, and then splices a plurality of data records and sends the data records to the data receiving server, based on the data records with the specific structure, the data receiving server can analyze each data record and further analyze the effective data transmitted by the application program server in the data records, so that the data acquisition method can perform aggregate transmission on the plurality of data records, reduce the communication between the application program server and the data receiving server, and reduce the response pressure of the data receiving server.
Optionally, in an embodiment, the step of adding the data content to be transmitted to the predefined data object to obtain the valid data includes: acquiring a common field and adding the common field to a data object to obtain an intermediate object; and adding the private field in the data to be transmitted to the intermediate object to obtain valid data.
Specifically, the data to be transmitted includes public fields and private fields, wherein the data to be transmitted corresponding to different application program clients have the same part of public fields and different part of public fields, and when the data to be transmitted is directly transmitted to the data receiving server as valid data, the data receiving server needs to adapt to different rules, so that the analysis cost is high. Based on this, in this embodiment, a uniform public field is obtained and added to a data object, for example, a model of a device, a version of a client application program, and the like, the fields are abstracted into the public field, written in the data object to obtain an intermediate object, and then a private field in data to be transmitted is added to the intermediate object to obtain valid data, so that the data receiving server can analyze the uniform public field when extracting the valid data for analysis, and different valid data only need to analyze the private field respectively. When the public field is acquired, the application server side can acquire the public field through the registration information of the application client side and the like.
By adopting the data acquisition method provided by the embodiment, the effective data of different data records can have uniform public fields, and the cost for the data receiving server side to analyze the data is further reduced.
Optionally, in an embodiment, the step of generating field information according to the field name and the field value of the field in the metadata includes: setting the length of the field name in the first x1 bytes of the field information; setting a field name in a byte which is behind the first x1 bytes and adjacent to the first x1 bytes; setting a second identification at the front y1 bit of x2 bytes which are adjacent to the field name and behind the field name; setting the length of a field value to the last y2 bits of x2 bytes, wherein the y1 bit and the y2 bit form x2 bytes; the field value is set to be x2 bytes and adjacent to x2 bytes, wherein x1, x2, y1 and y2 are all natural numbers.
Specifically, the length of the field name, the second flag, and the length of the field value are set to a fixed number of bytes, respectively, and the field name whose number of bytes is not fixed is located between the length of the field name and the second flag. When constructing field information, firstly calculating the length of a field name and setting the length of the field name in the first x1 bytes; then setting the field name behind the first x1 bytes; setting a second identifier according to whether other field information exists after the field information, if other field information exists after the current field information, setting the second identifier as a value representing that other field information exists after the current field information, if other field information does not exist after the current field information, setting the second identifier as a value representing that other field information does not exist after the current field information, and setting the second identifier at the first y1 bits of x2 bytes after the field name; then calculating the length of the field value to be arranged at the rear y2 bits of the x2 bytes; finally, a field value is set after the x2 bytes.
Based on the field information of the data structure, when the data receiving service end analyzes the field information, the data receiving service end obtains the content of the length number of the field name, analyzes the field name, obtains the content of the length of the second identification and the field value, obtains the second identification by analyzing the first y1 bit, obtains the length of the field value by analyzing the second y2 bit, and finally obtains the content of the length of the field value, namely the field value can be analyzed, and the field name and the field value form the key value pair of the field.
In a specific embodiment, x1=1, x2=4, y1=1, y2=31 are set as shown in table 1 below:
table 1 field information structure table
Setting the length name _ size of the field name to 1 byte, specifically to 8-bit unsigned number, with a minimum value of 1 and a maximum value of 256, representing the length of meta _ name; meta _ name corresponds to a field name, and may contain any character (supporting utf-8 name), but at least 1 character; the second identifier, moremeta and the length meta _ size of the field value, set 4 bytes, and the highest bit of the 4 bytes is the second identifier, where when the second identifier is 1, other field information still exists after representing the current field information, and when the second identifier is 0, other field information does not exist after representing the current field information. The last 31 bits are the length meta _ size of the field value, no sign, minimum value of 0, maximum value of 0x7FFFFFFF, which represents the number of bytes of meta data, and when the length meta _ size of the field value is 0, it can indicate that the subsequent meta data segment is not present (or understood as empty).
When analyzing the field information, firstly obtaining the content of the 1 st byte, analyzing the length of the field name, then obtaining the content of the length number of the field name, analyzing the field name Meta _ name, then obtaining the content of 4 bytes, analyzing the highest bit of the 4 bytes to obtain a second identifier, analyzing the last 31 bits of the 4 bytes to obtain the length of the field value, and finally obtaining the content of the length of the field value, namely analyzing the field value Meta.
By adopting the data acquisition method provided by the embodiment, based on the field information of the data structure, the field name and the field value do not need to be provided with identifiers, and the field name and the field value can be obtained by sequentially analyzing all parts, so that the data description information in the data records can be reduced, and the data transmission quantity can be reduced.
Optionally, in an embodiment, the step of generating the body data according to the valid data and the field information includes: sequentially arranging field information corresponding to each field in the metadata; valid data is set after all field information to generate body data.
Specifically, when the second identifier represents that other field information exists after the current field information, the length of the field name of the next field information is set after the field value of the current field information, when the second identifier represents that other field information does not exist after the current field information, effective data is set after the field value of the current field information, and after all the field information is sequentially arranged, the effective data is spliced to form main data.
By adopting the data acquisition method provided by the embodiment, the plurality of field information are sequentially arranged and then spliced with the effective data to form the main data, identifiers are not required to be set for the main data and the field information, and the field name and the field value corresponding to each field information and the effective data can be obtained by sequentially analyzing each part, so that the data description information in the data records can be reduced, and the data transmission quantity can be reduced.
Optionally, in an embodiment, the description information of the main data further includes a check code of the length of the main data; the step of determining the description information of the subject data includes: setting a preset character string as a data identifier in the first x3 bytes of the description information; setting a first identification on the first y3 bits of x4 bytes after x3 bytes and adjacent to the x3 bytes; setting the length of the main data to the last y4 bits of the x4 bytes, wherein the y3 bits and the y4 bits form the x4 bytes; setting a check code in the last x5 bytes of the description information, wherein x3, x4, x5, y3 and y4 are all natural numbers; the step of generating the data record according to the main data and the description information of the main data comprises the following steps: after the body data is set to the description information of the body data, a data record is generated.
Specifically, the description information of the main data further comprises a check code of the length of the main data, when the application program server side assembles the data, the length of the main data is calculated after the main data is formed, then the length of the main data is calculated by using a preset algorithm to obtain the check code, the check code is placed in a data record for transmission, one end of the data receiving server side can check the received main data according to the check code, and the validity of the transmitted data is improved.
In the embodiment, a data identifier, a first identifier, the length of main data and a check code are respectively set to be a fixed number of bytes, and the main data with unfixed bytes is positioned behind the check code, when the description information of the main data is constructed, a predetermined character string is firstly used as the data identifier and is arranged at the first x3 bytes of the description information for identifying data records, then the first identifier is set according to whether effective data has metadata, when the effective data has the metadata, the first identifier is set to be a value for representing that the main data comprises the metadata, when the effective data does not have the metadata, the first identifier is set to be a value for representing that the main data does not comprise the metadata, and when the effective data does not have the metadata, the first identifier is set to be the first y3 bits of x4 bytes behind the data identifier; when the effective data has metadata, calculating the length of the effective data and the length of the field information to obtain the length of the main data, and setting the length of the main data at the rear y4 bits of the x4 bytes, and when the effective data does not have the metadata, calculating the length of the effective data to obtain the length of the main data, and setting the length of the main data at the rear y4 bits of the x4 bytes; and calculating according to the length of the main data to obtain a check code, setting x5 bytes after the length of the main data to complete the setting of the description information of the main data, and setting the main data after the description information to generate a data record.
Based on the data record of the data structure, after the data receiving server receives the data record, the data record is identified based on a specific character string in a data identifier, then the content of x5 bytes is extracted at an interval of x4 bytes after the data identifier to obtain a check code, main data is extracted from the position after the check code to the next data identifier, the main data can be checked according to the check code, after the check is passed, the content of the first y3 bits of the x4 bytes is extracted after the data identifier to obtain a first identifier, the structure of the main data is determined according to the first identifier, when the first identifier indicates that the main data comprises metadata, field information in the main data is analyzed according to the process of analyzing the field information to obtain metadata, and then effective data is extracted; when the main data does not include metadata, the main data is directly extracted to obtain valid data.
In a specific embodiment, x3=4, x4=4, x5=1, y3=1, y4=31 are set, as shown in table 2 below:
table 2 data record structure table
| magic_num(4bytes) | metabit_and_size(4bytes) | size_checksum(1byte) | data(size bytes) |
Setting a preset section character string as a data identifier in the first 4 bytes of the description information, wherein the data identifier is a 4-byte character string without considering byte order; setting a first identifier at the highest position of 4 bytes in the middle for indicating whether meta data exists in the data, wherein when the first identifier is 1, the main data is represented to have meta data, the specific main data contains one or more meta data, the specific structure of the main data can be referred to above, when the first identifier is 0, the main data is represented to have no meta data, the main data is valid data payload, and the length size of the main data is the byte number of the valid data payload; setting the size of the main data to the last 31 bits of the middle 4 bytes, wherein the size is unsigned, the minimum value is 0, the maximum value is 0x7FFFFFFF, the size is used for representing the number of bytes of the data, and when the size is 0, the payload section of the valid data does not exist (or is understood to be empty); the check code is arranged in the 5 th byte, and the receiving device can check the main data according to the check code.
Optionally, in an embodiment, the step of sending the data record to the data receiving server includes: and sending the data record to a data receiving server side in an RPC calling mode.
Specifically, a data receiving server interface is called through a PRC (Remote Procedure Call Protocol) framework between the servers to perform data transfer, and since this method has the advantages of retry, load balancing, and the like, the data loss rate can be reduced.
Example two
Corresponding to the first embodiment, the second embodiment of the present invention provides a data collection device, which is disposed at an application server, and the related technical features and technical effects can be referred to the first embodiment. Fig. 2 is a block diagram of a data acquisition apparatus according to a second embodiment of the present invention, and as shown in fig. 2, the apparatus includes: a receiving module 201, an obtaining module 202, a processing module 203 and a sending module 204.
The receiving module 201 is configured to receive a resource request sent by an application client; the obtaining module 202 is configured to obtain, according to the resource request, data content to be transmitted corresponding to the application client; the processing module 203 is configured to generate a data record according to the data content to be transmitted; the sending module 204 is configured to send the data record to the data receiving server.
Optionally, in an embodiment, the processing module 203 includes a first processing unit and/or a second processing unit, where the first processing unit is configured to parse the resource request to obtain the data content carried in the resource request, and the second processing unit is configured to query the data content corresponding to the resource request at the application server.
Optionally, in an embodiment, the sending module 204 is further configured to sequentially splice data records corresponding to the application program clients when the receiving module 201 receives resource requests sent by multiple application program clients; and sending the spliced data records to a data receiving server.
Optionally, in an embodiment, the sending module 204 is further configured to serialize data in the data records before sequentially splicing the data records corresponding to each application client, and sequentially splice the serialized data records.
Optionally, in an embodiment, the data record includes main data and description information of the main data, and when the processing module 203 executes the step of generating the data record according to the data content to be transmitted, the specific executed steps include: adding the data content to be transmitted to a predefined data object to obtain effective data; determining metadata corresponding to the valid data, wherein the metadata comprises a plurality of fields; generating field information according to the field name and the field value of the field in the metadata; generating main data according to the valid data and the field information; determining description information of the main data; and generating a data record according to the main data and the description information of the main data.
Optionally, in an embodiment, the description information of the main data includes a data identifier, a first identifier characterizing whether the main data includes metadata, and a length of the main data; the field information comprises the length of the field name, a second identifier for representing whether other field information exists after the current field information, the length of the field value and the field value.
Optionally, in an embodiment, when the processing module 203 adds the data content to be transmitted to a predefined data object to obtain valid data, the specific steps executed include: acquiring a common field and adding the common field to a data object to obtain an intermediate object; and adding the private field in the data to be transmitted to the intermediate object to obtain valid data.
Optionally, in an embodiment, when the processing module 203 generates field information according to a field name and a field value of a field in the metadata, the specifically executed steps include: setting the length of the field name in the first x1 bytes of the field information; setting a field name in a byte which is behind the first x1 bytes and adjacent to the first x1 bytes; setting a second identification at the front y1 bit of x2 bytes which are adjacent to the field name and behind the field name; setting the length of a field value to the last y2 bits of x2 bytes, wherein the y1 bit and the y2 bit form x2 bytes; the field value is set to be x2 bytes and adjacent to x2 bytes, wherein x1, x2, y1 and y2 are all natural numbers.
Optionally, in an embodiment, when the processing module 203 generates the main data according to the valid data and the field information, the specifically executed steps include: sequentially arranging field information corresponding to each field in the metadata; valid data is set after all field information to generate body data.
Optionally, in an embodiment, the description information of the volume data further includes a check code of the length of the main body data; when the processing module 203 determines the description information of the main data, the specific steps executed include: setting a preset character string as a data identifier in the first x3 bytes of the description information; setting a first identification on the first y3 bits of x4 bytes after x3 bytes and adjacent to the x3 bytes; setting the length of the main data to the last y4 bits of the x4 bytes, wherein the y3 bits and the y4 bits form the x4 bytes; setting a check code in the last x5 bytes of the description information, wherein x3, x4, x5, y3 and y4 are all natural numbers; when the processing module 203 generates a data record according to the main data and the description information of the main data, the specific steps executed include: after the body data is set to the description information of the body data, a data record is generated.
Optionally, in an embodiment, the sending module 204 is further configured to send the data record to the data receiving server in a manner of RPC call.
EXAMPLE III
The embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers) capable of executing programs, and the like. As shown in fig. 3, the computer device 01 of the present embodiment at least includes but is not limited to: a memory 011 and a processor 012, which are communicatively connected to each other via a system bus, as shown in fig. 3. It is noted that fig. 3 only shows the computer device 01 having the component memory 011 and the processor 012, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
In this embodiment, the memory 011 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 011 can be an internal storage unit of the computer device 01, such as a hard disk or a memory of the computer device 01. In other embodiments, the memory 011 can also be an external storage device of the computer device 01, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a flash Card (FlashCard), etc. provided on the computer device 01. Of course, the memory 011 can also include both internal and external memory units of the computer device 01. In this embodiment, the memory 011 is generally used for storing an operating system installed in the computer device 01 and various application software, such as a program code of the data acquisition apparatus in the second embodiment. Further, the memory 011 can also be used to temporarily store various types of data that have been output or are to be output.
The processor 012 may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data Processing chip in some embodiments. The processor 012 is generally used to control the overall operation of the computer device 01. In this embodiment, the processor 012 is configured to run a program code stored in the memory 011 or process data, such as a data acquisition method.
Example four
The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of this embodiment is used to store a data acquisition apparatus, and when executed by a processor, the data acquisition method of the first embodiment is implemented.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one of 8230, and" comprising 8230does not exclude the presence of additional like elements in a process, method, article, or apparatus comprising the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (9)
1. A data acquisition method is characterized by being applied to an application program server side, and comprises the following steps:
receiving a resource request sent by an application program client;
acquiring data content to be transmitted corresponding to the application program client according to the resource request, wherein parameters carried in the resource request comprise the data content to be transmitted, or the application program server acquires the data content to be transmitted according to the parameters carried in the resource request;
generating a data record according to the data content to be transmitted;
sending the data record to a data receiving server,
the data record comprises main data and description information of the main data, the description information of the main data comprises a data identifier, a first identifier representing whether the main data comprises metadata or not and the length of the main data, and the step of generating the data record according to the data content to be transmitted comprises the following steps: adding the data content to be transmitted to a predefined data object to obtain effective data; determining metadata corresponding to the valid data, wherein the metadata comprises a plurality of fields, and the metadata is information describing data attributes; generating field information according to the field name and the field value of a field in the metadata, wherein the field information comprises the length of the field name, a second identifier for representing whether other field information exists after the current field information, the length of the field value and the field value; generating the main data according to the effective data and the field information; determining description information of the main data; generating the data record according to the main data and the description information of the main data,
when receiving resource requests sent by a plurality of application program clients, the step of sending the data records to a data receiving server comprises the following steps: sequentially splicing the data records corresponding to the application program clients; and sending the spliced data records to the data receiving server.
2. The data acquisition method according to claim 1,
before sequentially splicing the data records corresponding to the application program clients, the step of sending the data records to a data receiving server further comprises: serializing data in the data records;
the step of sequentially splicing the data records corresponding to the application program clients comprises the following steps: and sequentially splicing the serialized data records.
3. The data collection method of claim 1, wherein the step of adding data content to be transmitted to a predefined data object to obtain the valid data comprises:
obtaining a common field and adding the common field to the data object to obtain an intermediate object;
and adding the private field in the data to be transmitted to the intermediate object to obtain the effective data.
4. The data collection method of claim 1, wherein generating the field information based on the field name and the field value of the field in the metadata comprises:
setting the length of the field name to the first x1 bytes of the field information;
setting the field name to a byte that is subsequent to the first x1 bytes and adjacent to the first x1 bytes;
setting the second identifier at the first y1 bits of x2 bytes after the field name and adjacent to the field name;
setting the length of the field value to the last y2 bits of the x2 bytes, wherein the y1 bits and the y2 bits constitute the x2 bytes;
and setting the field value behind the x2 bytes and adjacent to the x2 bytes, wherein x1, x2, y1 and y2 are all natural numbers.
5. The data collection method of claim 4, wherein generating the body data from the valid data and the field information comprises:
sequentially arranging field information corresponding to each field in the metadata;
setting the valid data after all the field information to generate the body data.
6. The data acquisition method according to claim 1,
the description information of the main data also comprises a check code of the length of the main data;
the step of determining the description information of the main body data comprises:
setting a predetermined character string as the data identifier in the first x3 bytes of the description information;
setting the first identification to the first y3 bits of x4 bytes that are subsequent to and adjacent to the x3 bytes;
setting the length of the body data to the last y4 bits of the x4 bytes, wherein the y3 bits and the y4 bits constitute the x4 bytes; and
setting the check code in the last x5 bytes of the description information, wherein x3, x4, x5, y3 and y4 are all natural numbers;
the step of generating the data record according to the main data and the description information of the main data comprises: and generating the data record after the main data is set in the description information of the main data.
7. The data acquisition device is characterized by being arranged at an application program server side and comprising:
the receiving module is used for receiving a resource request sent by an application program client;
an obtaining module, configured to obtain, according to the resource request, data content to be transmitted corresponding to the application client, where parameters carried in the resource request include the data content to be transmitted, or the application server obtains the data content to be transmitted according to the parameters carried in the resource request;
the processing module is used for generating a data record according to the data content to be transmitted;
a sending module for sending the data record to a data receiving server,
the data record comprises main data and description information of the main data, the description information of the main data comprises a data identifier, a first identifier representing whether the main data comprises metadata or not and the length of the main data, and when the processing module executes the step of generating the data record according to the data content to be transmitted, the specifically executed step comprises: adding the data content to be transmitted to a predefined data object to obtain effective data; determining metadata corresponding to the valid data, wherein the metadata comprises a plurality of fields, and the metadata is information describing data attributes; generating field information according to field names and field values of fields in the metadata, wherein the field information comprises the length of the field names, second identifications for representing whether other field information exists after the current field information, the length of the field values and the field values; generating the main data according to the effective data and the field information; determining description information of the main data; generating the data record according to the main data and the description information of the main data,
the sending module is further configured to sequentially splice the data records corresponding to the application program clients when the receiving module 1 receives resource requests sent by multiple application program clients; and sending the spliced data records to the data receiving server.
8. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program when executed by a processor implements the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911082189.2A CN112788077B (en) | 2019-11-07 | 2019-11-07 | Data acquisition method and device, computer equipment and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911082189.2A CN112788077B (en) | 2019-11-07 | 2019-11-07 | Data acquisition method and device, computer equipment and computer-readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112788077A CN112788077A (en) | 2021-05-11 |
| CN112788077B true CN112788077B (en) | 2023-04-07 |
Family
ID=75747899
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911082189.2A Active CN112788077B (en) | 2019-11-07 | 2019-11-07 | Data acquisition method and device, computer equipment and computer-readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112788077B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858458B (en) * | 2023-02-22 | 2023-05-09 | 华中科技大学同济医学院附属同济医院 | Financial accounting data archiving system based on data statistics |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107169796A (en) * | 2017-05-12 | 2017-09-15 | 深圳市浩天投资有限公司 | A kind of analysis method of user behavior data, system and computer-readable recording medium |
| CN108182626A (en) * | 2017-12-28 | 2018-06-19 | 努比亚技术有限公司 | Service push method, information acquisition terminal and computer readable storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102253980A (en) * | 2011-06-23 | 2011-11-23 | 上海西本网络科技有限公司 | Data processing method and data processing system |
| CN109961836A (en) * | 2017-12-26 | 2019-07-02 | 深圳市智游人科技有限公司 | Workout data processing method, device, computer equipment and storage medium |
| CN108363658B (en) * | 2018-02-06 | 2021-06-11 | 四川虹美智能科技有限公司 | User behavior data acquisition method, intelligent terminal, server and system |
| CN109948077B (en) * | 2018-08-20 | 2023-09-01 | 中国印刷有限公司 | User behavior data acquisition method, device, equipment and computer storage medium |
-
2019
- 2019-11-07 CN CN201911082189.2A patent/CN112788077B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107169796A (en) * | 2017-05-12 | 2017-09-15 | 深圳市浩天投资有限公司 | A kind of analysis method of user behavior data, system and computer-readable recording medium |
| CN108182626A (en) * | 2017-12-28 | 2018-06-19 | 努比亚技术有限公司 | Service push method, information acquisition terminal and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112788077A (en) | 2021-05-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111555963B (en) | Message pushing method and device, electronic equipment and storage medium | |
| CN110245069B (en) | Page version testing method and device and page display method and device | |
| CN110704521A (en) | Interface data access method and system | |
| CN111427613B (en) | Application program interface API management method and device | |
| CN103428042A (en) | Method and system for carrying out stress test on server | |
| CN110858192A (en) | Log query method and system, log checking system and query terminal | |
| CN110838071A (en) | Policy data processing method and device and server | |
| CN112702228B (en) | Service flow limit response method, device, electronic equipment and readable storage medium | |
| CN104580376A (en) | Method, device and system for constructing connection between terminals in local area network | |
| CN112788077B (en) | Data acquisition method and device, computer equipment and computer-readable storage medium | |
| CN113656098A (en) | Configuration acquisition method and system | |
| CN112788078B (en) | Data transmission method, receiving device, sending device and computer equipment | |
| CN112787978B (en) | Data acquisition method and device, computer equipment and computer-readable storage medium | |
| CN114039801B (en) | Short link generation method, short link analysis system, short link analysis equipment and storage medium | |
| CN108614844B (en) | Multimedia file generation method and device | |
| CN112948733B (en) | Interface maintenance method, device, computing equipment and medium | |
| CN110941711A (en) | Electronic search report acquisition method and apparatus, storage medium, and electronic apparatus | |
| CN112416875B (en) | Log management method, device, computer equipment and storage medium | |
| CN108600255A (en) | Mock service management method, device, computer equipment and storage medium | |
| CN114390015A (en) | Data pushing system, method and equipment based on object model and storage medium | |
| CN107707328B (en) | Abstract information transmission method and device | |
| CN111949472A (en) | Method and device for recording application logs | |
| CN113132324B (en) | Sample identification method and system | |
| CN113141366B (en) | Message submitting method and device, storage medium and electronic equipment | |
| CN113485942B (en) | Automatic testing method and device based on independent modules |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |