CN109547234A - A kind of data processing, report method, device, equipment and storage medium - Google Patents
A kind of data processing, report method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN109547234A CN109547234A CN201811214237.4A CN201811214237A CN109547234A CN 109547234 A CN109547234 A CN 109547234A CN 201811214237 A CN201811214237 A CN 201811214237A CN 109547234 A CN109547234 A CN 109547234A
- Authority
- CN
- China
- Prior art keywords
- data
- reported
- custom protocol
- uri
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a kind of data processing, report method, device, equipment and storage mediums.This method comprises: receiving the reported data from data terminal, the reported data includes uniform resource identifier URI and data type；Reported data is parsed according to custom protocol, obtains target data, wherein custom protocol is corresponding with URI, and the configuration field in custom protocol is inputted by user；Target data is sent to data-base cluster.The present invention, which solves, frequently to modify and upgrades to the program of data processing end, the technical issues of to be parsed to reported data, it realizes using general protocol specification, relevant configuration field in agreement is managed by being customized of web front-end, improves the access efficiency of reported data.
Description
Technical field
The present embodiments relate to the communication technology more particularly to a kind of data processing, report method, device, equipment and deposit
Storage media.
Background technique
The basis that data are relied on as big data platform, general data amount is bigger, type is abundanter, and the value that can be provided is just
Higher, the potentiality for having excavation are also bigger.During data report server-side (data processing end) from client (data terminal),
May include following two mode:
First way arranges the corresponding data protocol of numerous types of data, visitor between server-side and client in advance
Family end carries out bottom code programming to reported data and encapsulates, client pair according to the corresponding data protocol of different types of data
Bottom code after encapsulation is tested, to guarantee the accuracy of reported data；Further, server-side carries out reported data
Test, to receive the reported data of corresponding data type.
The second way arranges the corresponding data protocol of specific data type, example between server-side and client in advance
Such as, arrange JS object numbered musical notation (JavaScript Object Notation, JSON) corresponding data protocol of data format, client
Reported data is directly sent to server-side according to the data protocol by end, without testing in client；Correspondingly, server-side
Reported data is not tested, and after only carrying out the verification of legitimacy to reported data, receive the qualified reported data of verification.
The above first way, the test by client to the test and server-side of reported data to reported data,
It ensure that the accuracy of reported data, but every a kind of business datum reports, server-side and client need customization to match
Data protocol needs to customize more and more data protocols with the development of business, the data type reported, can not keep away
Exempt to need frequently to modify serve end program and upgrading, increase more workload, the system for also affecting server is stablized
Property.
The above second way avoids the more and more data protocols of customization, however, only legal to reported data progress
Property verification, can not determine the accuracy of reported data, bring biggish hidden danger using reported data to be subsequent.
Summary of the invention
In view of this, the present invention provides a kind of data processing, report method, device, equipment and storage medium, it is existing to solve
Having needs that frequently the program of data processing end is modified and upgraded in technology, and causes the system at data processing end unstable,
And the technical issues of legitimacy verifies only are carried out to reported data and not can guarantee reported data accuracy.
In a first aspect, the embodiment of the invention provides a kind of data processing methods, comprising:
The reported data from data terminal is received, the reported data includes uniform resource identifier URI and data type；
The reported data is parsed according to custom protocol, obtains target data, wherein the custom protocol and the URI
Corresponding, the configuration field in the custom protocol is inputted by user；
The target data is sent to data-base cluster.
Further, before the reported data of the reception from data terminal, further includes:
The configuration field of custom protocol is obtained by target web interface；
It obtains agreement and generates instruction, custom protocol is generated according to the configuration field.
Further, described that the reported data is parsed according to custom protocol, obtain target data, comprising:
Extract the URI of reported data；
The configuration field in corresponding custom protocol is searched according to the URI；
The reported data is parsed according to the configuration field sequence, obtains target data.
Further, the target data is JSON data format.
It is further, described that the target data is sent to data-base cluster, comprising:
The target data is sent to Hadoop cluster by flume process.
It is further, described that the target data is sent to by Hadoop cluster by flume process, comprising:
The target data is sent to kafka cluster；
Long-range starting flume process；
The target data is sent to Hadoop cluster by the flume process by the kafka cluster.
It is further, described to be sent to Hadoop cluster, comprising:
Hive table in the Hadoop cluster includes the data type in the target data, then by the target data
It is stored in corresponding Hive table；
Hive table in the Hadoop cluster does not include the data type in the target data, then creates Hive table,
And the Hive table that target data deposit is newly-built.
Further, the configuration field includes at least one of following: URI, event title, topic, reporting schemes, word
Name section, data type and remark information；The reporting schemes include hypertext transfer protocol HTTP reporting schemes and transmission control
Agreement TCP reporting schemes processed.
Second aspect, the embodiment of the invention also provides a kind of data reporting methods, comprising:
Initial data is obtained, the initial data includes uniform resource identifier URI；
Encapsulate the initial data according to custom protocol, obtain the reported data of standard sequence, the custom protocol with
The URI is corresponding, and the configuration field in the custom protocol is inputted by user；
The reported data is sent to data processing end.
The third aspect, the embodiment of the invention also provides a kind of data processing equipments, comprising:
Receiving module, for receiving the reported data from data terminal, the reported data includes uniform resource identifier
URI and data type；
Parsing module obtains target data, wherein the customization for parsing the reported data according to custom protocol
Agreement is corresponding with the URI, and the configuration field in the custom protocol is inputted by user；
First sending module, for the target data to be sent to data-base cluster.
Further, the data processing equipment, further includes:
Module is obtained, for before the reported data of the reception from data terminal, it to be fixed to obtain by target web interface
The configuration field of agreement processed；
Generation module generates instruction for obtaining agreement, generates custom protocol according to the configuration field.
Further, parsing module includes:
Extraction unit, for extracting the URI of reported data；
Searching unit, for searching the configuration field in corresponding custom protocol according to the URI；
Resolution unit obtains target data for parsing the reported data according to the configuration field sequence.
Further, the target data is JSON data format.
Further, the first sending module includes:
Transmission unit, for the target data to be sent to Hadoop cluster by flume process.
Further, the transmission unit includes:
Transmission sub-unit, for the target data to be sent to kafka cluster；
Promoter unit, for remotely starting flume process；
The target data is sent to for the kafka cluster by the flume process by transmission sub-unit
Hadoop cluster.
It is further, described to be sent to Hadoop cluster, comprising:
Hive table in the Hadoop cluster includes the data type in the target data, then by the target data
It is stored in corresponding Hive table；
Hive table in the Hadoop cluster does not include the data type in the target data, then creates Hive table,
And the Hive table that target data deposit is newly-built.
Further, the configuration field includes at least one of following: URI, event title, topic, reporting schemes, word
Name section, data type and remark information；The reporting schemes include hypertext transfer protocol HTTP reporting schemes and transmission control
Agreement TCP reporting schemes processed.
Fourth aspect, the embodiment of the invention also provides a kind of data reporting devices, comprising:
Module is obtained, for obtaining initial data, the initial data includes uniform resource identifier URI；
Package module obtains the reported data of standard sequence, institute for encapsulating the initial data according to custom protocol
State that custom protocol is corresponding with the URI, the configuration field in the custom protocol is inputted by user；
Second sending module, for the reported data to be sent to data processing end.
5th aspect, the embodiment of the invention also provides a kind of data processing systems, comprising: data terminal, data processing end
And data-base cluster；
The data terminal is sent to the data processing end for generating reported data, and by reported data；
The data processing end for receiving the reported data, and carries out dissection process to the reported data and obtains
Target data, and target data is sent to the data-base cluster；
The data-base cluster, for receiving and storing the target data.
6th aspect, the embodiment of the invention also provides a kind of data processing equipments, comprising: memory and one are more
A processor；
The memory, for storing one or more programs；
When one or more of programs are executed by one or more of processors, so that one or more of processing
Device realizes data processing method as described in relation to the first aspect.
7th aspect, it is described the embodiment of the invention also provides a kind of storage medium comprising computer executable instructions
Computer executable instructions by computer processor when being executed for executing data processing method as described in relation to the first aspect.
The present invention uses general protocol specification, and relevant configuration field in agreement is managed by being customized of web front-end
Reason, and reported data is parsed one by one according to custom protocol in the reported data for receiving data terminal, form Hive table
The target data for the JSON data format that can be identified, and report to Hadoop cluster, solve need in the prior art it is frequently right
Serve end program is modified and is upgraded, and causes the system of server unstable, and only carries out legitimacy to reported data
The technical issues of verifying and not can guarantee reported data accuracy realizes without repairing to the bottom code of data processing end
Change, so that it may reported data is received, to improve the access efficiency of reported data, and by custom protocol to upper count off
According to being reported, the accuracy of reported data ensure that.
Detailed description of the invention
Fig. 1 is a kind of structure block diagram of data processing system provided in an embodiment of the present invention；
Fig. 2 is a kind of flow chart of data processing method provided in an embodiment of the present invention；
Fig. 3 is the flow chart of another data processing method provided in an embodiment of the present invention；
Fig. 4 is a kind of display schematic diagram of target web interface provided in an embodiment of the present invention
Fig. 5 is a kind of flow chart of data reporting method provided in an embodiment of the present invention；
Fig. 6 is the flow chart of another data processing method provided in an embodiment of the present invention；
Fig. 7 is a kind of structural block diagram of data processing equipment provided in an embodiment of the present invention；
Fig. 8 is a kind of structural block diagram of data reporting device provided in an embodiment of the present invention；
Fig. 9 is the structural block diagram of another data processing system provided in an embodiment of the present invention；
Figure 10 is the structural block diagram of another data processing system provided in an embodiment of the present invention；
Figure 11 is a kind of structural schematic diagram of data processing equipment provided in an embodiment of the present invention.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention rather than limiting the invention.It also should be noted that in order to just
Only the parts related to the present invention are shown in description, attached drawing rather than entire infrastructure.
Fig. 1 is a kind of structure block diagram of data processing system provided in an embodiment of the present invention.With reference to Fig. 1, the number
It include: data terminal 110, data processing end 120 and data-base cluster 130 according to processing system.
Wherein, data terminal 110 are sent to data processing end 120 for generating reported data, and by reported data；Data
Processing end 120 for receiving reported data, and carries out dissection process to reported data and obtains target data, and by target data
It is sent to data-base cluster 130；Data-base cluster 130, for receiving and storing target data.
It should be noted that data processing method in this programme is applied to data processing end, and the data side of reporting
Method is applied to data terminal.Wherein, the detailed description that data processing method and the specific embodiment of data reporting method see below.
It also is understood as, the data processing method and data reporting method in this programme are applied in data processing system as shown in Figure 1
In.Specifically, Fig. 2 is a kind of flow chart of data processing method provided in an embodiment of the present invention, the number provided in the present embodiment
It can be executed by data processing equipment according to processing method, which can be real by way of software and/or hardware
Existing, which can be two or more physical entities and constitutes, and is also possible to a physical entity and constitutes.This implementation
Data processing equipment is data processing end in example, is used to carry out data processing to reported data.
With reference to Fig. 2, which specifically comprises the following steps:
Step 210 receives the reported data from data terminal.
Wherein, reported data includes uniform resource identifier (Uniform Resource Identifier, URI) sum number
According to type.
Specifically, data terminal is the data side of reporting, for reported data to be sent to data processing end.In embodiment
In, it can also be background server that data terminal, which can be cell phone client,.Certainly, when the data side of reporting is different data terminal, institute
The reporting schemes of use are also different.When data terminal is cell phone client, hypertext transfer protocol can be used
Reported data is sent to data processing end by (HyperText Transfer Protocol, HTTP) reporting schemes；And work as data
When end is background server, the side of reporting transmission control protocol (Transmission Control Protocol, TCP) can be used
Reported data is sent to data processing end by formula.
In general, URI is one for identifying the character string of a certain Internet resources title.The mark allows user couple
The resource of any (including local and internet) interacts operation by specific agreement.In embodiment, URI is used to identify
Different reported events, the corresponding custom protocol of each reported event.Specifically, receiving the upper count off from data terminal
According to when, can identify corresponding custom protocol according to the URI in reported data, and carry out to reported data by the custom protocol
Parsing.
Step 220 parses reported data according to custom protocol, obtains target data.
Wherein, custom protocol is corresponding with URI, and the configuration field in custom protocol is inputted by user.
In embodiment, custom protocol adjusts on the basis of data interchange format according to the data type of reported data
Configuration field obtains.Wherein, data interchange format, which refers to, carries out data exchange between distinct program or different programming languages
When the format that uses, it describes data by text in a specified pattern.It should be noted that custom protocol not only includes
Configuration field should also have the function of dynamic analysis field.Wherein, the function of dynamic analysis field, it can be understood as to configuration
The function that configuration information in field is parsed one by one.
In embodiment, URI is used to distinguish mark to different custom protocols, meanwhile, each custom protocol
It is global variable that definition, which has the corresponding URI and URI, in order to be found according to URI pair in entire program file
The custom protocol answered.
Wherein, configuration field includes URI, event title, topic, reporting schemes, field name, data type and remarks
The basic informations such as information.Wherein, reporting schemes include HTTP reporting schemes or TCP reporting schemes.In embodiment, URI is used for
Reported event is identified, different URI corresponds to different reported events, also be understood as different URI correspond to it is different
Custom protocol；The corresponding title of the entitled each reported event of event, is checked, different reported events is equal for building table and log
A corresponding event title；Topic is sent to the topic queue of data cluster for the specified reported data collected.Meanwhile
When being configured to the configuration field in custom protocol, each reported event need to add at least one field definition, each field
Include field name, data type and remark information.Field name is the title of each field, for distinguishing different column, than
Such as, field name can be name, age etc.；Data type can be text, character string, number, character etc.；And remark information
It is just the Chinese of each field name, for example, corresponding remark information is just name when field name is name.
Wherein, target data is data information obtained from reported data is parsed and reconfigured, data letter
Breath can be JSON standard data format or other types data format, for example, the data information may be text formatting,
It is not limited thereto, as long as the data format of the target data can be identified by Hive table.In embodiment,
When target data is text formatting, then it can distinguish and come using separator between each field.Specifically, data processing
End passes through application programming interface (Application Programming Interface, API) interface to data terminal
The reported data of transmission parses the protocol header of custom protocol, therefrom extract in corresponding URI, and according to URI from data
All data informations in the corresponding configuration field of the custom protocol are obtained in library, and according to configuration field sequence, to upper count off
According to being parsed one by one, to obtain corresponding field contents, and according to JSON data format or other types data format to word
Section content is combined, with target data.
Specifically, when being customized management to custom protocol, the configuration field in custom protocol need to be configured.
Preferably, in the mode being configured to the configuration field in custom protocol by the way of user's input.Specifically, user's root
According to the customized demand of custom protocol configuration field is filled in, and after completing to fill in, by auditor to being filled in
Data information is audited, if after the approval, determining submission is clicked directly on, to generate custom protocol.
Target data is sent to data-base cluster by step 230.
In general, data-base cluster, is exactly to constitute a virtual list using at least two or more database servers
One data base logic image, and as single database system, transparent data service is provided to client.In embodiment
In, data-base cluster can be understood as the database server for storing target data.Preferably, data-base cluster includes
Kafka cluster, flume process and Hadoop cluster.Specifically, after obtaining target data, target data is sent to
In kafka cluster, and the target data in kafka cluster is uploaded in real time in Hadoop cluster in real time by flume process,
It is created in Hadoop cluster or updates Hive table, to provide the statistical query of data.Wherein, flume is a kind of log earliest
Collection system, can handle the daily record data of various data type/data formats, and according to the data transfer mode of definition by mesh
Mark data are transmitted in Hadoop cluster；Wherein, Hive is a data warehouse handling implement based on Hadoop, can pass through class
Structured query language (Structured Query Language, SQL) is realized to the number of targets stored in Hadoop cluster
According to inquiry.
Relevant configuration field in agreement is passed through web front-end using general protocol specification by the technical solution of the present embodiment
Being customized management, and reported data is solved one by one according to custom protocol in the reported data for receiving data terminal
Analysis obtains target data, and reports to data cluster, realizes without modifying to the bottom code of data processing end, just
Can receive reported data, to improve the access efficiency of reported data, and by custom protocol to reported data into
Row reports, and ensure that the accuracy of reported data.
On the basis of the above embodiments, Fig. 3 is that the embodiment of the invention provides the processes of another data processing method
Figure.Referring to Fig. 3, which specifically comprises the following steps:
Step 310, the configuration field that custom protocol is obtained by target web interface.
Wherein, target web interface is the software interface that can be checked under browser, is integrated in data processing end
Web front-end.Specifically, when user configures custom protocol, the browser of specified network address, and access target are opened
Web interface, to obtain the configuration field of custom protocol.User can according to customized demand to the corresponding the respective option of configuration field into
Row is filled in, to obtain corresponding custom protocol.Illustratively, Fig. 4 is a kind of target web interface provided in an embodiment of the present invention
Display schematic diagram.With reference to Fig. 4, shown in target web interface event title, URI, topic that configuration field included, on
The essential informations such as report mode, field name, data type and remark information, while on the right side of each configuration field corresponding one
A Option Box, user can directly fill in corresponding data information in Option Box, can also be by drop-down menu to configuration field
It is configured.For example, when filling in URI, due to the corresponding URI of each reported event be it is unique and different, then user can be straight
It connects and fills in corresponding character string in the right side Option Box of URI；When reporting schemes only have HTTP reporting schemes and TCP to report
When mode, then user can click drop-down menu (for example, as shown in Figure 4 three by corresponding Option Box directly on the right side of reporting schemes
Angular corresponding drop-down menu), to select corresponding reporting schemes.
It should be noted that non-limiting as example, have included in configuration field event title, URI,
The information such as topic, reporting schemes, field name, data type and remark information, certainly, user can also be according to customized demand pair
The corresponding bottom code of configuration information in configuration field is modified, in order to it is subsequent can be according to the configuration words after remodifying
Section, fills in the configuration information of configuration field in target web interface.In the present embodiment, it only illustratively uses
Web front-end carries out the mode that choosing is filled out and configures to configuration field, and saves into database, and also can according to need directly makes
Use the mode of configuration file as the preserving type to configuration field.
Step 320 obtains agreement generation instruction, generates custom protocol according to configuration field.
Wherein, agreement generates instruction, it can be understood as the valid instruction of custom protocol.Specifically, in user according to customization
When demand completes the setting to configuration field, system manager need to audit the configuration information of the custom protocol, if configuration
Information checks errorless and is when meeting the corresponding configuration rule of configuration field, then it is assumed that the custom protocol is legal, user
It can trigger submitting button, so that data processing end gets agreement and generates instruction.Meanwhile data processing end is receiving agreement life
When at instruction, the corresponding configuration information of configuration field filled according to user generates custom protocol.As shown in figure 4, in target
The lower right corner of web interface is provided with " submission " button, is exactly that agreement generates instruction when which is triggered.
It should be noted that only having audit permission when the configuration information to custom protocol is audited
System manager can just audit configuration information.It is auditing simultaneously, data processing end can generate one to deployment task
Record, to carry out relevant deployment operation according to the record.
Step 330 receives the reported data from data terminal.
Wherein, reported data includes uniform resource identifier URI and data type.
Step 340, the URI for extracting reported data.
In embodiment, reported data is received by api interface, and obtains the data content of custom protocol.Wherein, for
HTTP reporting schemes remove HTTP head, and the remaining part body is just the corresponding data content of custom protocol；For on TCP
Report mode removes TCP and establishes or disconnect etc. except information, and remainder data is the corresponding data content of custom protocol.To fixed
The protocol header of agreement processed is parsed, to identify the URI for extracting reported data.Specifically, the protocol header to custom protocol carries out
Parsing, to obtain corresponding URI.Wherein, preceding 32 data are data packet length mark in protocol header, and 32 are URI later, then
32 are return code information later, then the corresponding URI of the reported event can be directly extracted from protocol header, wherein Mei Geshang
Report event corresponds to a URI.
Step 350 searches the configuration field in corresponding custom protocol according to URI.
Specifically, after obtaining the corresponding URI of the reported event, data processing end is known from database according to URI
Corresponding custom protocol is not obtained, and obtains the corresponding configuration information of configuration field in custom protocol.
Step 360 parses reported data according to configuration field sequence, obtains target data.
It should be noted that being saved in the database at data processing end before being parsed to reported data
There is the basic field type being often used in reported data, the basis of parsing operation is customized with the support that data processing end has been realized
Field type is as foundation, while the web front-end for displaying target web interface provides the typing of relevant basic field type
And maintenance function.It is non-limiting as example, the corresponding meaning of field type basic in data processing end is illustrated.
For example, typonym is uintA_t, A unsigned ints are indicated, specifically, when typonym is uint8_t, then it represents that 8
Position unsigned int；When typonym is uint16_t, then it represents that 16 unsigned ints, in general, the value of A is 8,
16,32 or 64；For another example, if typonym is std::string, then it represents that character string；For another example, if typonym is std::
Vector<std::string>, then it represents that the array container containing character string.Certainly, in embodiment, to used type
Title is not defined.In embodiment, after getting the corresponding configuration information of configuration field, according to configuration words number of segment
It is parsed, also is understood as one by one according to reported data, be exactly to upper count off to the process that reported data is parsed one by one
According to the process for carrying out unserializing, then the data that unserializing obtains are reconfigured again, to obtain target data.Together
When, the corresponding relationship between the typonym of preset reported data and Hive table map type can be used to reported data
Carry out dissection process.It is non-limiting as example, when typonym is uintA_t, then corresponding Hive table map type institute
It indicates, A data is taken from data packet, be converted to no symbol A integers as output, specifically, working as typonym
When for uint32_t, then represented by Hive table map type, 31 data are taken from data packet, are converted to no symbol 32
Integer is as output, and in general, the value of A can be 8,16,32 or 64；For another example, when typonym be std::string, then
Represented by corresponding Hive table map type, length of preceding 16 data as character string is first taken from data packet, then
The data content of the length is taken to change into string data as output.Certainly, only illustratively to type name in the present embodiment
Claim the corresponding relationship between Hvie table map type to carry out citing description, is not limited thereto.
Target data is sent to Hadoop cluster by flume process by step 370.
Specifically, the step specifically includes step 371- step 373:
Target data is sent to kafka cluster by step 371.
Specifically, reported data is being subjected to unserializing to obtain the mesh of JSON data format or other data formats
After marking data, target data is directly sent to kafka cluster, and be stored temporarily in kafka cluster.
Step 372, long-range starting flume process.
In embodiment, while configuration information of the system manager to custom protocol is audited, data processing end
Can automatically generate and be recorded to deployment task, and is stored in the database at data processing end, at the same data processing end according to this to
The long-range starting flume process of deployment task record, to be uploaded to the target data in kafka cluster by flume process
In Hadoop cluster.
Target data is sent to Hadoop cluster by flume process by step 373, kafka cluster.
Specifically, being transferred from kafka cluster after starting flume process according to the corresponding URI of different reported events
Corresponding target data, target data is sent to Hadoop cluster by flume process.
It should be noted that then will when the Hive table in Hadoop cluster includes the data type in target data
Target data is stored in corresponding Hive table；When the Hive table in Hadoop cluster does not include the data type in target data, then
Newly-built Hive table, and the Hive table that target data deposit is newly-built.Specifically, will be in the target data in kafka cluster
When reaching Hadoop cluster, judge in the Hive table in Hadoop cluster whether to include the mesh according to the data type of target data
All data types in data are marked, if in Hive table including all data types in the target data, direct basis
The modification of Hive table orders corresponding Hive table to be modified；If in Hive table not including all numbers in the target data
According to type, then need according to the transformational relation table between typonym and Hive table map type, to all numbers in target data
It is converted according to type, to obtain Hive table map type, composition builds table statement and executes the order of newly-built Hive table.Wherein,
Corresponding transformational relation between the typonym of target data and Hvie table map type can be used and carry out conversion process.As showing
Example and it is non-limiting, when typonym be uintA_t when, then corresponding Hive table map type be int, wherein A be 8 or 16；When
When typonym is uint32_t, corresponding Hive table map type is long；And when typonym is uint64_t, then it is right
The Hive table map type answered is string；When typonym is std::vector<std::string>；It is then corresponding
Hive table map type is array<string>.Certainly, in the present embodiment, used typonym and Hive table are reflected
The transformational relation penetrated between type is not defined, and can limit typonym and Hive table map type according to the actual situation
Between transformational relation.The technical solution of the present embodiment is further advanced by web front-end based on the above technical solution
Management is customized to the configuration field of custom protocol, and after data processing termination receives reported data, is assisted according to customization
The configuration field of view parses reported data one by one, the corresponding number of targets of data format that composition Hive table can identify
According to, and target data is uploaded to Hadoop cluster, reach target data in the available purpose of Hive table.
Fig. 5 is a kind of flow chart of data reporting method provided in an embodiment of the present invention.The data provided in the present embodiment
Report method can be executed by data processing equipment, which can be realized by way of software and/or hardware,
The data processing equipment can be two or more physical entities and constitute, and is also possible to a physical entity and constitutes.The present embodiment
Middle data processing equipment is that data terminal to obtain reported data, and reported data is sent out for serializing initial data
It send to data processing end.
Step 410 obtains initial data.
Wherein, initial data includes uniform resource identifier URI.
In embodiment, the bottom code that initial data is write by developer in data terminal.Specifically, exploit person
Member completes the custom protocol pair in custom protocol after the configuration information of configuration field, passed through according to audit in target web interface
The configuration information answered carries out the code development of cell phone client or background server, and will write the code of completion as original number
According to.
Step 420 encapsulates initial data according to custom protocol, obtains the reported data of standard sequence.
Wherein, custom protocol is corresponding with URI, and the configuration field in custom protocol is inputted by user.
Specifically, after obtaining initial data, sequence is carried out to initial data according to the configuration field of custom protocol
Change, to convert raw data into the reported data that can store or transmit.Wherein, serializing is exactly by the status information of object
Be converted to the process for the form that can store or transmit.
Reported data is sent to data processing end by step 430.
In embodiment, after obtaining the reported data of standard sequence, data terminal is according to customized in custom protocol
Reporting schemes reported data is sent to data processing end.In general, when data terminal is cell phone client, using HTTP
Reporting schemes carry out reporting for data；And when data terminal is background server, the upper of data is carried out using TCP reporting schemes
Report.
The technical solution of this implementation serializes initial data by custom protocol, to obtain standard sequence
Reported data, and reported data is sent to data processing end by the reporting schemes in custom protocol, thus using general side
Formula carries out data and reports, and realizes standardization and systematization that data report, and then put on the access efficiency of reported data.
Fig. 6 is the flow chart of another data processing method provided in an embodiment of the present invention, is applied to data processing system.
With reference to Fig. 6, data processing method provided in an embodiment of the present invention is specifically included:
Step 510, the configuration field that custom protocol is obtained by the target web interface at data processing end.
It in embodiment, need to be according to customized demand in number before reported data is sent to data processing end by data terminal
It is configured accordingly according to configuration field of the processing end to custom protocol.
Step 520, data processing end obtain agreement and generate instruction, generate custom protocol according to configuration field.
Specifically, after the configuration field for completing custom protocol corresponds to the filling in of configuration information, to the configuration information into
Row audit, and after the approval, directly confirmation is submitted, to generate custom protocol.
Step 530, data terminal obtain initial data.
Wherein, which includes URI.
Specifically, in user after the configuration that target web interface completes custom protocol, pass through mobile phone visitor in data terminal
Family end or background server, and corresponding code development is carried out according to the corresponding configuration information of the configuration field of custom protocol, with
The URI of the code developed and custom protocol is corresponded.
Step 540, data terminal encapsulate initial data according to custom protocol, obtain the reported data of standard sequence.
Wherein, the custom protocol is corresponding with URI, and the configuration field in the custom protocol is inputted by user.
In embodiment, initial data is serialized, to obtain the reported data of standard sequence, wherein report
The corresponding reported event of URI in data.
Reported data is sent to data processing end by step 550, data terminal.
Specifically, reported data is sent to data processing end by the reporting schemes in custom protocol, so that at data
It manages end and dissection process is carried out to reported data.
Step 560, data processing end parse reported data according to custom protocol, obtain target data.
Wherein, custom protocol is corresponding with URI, and the configuration field in custom protocol is inputted by user.Specifically, in number
After receiving reported data according to processing end, identification extracts URI in reported data, and according to URI from data processing end
Corresponding custom protocol is searched in database, to be solved one by one according to the configuration field of custom protocol sequence to reported data
Analysis, to obtain the corresponding data content of reported data, and reconfigures data content, to obtain target data.
Target data is sent to data-base cluster by step 570.
After obtaining target data, target data is directly uploaded to kafka cluster, and by remotely starting
Target data in kafka cluster is uploaded to Hadoop cluster by flume process in real time.
Step 580, data-base cluster execute according to the data type in target data and create or modify corresponding Hive
Table.
Specifically, if the Hive table in Hadoop cluster includes the data type in target data, by target data
It is stored in corresponding Hive table；If the Hive table in Hadoop cluster does not include the data type in the target data, create
Hive table, and the Hive table that target data deposit is newly-built.
The technical solution of the present embodiment carries out reporting for data using generic way, and supports the mode of customizable
It is accessed, so that the reception that data report more standardizes and systematization, realizes dynamic autoization, be obviously improved reported data
Access efficiency, for subsequent data receiver monitoring optimization etc. functions lay the foundation.
Fig. 7 is a kind of structural block diagram of data processing equipment provided in an embodiment of the present invention.With reference to Fig. 7, the data processing
Device includes: receiving module 610, parsing module 620 and the first sending module 630.
Wherein, receiving module 610, for receiving the reported data from data terminal, which includes unified resource
Identifiers, URIs and data type；
Parsing module 620, for according to custom protocol parse reported data, obtain target data, wherein custom protocol with
URI is corresponding, and the configuration field in the custom protocol is inputted by user；
First sending module 630, for target data to be sent to data-base cluster.
Relevant configuration field in agreement is passed through web using general protocol specification by technical solution provided in this embodiment
The management of being customized of front end, and reported data is carried out one by one according to custom protocol in the reported data for receiving data terminal
Parsing, the target data that composition Hive can be identified, and data-base cluster is reported to, it realizes without the bottom to data processing end
Layer identification code is modified, so that it may be received reported data, to improve the access efficiency of reported data, and be passed through customization
Agreement reports reported data, ensure that the accuracy of reported data.
Further, the data processing equipment, further includes:
Module is obtained, for obtaining customization association by target web interface before receiving the reported data from data terminal
The configuration field of view；
Generation module generates instruction for obtaining agreement, generates custom protocol according to the configuration field.
Further, parsing module includes:
Extraction unit, for extracting the URI of reported data；
Searching unit, for searching the configuration field in corresponding custom protocol according to URI；
Resolution unit obtains target data for parsing the reported data according to configuration field sequence.
Further, the target data is JSON data format.
Further, the first sending module includes:
Transmission unit, for target data to be sent to Hadoop cluster by flume process.
Further, the transmission unit includes:
Transmission sub-unit, for target data to be sent to kafka cluster；
Promoter unit, for remotely starting flume process；
Target data is sent to Hadoop cluster by flume process for kafka cluster by transmission sub-unit.
It is further, described to be sent to Hadoop cluster, comprising:
Hive table in Hadoop cluster includes the data type in target data, then is stored in target data corresponding
Hive table；
Hive table in Hadoop cluster does not include the data type in target data, then creates Hive table, and by target
The newly-built Hive table of data deposit.
Further, configuration field includes at least one of following: URI, event title, topic, reporting schemes, field name
Title, data type and remark information；The reporting schemes include hypertext transfer protocol HTTP reporting schemes and transmission control protocol
TCP reporting schemes.
Data processing method provided by any embodiment of the invention can be performed in above-mentioned data processing equipment, has the side of execution
The corresponding functional module of method and beneficial effect.
Fig. 8 is a kind of structural block diagram of data reporting device provided in an embodiment of the present invention.With reference to Fig. 8, which is reported
Device includes: to obtain module 710, package module 720 and the second sending module 730.
Wherein, module 710 is obtained, for obtaining initial data, which includes uniform resource identifier URI；
Package module 720 obtains the reported data of standard sequence, is somebody's turn to do for encapsulating initial data according to custom protocol
Custom protocol is corresponding with URI, and the configuration field in the custom protocol is inputted by user；
Second sending module 730, for reported data to be sent to data processing end.
Data reporting method provided by any embodiment of the invention can be performed in above-mentioned data reporting device, has the side of execution
The corresponding functional module of method and beneficial effect.
On the basis of the above embodiments, Fig. 9 is the structure of another data processing system provided in an embodiment of the present invention
Block diagram.Referring to Fig. 9, which is specifically included: data terminal 810, data processing end 820 and data-base cluster 830.
Wherein, data terminal 810 are sent to data processing end 820 for generating reported data, and by reported data；Data
Processing end 820 for receiving reported data, and carries out dissection process to reported data and obtains target data, and by target data
It is sent to data-base cluster 830；Data-base cluster 830, for receiving and storing target data.
Specifically, data terminal 810 includes: cell phone client 8101 and/or background server 8102；Cell phone client
8101 for being sent to background server 8202 for reported data by HTTP reporting schemes；Background server 8102 is for passing through
Reported data is sent to background server 8202 by TCP reporting schemes.
Data processing end includes: web front-end 8201 and background server 8202；Web front-end 8201 is used for displaying target web
Interface, so that user carries out filling in for corresponding configuration information in configuration field of the target web interface to custom protocol；Background service
Device 8202 is used to carry out dissection process to reported data, to obtain the target data of the identifiable data format of Hive table, and will
Target data is sent in kafka cluster 8301 and is stored.
Data-base cluster 830 includes: kafka cluster 8301, flume cluster 8302 and Hadoop cluster 8303.Kafka collection
Group 8301 for receiving target data, and the flume process by being run in flume cluster 8302 by target data it is real-time on
Reach Hadoop cluster 8303.It should be noted that kafka cluster 8301, flume cluster 8302 and Hadoop cluster
8303 be background server, but it is database server.
On the basis of the above embodiments, Figure 10 is the structure of another data processing system provided in an embodiment of the present invention
Block diagram.With reference to Figure 10, which is specifically included: data terminal 910, data processing end 920 and data-base cluster 930.
Wherein, data terminal 910 includes: cell phone client 9101 and background server 9102；Data processing end 920 includes:
Web front-end 9201 and background server 9202；Wherein, web front-end 9201 includes: type management module and event manager module；
Background server 9202 includes: automatic deployment service module and data receiver service module.Specifically, 9201 He of web front-end
By database as medium between background server 9202, wherein in the configuration words of 9201 typing custom protocol of web front-end
Section, to generate corresponding custom protocol, and custom protocol is saved into database, then 9202 timing scan of background server
Database, to load deployment task caused by automatic deployment service module, and after generating deployment task, automatic deployment clothes
Business module remotely starts flume process.Meanwhile after reported data is sent to data processing end 920 by data terminal 910, data
Data reception service module in processing end 920 receives reported data, and according to the protocol format of custom protocol to reported data
It is parsed, to form the target data of the identifiable data format of Hive table, and target data is sent to kafka cluster
9301.After kafka cluster 9301 receives target data, can by the flume process that is started in flume cluster 9302,
Target data in kafka cluster 9301 is uploaded to Hadoop cluster 9303 in real time, and executes creation Hive table, to be counted
According to statistical query.
Figure 11 is a kind of structural schematic diagram of data processing equipment provided in an embodiment of the present invention.With reference to Figure 11, the data
Processing equipment includes: processor 1010, memory 1020, input unit 1030 and output device 1040.The data processing is set
The quantity of standby middle processor 1010 can be one or more, in Figure 11 by taking a processor 1010 as an example.The data processing
The quantity of memory 1020 can be one or more in equipment, in Figure 11 by taking a memory 1020 as an example.At the data
Manage equipment processor 1010, memory 1020, input unit 1030 and output device 1040 can by bus or its
He connects mode, in Figure 11 for being connected by bus.In embodiment, data processing equipment can be computer and plate
Deng.
Memory 1020 is used as a kind of computer readable storage medium, and it is executable to can be used for storing software program, computer
Program and module, the corresponding program instruction/module of data processing equipment as described in any embodiment of that present invention is (for example, number
According to receiving module 610, parsing module 620 and the first sending module 630 in processing unit).Memory 1020 can be wrapped mainly
Include storing program area and storage data area, wherein storing program area can application needed for storage program area, at least one function
Program；Storage data area, which can be stored, uses created data etc. according to equipment.In addition, memory 1020 may include high speed
Random access memory, can also include nonvolatile memory, a for example, at least disk memory, flush memory device or
Other non-volatile solid state memory parts.In some instances, memory 1020 can further comprise relative to processor 1010
Remotely located memory, these remote memories can pass through network connection to equipment.The example of above-mentioned network includes but not
It is limited to internet, intranet, local area network, mobile radio communication and combinations thereof.
Input unit 1030 can be used for receiving the number or character information of input, and generate and data processing equipment
User setting and the related key signals input of function control, can also be the camera for obtaining image and obtain audio
The pick up facility of data.Output device 1040 may include the audio frequency apparatuses such as loudspeaker.It should be noted that input unit 1030
It may be set according to actual conditions with the concrete composition of output device 1040.
Software program, instruction and the module that processor 1010 is stored in memory 1020 by operation, thereby executing
The various function application and data processing of equipment, that is, realize above-mentioned data processing method.
The data processing equipment of above-mentioned offer can be used for executing that above-mentioned any embodiment provides is applied to data processing end
Data processing method, have corresponding function and beneficial effect.
In addition, hardware configuration can be found in interior when data reporting equipment is data terminal when equipment is data reporting equipment
Hold and explains.It should be noted that when data reporting equipment is data terminal, the program that stores in corresponding memory 1020 can be with
It is that the embodiment of the present invention provides the corresponding program instruction/module of data reporting method applied to data terminal, processor 1010
By running the software program, instruction and the module that are stored in memory 1020, thereby executing the various function of computer equipment
It can apply and data processing, i.e., be applied to the data reporting method of data terminal in realization above method embodiment.It is understood that
, when above-mentioned data reporting equipment is data terminal, can be performed provided by any embodiment of the invention applied to data terminal
Data reporting method, and have corresponding function and beneficial effect.
The embodiment of the present invention also provides a kind of storage medium comprising computer executable instructions, and the computer is executable
Instruction by computer processor when being executed for executing a kind of data processing method, comprising:
The reported data from data terminal is received, which includes uniform resource identifier URI and data type；
Reported data is parsed according to custom protocol, obtains target data, wherein custom protocol is corresponding with URI, customization association
Configuration field in view is inputted by user；
Target data is sent to data-base cluster.
Certainly, a kind of storage medium comprising computer executable instructions, computer provided by the embodiment of the present invention
The data processing method operation that executable instruction is not limited to the described above, can also be performed provided by any embodiment of the invention
Relevant operation in data processing method, and have corresponding function and beneficial effect.
By the description above with respect to embodiment, it is apparent to those skilled in the art that, the present invention
It can be realized by software and required common hardware, naturally it is also possible to which by hardware realization, but in many cases, the former is more
Good embodiment.Based on this understanding, technical solution of the present invention substantially in other words contributes to the prior art
Part can be embodied in the form of software products, which can store in computer readable storage medium
In, floppy disk, read-only memory (Read-Only Memory, ROM), random access memory (Random such as computer
Access Memory, RAM), flash memory (FLASH), hard disk or CD etc., including some instructions are with so that a computer is set
Standby (can be robot, personal computer, server or the network equipment etc.) executes number described in any embodiment of that present invention
According to processing method.
It is worth noting that, included each unit and module are only patrolled according to function in above-mentioned data processing equipment
It volume is divided, but is not limited to the above division, as long as corresponding functions can be realized；In addition, each function list
The specific name of member is also only for convenience of distinguishing each other, the protection scope being not intended to restrict the invention.
The embodiment of the invention also provides another computer readable storage medium, the computer executable instructions by
For executing a kind of data reporting method applied to data terminal when computer processor executes, this method comprises: obtaining original
Data, the initial data include uniform resource identifier URI；Initial data is encapsulated according to custom protocol, obtains standard sequence
Reported data, the custom protocol is corresponding with URI, and the configuration field in custom protocol is inputted by user；Reported data is sent out
Give data processing end.
It can be found in the content in above-described embodiment to the introduction of storage medium to explain.
It should be appreciated that each section of the invention can be realized with hardware, software, firmware or their combination.Above-mentioned
In embodiment, software that multiple steps or method can be executed in memory and by suitable instruction execution system with storage
Or firmware is realized.It, and in another embodiment, can be under well known in the art for example, if realized with hardware
Any one of column technology or their combination are realized: having a logic gates for realizing logic function to data-signal
Discrete logic, with suitable combinational logic gate circuit specific integrated circuit, programmable gate array (PGA), scene
Programmable gate array (FPGA) etc..
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
Centainly refer to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be any
One or more embodiment or examples in can be combined in any suitable manner.
Note that the above is only a better embodiment of the present invention and the applied technical principle.It will be appreciated by those skilled in the art that
The invention is not limited to the specific embodiments described herein, be able to carry out for a person skilled in the art it is various it is apparent variation,
It readjusts and substitutes without departing from protection scope of the present invention.Therefore, although being carried out by above embodiments to the present invention
It is described in further detail, but the present invention is not limited to the above embodiments only, without departing from the inventive concept, also
It may include more other equivalent embodiments, and the scope of the invention is determined by the scope of the appended claims.
Claims (14)
1. a kind of data processing method characterized by comprising
The reported data from data terminal is received, the reported data includes uniform resource identifier URI and data type；
The reported data is parsed according to custom protocol, obtains target data, wherein the custom protocol is opposite with the URI
It answers, the configuration field in the custom protocol is inputted by user；
The target data is sent to data-base cluster.
2. the method according to claim 1, wherein before the reported data of the reception from data terminal,
Further include:
The configuration field of custom protocol is obtained by target web interface；
It obtains agreement and generates instruction, custom protocol is generated according to the configuration field.
3. according to the method described in claim 2, obtaining it is characterized in that, described parse the reported data according to custom protocol
To target data, comprising:
Extract the URI of reported data；
The configuration field in corresponding custom protocol is searched according to the URI；
The reported data is parsed according to the configuration field sequence, obtains target data.
4. according to the method described in claim 3, it is characterized in that, the target data is JSON data format.
5. according to the method described in claim 4, it is characterized in that, described be sent to data-base cluster for the target data,
Include:
The target data is sent to Hadoop cluster by flume process.
6. according to the method described in claim 5, it is characterized in that, described sent the target data by flume process
To Hadoop cluster, comprising:
The target data is sent to kafka cluster；
Long-range starting flume process；
The target data is sent to Hadoop cluster by the flume process by the kafka cluster.
7. method according to claim 5 or 6, which is characterized in that described to be sent to Hadoop cluster, comprising:
Hive table in the Hadoop cluster includes the data type in the target data, then is stored in the target data
Corresponding Hive table；
Hive table in the Hadoop cluster does not include the data type in the target data, then creates Hive table, and will
The newly-built Hive table of the target data deposit.
8. method according to any one of claims 1 to 7, which is characterized in that the configuration field include at least it is following it
One: URI, event title, topic, reporting schemes, field name, data type and remark information；The reporting schemes include super
Text transfer protocol HTTP reporting schemes and transmission control protocol TCP reporting schemes.
9. a kind of data reporting method characterized by comprising
Initial data is obtained, the initial data includes uniform resource identifier URI；
Encapsulate the initial data according to custom protocol, obtain the reported data of standard sequence, the custom protocol with it is described
URI is corresponding, and the configuration field in the custom protocol is inputted by user；
The reported data is sent to data processing end.
10. a kind of data processing equipment characterized by comprising
Receiving module, for receiving the reported data from data terminal, the reported data include uniform resource identifier URI and
Data type；
Parsing module obtains target data, wherein the custom protocol for parsing the reported data according to custom protocol
Corresponding with the URI, the configuration field in the custom protocol is inputted by user；
First sending module, for the target data to be sent to data-base cluster.
11. a kind of data reporting device characterized by comprising
Module is obtained, for obtaining initial data, the initial data includes uniform resource identifier URI；
Package module, it is described fixed for obtaining the reported data of standard sequence according to the custom protocol encapsulation initial data
Agreement processed is corresponding with the URI, and the configuration field in the custom protocol is inputted by user；
Second sending module, for the reported data to be sent to data processing end.
12. a kind of data processing system characterized by comprising data terminal, data processing end and data-base cluster；
The data terminal is sent to the data processing end for generating reported data, and by reported data；
The data processing end for receiving the reported data, and carries out dissection process to the reported data and obtains target
Data, and target data is sent to the data-base cluster；
The data-base cluster, for receiving and storing the target data.
13. a kind of data processing equipment characterized by comprising memory and one or more processors；
The memory, for storing one or more programs；
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
Existing data processing method a method as claimed in any one of claims 1-8.
14. a kind of storage medium comprising computer executable instructions, which is characterized in that the computer executable instructions by
For executing data processing method a method as claimed in any one of claims 1-8 when computer processor executes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811214237.4A CN109547234A (en) | 2018-10-18 | 2018-10-18 | A kind of data processing, report method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811214237.4A CN109547234A (en) | 2018-10-18 | 2018-10-18 | A kind of data processing, report method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109547234A true CN109547234A (en) | 2019-03-29 |
Family
ID=65843915
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811214237.4A Pending CN109547234A (en) | 2018-10-18 | 2018-10-18 | A kind of data processing, report method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109547234A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110753050A (en) * | 2019-10-22 | 2020-02-04 | 网易（杭州）网络有限公司 | Method and device for generating protocol document, computer storage medium and electronic equipment |
| CN111177246A (en) * | 2019-12-27 | 2020-05-19 | 腾讯云计算（北京）有限责任公司 | Service data processing method and device |
| CN111752803A (en) * | 2020-06-28 | 2020-10-09 | 厦门美柚股份有限公司 | Method, device and medium for collecting and reporting buried point data |
| CN111782467A (en) * | 2020-06-28 | 2020-10-16 | 厦门美柚股份有限公司 | Method, device and medium for collecting and reporting buried point data |
| CN112579401A (en) * | 2019-09-30 | 2021-03-30 | 北京国双科技有限公司 | Data analysis method and device, storage medium and electronic equipment |
| CN113810340A (en) * | 2020-06-12 | 2021-12-17 | 腾讯科技（深圳）有限公司 | Data reporting method, device, equipment and computer readable storage medium |
| CN114553479A (en) * | 2022-01-13 | 2022-05-27 | 广东化一环境科技有限公司 | Protocol receiving and processing system, method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102546557A (en) * | 2010-12-17 | 2012-07-04 | 北京中水科水电科技开发有限公司 | Method and device for customizing protocol |
| US20160321308A1 (en) * | 2015-05-01 | 2016-11-03 | Ebay Inc. | Constructing a data adaptor in an enterprise server data ingestion environment |
| CN106470123A (en) * | 2016-09-29 | 2017-03-01 | 北京三快在线科技有限公司 | Log collecting method, client, server and electronic equipment |
| CN107818120A (en) * | 2016-09-14 | 2018-03-20 | 博雅网络游戏开发（深圳）有限公司 | Data processing method and device based on big data |
| CN108040040A (en) * | 2017-11-30 | 2018-05-15 | 北京锐安科技有限公司 | A kind of automation analysis method and device of application protocol message |
-
2018
- 2018-10-18 CN CN201811214237.4A patent/CN109547234A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102546557A (en) * | 2010-12-17 | 2012-07-04 | 北京中水科水电科技开发有限公司 | Method and device for customizing protocol |
| US20160321308A1 (en) * | 2015-05-01 | 2016-11-03 | Ebay Inc. | Constructing a data adaptor in an enterprise server data ingestion environment |
| CN107818120A (en) * | 2016-09-14 | 2018-03-20 | 博雅网络游戏开发（深圳）有限公司 | Data processing method and device based on big data |
| CN106470123A (en) * | 2016-09-29 | 2017-03-01 | 北京三快在线科技有限公司 | Log collecting method, client, server and electronic equipment |
| CN108040040A (en) * | 2017-11-30 | 2018-05-15 | 北京锐安科技有限公司 | A kind of automation analysis method and device of application protocol message |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579401A (en) * | 2019-09-30 | 2021-03-30 | 北京国双科技有限公司 | Data analysis method and device, storage medium and electronic equipment |
| CN110753050A (en) * | 2019-10-22 | 2020-02-04 | 网易（杭州）网络有限公司 | Method and device for generating protocol document, computer storage medium and electronic equipment |
| CN110753050B (en) * | 2019-10-22 | 2022-03-04 | 网易（杭州）网络有限公司 | Method and device for generating protocol document, computer storage medium and electronic equipment |
| CN111177246A (en) * | 2019-12-27 | 2020-05-19 | 腾讯云计算（北京）有限责任公司 | Service data processing method and device |
| CN113810340A (en) * | 2020-06-12 | 2021-12-17 | 腾讯科技（深圳）有限公司 | Data reporting method, device, equipment and computer readable storage medium |
| CN113810340B (en) * | 2020-06-12 | 2024-04-09 | 腾讯科技（深圳）有限公司 | Data reporting method, device, equipment and computer readable storage medium |
| CN111752803A (en) * | 2020-06-28 | 2020-10-09 | 厦门美柚股份有限公司 | Method, device and medium for collecting and reporting buried point data |
| CN111782467A (en) * | 2020-06-28 | 2020-10-16 | 厦门美柚股份有限公司 | Method, device and medium for collecting and reporting buried point data |
| CN114553479A (en) * | 2022-01-13 | 2022-05-27 | 广东化一环境科技有限公司 | Protocol receiving and processing system, method and device |
| CN114553479B (en) * | 2022-01-13 | 2024-01-02 | 广东化一环境科技有限公司 | Protocol receiving and processing system, method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109547234A (en) | A kind of data processing, report method, device, equipment and storage medium | |
| CN104965691B (en) | The method, apparatus and system of the page elements of the configuration webpage page | |
| CN109302522A (en) | Test method, device and computer system and medium | |
| US10997531B2 (en) | System, method and graphical user interface for workflow generation, deployment and/or execution | |
| CN104298604B (en) | Cloud service robustness testing system and method for testing | |
| CN107704395A (en) | One kind is based on cloud platform automatic test implementation and system under Openstack | |
| CN106357457B (en) | A kind of alarm test method, apparatus and system | |
| CN108984388A (en) | A kind of method and terminal device generating automatic test cases | |
| CN108964968A (en) | Service access management method and system under a kind of container cloud environment | |
| EP4161015A1 (en) | Network management system, method, and apparatus, and electronic device | |
| CN101145163A (en) | Method and system for obtaining data from a plurality of data pool | |
| CN110321154A (en) | A kind of interface message methods of exhibiting, device and the electronic equipment of micro services | |
| CN111694561A (en) | Interface management method, device, equipment and storage medium | |
| CN114816499A (en) | Electronic resource sharing method and device and storage medium | |
| CN111752820B (en) | gRPC interface pressure test method, computer equipment and storage medium | |
| CN114356704A (en) | Software definition method and system for cloud network terminal streaming data | |
| CN110069321A (en) | Business containerization operation method, equipment, storage medium and device | |
| CN110457010B (en) | Method and device for batch changing task list based on collaboration platform | |
| CN106487855B (en) | File uploading method, file accessing method, file uploading device, file accessing device and equipment | |
| CN112559278B (en) | Method and device for acquiring operation data | |
| CN114253546A (en) | Code generation method and device, storage medium and electronic device | |
| CN109947476A (en) | A kind of statistical method and system in run time stack space | |
| CN112231109B (en) | Buffer resource management system, method and device | |
| CN109167826A (en) | The restocking method, apparatus and system of WEB application | |
| CN108984701A (en) | Data managing method and device in cloud data system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190329 |