CN106648912A - Modular method and apparatus for data processing in data acquisition platform - Google Patents
Modular method and apparatus for data processing in data acquisition platform Download PDFInfo
- Publication number
- CN106648912A CN106648912A CN201510729333.2A CN201510729333A CN106648912A CN 106648912 A CN106648912 A CN 106648912A CN 201510729333 A CN201510729333 A CN 201510729333A CN 106648912 A CN106648912 A CN 106648912A
- Authority
- CN
- China
- Prior art keywords
- data
- pending
- activation
- configuration information
- pending data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/541—Interprogram communication via adapters, e.g. between incompatible applications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a modular method and apparatus for data processing in a data acquisition platform, relates to the technical field of computers, and mainly aims to improve the flexibility of adding, modifying and deleting data acquisition indexes in the data acquisition platform. According to the main technical scheme, the method comprises the steps of acquiring to-be-processed data according to a data acquisition instruction, wherein the data acquisition instruction is generated by a server, the data acquisition instruction contains configuration information of the to-be-processed data, and the configuration information contains acquisition of a to-be-processed data quantity, a data type, and a data sending protocol; analyzing the data acquisition instruction and obtaining the configuration information of the to-be-processed data; and performing modular processing on the to-be-processed data based on the configuration information, wherein the modular processing comprises data formatting, addition of data to a data sending queue, and data sending. The method and the apparatus are mainly applied to the data acquisition process.
Description
Technical field
The present invention relates to field of computer technology, in more particularly to a kind of data acquisition platform at data
The modular method and device of reason.
Background technology
Data analysis is to organize purposefully collect data, data processing, analyze data, is made
The process of information.The premise of data analysis is data acquisition and data processing, in data acquisition
Data source may be different, and form is likely to difference, therefore when data acquisition is carried out, it is possible to use
The data acquisition software provided in prior art, for example, when video web page played data is gathered, can
With using collection video web page played data software development kit (Software Development Kit,
SDK)；When the user behavior data of mobile terminal is gathered, it is possible to use the user behavior data of mobile terminal
SDK etc..SDK used in data acquisition, there is specific source-information, specific
Data form, specific data transfer format, specific source letter will be respectively directed in data processing
Breath, specific data form, specific data transfer format carry out data processing.
When Usage data collection software carries out data acquisition, inventor has found it, and there are the following problems：
When the two kinds of data acquisition indexs of user behavior data for gathering video web page played data and mobile terminal simultaneously
When, need to dispose different data respectively to video web page played data and the user behavior of mobile terminal and adopt
Collection platform, could realize while collecting the user behavior data of video web page played data and mobile terminal；
Or, one collection movement of addition in a data acquisition platform for gathering video web page played data
The user behavior at end.But, addition, modification and deletion data acquisition in a data acquisition platform
Index, needs make more modification to existing code in the data acquisition platform, therefore, available data
In addition, modification and deletion data acquisition index, flexibility is relatively low in acquisition platform.
The content of the invention
In view of this, the modular method of data processing in a kind of data acquisition platform that the present invention is provided
And device, main purpose is to improve addition in data acquisition platform, modification and delete data acquisition to refer to
Target flexibility.
In order to solve the above problems, present invention generally provides following technical scheme：
On the one hand, the invention provides in a kind of data acquisition platform data processing modular method,
The method includes：
Receiving data acquisition instructions, and gather pending data according to the data acquisition instructions；Wherein,
The data acquisition instructions are generated by server, and pending data is included in the data acquisition instructions
Configuration information, the configuration information includes：Quantity, data type, the number of collection pending data
According to transmission agreement；
The data acquisition instructions are parsed, the configuration information of the pending data is obtained；
Modularized processing is carried out to the pending data based on the configuration information；Wherein, the mould
Blockization process includes data form, adds data is activation queue, data is activation.
On the other hand, the present invention also provides a kind of modular unit of data processing in data acquisition platform,
The device includes：
Receiving unit, for receiving data acquisition instructions, wherein, the data acquisition instructions are by servicing
Device is generated, and the configuration information comprising pending data in the data acquisition instructions, described with confidence
Breath is included：Quantity, data type, the data is activation agreement of collection pending data；
Collecting unit, the data acquisition instructions for being received according to the receiving unit are gathered to be waited to locate
Reason data；
Resolution unit, for parsing the data acquisition instructions that the receiving unit is received；
Acquiring unit, after parsing the data acquisition instructions in the resolution unit, obtains institute
State the configuration information of pending data；
Processing unit, for the configuration information based on acquiring unit acquisition to described pending
Data carry out modularized processing；Wherein, the modularized processing includes data form, adds data
Transmit queue, data is activation.
By above-mentioned technical proposal, the technical scheme that the present invention is provided at least has following advantages：
The modular method and device of data processing, receives number in the data acquisition platform that the present invention is provided
According to acquisition instructions, and according to data acquisition instructions collection pending data, the data acquisition instructions by
Server is generated, and the data acquisition instructions include the configuration information of pending data, parse the number
According to acquisition instructions, and the configuration information of pending data is obtained, based on the configuration information to pending number
According to carrying out modularized processing, wherein, the modularized processing includes：Data form, addition data are sent out
Send queue, data is activation；Compared with prior art, the present invention is by the configuration in data acquisition instructions
Information, configures pending data, and carries out modularized processing to pending data based on the configuration information,
Improve the flexibility for adding, change and delete data acquisition index based on a data acquisition platform.
Described above is only the general introduction of technical solution of the present invention, in order to better understand the present invention's
Technological means, and being practiced according to the content of specification, and in order to allow the above-mentioned of the present invention and
Other objects, features and advantages can become apparent, below especially exemplified by the specific embodiment of the present invention.
Description of the drawings
By the detailed description for reading hereafter preferred embodiment, various other advantage and benefit for
Those of ordinary skill in the art will be clear from understanding.Accompanying drawing is only used for illustrating the mesh of preferred embodiment
, and it is not considered as limitation of the present invention.And in whole accompanying drawing, with identical with reference to symbol
Number represent identical part.In the accompanying drawings：
Fig. 1 shows the module of data processing in a kind of data acquisition platform provided in an embodiment of the present invention
The flow chart of change method；
Fig. 2 shows a kind of schematic diagram of modularized processing provided in an embodiment of the present invention；
Fig. 3 shows one kind provided in an embodiment of the present invention based on the configuration information to described pending
Data carry out the flow chart of modularized processing method；
Fig. 4 shows the module of data processing in a kind of data acquisition platform provided in an embodiment of the present invention
The composition frame chart that makeup is put；
Fig. 5 shows the mould of data processing in another kind of data acquisition platform provided in an embodiment of the present invention
The composition frame that block makeup is put.
Specific embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although showing in accompanying drawing
The exemplary embodiment of the disclosure is shown, it being understood, however, that may be realized in various forms the disclosure
And should not be limited by embodiments set forth here.On the contrary, there is provided these embodiments are able to more
Thoroughly understand the disclosure, and can be by the complete technology for conveying to this area of the scope of the present disclosure
Personnel.
The embodiment of the present invention provides a kind of modular method of data processing in data acquisition platform, such as Fig. 1
Shown, the method includes：
101st, receiving data acquisition instructions, and gather pending data according to the data acquisition instructions.
In embodiments of the present invention, the acquisition logic that pending data is gathered in data acquisition platform is entered
Row is extracted, and the acquisition logic for extracting is made into an interface, and the function of the interface is：To collect
Pending data transfer to encapsulate after different Index modules, by those Index modules to pending data
Processed；Wherein, the Index module is process pending data original to data acquisition platform
Method carry out modularized encapsulation formed Index module.For in user level, when being connect based on this
Mouthful collection pending data after, without the need for being processed pending data can by gather it is pending
Data is activation is to server；For being realized above based on technology, after collection pending data, will treat
Processing data is transferred in different Index modules, by different Index modules respectively to pending data
Modularized processing is carried out, modularized processing includes：Data form, addition data is activation queue, number
According to operating process such as transmissions.
When pending data is gathered, pending data is gathered by preset data acquisition interface, default
When data acquisition interface collection is pending, the data acquisition instructions that the reception server sends, and according to this
Data acquisition instructions gather pending data, the configuration comprising pending data in the data acquisition instructions
Information, the configuration information is included：Quantity, data type, the data is activation of collection pending data
Agreement.
It is exemplary, it is assumed that the configuration information in data acquisition instructions is：Collection pending data
Quantity is 2, respectively vido play begin time and vido play end time, and this is pending
The data type of data is numeric type, then preset data acquisition interface only gathers vido play begin
The corresponding pending datas of time and vido play end time.
102nd, the data acquisition instructions are parsed, and obtains the configuration information of the pending data.
Data acquisition instructions are parsed, pending data is corresponding in acquisition data acquisition instructions matches somebody with somebody
Confidence ceases, it is intended that pending data is processed based on configuration information, meanwhile, inspection
Whether the pending data of collection is consistent with the content recorded in configuration information.
Used as a kind of implementation of the embodiment of the present invention, the configuration information is stored in form, is made
For another kind of implementation of the embodiment of the present invention, the configuration information is stored in file.The present invention
Embodiment is not defined to the storage form of configuration information.
103rd, modularized processing is carried out to the pending data based on the configuration information.
In the embodiment of the present invention, modularized processing, the modularized processing bag are carried out to pending data
Include data form, add data is activation queue, data is activation.It should be noted that at modularization
Reason will pending data be transferred to respectively data form Index module, add data is activation queue refer to
In mark module, data is activation Index module, data form Index module, addition data is activation queue
Index module, data is activation Index module are carried out respectively according to the function in its module to pending data
Data processing.
Data form Index module when processing pending data, based in configuration file
The quantity of pending data, data type are formatted process to pending data；Data are added to send out
Pending data after sending queue Index module to process formatting is performed and added in data is activation queue,
To monitor whether pending data has been successfully transmitted to server；Data is activation Index module is based on configuration
Data is activation agreement in information sends the pending data in data is activation queue to server.
For the ease of carrying out modularized processing to pending data, below will in exemplary fashion to module
Change is processed and is described in detail.As shown in Fig. 2 Fig. 2 shows one kind provided in an embodiment of the present invention
The schematic diagram of modularized processing；When pending data is transferred to data form Index module, data
Format Index module and row format is entered based on the quantity of the pending data in configuration information, data type
Change is processed；Formatting is transferred to pending data after processing to add data is activation queue Index module,
Pending data is performed and adds data is activation queue；Pending data is transferred to into data is activation index
Module, data is activation Index module is sent out pending data based on the data is activation agreement in configuration information
Deliver to server.Wherein, Fig. 2 is only to readily appreciate that carrying out modularized processing to pending protector takes out
As the schematic diagram for being given, the specific embodiment of the present invention carries out the detailed of modularized processing to pending data
Thin process is not defined.
It should be noted that because the data type of the pending data described in the embodiment of the present invention is non-
The data of specific data type, therefore, in data form Index module, add data is activation queue
When Index module, data is activation Index module are processed pending data, each other mutually solely
It is vertical, i.e., when pending data is transferred to data form Index module to be processed, the data form
Changing Index module carries out data form process, the pending number after formatting process to pending data
Carry out at data according to addition data is activation queue Index module, data is activation Index module is fed in transmission
During reason, also because of the pending data because being processed by data form Index module, and will not produce
Others affect.It is simpler to be interpreted as：Pending data is used as a string characters without Special Significance
String, data form Index module, addition data is activation queue Index module, data is activation index mould
At the character string that block is formed according to the mode of its inter-process character string respectively to pending data
Reason.
The modular method of data processing in data acquisition platform provided in an embodiment of the present invention, according to number
Pending data is gathered according to acquisition instructions, the data acquisition instructions are generated by server, and the data are adopted
Collection instruction includes the configuration information of pending data, parses the data acquisition instructions, and obtains and wait to locate
The configuration information of reason data, modularized processing is carried out based on the configuration information to pending data, wherein,
The modularized processing includes：Data form, addition data is activation queue, data is activation；With it is existing
Technology is compared, and the embodiment of the present invention configures pending number by the configuration information in data acquisition instructions
According to, and modularized processing is carried out to pending data based on the configuration information, improve based on a number
According to acquisition platform addition, modification and the flexibility of deletion data acquisition index.
Further, as the refinement and extension to above-described embodiment, in execution step 103 institute is based on
When stating configuration information and carrying out modularized processing to the pending data.Can including but not limited to
Under method, as shown in figure 3, the method includes：
301st, the pending data is obtained.
In the embodiment of the present invention, the number of the pending data is determined by configuration information, described to wait to locate
Reason data can be one, or at least two, the embodiment of the present invention in configuration information to configuring
The concrete number of pending data is not defined.
302nd, data form process is carried out to the pending data based on the configuration information.
Data form Index module is obtained before processing pending data, first matches somebody with somebody confidence
It is pending in the quantity and data type of the pending data in breath, and the configuration information for judging to obtain
Whether the quantity and data type of data obtain the quantity and data class of pending data with step 301
Type is consistent.If consistent, the pending data for continuing step 303 pair acquisition is formatted process；
If inconsistent, illustrate that the pending data that step 301 is obtained is wrong, therefore by step 301
The pending data of acquisition is abandoned.
In the embodiment of the present invention, place is formatted to pending data in data form Index module
During reason, the pending data is formatted by process, the preset rule according to preset specification rule
Model rule is used for the form of pending data described in specification；Wherein, preset specification rule is artificial setting
, including：The length information of pending data, the specific format of pending data are changed, work as presence
During multiple pending datas, connector used when splicing to those pending datas etc..
Exemplary, carry out by taking the specific format conversion of the pending data in preset specification rule as an example
Explanation.If pending data is video buffer duration vido play buffer time=800 milliseconds, and pre-
Put specification rule in require pending data unit be the second, then need video buffer duration vido play
Buffer time=800 milliseconds are formatted and are processed as video buffer duration vido play buffer time=0.8
Second, exemplary citing is these are only, the embodiment of the present invention is not specifically limited preset specification rule
It is fixed.
303rd, the pending data after data formization process is performed and adds data is activation queue.
The final purpose of the embodiment of the present invention is that the pending data that will be collected is sent to server, is
Be able to confirm that whether pending data to be sent to server, data is activation Index module to
Server is sent before pending data, adds data is activation queue Index module by the pending data
Send to perform and add in data is activation queue, the data is activation queue is used for after storage formatting process
Pending data；The transmission waiting list can control to send the transmission time of pending data, monitoring
Whether pending data sends successfully, specifically includes：Data is activation queue Index module is added by form
Pending data after change is processed is added in transmission waiting list, sends and arrange in waiting list transmission
A length of 10s during the interval of pending data, that is, send pending every 10s transmissions one in waiting list
Data, and monitor whether the pending data sends successfully, if the pending data sends successfully,
Successfully mark is transmitted to the pending data；If the pending data sends failure, to this
Pending data is transmitted the mark of failure.It is relevant that pending data is entered in the embodiment of the present invention
Row sends the identification method of successfully/failure and refer to related realization of the prior art, the embodiment of the present invention
Here is no longer repeated.It should be noted that the embodiment of the present invention in transmission waiting list to sending
The interval duration of pending data is not defined.
304th, the pending data in the data is activation queue is sent to server.
When data is activation Index module sends pending data to server, configuration information is solved
Analysis, it is determined that sending the data is activation agreement of pending data, if not indicating in configuration information, transmission is treated
The data is activation agreement of processing data, then using the data is activation association given tacit consent in data is activation Index module
View.In the embodiment of the present invention, the data is activation agreement including but not limited in the following manner, for example,
GET methods, the POST side of HTTP (Hypertext transfer protocol, HTTP)
Method and PUT methods；Or, transmission control protocol (Transmission Control Protocol,
TCP)；The embodiment of the present invention is not defined to the particular content for sending agreement.
According to said method embodiment, a kind of device embodiment is provided below, the device embodiment with it is front
Embodiment of the method correspondence is stated, for ease of reading, this device embodiment is no longer in preceding method embodiment
Detail content repeated one by one, it should be understood that the device in the present embodiment can be realized correspondingly
Full content in preceding method embodiment.
Further, the present invention implements the modular unit of data processing in also a kind of data acquisition platform,
As shown in figure 4, the device includes：
Receiving unit 41, for receiving data acquisition instructions, wherein, the data acquisition instructions are by taking
Business device is generated, and the configuration information comprising pending data, the configuration in the data acquisition instructions
Packet contains：Quantity, type, the data is activation agreement of collection pending data；
Collecting unit 42, for the data acquisition instructions collection received according to the receiving unit 41
Pending data；
Resolution unit 43, for parsing the data acquisition instructions of the reception of the receiving unit 41；
Acquiring unit 44, after parsing the data acquisition instructions in the resolution unit 43, obtains
Take the configuration information of the pending data；
Processing unit 45, the configuration information for being obtained based on the acquiring unit 44 is treated to described
Processing data carries out modularized processing；Wherein, the modularized processing includes data form, adds
Data is activation queue, data is activation.
Further, as shown in figure 5, the processing unit 45 includes：
Acquisition module 451, for obtaining the pending data；
Processing module 452, described in being obtained to the acquisition module 451 based on the configuration information
Pending data carries out data form process；
Add module 453, for by the data formization of the processing module 452 process after described in wait to locate
Reason data are performed and add data is activation queue；
Sending module 454, for by it is described addition module 453 add in data is activation queue described in treat
Processing data is sent to server.
Further, the processing module 452, being additionally operable to will be described pending according to preset specification rule
Data are formatted process, and the preset specification rule is used for the form of pending data described in specification.
Further, as shown in figure 5, the processing unit 45 also includes：
Determining module 455, in the sending module 454 by described in the data is activation queue
Pending data is sent to before server, it is determined that sending the data is activation agreement of the pending data；
The sending module 454, is additionally operable to the data is activation determined based on the determining module 455
Agreement sends the pending data in the data is activation queue to the server.
Further, in the processing module 452 configuration information is based on to the pending data
When carrying out modularized processing, the data form, addition data is activation queue and described are performed
It is separate between data is activation.
The modular unit of data processing in data acquisition platform provided in an embodiment of the present invention, according to number
Pending data is gathered according to acquisition instructions, the data acquisition instructions are generated by server, and the data are adopted
Collection instruction includes the configuration information of pending data, parses the data acquisition instructions, and obtains and wait to locate
The configuration information of reason data, modularized processing is carried out based on the configuration information to pending data, wherein,
The modularized processing includes：Data form, addition data is activation queue, data is activation；With it is existing
Technology is compared, and the embodiment of the present invention configures pending number by the configuration information in data acquisition instructions
According to, and modularized processing is carried out to pending data based on the configuration information, improve based on a number
According to acquisition platform addition, modification and the flexibility of deletion data acquisition index.
The modular unit of data processing includes processor and memory in the data acquisition platform, on
State collecting unit, resolution unit, acquiring unit and processing unit etc. to be stored in as program unit
In reservoir, corresponding function is realized by computing device storage said procedure unit in memory.
Kernel is included in processor, is gone in memory to transfer corresponding program unit by kernel.Kernel can
With arrange one or more, by adjust kernel parameter to improve data acquisition platform in addition, modification
And the flexibility of deletion data acquisition index.
Memory potentially includes the volatile memory in computer-readable medium, random access memory
The form such as device (RAM) and/or Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash
RAM), memory includes at least one storage chip.
Present invention also provides a kind of computer program, when performing on data processing equipment,
It is adapted for carrying out initializing the program code of there are as below methods step：(treated according to data acquisition instructions collection
Processing data；Wherein, the data acquisition instructions are generated by server, and the data acquisition instructions
In the configuration information comprising pending data, the configuration information includes：The number of collection pending data
Amount, data type, data is activation agreement；The data acquisition instructions are parsed, and waits to locate described in obtaining
The configuration information of reason data；Modularization is carried out to the pending data based on the configuration information
Process；Wherein, the modularized processing includes data form, adds data is activation queue, data
Send.In the above embodiment of the present invention, the description to each embodiment all emphasizes particularly on different fields, certain
Without the part described in detail in embodiment, the associated description of other embodiment is may refer to.
Those skilled in the art it should be appreciated that embodiments herein can be provided as method, system,
Or computer program.Therefore, the application can be implemented using complete hardware embodiment, complete software
Example or with reference to the form of the embodiment in terms of software and hardware.And, the application can be adopted at one
Or it is multiple wherein include computer usable program code computer-usable storage medium (including but not
Be limited to magnetic disc store, CD-ROM, optical memory etc.) on the computer program implemented
Form.
The application is with reference to the method according to the embodiment of the present application, equipment (system) and computer program
The flow chart and/or block diagram of product is describing.It should be understood that can be realized flowing by computer program instructions
In each flow process and/or square frame and flow chart and/or block diagram in journey figure and/or block diagram
Flow process and/or square frame combination.Can provide these computer program instructions to all-purpose computer, specially
With the processor of computer, Embedded Processor or other programmable data processing devices producing one
Machine so that produced by the instruction of computer or the computing device of other programmable data processing devices
It is raw to be used to realize in one flow process of flow chart or one square frame of multiple flow processs and/or block diagram or multiple sides
The device of the function of specifying in frame.
These computer program instructions may be alternatively stored in can guide computer or other programmable datas to process
In the computer-readable memory that equipment works in a specific way so that be stored in the computer-readable and deposit
Instruction in reservoir is produced and includes the manufacture of command device, and command device realization is in flow chart one
The function of specifying in flow process or one square frame of multiple flow processs and/or block diagram or multiple square frames.
These computer program instructions can also be loaded into computer or other programmable data processing devices
On so that series of operation steps is performed on computer or other programmable devices to produce computer
The process of realization, so as to the instruction performed on computer or other programmable devices is provided for realizing
Specify in one flow process of flow chart or one square frame of multiple flow processs and/or block diagram or multiple square frames
The step of function.
In a typical configuration, computing device include one or more processors (CPU), input/
Output interface, network interface and internal memory.
Memory potentially includes the volatile memory in computer-readable medium, random access memory
The form such as device (RAM) and/or Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash
RAM).Memory is the example of computer-readable medium.
Computer-readable medium includes that permanent and non-permanent, removable and non-removable media can be with
Information Store is realized by any method or technique.Information can be computer-readable instruction, data knot
Structure, the module of program or other data.The example of the storage medium of computer includes, but are not limited to phase
Become internal memory (PRAM), static RAM (SRAM), dynamic random access memory
(DRAM), other kinds of random access memory (RAM), read-only storage (ROM), electricity can
Erasable programmable read-only memory (EPROM) (EEPROM), fast flash memory bank or other memory techniques, read-only light
Disk read-only storage (CD-ROM), digital versatile disc (DVD) or other optical storages, magnetic
Cassette tape, the storage of tape magnetic rigid disk or other magnetic storage apparatus or any other non-transmission medium,
Can be used to store the information that can be accessed by a computing device.Define according to herein, computer-readable
Medium does not include temporary computer readable media (transitory media), the such as data-signal and load of modulation
Ripple.
Also, it should be noted that term " including ", "comprising" or its any other variant are intended to contain
Lid nonexcludability is included, so that process, method, commodity including a series of key elements or setting
It is standby not only to include those key elements, but also including other key elements being not expressly set out, or also wrap
Include the key element intrinsic for this process, method, commodity or equipment.In the feelings without more restrictions
Under condition, the key element limited by sentence "including a ...", it is not excluded that including key element process,
Also there is other identical element in method, commodity or equipment.
It will be understood by those skilled in the art that embodiments herein can be provided as method, system or calculating
Machine program product.Therefore, the application can adopt complete hardware embodiment, complete software embodiment or knot
Close the form of the embodiment in terms of software and hardware.And, the application can adopt at one or more it
In include computer-usable storage medium (the including but not limited to disk of computer usable program code
Memory, CD-ROM, optical memory etc.) on implement computer program form.
Embodiments herein is these are only, the application is not limited to.For this area skill
For art personnel, the application can have various modifications and variations.It is all spirit herein and principle it
Interior made any modification, equivalent substitution and improvements etc., should be included in claims hereof model
Within enclosing.
Claims (10)
1. in a kind of data acquisition platform data processing modular method, it is characterised in that include：
Receiving data acquisition instructions, and gather pending data according to the data acquisition instructions；Wherein,
The data acquisition instructions are generated by server, and pending data is included in the data acquisition instructions
Configuration information, the configuration information includes：Quantity, data type, the number of collection pending data
According to transmission agreement；
The data acquisition instructions are parsed, and obtains the configuration information of the pending data；
Modularized processing is carried out to the pending data based on the configuration information；Wherein, the mould
Blockization process includes data form, adds data is activation queue, data is activation.
2. method according to claim 1, it is characterised in that based on the configuration information to institute
Stating pending data and carrying out modularized processing includes：
Obtain the pending data；
Data form process is carried out to the pending data based on the configuration information；
The pending data after data formization process is performed and adds data is activation queue；
The pending data in the data is activation queue is sent to server.
3. method according to claim 2, it is characterised in that carry out the pending data
Data formization process includes：
The pending data is formatted by process, the preset specification according to preset specification rule
Rule is used for the form of pending data described in specification.
4. method according to claim 3, it is characterised in that by the data is activation queue
In the pending data send to before server, methods described also includes：
It is determined that sending the data is activation agreement of the pending data；
The pending data in the data is activation queue is sent to server to be included：
The pending data in the data is activation queue is sent based on the data is activation agreement
To the server.
5. the method according to any one of claim 1-4, it is characterised in that based on described
Configuration information is carried out during modularized processing to the pending data, the data form, institute
State separate between addition data is activation queue and the data is activation.
6. in a kind of data acquisition platform data processing modular unit, it is characterised in that include：
Receiving unit, for receiving data acquisition instructions, wherein, the data acquisition instructions are by servicing
Device is generated, and the configuration information comprising pending data in the data acquisition instructions, described with confidence
Breath is included：Quantity, data type, the data is activation agreement of collection pending data；
Collecting unit, the data acquisition instructions for being received according to the receiving unit are gathered to be waited to locate
Reason data；
Resolution unit, for parsing the data acquisition instructions that the receiving unit is received；
Acquiring unit, after parsing the data acquisition instructions in the resolution unit, obtains institute
State the configuration information of pending data；
Processing unit, for the configuration information based on acquiring unit acquisition to described pending
Data carry out modularized processing；Wherein, the modularized processing includes data form, adds data
Transmit queue, data is activation.
7. device according to claim 6, it is characterised in that the processing unit includes：
Acquisition module, for obtaining the pending data；
Processing module, it is described pending for what is obtained to the acquisition module based on the configuration information
Data carry out data form process；
Module is added, for the pending data after the processing module data formization is processed
Perform and add data is activation queue；
Sending module, for the addition module to be added the described pending number in data is activation queue
According to transmission to server.
8. device according to claim 7, it is characterised in that the processing module, is additionally operable to
The pending data is formatted by process, the preset specification rule according to preset specification rule
For the form of pending data described in specification.
9. device according to claim 8, it is characterised in that the processing unit also includes：
Determining module, for will be described pending in the data is activation queue in the sending module
Data is activation is to before server, it is determined that sending the data is activation agreement of the pending data；
The sending module, the data is activation agreement for being additionally operable to determine based on the determining module will
The pending data in the data is activation queue is sent to the server.
10. the device according to any one of claim 6-9, it is characterised in that in the process
Module is carried out during modularized processing based on the configuration information to the pending data, the number
According to separate between formatting, the addition data is activation queue and the data is activation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510729333.2A CN106648912B (en) | 2015-10-30 | 2015-10-30 | Modularization method and device for data processing in data acquisition platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510729333.2A CN106648912B (en) | 2015-10-30 | 2015-10-30 | Modularization method and device for data processing in data acquisition platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106648912A true CN106648912A (en) | 2017-05-10 |
| CN106648912B CN106648912B (en) | 2020-11-03 |
Family
ID=58810478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510729333.2A Active CN106648912B (en) | 2015-10-30 | 2015-10-30 | Modularization method and device for data processing in data acquisition platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106648912B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145194A (en) * | 2017-06-27 | 2019-01-04 | 北京国双科技有限公司 | The acquisition method and device of user behavior data |
| CN110262329A (en) * | 2019-06-11 | 2019-09-20 | 华强方特文化科技集团股份有限公司 | Manned device data acquisition system and data format storage method |
| CN110320842A (en) * | 2019-07-16 | 2019-10-11 | 东北大学 | Multi-Scale Data acquisition and processing unit and method for aluminum oxide production process |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101286895A (en) * | 2008-05-22 | 2008-10-15 | 上海交通大学 | Dynamic configurable data monitoring system and method for distributed network |
| CN102270141A (en) * | 2011-08-21 | 2011-12-07 | 中国海洋大学 | Configurable data acquisition software system and designing method |
| CN103491059A (en) * | 2012-06-12 | 2014-01-01 | 中国科学院沈阳自动化研究所 | Data collection implementation method of semiconductor manufacturing equipment |
| CN104123134A (en) * | 2014-07-07 | 2014-10-29 | 四川中电启明星信息技术有限公司 | Intelligent electricity use data management method and system based on AMI and J2EE |
| CN104954242A (en) * | 2014-03-26 | 2015-09-30 | 洛克威尔自动控制技术股份有限公司 | Unified data ingestion adapter for migration of industrial data to a cloud platform |
-
2015
- 2015-10-30 CN CN201510729333.2A patent/CN106648912B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101286895A (en) * | 2008-05-22 | 2008-10-15 | 上海交通大学 | Dynamic configurable data monitoring system and method for distributed network |
| CN102270141A (en) * | 2011-08-21 | 2011-12-07 | 中国海洋大学 | Configurable data acquisition software system and designing method |
| CN103491059A (en) * | 2012-06-12 | 2014-01-01 | 中国科学院沈阳自动化研究所 | Data collection implementation method of semiconductor manufacturing equipment |
| CN104954242A (en) * | 2014-03-26 | 2015-09-30 | 洛克威尔自动控制技术股份有限公司 | Unified data ingestion adapter for migration of industrial data to a cloud platform |
| CN104123134A (en) * | 2014-07-07 | 2014-10-29 | 四川中电启明星信息技术有限公司 | Intelligent electricity use data management method and system based on AMI and J2EE |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145194A (en) * | 2017-06-27 | 2019-01-04 | 北京国双科技有限公司 | The acquisition method and device of user behavior data |
| CN110262329A (en) * | 2019-06-11 | 2019-09-20 | 华强方特文化科技集团股份有限公司 | Manned device data acquisition system and data format storage method |
| CN110320842A (en) * | 2019-07-16 | 2019-10-11 | 东北大学 | Multi-Scale Data acquisition and processing unit and method for aluminum oxide production process |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106648912B (en) | 2020-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103034483A (en) | Web page script management method and system | |
| CN106550052A (en) | A kind of data acquisition unit and method based on OPC UA | |
| CN103116645B (en) | The method and device of browsing page by mobile equipment | |
| CN110309065B (en) | Concurrent execution method, device and equipment of test tasks and storage medium | |
| CN107370806A (en) | HTTP conditional codes monitoring method, device, storage medium and electronic equipment | |
| CN107609004A (en) | Application program buries point methods and device, computer equipment and storage medium | |
| WO2014176919A1 (en) | Method and system for displaying recommended pages by terminal device | |
| CN106648912A (en) | Modular method and apparatus for data processing in data acquisition platform | |
| CN108182217A (en) | A kind of APP development systems and dissemination method based on mobile Internet | |
| CN107370628B (en) | Log processing method and system based on embedded points | |
| CN112286815A (en) | Interface test script generation method and related equipment thereof | |
| CN104202328B (en) | A kind of method, configuration module and the subscription end of subscription GOOSE/SMV messages | |
| CN105550179A (en) | Webpage collection method and browser plug-in | |
| CN115543479A (en) | Interface calling analysis method and device suitable for dynamic parameters | |
| CN105468412A (en) | Dynamic packaging method and dynamic packaging device | |
| CN101477539B (en) | Information acquisition method and device | |
| CN103354546A (en) | Message filtering method and message filtering apparatus | |
| CN108063746A (en) | Processing method, client, server and the system of data | |
| CN104811418B (en) | The method and device of viral diagnosis | |
| CN108243238B (en) | Method and device for acquiring performance data | |
| CN106776614A (en) | The display methods and device of sharing platform | |
| CN105245394A (en) | Method and equipment for analyzing network access log based on layered approach | |
| CN109558129A (en) | Progress bar implementation method and device | |
| CN105094810B (en) | Data processing method and device based on CGI(Common gateway interface) plug-in unit | |
| EP4156009A1 (en) | Systematic identification and masking of private data for replaying user sessions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Applicant after: Beijing Guoshuang Technology Co.,Ltd. Address before: 100086 Cuigong Hotel, 76 Zhichun Road, Shuangyushu District, Haidian District, Beijing Applicant before: Beijing Guoshuang Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |