CN106227855A - A kind of transacter, system and method - Google Patents
A kind of transacter, system and method Download PDFInfo
- Publication number
- CN106227855A CN106227855A CN201610608390.XA CN201610608390A CN106227855A CN 106227855 A CN106227855 A CN 106227855A CN 201610608390 A CN201610608390 A CN 201610608390A CN 106227855 A CN106227855 A CN 106227855A
- Authority
- CN
- China
- Prior art keywords
- data
- dynamically
- analysis
- class
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000004458 analytical method Methods 0.000 claims abstract description 94
- 238000013480 data collection Methods 0.000 claims abstract description 57
- 238000012544 monitoring process Methods 0.000 claims abstract description 24
- 230000007246 mechanism Effects 0.000 claims description 17
- 230000000694 effects Effects 0.000 abstract description 5
- 238000012545 processing Methods 0.000 description 24
- 238000007405 data analysis Methods 0.000 description 12
- 238000013500 data storage Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of transacter, system and method.This device includes: monitoring unit, for monitoring whether be cached with the data that client reports in data buffer storage；Resolution unit, during for being cached with data that client reports in described monitoring unit monitors described data buffer storage, calls the parsing class of dynamically setting, described data is carried out dynamic analysis；Described data, for the analysis result according to described resolution unit, are inserted in the database table of correspondence by collector unit.The present invention can be by dynamically arranging parsing class, realize dynamically data parsing, and then the fast resolving of data and storage can be improve data collection effect, can also be by dynamically arranging the dynamic analysis resolving class realization to the data of newly-increased kind, it is not necessary to restart data gathering system.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data collection device, system, and method.
Background
At present, with the rapid development of computers and information technologies, data generated by application systems are explosively increased, the data amount of the data which reaches hundreds of TB or even hundreds of PB scale at all times is far beyond the processing capacity of the traditional computing technology and information systems, and therefore, the search for effective big data processing technology, especially the search for effective big data statistical analysis technology, is an urgent problem to be solved in the field.
With the development of mobile terminal services and the abundance of functions, various applications (APPs, abbreviated as APPs) applied to mobile terminals are increasing, and large data statistics and analysis are required for large data volumes generated by various APPs.
In order to achieve large data statistical analysis, it is necessary to be able to collect data of clients over a wide range. In a traditional data collection mode, a client uploads data, and a server stores the data uploaded by the client into a corresponding data table according to a corresponding interface or a type of the data. When the server stores data, for the newly added type of data, the analysis class corresponding to the type needs to be newly added, but the new analysis class cannot be dynamically updated, the new analysis class needs to be validated by restarting the server, and if a relational database is used, a database table corresponding to the type needs to be newly added every time a new type is added, so that data storage can be realized. Therefore, the traditional data collection mode is low in reusability of data of new types, and high in labor cost and maintenance cost.
Disclosure of Invention
The invention mainly aims to provide a data collection device, a system and a method, aiming at solving the problem that the newly added data can not be dynamically analyzed in the traditional data collection mode.
Aiming at the technical problems, the invention solves the technical problems by the following technical scheme:
the present invention provides a data collection device, comprising: the monitoring unit is used for monitoring whether the data reported by the client side is cached in the data cache; the analysis unit is used for calling a dynamically set analysis class to dynamically analyze the data when the monitoring unit monitors that the data reported by the client side is cached in the data cache; and the collection unit is used for inserting the data into a corresponding database table according to the analysis result of the analysis unit.
The analysis unit is also used for dynamically setting an analysis class in a hot deployment mode; wherein dynamically setting the analytic class comprises: and dynamically adding and/or updating the analysis classes.
The analysis unit is used for calling a dynamically set analysis class according to a JAVA reflection mechanism and dynamically analyzing the data.
The data cache is a message queue or a cache database; the database table is located in a non-relational database.
The present invention also provides a data collection system, comprising: the system comprises an interface server, a cache server, an asynchronous storage middleware and a database which are connected in sequence; the interface server receives data reported by a client and caches the data in the cache server; and when monitoring that the cache server caches the data reported by the client, the asynchronous storage middleware calls a dynamically set analysis class to dynamically analyze the data, and inserts the data into a corresponding database table in the database according to an analysis result.
The asynchronous storage middleware dynamically sets the analysis class in a hot deployment mode; wherein dynamically setting the analytic class comprises: dynamically adding and/or updating the analytic classes; and calling a dynamically set analysis class by the asynchronous storage middleware according to a JAVA reflection mechanism, and dynamically analyzing the newly reported data.
The invention also provides a data collection method, which comprises the following steps: monitoring whether data reported by a client side is cached in a data cache; if so, calling a dynamically set analysis class to dynamically analyze the data; and inserting the data into a corresponding database table according to the analysis result.
Wherein, dynamically setting the analytic class comprises: dynamically setting an analytic class by adopting a hot deployment mode; wherein dynamically setting the analytic class comprises: and dynamically adding and/or updating the analysis classes.
The calling of the dynamically set analysis class is used for dynamically analyzing the data, and the method comprises the following steps: and calling the dynamically set analysis class according to a JAVA reflection mechanism, and dynamically analyzing the data.
The data cache is a message queue or a cache database; the database table is located in a non-relational database.
The invention has the following beneficial effects:
the invention can realize dynamic data analysis by dynamically setting the analysis class, further can quickly analyze and store the data, improves the data collection effect, can realize dynamic analysis of the newly added class of data by dynamically setting the analysis class, does not need to restart the data collection system, and avoids a series of problems caused by restarting the data collection system when the analysis class is newly added.
Drawings
FIG. 1 is a block diagram of a data collection system according to one embodiment of the present invention;
FIG. 2 is a detailed block diagram of a data collection system according to one embodiment of the present invention;
FIG. 3 is a block diagram of a data collection device according to one embodiment of the present invention;
FIG. 4 is a flow chart of a data collection method according to an embodiment of the invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
A mobile terminal implementing various embodiments of the present invention will now be described with reference to the accompanying drawings. In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in themselves. Thus, "module" and "component" may be used in a mixture.
The embodiment of the invention provides a data collection system. FIG. 1 is a block diagram of a data collection system according to one embodiment of the present invention.
In the present embodiment, the data collection system includes: interface server 110, cache server 120, asynchronous storage middleware 130, and collection database 140, connected in series.
The interface server 110 is configured to receive data reported by the client, and cache the data in the cache server 120. Wherein the client is a client pre-installed in the terminal. A communication connection may be established between the terminal and the interface server 110, thereby enabling a communication connection to be established between the client and the interface server 110.
And the cache server 120 is used for caching data. In this embodiment, the cache server 120 includes a data cache. The message queue or cache database may act as a data cache. The data cache is mainly used for caching data reported by the client. The data reported by the client is the data generated by the user operating the client.
The asynchronous storage middleware 130 is configured to monitor whether data reported by the client is cached in the cache server 120, call a dynamically set parsing class when the data reported by the client is monitored to be cached in the cache server 120, perform dynamic parsing on the data, and insert the data into a corresponding database table in the collection database 140 according to a parsing result. The analysis result includes a database table name in which data should be stored, a field name in which data should be stored in the database table, and the like.
A collection database 140 for storing corresponding data in a plurality of database tables under the control of the asynchronous storage middleware 130. The data stored in the collection database 140 may be used as a data base for subsequent data viewing, data processing, and data analysis of the data.
In this embodiment, the asynchronous storage middleware 130 dynamically sets the analysis class to implement dynamic analysis of the data, and by dynamically analyzing the data, the data is quickly analyzed and stored, thereby improving the data collection effect.
In this embodiment, the asynchronous storage middleware 130 dynamically sets the analysis class, so that dynamic analysis of the data of the new class is realized, the data collection system does not need to be restarted, and a series of problems caused by restarting the data collection system when the analysis class is newly added are avoided.
In this embodiment, the interface server 110, the cache server 120, the asynchronous storage middleware 130, and the collection database 140 may complete collection of data reported by the client. It should be understood by those skilled in the art that the interface server 110, the cache server 120, the asynchronous storage middleware 130, and the collection database 140 may be separate servers, or may be integrated into a server, such as a data collection server, and may be configured according to actual requirements.
In the present embodiment, the terminal for installing the client may be implemented in various forms. For example, the terminal described in the embodiments of the present invention may include a mobile terminal such as a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, and the like, and a fixed terminal such as a digital TV, a desktop computer, a smart home appliance, and the like.
The terminal may include: a communication unit for establishing a communication connection with the interface server 110, an input unit for receiving a user operation instruction, an output unit for outputting information, a display unit for displaying a client user interface, a memory for storing data, an interface unit for connecting and/or communicating with an external device, a controller for controlling the terminal, and a power supply unit for supplying power to the terminal, and the like. Wherein, the data stored in the memory comprises data generated by user operation.
It is to be understood that the terminal of the present embodiment may alternatively implement more or fewer components.
Specifically, FIG. 2 is a detailed block diagram of a data collection system according to one embodiment of the invention.
The interface server 110 includes: a statistical interface module 111 and a queue locating module 112 connected to each other.
And the statistical interface module 111 is configured to receive data reported by the client 180.
The number of clients 180 is one or more. Establishing a communication connection between the client 180 and the interface server 110; the client 180 performs data collection processing on data generated by user operation, acquires data to be reported to the interface server 110, and uploads the data to be reported to the statistical interface module 111, so as to realize data reporting.
The data reported by the client 180 includes: user behavior data. The user behavior data includes: the page browsed by the user, the browsed object, the browsing time length, the operation of the user, the operated object and the like.
The queue positioning module 112 is configured to monitor the statistics interface module 111, and when it is monitored that the statistics interface module 111 receives the data reported by the client 180, obtain the data reported by the client 180, and cache the data reported by the client 180 in the cache server 120. Further, the queue positioning module 112 needs to send each data received by the statistical interface module 111 to the cache server 120 in sequence according to the sequence of the data received by the statistical interface module 111.
The cache server 120 is connected to the queue locator module 112 in the interface server 110.
The cache server 120 includes: a message queue; alternatively, the cache server 120 includes: a message queue and a cache database.
In this embodiment, the message queue may be a first-in-first-out message queue.
In this embodiment, the cache database may be a Redis database.
If the cache server 120 includes a message queue, the message queue is used as a data cache, the queue positioning module 112 sequentially sends data reported by the client into the message queue, and the asynchronous storage middleware 130 sequentially takes out the data for data analysis.
If the cache server 120 includes a message queue and a cache database, the cache database is used as a data cache, after the queue positioning module 112 sequentially sends data into the message queue, the cache server 120 sequentially sends data in the message queue into the cache database, and then the asynchronous storage middleware 130 sequentially takes data out of the cache database for data analysis.
The asynchronous storage middleware 130 includes: a data storage processing module 131. The data storage processing module 131 is connected to the cache server 120 and the collection database 140 respectively.
And the data storage processing module 131 is configured to dynamically set the analysis class in a hot deployment manner.
The analytic classes include: the type of data, the analysis method of the data of the type, the name of the database table corresponding to the data of the type, the field information of the database table, the configuration information and the like.
In this embodiment, dynamically setting the parsing class includes: additions and/or updates to the resolution class, for example: a new class of resolution is added. For another example: the field information of the database table in a certain analytic class is updated. Further, each time a kind of data is added, an analysis class corresponding to the kind of data is added, a newly added analysis class or data to be updated in the analysis class may be preset, and the data is uploaded to the asynchronous storage middleware 130 in a hot deployment manner and becomes effective in the asynchronous storage middleware 130.
The data storage processing module 131 is further configured to monitor whether the data cache of the cache server 120 has data reported by the client 180 in real time, and when the data cache of the cache server 120 has the data reported by the client 180, call a dynamically set analysis class according to a JAVA reflection mechanism, perform dynamic analysis on the data, and store the newly reported data into the collection database 140 according to an analysis result. Further, the data storage processing module 131 parses the data in the message queue or the cache database of the cache server 120 one by one according to the first-in first-out rule.
The collection database 140 is a non-relational database. Further, the collection database 140 is a MongoDB, which is a database based on distributed file storage. After analyzing the data of the new category, the MongoDB may add a database table corresponding to the category to the database and/or update a field corresponding to the category in the database table. If the collection database 140 is not a MongoDB and does not have the function of MongoDB, the database table can be added or updated every time a kind of data is added, so that the data of the kind has a corresponding storage position and is normally stored.
Specifically, the collection database 140 includes a plurality of database tables; in the system operation process, the data storage processing module 131 dynamically calls an analysis class corresponding to a type of data reported by the client 180 by using a JAVA reflection mechanism according to the type of the data, analyzes the data by using an analysis method in the analysis class, determines a name of a database table corresponding to the data, that is, in a plurality of database tables in the collection database 140, a database table into which the data should be stored, determines a field corresponding to the data according to field information of the database table in the analysis class, and stores the data into a corresponding field of a corresponding database table in the collection database 140 according to the database table and the field corresponding to the data.
In this embodiment, the parsing class is set in the asynchronous storage middleware 130 by means of thermal deployment, which is to upgrade when the system is running, and does not need to restart the system. Therefore, the embodiment does not need to store the middleware 130 asynchronously again, and does not need to store other servers and modules in the data collection system again, thereby realizing dynamic setting of analysis classes, facilitating data analysis and having strong reusability.
In the embodiment, the analysis class is dynamically called through a JAVA reflection mechanism to dynamically analyze the data, so that the analysis efficiency is high. The JAVA reflection mechanism refers to: in the system running state, for any class, all attributes and methods of the class can be called; any method and property can be called for any object.
In this embodiment, in order to perform subsequent data viewing, data processing and data analysis on the data in the collected database 140, the data collection system may further include: a background operations management server 150 and data statistics processing middleware 160.
The background operation management server 150 includes: a data statistics platform 151. The data statistics platform 151 is connected to the collection database 140.
A data statistics platform 151 in the background operations management server 150 for viewing and/or processing the data stored in the collection database 140. The operator can call the data in the collection database 140 through the data statistics platform 151 in the background operation management server 150, and view and operate the called data. The operations include modification, addition, and deletion. For example: and calling the financial statement of the year from the collection database and checking the financial statement.
The data statistics processing middleware 160 includes: a data statistics platform 161. The data statistics platform 161 is connected to the collection database 140 and the processing database 170, respectively.
The process database 170 is used to store data after data processing and data analysis. The process database 170 may be a relational database, such as the Mysql database.
And a data statistics platform 161 in the data statistics processing middleware 160, configured to perform data processing and data analysis on the data stored in the collection database 140, and store the processed and analyzed data in the processing database 170. Further, data collection is mainly used for data analysis, and the data statistics platform 161 in the data statistics processing middleware 160 is mainly used for data analysis. The data processing comprises the following steps: and (5) clustering processing of data.
Based on the data collection system, the embodiment of the invention provides a data collection device. The data collection means may be provided in the asynchronous storage middleware 130. Further, the data collection means may be provided in the data storage processing module 131.
FIG. 3 is a block diagram of a data collection device according to an embodiment of the present invention.
In this embodiment, the data collection apparatus includes: a monitoring unit 310, a parsing unit 320, and a collecting unit 330 connected in sequence.
The monitoring unit 310 is configured to monitor whether data reported by the client is cached in the data cache.
The parsing unit 320 is configured to, when the monitoring unit 310 monitors that the data reported by the client is cached in the data cache, invoke a dynamically set parsing class to perform dynamic parsing on the data.
The collecting unit 330 is configured to insert the data into a corresponding database table according to the analysis result of the analyzing unit 320.
The embodiment of the invention adopts a mode of dynamically setting the analysis class and dynamically analyzing the data, thereby avoiding the adverse effect of restarting a server on the data set and improving the reusability of data collection.
Specifically for the monitoring unit 310:
the monitoring unit 310 is configured to monitor the data cache.
The data cache may be internal or external to the data collection device of this embodiment. The data cache may be a message queue or a cache database. Further, the data cache may be a message queue or a cache database in the cache server 120 described above. The message queue may be a first-in-first-out message queue.
The built-in case may be: the cache server 120 described above is integrated with the data collection device of the present embodiment.
The external case may be: the cache server 120 and the data collection device of the present embodiment are respectively regarded as independent entities.
The monitoring unit 310 is specifically configured to monitor whether there is cached data in the data cache, monitor whether there is new data entering the data cache, and control the data cache to receive data, suspend receiving data, or stop receiving data.
The monitoring unit 310 may notify the parsing unit 320 to parse the data in the data cache after monitoring that the data cache has cached data or new data enters the data cache.
Specifically, parsing section 320:
in order to ensure fairness of data analysis, the analyzing unit 320 is configured to, when the monitoring unit 310 monitors that data reported by the client exists in the data cache, sequentially invoke an analysis class corresponding to each data according to a sequence of the data entering the data cache, perform dynamic analysis on each data, and delete the analyzed data in the data cache.
For example: the data buffer is a message queue, such as a first-in-first-out message queue; and for the data cached in the message queue, sequentially calling the analysis class corresponding to each data according to the sequence of the data entering the message queue, and dynamically analyzing each data. That is, data that enters the message queue first is analyzed first, and data that enters the message queue later is analyzed.
For dynamically setting the parse class:
the analysis unit 320 is further configured to dynamically set an analysis class in a hot deployment manner; wherein dynamically setting the analytic class comprises: and dynamically adding and/or updating the analysis classes.
The hot deployment refers to the dynamic setting of the analytic classes in the data collection process, and the method does not need to restart the server and influence the execution of normal data collection.
When a new type of data is added, the parsing unit 320 sets the parsing class corresponding to the type, and submits the parsing class in a hot-deployment manner, so that the parsing class becomes effective, and then the parsing unit 320 may call the parsing class to parse the type of data.
The analytic classes include: the type of data, the analysis method of the data of the type, the name of the database table corresponding to the data of the type, the field information of the database table, the configuration information and the like.
Increasing the kind of data is to increase the database tables or fields not present in the database.
The data format of each kind of data may be different, so that a corresponding parsing method needs to be set for each kind in order to correctly parse out the data.
The name of the database table where the data corresponds is the name of the database table where the data should be stored. For example: the name of the database table is a financial statement database table.
The field information of the database table is the field contained in the database table. For example: the financial statement database table includes an expenditure field, an income field, and a loss field. From the field information of the database table, it can be determined into which field or fields the data should be stored.
The configuration information is, for example, information of a timing task, a storage path of data, and the like.
Specifically, for the dynamic analysis data:
and the analysis unit 320 is configured to invoke a dynamically set analysis class according to a JAVA reflection mechanism, and dynamically analyze the data reported by the client.
Based on the JAVA reflection mechanism, any one of the dynamically set analysis classes may be loaded during the operation of the data collection device in this embodiment, and an analysis method in the analysis class is called to obtain information in the analysis class. Therefore, the JAVA reflection mechanism is a mechanism capable of dynamically acquiring information and dynamically calling methods, and dynamic analysis of data can be achieved through the mechanism.
Specifically, when the monitoring unit 310 monitors that data reported by the client is cached in the data cache, based on the JAVA reflection mechanism, the parsing unit 320 may find the parsing class corresponding to the data according to the type of the cached data, parse the data by using an parsing method in the parsing class, determine the name of the database table corresponding to the data and the field information in the database table of the name of the database table, and store the data into the corresponding field in the corresponding database table according to the name of the database table corresponding to the data and the field information in the database table of the name of the database table.
Specifically, for the collection unit 330:
the parsing unit 320 outputs the parsing result to the collection unit 330. The analysis result comprises: the name of the database table where the data should be stored, and the field and other information in the database table of the name of the database table where the data should be stored.
The collecting unit 330 is configured to insert the data into a corresponding database table in a database according to the analysis result of the analyzing unit 320.
The database table is located in a non-relational database. The non-relational data is used to store the parsed data. The non-relational data is, for example, MongoDB.
The non-relational database is built in or out of the data collection device of the present embodiment. If the database is external to the data collection device of this embodiment, the database is similar to the collection database 140 described above.
Based on the data collection system and the data collection device, the invention provides a data collection method. Fig. 4 is a flow chart of a data collection method according to an embodiment of the invention.
Step S410, monitoring whether data reported by the client side is cached in the data cache; if yes, step S420 is performed, and if no, step S410 is continued.
The data cache may be a message queue or a cache database. Further, the message queue may be a first-in-first-out message queue.
Whether the data reported by the client side is cached in the message queue can be monitored in real time.
Step S420, calling a dynamically set parsing class to dynamically parse the data.
In this embodiment, a hot deployment manner is adopted to dynamically set the analytic classes; wherein dynamically setting the analytic class comprises: and dynamically adding and/or updating the analysis classes.
In this embodiment, according to a JAVA reflection mechanism, a dynamically set parsing class is called to perform dynamic parsing on the data.
And according to the condition that a plurality of data are cached in the data cache, calling the analysis class corresponding to each data in sequence according to the sequence of the plurality of data entering the data cache, dynamically analyzing each data, and deleting the analyzed data in the data cache. Furthermore, for the data cached in the message queue, according to the sequence of the data entering the message queue, the analysis class corresponding to each data is sequentially called, and each data is dynamically analyzed. That is, data that enters the message queue first is analyzed first, and data that enters the message queue later is analyzed.
And step S430, inserting the data into a corresponding database table according to the analysis result.
The resolution result includes the database table name where the data should be stored, the field in the database table of the database table name where the data should be stored.
The database table is located in a non-relational database. The non-relational data is used to store the parsed data. The non-relational data is, for example, MongoDB.
The data collection method described in this embodiment corresponds to the data collection device, and since the data collection device has been described in the embodiment shown in fig. 1 to fig. 3, details of this embodiment are not described in detail, and reference may be made to relevant descriptions in the foregoing embodiments, which are not described herein again.
The invention uses the non-relational database, can realize dynamic data analysis by dynamically setting the analysis class, further can quickly analyze and store the data, improves the data collection effect, can realize dynamic analysis of newly added data by dynamically setting the analysis class, does not need to restart the data collection system, avoids a series of problems caused by restarting the data collection system when newly adding the analysis class, improves the reusability of the whole system, and reduces the development and maintenance cost.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A data collection device, comprising:
the monitoring unit is used for monitoring whether the data reported by the client side is cached in the data cache;
the analysis unit is used for calling a dynamically set analysis class to dynamically analyze the data when the monitoring unit monitors that the data reported by the client side is cached in the data cache;
and the collection unit is used for inserting the data into a corresponding database table according to the analysis result of the analysis unit.
2. The apparatus of claim 1,
the analysis unit is also used for dynamically setting analysis classes in a hot deployment mode;
wherein dynamically setting the analytic class comprises: and dynamically adding and/or updating the analysis classes.
3. The apparatus of claim 1,
and the analysis unit is used for calling the analysis class which is dynamically set according to a JAVA reflection mechanism and carrying out dynamic analysis on the data.
4. The apparatus of claim 1,
the data cache is a message queue or a cache database;
the database table is located in a non-relational database.
5. A data collection system, comprising: the system comprises an interface server, a cache server, an asynchronous storage middleware and a database which are connected in sequence;
the interface server receives data reported by a client and caches the data in the cache server;
and when monitoring that the cache server caches the data reported by the client, the asynchronous storage middleware calls a dynamically set analysis class to dynamically analyze the data, and inserts the data into a corresponding database table in the database according to an analysis result.
6. The system of claim 5,
the asynchronous storage middleware dynamically sets an analytic class in a hot deployment mode; wherein dynamically setting the analytic class comprises: dynamically adding and/or updating the analytic classes; and the number of the first and second groups,
and the asynchronous storage middleware calls a dynamically set analysis class according to a JAVA reflection mechanism and dynamically analyzes the newly reported data.
7. A method of data collection, comprising:
monitoring whether data reported by a client side is cached in a data cache;
if so, calling a dynamically set analysis class to dynamically analyze the data;
and inserting the data into a corresponding database table according to the analysis result.
8. The method of claim 7, wherein dynamically setting a parsing class comprises:
dynamically setting an analytic class by adopting a hot deployment mode; wherein,
the dynamically setting the parsing class includes: and dynamically adding and/or updating the analysis classes.
9. The method of claim 7, wherein said invoking a dynamically set parsing class to dynamically parse the data comprises:
and calling the dynamically set analysis class according to a JAVA reflection mechanism, and dynamically analyzing the data.
10. The method of claim 7,
the data cache is a message queue or a cache database;
the database table is located in a non-relational database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610608390.XA CN106227855A (en) | 2016-07-28 | 2016-07-28 | A kind of transacter, system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610608390.XA CN106227855A (en) | 2016-07-28 | 2016-07-28 | A kind of transacter, system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106227855A true CN106227855A (en) | 2016-12-14 |
Family
ID=57535108
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610608390.XA Pending CN106227855A (en) | 2016-07-28 | 2016-07-28 | A kind of transacter, system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106227855A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790629A (en) * | 2017-01-03 | 2017-05-31 | 努比亚技术有限公司 | Data synchronization unit and its realize the method for data syn-chronization, client access system |
| CN111880838A (en) * | 2020-08-03 | 2020-11-03 | 北京神舟航天软件技术有限公司 | Data analysis method based on template matching technology |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5642477A (en) * | 1994-09-22 | 1997-06-24 | International Business Machines Corporation | Method and apparatus for selectably retrieving and outputting digitally stored multimedia presentations with real-time non-interrupting, dynamically selectable introduction of output processing |
| US20080228704A1 (en) * | 2007-03-16 | 2008-09-18 | Expanse Networks, Inc. | Expanding Bioattribute Profiles |
| CN101958987A (en) * | 2009-07-14 | 2011-01-26 | 中国电信股份有限公司 | Method and system for dynamically converting telecommunications service data |
| CN103150380A (en) * | 2013-03-13 | 2013-06-12 | 河海大学 | Table format customizable Excel table analysis method |
| CN105447146A (en) * | 2015-11-26 | 2016-03-30 | 陕西艾特信息化工程咨询有限责任公司 | Massive data collecting and exchanging system and method |
-
2016
- 2016-07-28 CN CN201610608390.XA patent/CN106227855A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5642477A (en) * | 1994-09-22 | 1997-06-24 | International Business Machines Corporation | Method and apparatus for selectably retrieving and outputting digitally stored multimedia presentations with real-time non-interrupting, dynamically selectable introduction of output processing |
| US20080228704A1 (en) * | 2007-03-16 | 2008-09-18 | Expanse Networks, Inc. | Expanding Bioattribute Profiles |
| CN101958987A (en) * | 2009-07-14 | 2011-01-26 | 中国电信股份有限公司 | Method and system for dynamically converting telecommunications service data |
| CN103150380A (en) * | 2013-03-13 | 2013-06-12 | 河海大学 | Table format customizable Excel table analysis method |
| CN105447146A (en) * | 2015-11-26 | 2016-03-30 | 陕西艾特信息化工程咨询有限责任公司 | Massive data collecting and exchanging system and method |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790629A (en) * | 2017-01-03 | 2017-05-31 | 努比亚技术有限公司 | Data synchronization unit and its realize the method for data syn-chronization, client access system |
| CN111880838A (en) * | 2020-08-03 | 2020-11-03 | 北京神舟航天软件技术有限公司 | Data analysis method based on template matching technology |
| CN111880838B (en) * | 2020-08-03 | 2024-04-12 | 北京神舟航天软件技术有限公司 | Data analysis method based on template matching technology |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110262807B (en) | Cluster creation progress log acquisition system, method and device | |
| CN111352921A (en) | ELK-based slow query monitoring method and device, computer equipment and storage medium | |
| CN112162965B (en) | Log data processing method, device, computer equipment and storage medium | |
| US20210081263A1 (en) | System for offline object based storage and mocking of rest responses | |
| CN110688096B (en) | Method and device for constructing application program containing plug-in, medium and electronic equipment | |
| CN108415998B (en) | Application dependency relationship updating method, terminal, device and storage medium | |
| US20100076937A1 (en) | Feed processing | |
| CN112416991A (en) | Data processing method and device and storage medium | |
| CN111241189B (en) | Method and device for synchronizing data | |
| CN112732663A (en) | Log information processing method and device | |
| CN113760722A (en) | Test system and test method | |
| CN111427899A (en) | Method, device, equipment and computer readable medium for storing file | |
| CN113204558B (en) | Automatic data table structure updating method and device | |
| CN106227855A (en) | A kind of transacter, system and method | |
| US9679262B2 (en) | Image index routing | |
| CN109783440B (en) | Data storage method, data retrieval method, data storage device, medium and electronic equipment | |
| CN115080114B (en) | Application program transplanting processing method, device and medium | |
| CN111159207A (en) | Information processing method and device | |
| CN113761433B (en) | Service processing method and device | |
| CN115496544A (en) | Data processing method and device | |
| CN111368039B (en) | Data management system | |
| CN104125100A (en) | Method for real-time monitoring of dashboards in communication network management | |
| CN114238438A (en) | Method, device, equipment and medium for real-time calculation and statistics of data | |
| CN111552674B (en) | Log processing method and equipment | |
| CN107888445B (en) | Method and device for analyzing performance state, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161214 |
|
| RJ01 | Rejection of invention patent application after publication |