CN105635298A - Service isolation principle based data acquisition equipment unified access system - Google Patents
Service isolation principle based data acquisition equipment unified access system Download PDFInfo
- Publication number
- CN105635298A CN105635298A CN201610017431.8A CN201610017431A CN105635298A CN 105635298 A CN105635298 A CN 105635298A CN 201610017431 A CN201610017431 A CN 201610017431A CN 105635298 A CN105635298 A CN 105635298A
- Authority
- CN
- China
- Prior art keywords
- data
- thread
- module
- acquisition equipment
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/566—Grouping or aggregating service requests, e.g. for unified processing
Abstract
The invention provides a service isolation principle based data acquisition equipment unified access system. The service isolation principle based data acquisition equipment unified access system comprises a configuration initialization module, a thread pool initialization module, a communication service module, an isolation module, an analysis service module and a buffer module. Theoretically, the service isolation principle based data acquisition equipment unified access system does not depend on industry natures, specific data acquisition equipment types and data acquisition equipment manufacturers; and generally, any data acquisition equipment which supports current mainstream data software communication protocols and international conventional data interaction formats can be accessed. At the same time, the service isolation principle based data acquisition equipment unified access system reduces work during access of the data acquisition equipment and huge cost of latter maintenance, is reliable in technology and convenient to operate. The service isolation principle based data acquisition equipment unified access system is not only suitable for traditional industries such as traffic, banks and operators, but also applicable for novel Internet industries such as O2O and e-commerce.
Description
Technical field
The present invention relates to data acquisition equipment and access field of terminal technology, specifically a kind of data acquisition equipment uniform access system based on business separation principle.
Background technology
The data access of existing data acquisition equipment is usually certain or certain several device data of particular area and data is stored by volume of data access process after equipment collection, is generally of certain business threshold, and the scope of application is not extensive yet.
Chinese invention patent application CN104753776A discloses a kind of business isolation processing method, device, system, DPU and network adapter, the Plain Old Telephone Service POTS transmitted on the twisted-pair that presently, there are by using business isolating means to solve and user side equipment are to the DPU incompatible problem of reverse power supply business carried out, and then have reached to improve the effect of Consumer's Experience.
Chinese invention patent application CN102830281A discloses a kind of intelligent electric energy meter and utility meter kilowatt meter reading-out system, and intelligent electric energy meter, by carrying out communication between wireless network and terminal system, solves the data transmission problems of intelligent electric meter.
Chinese invention patent application CN103685467A discloses a kind of Internet of Things and interconnects platform and communication means thereof, uses internet communication mode to solve multiple data transmission problems.
There is following defect in prior art:
(1) different equipment sources needs corresponding different equipment cut-in methods, all indicates a certain special communication apparatus in above-mentioned three sections of patent applications, and therefore its hardware and software cost is high, is also unfavorable for that the later stage expands.
(2) device type is varied, if realizing different services for different equipment, then not only R&D work amount is huge, is also unfavorable for management and later maintenance.
(3) current data access process regimes underuses the resource allocation rule of incoming end, and during some equipment access-in resource deficiencies, some resource is then just idle.
(4) it is unfavorable for expanding, is difficult to new device type or new data type support or cannot support at all.
Summary of the invention
It is an object of the invention to provide a kind of data acquisition equipment uniform access system based on business separation principle, after this system is run, dynamically can access the relevant configuration of concrete communication mode and data form by the equipment in reading database or configuration file, the whole flow process that device data accesses can be completed; Without revising any system implementations when follow-up new equipment accesses, the data access of multiple multiple equipment can be realized.
The technical scheme is that
A kind of data acquisition equipment uniform access system based on business separation principle, this system includes configuration initialization module, thread pool initialization module, Communications service module, isolation module, analysis service module and cache module;
Described configuration initialization module, for using the initial configuration information of scene to be loaded in system when system start-up by difference;
Described thread pool initialization module, for directly initiating each service in the way of single-threaded, including NETTY service, WebService service, HTTP service and file service, it is additionally operable to start communication thread pond with the capacity less than certain threshold value and resolve thread pool;
Described Communications service module, data access for responding collecting device is asked, when there is idle thread in communication thread pond, directly distribution thread response data access request, when communication thread pond is absent from idle thread, if the capacity in communication thread pond not yet reaches predetermined threshold value, then increase the capacity in communication thread pond, reallocation thread response data access request, if the capacity in communication thread pond has reached predetermined threshold value, then control data access request enter thread queuing queue with etc. thread to be allocated;
Described isolation module, is used for isolating Communications service module and analysis service module, after the end-of-job of Communications service module, the data that interim memory communicating service module accesses, and certain pre-conditioned under, start analysis service module;
Described analysis service module, for response data analysis request, when there is idle thread in parsing thread pool, directly distribution thread response data analysis request, when being absent from idle thread in parsing thread pool, if the capacity resolving thread pool not yet reaches predetermined threshold value, then increase the capacity resolving thread pool, reallocation thread response data analysis request, if resolve thread pool capacity reached predetermined threshold value, then control data parsing request enter thread queuing queue with etc. thread to be allocated;
Described cache module, for being completed the storage to the data that analysis service module resolves by two-stage data buffer storage, the temporal cache of the complete paired data of first order data buffer storage, Tong Bu with real data resolving, the final buffer memory of the complete paired data of second level data buffer storage also stores data into data base, asynchronous with real data resolving.
The described data acquisition equipment uniform access system based on business separation principle, this system also includes configuration injection module;
Described configuration injection module, for completing the renewal to configuration information by two stage arrangement buffer memory, first order allocating cache completes the temporal cache to configuration information, request is changed Tong Bu with actual disposition information, second level allocating cache completes the final buffer memory to configuration information and selects suitable time and scene to complete the renewal to configuration information, asynchronous with the change request of actual disposition information.
The described data acquisition equipment uniform access system based on business separation principle, this system also includes daily record monitoring management module;
Described daily record monitoring management module, for completing the storage to daily record by two-stage log buffer, first order log buffer completes the temporal cache to daily record, Tong Bu with actual log production process, second level log buffer completes the final buffer memory to daily record and daily record is stored data base, asynchronous with actual log production process.
The described data acquisition equipment uniform access system based on business separation principle, described Communications service module, it is additionally operable to, when thread queuing queue is absent from the data access request queued up, from communication thread pond, reclaim a certain proportion of idle thread.
The described data acquisition equipment uniform access system based on business separation principle, described analysis service module, it is additionally operable to, when thread queuing queue is absent from the data parsing request queued up, thread pool analytically reclaim a certain proportion of idle thread.
The invention have the benefit that
As shown from the above technical solution, the present invention compared with prior art, has the advantage that
(1) present invention is without secondary development, once realizes, recycles, puts things right once and for all;
(2) present invention is unrelated with various industrial natures, also substantially unrelated with various data acquisition equipment types, can fully be applied in each traditional industries containing model of data service and novel industry;
(3) optimized integration of the present invention is theoretical comprehensive and extensive, and thinking is novel;
(4) each module of the present invention realizes each through software, the problem solving current every profession and trade ubiquitous " to increase hardware raising performance ", make full use of the software approach such as data buffer storage, thread distribution, function isolation, asynchronous process, be conducive to the later stage to expand.
Accompanying drawing explanation
Fig. 1 is the structural representation of the present invention.
Detailed description of the invention
The present invention is further illustrated below in conjunction with the drawings and specific embodiments.
As shown in Figure 1, a kind of data acquisition equipment uniform access system based on business separation principle, including configuration initialization module 1, thread pool initialization module 2, Communications service module 3, isolation module 4, analysis service module 5, cache module 6, configuration injection module 7 and daily record monitoring management module 8.
Described configuration initialization module 1, for using the initial configuration information of scene to be loaded in system when system start-up by difference.
One emphasis thinking of the present invention configures completion system " various applicable scenes are inconsistent " function by early stage, it is believed that " configuration decision systems ", when, after the various content needing configuration of configuration, system just can start and these configuration informations are rationally loaded. In configuration initializes, for needing the configuration information of life-time service to adopt the mode of overall situation Cache to store, for needing the configuration information of medium-term and long-term use to adopt the mode of global variable to store, the mode disposable configuration information being adopted to local variable stores, this kind of realization is advantageous in that different configurations has different life cycle, then the configuration information classification different in size for these life cycle processes internal memory and the IO expense that just can greatly reduce system.
Thread pool initialization module 2, for directly initiating each service in the way of single-threaded, including NETTY service, WebService service, HTTP service and file service, is additionally operable to start communication thread pond with the capacity less than certain threshold value and resolve thread pool.
Service single-threaded startup: any one communication of middle finger of the present invention and request of data all adopt the mode of single-threaded service encapsulation to carry out completion system service, this single-threaded service and main service. In the present invention, the access procedure of all data acquisition equipments has relied on service, and service externally provides the unique interface mode after encapsulation, and data acquisition equipment calls this interface. The data acquisition equipment of current main flow suffers from least one of which and meets the data access standard of international norm, and the service targetedly that is therefore provided with is the optimized integration place of the present invention. The present invention does not rely on concrete data acquisition equipment type, and also not dependent on concrete industrial nature, any data acquisition equipment with current international norm can both carry out data access in the form of services. The main services of the present invention has two classes: the first kind refers to Communications service, namely provide the Netty for bottom hardware communication apparatus such as transport information equipment to service, such as WebService between internet device and HTTP service for general network communication protocol, for the File Transfer Protocol service etc. that all kinds of mass file of every profession and trade such as reconciliation file etc. transmits; Equations of The Second Kind refers to analysis service, namely provides the data parsing of fixed length, the traditional data format such as elongated, it is provided that the information analysis of the WEB such as XML and JSON and non-internet data format, and the mass data additionally providing various file type resolves. The system resource of ample resources or consuming that these services will not expend system when starting is negligible on the whole, and this is the theoretical basis that the present invention realizes.
Communications service module 3, data access for responding collecting device is asked, when there is idle thread in communication thread pond, directly distribution thread response data access request, when communication thread pond is absent from idle thread, if the capacity in communication thread pond not yet reaches predetermined threshold value, then increase the capacity in communication thread pond, reallocation thread response data access request, if the capacity in communication thread pond has reached predetermined threshold value, then control data access request enter thread queuing queue with etc. thread to be allocated; It is additionally operable to, when thread queuing queue is absent from the data access request queued up, from communication thread pond, reclaim the idle thread of certain proportion (such as 2/3).
Communications service is regular based on distribution and the reclaim mechanism of thread pool, it is achieved the various scheduling of various Communications services and function, with the data access communication request of completion system Yu data acquisition equipment. Communications service is one of main service of system, Communications service due to the data acquisition equipment that relates to is many and request frequently, therefore require higher for real-time responsiveness, it is first mutual that Communications service completes between data acquisition equipment and system, after communication request completes, this communication thread terminates, and Communications service terminates.
The server access capability that communication thread distribution and the purpose reclaimed cause when being in that to solve substantial amounts of Data Concurrent access request declines rapidly thus causing main frame to be delayed the problem of machine or system loss data, the stability weakening the whole data access request of direct relation of server access capability and integrity. System can load an a small amount of communication thread tankage when starting, and is traditionally arranged to be heap(ed) capacity 1/10th even less in communication thread pond, and the principle of this capacity setting has been able to the system on basis and has run or a small amount of data access request. When the number of data access service thread cannot meet data access request, system can increase the capacity in communication thread pond until reaching the heap(ed) capacity in the communication thread pond that system is arranged, this heap(ed) capacity depends on CPU core calculation and the memory size of system place server, and in general the heap(ed) capacity in communication thread pond is about the twice of CPU core calculation; If data access request, then the thread queuing queue designed by entrance system still cannot be met, wait thread to be allocated to process follow-up data access request; Meanwhile, when in system, the thread free time in communication thread pond exceedes certain threshold value, system will reclaim the thread of these free time under certain condition until the minimum capacity in communication thread pond that allows of system. Time the scene that communication distribution service realizes is in that data access request is quickly or frequently.
Isolation module 4, is used for isolating Communications service module 3 and analysis service module 5, after the end-of-job of Communications service module 3, the data that interim memory communicating service module 3 accesses, and certain pre-conditioned under, start analysis service module 5.
The isolation of Communications service and analysis service is realized by maximum decoupling in the present invention. In the present invention, Communications service and analysis service are substantially free of common factor, namely the way resolved is carried out after changing traditional communication, in the present invention, communication terminates not represent parsing and starts, resolve and terminate also not represent wait communication request next time, namely between the two without substantial dependence, the isolation module 4 of system only can be illustrated after completing data access request by Communications service module 3, analysis service module 5 also only can wait the scheduling etc. of isolation module 4.
The startup of analysis service is to rely on default certain or certain the several conditions of system and completes, and before this, the data that Communications service produces are temporarily stored, and the entry condition of analysis service mainly has three classes, is described in detail below:
A, time-based condition, refer to analysis service and start according to the time rule that system configuration is set, as completed by every 5 minutes, within every 3 minutes and 5 minutes, be alternately accomplished.
B, condition based on event, refer to that analysis service reaches certain magnitude in data, data burst formula increases, starting under the situation such as more exception occur suddenly in data.
C, the condition that both are comprehensive, refer to analysis service and start under one or the integrated condition that all meets in time-based condition and meeting based on the condition of event.
After analysis service starts, the data of interim storage being carried out extensive batch and resolve, its execution efficiency is much higher when can resolve than single.
Analysis service module 5, for response data analysis request, when there is idle thread in parsing thread pool, directly distribution thread response data analysis request, when being absent from idle thread in parsing thread pool, if the capacity resolving thread pool not yet reaches predetermined threshold value, then increase the capacity resolving thread pool, reallocation thread response data analysis request, if resolve thread pool capacity reached predetermined threshold value, then control data parsing request enter thread queuing queue with etc. thread to be allocated; It is additionally operable to, when thread queuing queue is absent from the data parsing request queued up, thread pool analytically reclaim a certain proportion of idle thread.
Analysis service also relies on distribution and the reclaim mechanism rule of thread pool, but it is in that data parsing process will not must carry out the release work of service ability as Communications service within very short time with the fundamental difference of Communications service, although therefore data parsing function expense is relatively big, but will not be too high to requirement of real-time. Therefore in this case, the distribution of thread is metastable. Analysis service does not generally have resolving problem, unless there are the extreme cases such as server power-off. The secondary that analysis service completes between data acquisition equipment and system is mutual. After analysis request completes, this parsing thread terminates, and analysis service terminates.
Parsing thread distribution and the purpose reclaimed are in that the problem that parsing is delayed or data parsing ability does not catch up with solving to cause owing to resolving is relatively slow in actual environment, this result also result in server access capability decline rapidly thus cause the machine of delaying or lose data problem, server data analytic ability weaken the stability and integrity that also can be related to whole server. System can load an a small amount of parsing thread pool capacity when starting, and is traditionally arranged to be the 1/10th even less of the heap(ed) capacity that resolves thread pool, and the principle of this capacity setting has been able to the system on basis and has run or basic data parsing function. When the number of data parsing service thread cannot meet data parsing request, system can increase the capacity resolving thread pool until reaching the heap(ed) capacity resolving thread pool of system setting, this heap(ed) capacity depends on CPU core calculation and the memory size of system place server, and the heap(ed) capacity in general resolving thread pool is about the twice of CPU core calculation; If data parsing request, then the thread queuing queue designed by entrance system still cannot be met, wait thread to be allocated to process follow-up data parsing request; Meanwhile, when the thread free time resolving thread pool in system exceedes certain threshold value, system will reclaim the thread of these free time under certain condition until system allow resolve thread pool minimum capacity.
Cache module 6, for being completed the storage to the data that analysis service module 5 resolves by two-stage data buffer storage, the temporal cache of the complete paired data of first order data buffer storage, Tong Bu with real data resolving, the final buffer memory of the complete paired data of second level data buffer storage also stores data into data base 9, asynchronous with real data resolving.
The purpose of data buffer storage is in that to reduce the IO expense in data base's CRUD process. It is known that the IO ability of global computer system does not have substantial change at present, I O process is often topmost overhead problem in system, then access for the mass data in the present invention and storage problem seems important especially. The present invention uses two-level cache mechanism, two-stage synchronization mechanism to solve this problem. Two-level cache mechanism refers to that data are carried out two-level cache by the data buffer storage by two ranks and affair mechanism, first order buffer memory is the interim storage to resolution data, second level buffer memory completes the real cache to the data being about to formal storage and final storage, two-level cache achieves data exchange agreement namely the data distribution mechanism of buffer memory. Two-stage synchronization mechanism refers to that finally the data cached mode by two dimensions of time and space carries out data storage, and time aspect is to rely on what configuration timing data task completed, and space aspect is to rely on what configuration quantitative data task completed. By the two mechanism, the IO expense of system can be reduced to minimum.
Configuration injection module 7, for completing the renewal to configuration information by two stage arrangement buffer memory, first order allocating cache completes the temporal cache to configuration information, request is changed Tong Bu with actual disposition information, second level allocating cache completes the final buffer memory to configuration information and selects suitable time and scene to complete the renewal to configuration information, asynchronous with the change request of actual disposition information.
The present invention adopts configuration to inject mechanism, the i.e. active synchronization of configuration information and asynchronous implement mechanism, due to configuration information in systems life cycle it may happen that bigger change, so the alteration of these information is accomplished by proactive notification to system, completing configuration by two-stage data synchronization mechanism in the present invention to inject, namely the active synchronization interface of data realizes the asynchronous middleware processes with data isolation. Active synchronization interface refers to when configuration information changes, and system actively calls the method for synchronization to inform that configuration information changes request; The asynchronous middleware of isolation refers to configuration modification and asks can not immediately configuration information be come into force when synchronizing; the existing task run exception caused to prevent configuration information from changing immediately or fault, system problem and other unknown situation occur; therefore the purpose of this middleware is in that to select suitable time and scene to complete the renewal of configuration information, and this is asynchronous middleware processes. Using two-level cache mechanism to coordinate realization configuration to inject mechanism in the present invention, first order buffer memory keeps the synchronization request work that configuration information changes, and second level buffer memory realizes configuration information and changes the asynchronous process work of request.
Daily record monitoring management module 8, for completing the storage to daily record by two-stage log buffer, first order log buffer completes the temporal cache to daily record, Tong Bu with actual log production process, second level log buffer completes the final buffer memory to daily record and daily record stores data base 9, asynchronous with actual log production process.
Access due to mass data, the storage of log information also becomes more and more huger, the daily record storage in past generally uses pattern that a daily record once stores or due to the huge clear-cut thoroughly abort log storage of daily record, and its consequence brought is that the expense pressure of computer IO is very big or the problem such as the record disappearance of data procedures. in the present invention, propose log buffer mechanism, according to this mechanism, the result of daily record will not be made directly data Layer persistence, but under meeting certain condition, just carry out persistence work, this condition mainly has three kinds of possibilities: on time, according to quantity or the daily record persistence work completed during other specific conditions such as system burst exception, also due to log buffer has resource competing problem, the present invention also using two-level cache mechanism and rationally solve this problem, wherein first order log buffer mechanism maintains the temporal cache purpose of daily record, this buffer memory carries out synchronize with actual log generation, second level log buffer mechanism just achieves final buffer memory and the persistence purpose of daily record, this buffer memory is all asynchronous operation with actual log production process and last persistence process.
The operation principle of the present invention:
Business separation principle refers to the present invention and does not rely on practical business link, and links is separate, and the present invention is also substantially unrelated with industry simultaneously. The present invention is compatible with all kinds hardware data collecting device of whole world main flow at present, and these data acquisition equipments are dispersed throughout the industry-by-industries such as such as traffic, bank, communication and the Internet. Data acquisition equipment can use ICP/IP protocol, udp protocol etc. based on the data communication mode of hardware bottom layer, it is also possible to uses and carries out communication based on the document communication mode of File Transfer Protocol and the data access end of the present invention. Any data acquisition equipment both can select any one communication mode simultaneously, has also selected the data access end of communication and the present invention to interact, independent and mutually unaffected between data communication. Type of data format mutual between data acquisition equipment and data access end comprises the multiple types of datas such as fixed length type, elongated type, XML-style, JSON type, file type, optional one or more data forms therein of any one data acquisition equipment and data access end carry out data interaction, and degree of flexibility is higher.
Simultaneously, the present invention is the mode adopting service encapsulation for response and the request of any one means of communication and data exchange format, take full advantage of multi-thread mechanism simultaneously, both ensure that the effectiveness of data, ensure again the safety of communication, more made whole system externally expand and become very simple. All communication modes in the present invention or data form all adopt the means of configuration, and namely user only needs, by the simple several steps operation of the external configuration feature in the present invention, to complete the access of new categorical data collecting device, and whole process is substantially Z-operation.
Assuming that the resource-constrained of data access end be assumed to 100, the way so pass by is: certain categorical data collecting device is distributed to the resource of 20, the resource of certain categorical data collecting device distribution 30, in this case, resource is distributed according to quantity in the access procedure of certain categorical data collecting device, namely resource and data acquisition equipment type are strictly bound, so due to the number of data acquisition equipment itself, network, the factor such as stable, the data acquisition equipment wasting of resources that may result in, but some data acquisition equipment resource wretched insufficiencies. In the present invention, also assume that data access end total resources is 100, when system initialization, only that the distribution of various data acquisition equipments is very little initialization stock number, such as certain data acquisition equipment Resources allocation is 1, this sub-fraction resource is only safeguarded or monitoring at the routine of data access end for this data acquisition equipment, and the real data being not involved in this data acquisition equipment and data access end accesses. Hereafter, system can initialize one " thread little Chi ", this pond comprises communication thread pond and resolves thread pool two types, this thread pool will not for the data acquisition equipment of a certain type, when there being data access, system can apply in thread pool wherein one or more resources for the mutual of data acquisition equipment and data access, after completing alternately, this thread discharges, and after meeting certain condition, this thread will be recovered in thread pool. In this process, thread pool can increase to the maximum tankage of system permission and access with the data burst adapting to high-volume data acquisition equipment, can also being reduced to minimum tankage to save system resource, the management of thread pool completes to distribute service. Simultaneity factor uses the problems such as the system failure, the loss of data that QUEUE mechanism prevents from may result in owing to sudden big request of data causes system thread pool explosive growth.
Along with the explosive growth of data, host resource increasingly seems shortage, and except making full use of each host resource except the various software frames used based on distributed treatment at present, reasonable employment and resource distribution to each host resource are also particularly important. The core distributing service in the present invention is distribution and management thread pool, distribution and management thread pool contain a series for the treatment of mechanisms such as distribution, recovery and monitoring, achieving the distribution of the configuration to system and management, data distribution and management and resource dissemination and management, all system services such as Communications service, analysis service are carried out overall resource dissemination and resource management and control by distribution service.
The Communications service of the present invention, refer to the general approach principle based on the data access of domestic and international main flow at present, these methods are packaged, finally in the form of services data access request is carried out supporting, the means of communication being directed to mainly comprise the socket communication based on TCP/IP and UDP, the seeervice level communication of sing on web Service etc., FTP and the SFTP communication based on file transmission and the WEB communication etc. based on http protocol.
The analysis service of the present invention, refers to the data format specifications that data communication is used and carries out " being not intended to property " data parsing. The implication of " being not intended to property " refers to general document format data is both provided analysis service, wherein both contained the fixed-length data format based on fixed length byte or character etc., based on traditional data exchange format such as separator or the elongated data forms separating mark, contain again the general XML data based on XML protocol, the internet data exchange format such as JSON data based on JSON agreement, further comprises the complex data exchange format of the types such as user-defined format data and file.
Caching mechanism can effectively solve the problem that current many number systems in actual scene owing to device category and number are various, performance bottleneck that the problems such as data volume is huge that equipment produces are brought and disaster, therefore to be uniformly accessed in process at the data acquisition equipment of the present invention essential for caching mechanism, the reasonable employment of buffer memory can also be greatly reduced the loss problem of data simultaneously, fully save main frame and database resource simultaneously.
Realize in process in the present invention, isolation mech isolation test is have employed between analysis service and Communications service, although namely the final purpose of communication is data parsing, the preposition dependence of data parsing is communication, really there is logical relation between the two, but in order to realize decoupling and high customizability, system carries out business isolation by the mode of " service isolation ", its result is communication and data parsing no longer exists obvious logical relation, so that communication and two processes of data parsing are isolated and come. In addition, several correlation functions of the present invention, as configuration, daily record, data, service all take business isolation principle, namely certain function will not depend upon additionally other function, this have the advantage that all types of user have only to when using and perform attended operation to be concerned about its actual wherein a certain piece function related to, and to other parts without too much understanding.
Data acquisition equipment purposes involved in the present invention is varied, comprise transport information class, Provider Equipment communication class, database purchase class, banking equipment class, the Internet O&M host data communication apparatus, these equipment are as peripheral data nucleus equipment, and being mainly the present invention provides data source to support. Completing data interaction by certain communication modes between the present invention and these equipment, these communication modes comprise the internet communication based on serial ports, GPRS, EDGE, 4G network etc. based on bluetooth, the wireless telecommunications of Wi-Fi and operator.
In theory, the present invention does not rely on industrial nature, does not rely on concrete data acquisition equipment type, does not rely on data acquisition equipment manufacturing firm. General, the data acquisition equipment of any support current mainstream data software communication agreement and international data interaction form can access, The present invention reduces work when data acquisition equipment accesses and the significant cost of later maintenance simultaneously, technically reliable, easy to operate. The present invention is applicable not only to the traditional industries such as traffic, bank, operator, applies also for the Novel Internet industry such as O2O, ecommerce.
The above embodiment is only that the preferred embodiment of the present invention is described; not the scope of the present invention is defined; under the premise designing spirit without departing from the present invention; various deformation that technical scheme is made by those of ordinary skill in the art and improvement, all should fall in the protection domain that claims of the present invention are determined.
Claims (5)
1. the data acquisition equipment uniform access system based on business separation principle, it is characterised in that: this system includes configuration initialization module, thread pool initialization module, Communications service module, isolation module, analysis service module and cache module;
Described configuration initialization module, for using the initial configuration information of scene to be loaded in system when system start-up by difference;
Described thread pool initialization module, for directly initiating each service in the way of single-threaded, including NETTY service, WebService service, HTTP service and file service, it is additionally operable to start communication thread pond with the capacity less than certain threshold value and resolve thread pool;
Described Communications service module, data access for responding collecting device is asked, when there is idle thread in communication thread pond, directly distribution thread response data access request, when communication thread pond is absent from idle thread, if the capacity in communication thread pond not yet reaches predetermined threshold value, then increase the capacity in communication thread pond, reallocation thread response data access request, if the capacity in communication thread pond has reached predetermined threshold value, then control data access request enter thread queuing queue with etc. thread to be allocated;
Described isolation module, is used for isolating Communications service module and analysis service module, after the end-of-job of Communications service module, the data that interim memory communicating service module accesses, and certain pre-conditioned under, start analysis service module;
Described analysis service module, for response data analysis request, when there is idle thread in parsing thread pool, directly distribution thread response data analysis request, when being absent from idle thread in parsing thread pool, if the capacity resolving thread pool not yet reaches predetermined threshold value, then increase the capacity resolving thread pool, reallocation thread response data analysis request, if resolve thread pool capacity reached predetermined threshold value, then control data parsing request enter thread queuing queue with etc. thread to be allocated;
Described cache module, for being completed the storage to the data that analysis service module resolves by two-stage data buffer storage, the temporal cache of the complete paired data of first order data buffer storage, Tong Bu with real data resolving, the final buffer memory of the complete paired data of second level data buffer storage also stores data into data base, asynchronous with real data resolving.
2. the data acquisition equipment uniform access system based on business separation principle according to claim 1, it is characterised in that: this system also includes configuration injection module;
Described configuration injection module, for completing the renewal to configuration information by two stage arrangement buffer memory, first order allocating cache completes the temporal cache to configuration information, request is changed Tong Bu with actual disposition information, second level allocating cache completes the final buffer memory to configuration information and selects suitable time and scene to complete the renewal to configuration information, asynchronous with the change request of actual disposition information.
3. the data acquisition equipment uniform access system based on business separation principle according to claim 1, it is characterised in that: this system also includes daily record monitoring management module;
Described daily record monitoring management module, for completing the storage to daily record by two-stage log buffer, first order log buffer completes the temporal cache to daily record, Tong Bu with actual log production process, second level log buffer completes the final buffer memory to daily record and daily record is stored data base, asynchronous with actual log production process.
4. the data acquisition equipment uniform access system based on business separation principle according to claim 1, it is characterized in that: described Communications service module, it is additionally operable to, when thread queuing queue is absent from the data access request queued up, from communication thread pond, reclaim a certain proportion of idle thread.
5. the data acquisition equipment uniform access system based on business separation principle according to claim 1, it is characterized in that: described analysis service module, it is additionally operable to, when thread queuing queue is absent from the data parsing request queued up, thread pool analytically reclaim a certain proportion of idle thread.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610017431.8A CN105635298B (en) | 2016-01-12 | 2016-01-12 | A kind of data acquisition equipment uniform access system based on business separation principle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610017431.8A CN105635298B (en) | 2016-01-12 | 2016-01-12 | A kind of data acquisition equipment uniform access system based on business separation principle |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105635298A true CN105635298A (en) | 2016-06-01 |
| CN105635298B CN105635298B (en) | 2018-12-18 |
Family
ID=56049769
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610017431.8A Active CN105635298B (en) | 2016-01-12 | 2016-01-12 | A kind of data acquisition equipment uniform access system based on business separation principle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105635298B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109101440A (en) * | 2018-08-01 | 2018-12-28 | 浪潮软件集团有限公司 | Method for processing concurrent request of trace data based on JVM (Java virtual machine) cache |
| CN109408240A (en) * | 2018-10-31 | 2019-03-01 | 四川长虹电器股份有限公司 | Terminal device data collection system based on Netty frame |
| CN109614412A (en) * | 2018-12-11 | 2019-04-12 | 江苏瑞中数据股份有限公司 | Data publication service two-stage shared buffer memory analysis strategy on cloud based on power industry |
| CN111314212A (en) * | 2020-01-16 | 2020-06-19 | 青木数字技术股份有限公司 | API gateway based on Netty and plug-in mechanism and control method |
| CN111580476A (en) * | 2020-04-30 | 2020-08-25 | 联宝(合肥)电子科技有限公司 | Intelligent production management method, equipment and computer readable medium |
| CN112731850A (en) * | 2020-12-30 | 2021-04-30 | 湖南嘿哈猫网络科技有限公司 | Intelligent home box based on Internet of things |
| CN113037710A (en) * | 2021-02-03 | 2021-06-25 | 南京物通物语科技有限公司 | Processing method for big data transmission for industrial Internet of things |
| CN114095537A (en) * | 2021-11-18 | 2022-02-25 | 重庆邮电大学 | Netty-based mass data access method and system in application of Internet of things |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859104A (en) * | 2005-09-28 | 2006-11-08 | 华为技术有限公司 | Service isolation transfer method |
| US20070104095A1 (en) * | 2005-11-08 | 2007-05-10 | Alcatel | Communication traffic isolation and control |
| CN102307224A (en) * | 2011-05-17 | 2012-01-04 | 华为软件技术有限公司 | Method and device for realizing service isolation in cloud computing system, and cloud server |
| CN104753776A (en) * | 2013-12-31 | 2015-07-01 | 中兴通讯股份有限公司 | Service isolation handling method, device, system, DPU (distributed processing unit) and network adapter |
-
2016
- 2016-01-12 CN CN201610017431.8A patent/CN105635298B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859104A (en) * | 2005-09-28 | 2006-11-08 | 华为技术有限公司 | Service isolation transfer method |
| US20070104095A1 (en) * | 2005-11-08 | 2007-05-10 | Alcatel | Communication traffic isolation and control |
| CN102307224A (en) * | 2011-05-17 | 2012-01-04 | 华为软件技术有限公司 | Method and device for realizing service isolation in cloud computing system, and cloud server |
| CN104753776A (en) * | 2013-12-31 | 2015-07-01 | 中兴通讯股份有限公司 | Service isolation handling method, device, system, DPU (distributed processing unit) and network adapter |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109101440A (en) * | 2018-08-01 | 2018-12-28 | 浪潮软件集团有限公司 | Method for processing concurrent request of trace data based on JVM (Java virtual machine) cache |
| CN109101440B (en) * | 2018-08-01 | 2023-08-04 | 浪潮软件集团有限公司 | Method for processing trace data concurrent request based on JVM (Java virtual machine) cache |
| CN109408240A (en) * | 2018-10-31 | 2019-03-01 | 四川长虹电器股份有限公司 | Terminal device data collection system based on Netty frame |
| CN109614412A (en) * | 2018-12-11 | 2019-04-12 | 江苏瑞中数据股份有限公司 | Data publication service two-stage shared buffer memory analysis strategy on cloud based on power industry |
| CN109614412B (en) * | 2018-12-11 | 2023-06-02 | 江苏瑞中数据股份有限公司 | Cloud data release service two-stage sharing cache analysis method based on power industry |
| CN111314212A (en) * | 2020-01-16 | 2020-06-19 | 青木数字技术股份有限公司 | API gateway based on Netty and plug-in mechanism and control method |
| CN111580476A (en) * | 2020-04-30 | 2020-08-25 | 联宝(合肥)电子科技有限公司 | Intelligent production management method, equipment and computer readable medium |
| CN111580476B (en) * | 2020-04-30 | 2021-05-11 | 联宝(合肥)电子科技有限公司 | Intelligent production management method, equipment and computer readable medium |
| CN112731850A (en) * | 2020-12-30 | 2021-04-30 | 湖南嘿哈猫网络科技有限公司 | Intelligent home box based on Internet of things |
| CN113037710A (en) * | 2021-02-03 | 2021-06-25 | 南京物通物语科技有限公司 | Processing method for big data transmission for industrial Internet of things |
| CN114095537A (en) * | 2021-11-18 | 2022-02-25 | 重庆邮电大学 | Netty-based mass data access method and system in application of Internet of things |
| CN114095537B (en) * | 2021-11-18 | 2023-07-14 | 重庆邮电大学 | Netty-based mass data access method and system in Internet of things application |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105635298B (en) | 2018-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105635298A (en) | Service isolation principle based data acquisition equipment unified access system | |
| CN110908879B (en) | Reporting method, reporting device, reporting terminal and recording medium of buried point data | |
| WO2014206289A1 (en) | Method and apparatus for outputting log information | |
| CN102323894B (en) | System and method for realizing non-blockage mutual calling in distributed application of enterprise | |
| CN103995755A (en) | Server optimizing method based on mobile terminal and server system | |
| CN103067425A (en) | Creation method of virtual machine, management system of virtual machine and related equipment thereof | |
| CN102467412B (en) | Method, device and business system for processing operation request | |
| CN103117895B (en) | A kind of services listen method of LXI instrument | |
| CN107682460B (en) | Distributed storage cluster data communication method and system | |
| CN104077212A (en) | Pressure test system and method | |
| CN103368986A (en) | Information recommendation method and information recommendation device | |
| CN109218100A (en) | Distributed objects storage cluster and its request responding method, system and storage medium | |
| CN107070709B (en) | NFV (network function virtualization) implementation method based on bottom NUMA (non uniform memory Access) perception | |
| WO2023046141A1 (en) | Acceleration framework and acceleration method for database network load performance, and device | |
| EP4293987A1 (en) | Information processing method based on internet-of-things device, and related device and storage medium | |
| US20140068165A1 (en) | Splitting a real-time thread between the user and kernel space | |
| CN109756461A (en) | A kind of remote procedure calling (PRC) method based on NETTY | |
| CN110290190A (en) | A kind of method, monitored device and monitoring server acquiring data | |
| CN106131162A (en) | A kind of method realizing network service agent based on IOCP mechanism | |
| CN112243033B (en) | Method, device and equipment for acquiring memory information and computer readable storage medium | |
| CN109241157A (en) | Data calling method, device, communication equipment and storage medium | |
| CN108075989B (en) | Extensible protocol-based load balancing network middleware implementation method | |
| CN104349506A (en) | Association management method, wireless access device and system | |
| CN207939557U (en) | A kind of statistics of data acquisition hardware platform | |
| CN112463167B (en) | Bank account dynamic account notification service system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A Unified Access System for Data Acquisition Equipment Based on Business Isolation Principle Effective date of registration: 20230807 Granted publication date: 20181218 Pledgee: China Construction Bank Hefei Zhonglou sub branch Pledgor: ANHUI KELI INFORMATION INDUSTRY Co.,Ltd. Registration number: Y2023980051050 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |