CN105912306B - A kind of method of the data processing of high concurrent Platform Server - Google Patents
A kind of method of the data processing of high concurrent Platform Server Download PDFInfo
- Publication number
- CN105912306B CN105912306B CN201610225491.9A CN201610225491A CN105912306B CN 105912306 B CN105912306 B CN 105912306B CN 201610225491 A CN201610225491 A CN 201610225491A CN 105912306 B CN105912306 B CN 105912306B
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- server
- internet
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000012545 processing Methods 0.000 title claims abstract description 79
- 238000000034 method Methods 0.000 title claims abstract description 15
- 230000009471 action Effects 0.000 claims description 2
- 230000001934 delay Effects 0.000 claims 1
- 230000003139 buffering effect Effects 0.000 abstract description 4
- 230000008569 process Effects 0.000 abstract description 3
- 230000005540 biological transmission Effects 0.000 abstract description 2
- 238000007667 floating Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000005314 correlation function Methods 0.000 description 2
- 239000003921 oil Substances 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000002826 coolant Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000010705 motor oil Substances 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/38—Concurrent instruction execution, e.g. pipeline or look ahead
- G06F9/3836—Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention relates to the high concurrent processing of the data to Internet of Things background server.The present invention will receive byte type data by JNI interfaces and be sent to the Transmit-Receive Unit of dynamic link libraries built in server, and current data is stored in idle data buffering area when fill up after by it, is marked as saturation and idle processing unit transmission processing notification;Alignment processing unit reads data simultaneously locks current data buffering area and by oneself state labeled as busy;Then parallel instruction is called to carry out data processing and handling result is stored in result buffer, after being disposed, by current data buffer tag for the free time and by oneself state labeled as the free time;Transmit-Receive Unit reading process is as a result, and pass through the physical quantity database that JNI interfaces send it to server.The method can significantly improve the data-handling capacities of the Internet of Things background server of SSH frameworks, have not only used the good network ability of java language, but also make use of the processing capacities of server CPU in itself.
Description
Technical field
The present invention relates to internet of things field, and in particular at the high concurrent of the data of Internet of Things background server
Reason.
Background technology
At present, for internet of things equipment, people's general concern be terminal acquisition and sending function, it is micro- due to terminal
Controller processing capacity is weaker, generally only does a little simple analysis, background server due to there is great amount of terminals to communicate, it
It stores after simply handling data, and is shown in a suitable manner to user.
For the framework of Internet of Things background server, generally using SSH (struts2+spring+hibernate) framework,
I.e. using the good network performance of java language, realize and carry out high concurrent processing to largely connecting, that is, pass through java language institute
After the network data interface write receives the data that Internet of Things collection terminal is sent, by transferring the JVM of java language, (java is virtual
Machine) correlation function high concurrent processing is carried out to the byte type data of reception and is stored in the predetermined physical of Internet of Things background server
It measures in database, obtains that the handling result that user shows can be used for.And the data of Internet of Things collection terminal it is general the characteristics of be use
Single byte, double byte or integer represent physical quantity, when the Internet of Things background server of SSH frameworks is connect by network data interface
Receive Internet of Things collection terminal transmission data after, it is necessary to by the byte type data conversion received into floating number, then do floating number
Co-efficient multiplication, be finally based on the data processings such as the add operation of offset and can just become real physical quantity and (can be directly used for
Shown to user), by taking double byte as an example, when being realized based on java language to receiving data progress data processing, pass through transfer
The correlation function of the JVM of java language realizes that conversion, multiplication and offset handle to obtain physical quantity, i.e.,:
For java language, its advantage is cross-platform and network performance, but data processing, especially at high concurrent data
Reason is its weakness, and when being related to high concurrent data, when big data quantity arrives, the processing of data is even more the bottleneck of system.
The content of the invention
The goal of the invention of the present invention is:In order to overcome the Internet of Things background server for using SSH frameworks based on java
Language carries out data processing, when obtaining corresponding actual physical amount, the insufficient skill of processing capacity of java language to receiving data
Art problem provides a kind of method of the data processing of high concurrent Platform Server, can use the good network energy of java language
Power, and the processing capacity of Platform Server can be improved.
A kind of method of the data processing of high concurrent Platform Server of the present invention, comprises the following steps:
When the Internet of Things background server of SSH frameworks is received by network data interface (being realized by java Programming with Pascal Language)
After the data that Internet of Things collection terminal is sent, the byte type data for representing physical quantity are passed through into JNI interfaces (Java Native
Interface, java local interface) it is sent to the dynamic link library built in Internet of Things background server, it is preferred that dynamic link
Storehouse is based on C language programming and realizes;Wherein, dynamic link library includes Transmit-Receive Unit, data buffer zone, processing unit and result buffering
Area, and the original state of processing unit and data buffer zone is the free time;
Transmit-Receive Unit selects state to be connect for idle data buffer zone deposit by JNI interface byte type data
Byte type data of receipts, while the data buffer zone of current accessed is locked, after data are filled up, Transmit-Receive Unit is by current accessed
Buffer tag for saturation, and send data processing notice to state for idle processing unit;
The processing unit for receiving data processing notice reads data from state for the data buffer zone of saturation, and will currently visit
The data buffer zone asked locks and by oneself state labeled as busy;Processing unit calls parallel instruction to carry out data processing simultaneously
Handling result is stored in result buffer;After data processing, by the data buffer zone of current accessed labeled as idle and
By oneself state labeled as the free time;Transmit-Receive Unit reads the handling result of result buffer, and passes through JNI interfaces by handling result
It is sent to the physical quantity database of Internet of Things background server.
Main idea is that data are become batch processing from single processing, it will be using the Internet of Things of SSH frameworks
The data processing of background server by call parallel instruction carry out data processing (such as by byte type data conversion into floating number,
To the multiplication of floating number, the add operation based on offset etc.), such as call SIMD (single-instruction multiple-data in compilation)
It instructs and data processing is carried out to the byte type data of reception.X86 or X64 frameworks are a powerful processors, Intel or
The mainstream processor of AMD all supports the instruction set of SSE (Streaming SIMD Extension, Streaming SIMD Extension), therefore
Data processing of the SSE instruction completions to byte type data may be employed, so as to significantly improve the data of Internet of Things background server
Processing capacity.By taking SSE is instructed as an example, the parallel instruction sequence for the data processing being related to is as shown in the table:
Instruction name | Instruct purposes |
Movd | The data of buffering area are moved in the xmm registers of SSE instructions |
Punpcklbw | Byte is expanded into word |
Punpcklwd | Word is expanded into 32 integers |
Cvtdq2ps | 4 integers are changed into 4 floating numbers |
Mulps | 4 floating numbers are carried out at the same time multiplication |
AddPs | 4 floating numbers are carried out at the same time addition |
movq | Result is stored in result buffer |
In conclusion by adopting the above-described technical solution, the beneficial effects of the invention are as follows:Significantly improve SSH frameworks
The data-handling capacity of Internet of Things background server, had not only used the good network ability of java language, but also make use of server
The processing capacities of CPU in itself.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, with reference to embodiment, the present invention is made into
One step it is described in detail.
First, in the Internet of Things background server using SSH frameworks, (server uses the platform of X86 or X64, can incite somebody to action
Substantial amounts of Internet of Things data (byte type data) deposit data buffer zone) in the built-in dynamic link library realized by C language programming.
The dynamic link library includes 1 Transmit-Receive Unit, n data buffer zone (can be based on currently processed object real-time aligned data to buffer
The number in area), m processing unit (can be adjusted according to processor number) and 1 result buffer (form of queue, to place
Manage result and carry out first in first out), and n > m > 1.During specific implementation, Transmit-Receive Unit, processing unit are realized in a manner of thread, are
Convenient for subsequent descriptions, Transmit-Receive Unit, processing unit are stated with receiving or sending thread, processing thread, original state is set to
It is idle.
In the present invention, after network data interface (being realized based on java programmings) receives byte type data, do not handle directly,
But the receiving or sending thread of dynamic link library is issued by JNI interfaces, receiving or sending thread in state is idle data buffer zone with
Machine selects one as current accessed data buffer zone, by currently received byte type data deposit current accessed data buffer zone
It is at this time saturation by the status indication of current accessed data buffer zone until being saturated, while to the processing thread of idle state
Send data processing notification message.When receiving or sending thread accesses a data buffer zone, need to be locked, i.e., except current operation
Outside thread, access of other threads to current accessed data buffer zone is limited.
The processing thread of data processing notification message is received, then the data buffer zone of saturation state is locked first, forbidden
Then other thread accesses are called parallel instruction processing, for the server of X86 or x64 platforms, can be referred to using MMX or SSE
Order carries out data processing parallel, after the completion of data processing, handling result is stored in result buffer, and is buffered to current data
Area unlocks, while marks its state as the free time.
Finally, reading order of the data of result buffer based on first in first out is read by receiving or sending thread and passes through JNI
Interface is stored in the physical quantity database of Internet of Things background server.
When the SSE of X86 or 64 bit platforms (Internet of Things background server) instructions is used to carry out data processing, due to java
Cross-platform characteristic, SSH high concurrents framework can be that windows platforms can also be linux platforms, and that C language is write is dynamic
State chained library is dll forms under windows, is so forms under linux, can be by JNI interfaces for java layers of calling.
SSE instructions can be that independent assembling file can also be embedded among C language.The form of instruction can be intel forms
Can be AT&T forms, depending on used system and compiler.
Embodiment
By taking linux platforms as an example, selection is 64 Ubuntu, and version 12.04, developing instrument is eclipse 4.3, C
CompilerTools is gcc (the C compiler under linux), version 4.8.4.Data processing object is with the CAN of car networking
Exemplified by (Controller Area Network) bus, vehicle-mounted OBD terminals (hereinafter referred to as terminal) believe original CAN message
Breath is dealt into Internet of Things background server (hereinafter referred to as server), and message content meets SAE J1939-71 agreements, and following table provides
With two message examples of engine temperature and oil pressure.
In two above message, each message has 8 bytes, and byte 1 and 2 is respectively cooling agent temperature in first message
Degree and fuel temperature, the physical quantity all represented for single byte, byte 3 and 4 are the engine temperature of double-byte representation.Second report
Text is all the oil pressure that single byte represents.The two messages have 8 bytes to need to handle.
MinaServer (common Network Communication Framework in java) in java first passes through NioSocketAcceptor
() method creates the socket at the Internet of Things background server end of a non-obstruction, and bind () method is called to monitor 8011 ends
Mouth receives the socket requests of car-mounted terminal.
After Socket connections foundation, the service logic thread (the existing processing thread of server) of server will receive
Message be saved in obstruction queue BlockingQueue<Byte>In.
Consumer thread INFODeal (the existing processing thread of server) in data processing threads pond in server
Case-based system obstruction queue in message data and pass through JNI interfaces call dynamic link library carry out data parsing.JNI modules
In interactive class DataInteract in 4 nation methods:
(1) data transfer:Array buf is passed into bottom c language (for storing the byte type data for representing physical quantity)
Byte type data transfer is given the dynamic link library realized based on C language programming by processing;
(2) initialize:Number (size) is connected according to terminal and returns to buffer length, the length of buffer is adjustable;
(3) result is read:From result buffer reading process result;
(4) exit:The buffer space of bottom distribution is discharged during server closing.
In the present embodiment, 8 data buffer zones of dynamic link lab setting, 6 processing threads, each data buffer zone has 8
A cell queue, each cell queue correspond to corresponding byte in CAN message, and the length of each unit queue adaptive can should determine that,
The length of cell queue is arranged to 4096 in the present embodiment, i.e. data in each unit queue are made of 4096 bytes, often
Terminal corresponding to a byte is recorded in preset group.By JNI interfaces, to server by network data interface be transmitted through come
Byte type data, be sequentially stored by the receiving thread of dynamic link library in each corresponding cell queue, work as unit number
During according to reaching 4096, receiving thread marks the state of the data buffer zone as saturation, while notifies that state is idle after locking
Thread is handled to handle.By taking the byte for handling engine oil temperature as an example, by the corresponding cell queue of the 3rd and 4 bytes of message 1
(being referred to as canEE_3 and canEE_4 queues) is read in the XMM register of SSE instructions successively, and coefficient and offset are also read into
In XMM register, then start SSE instructions and carry out data processing.In data handling procedure, by the 4 of canEE_3 and canEF_4
A byte is read in respectively in register cell xmm0 and xmm1, is then spliced into 4 shorts with punpcklbw instructions
Number, then 4 integers are broadened into punpcklwd instructions, floating number is then converted by cvtdq2ps instructions, it is most laggard
Row floating number multiplication and add operation.
After the data processing for completing 4 byte type data, it is recycled and is read in into xmm0 and xmm1, until unit team
Byte type data processing in row finishes.
For single byte physical quantity, xmm1 can be emptied, then mode of operation is consistent with aforesaid way;For integer
Physical quantity after must high 16 data be spliced in the manner described above, reuses punpcklwd instructions and low 16 splicings, remaining
Operation is just as.
Handling result after data processing is stored in result buffer, and java functions is waited to take, i.e., will by JNI interfaces
Handling result is read in physical database.
Performance compares:
Server is configured to:CPU (4 cores of AMD Athon), memory (8G), operating system (Ubuntu 12.04), application
Software (tomcat7.0).When carrying out data processing, 15-20 CAN message is chosen altogether, and average each terminal is per second to send 132
The data of a byte, when simulating multiple terminal tests, performance comparison is as follows:
As can be known from the above table, after number of terminals is more than 2000,20% is maintained using the utilization rate of the CPU of pure java modes
Left and right, but packet loss is serious at this time, many data are not come and processing.If therefore being handled only with java modes, work as terminal
Number more than 2000 when, system begins to packet drop occur, but the utilization rate of CPU only rests on 20% or so, this be because
It to be done for the floating point arithmetic in java by java virtual machines, to call a large amount of other resources, cause the long period of CPU
Wait, and the function of multinuclear is not brought into play.And in the scheme invented, java is only responsible for network and receives, data processing
It is instructed and completed by C+SSE, while be provided with multiple data buffer zones, and be assigned with multiprocessing thread to perform to byte type data
Data processing, can fully use CPU function, even if terminal number reaches 10000, system can still be handled.Meanwhile
The solution of the present invention also has stronger extended capability, if processing does not come terminal too much, can increase the check figure or a of CPU
Number increases data buffer zone to complete, and multiple processing threads can be distributed according to check figure and carry out data processings, make full use of system
Performance.
The above description is merely a specific embodiment, any feature disclosed in this specification, except non-specifically
Narration, can be replaced by other alternative features that are equivalent or have similar purpose;Disclosed all features or all sides
Method or in the process the step of, in addition to mutually exclusive feature and/or step, can be combined in any way.
Claims (2)
- A kind of 1. method of the data processing of high concurrent Platform Server, which is characterized in that comprise the following steps:When the Internet of Things background server of SSH frameworks receives Internet of Things by the network data interface that Java language programming is realized After the data that collection terminal is sent, the byte type data for representing physical quantity are sent to Internet of Things background server by JNI interfaces Built-in dynamic link library;The dynamic link library is based on C language programming and realizes, including 1 Transmit-Receive Unit, n data buffer zone, m processing unit With 1 result buffer, and n > m > 1;Wherein, the original state of processing unit and data buffer zone is the free time;Transmit-Receive Unit selects what state received for idle data buffer zone deposit by JNI interface byte type data Byte type data, while the data buffer zone of current accessed is locked, after data are filled up, Transmit-Receive Unit delays current accessed Area is rushed labeled as saturation, and data processing notice is sent for idle processing unit to state;The processing unit for receiving data processing notice reads data from state for the data buffer zone of saturation, and by current accessed Data buffer zone locks and by oneself state labeled as busy;Processing unit calls parallel instruction to carry out data processing and will locate Manage result deposit result buffer;After data processing, oneself labeled as the free time and is incited somebody to action in the data buffer zone of current accessed Body status indication is the free time;Transmit-Receive Unit reads the handling result of result buffer, and passes through JNI interfaces and handling result is sent to Internet of Things backstage The physical quantity database of server.
- 2. the method as described in claim 1, which is characterized in that the processing unit of dynamic link library is by calling the list in collecting Instruction multiple performs data processing according to SIMD instruction.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610225491.9A CN105912306B (en) | 2016-04-12 | 2016-04-12 | A kind of method of the data processing of high concurrent Platform Server |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610225491.9A CN105912306B (en) | 2016-04-12 | 2016-04-12 | A kind of method of the data processing of high concurrent Platform Server |
Publications (2)
Publication Number | Publication Date |
---|---|
CN105912306A CN105912306A (en) | 2016-08-31 |
CN105912306B true CN105912306B (en) | 2018-05-18 |
Family
ID=56746042
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610225491.9A Expired - Fee Related CN105912306B (en) | 2016-04-12 | 2016-04-12 | A kind of method of the data processing of high concurrent Platform Server |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN105912306B (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109714337B (en) * | 2018-12-26 | 2021-08-10 | 网宿科技股份有限公司 | Data encryption transmission method and equipment |
CN112394675B (en) * | 2019-08-12 | 2023-02-10 | 比亚迪股份有限公司 | Monitoring system, method and server for rail transit |
CN111212085B (en) * | 2020-01-16 | 2022-11-22 | 厦门网宿有限公司 | Method for synchronously calling Internet of things platform, internet of things system and network equipment |
CN111405015B (en) * | 2020-03-09 | 2022-09-30 | 中国建设银行股份有限公司 | Data processing method, device, equipment and storage medium |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102073547A (en) * | 2010-12-17 | 2011-05-25 | 国家计算机网络与信息安全管理中心 | Performance optimizing method for multipath server multi-buffer-zone parallel packet receiving |
CN103036959A (en) * | 2012-12-07 | 2013-04-10 | 武汉邮电科学研究院 | Realization method and realization system of distributed deployment application program based on input/output (IO) decoupling |
CN103279360A (en) * | 2013-06-09 | 2013-09-04 | 浪潮电子信息产业股份有限公司 | Method for obtaining memory information through Linux command based program |
CN105278922A (en) * | 2015-09-29 | 2016-01-27 | 新浪网技术(中国)有限公司 | Data distribution method, system and apparatus |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8863159B2 (en) * | 2006-07-11 | 2014-10-14 | Mcafee, Inc. | System, method and computer program product for inserting an emulation layer in association with a COM server DLL |
US20080244080A1 (en) * | 2007-03-29 | 2008-10-02 | James Thomas H | Prefetching Based on Streaming Hints |
-
2016
- 2016-04-12 CN CN201610225491.9A patent/CN105912306B/en not_active Expired - Fee Related
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102073547A (en) * | 2010-12-17 | 2011-05-25 | 国家计算机网络与信息安全管理中心 | Performance optimizing method for multipath server multi-buffer-zone parallel packet receiving |
CN103036959A (en) * | 2012-12-07 | 2013-04-10 | 武汉邮电科学研究院 | Realization method and realization system of distributed deployment application program based on input/output (IO) decoupling |
CN103279360A (en) * | 2013-06-09 | 2013-09-04 | 浪潮电子信息产业股份有限公司 | Method for obtaining memory information through Linux command based program |
CN105278922A (en) * | 2015-09-29 | 2016-01-27 | 新浪网技术(中国)有限公司 | Data distribution method, system and apparatus |
Non-Patent Citations (1)
Title |
---|
Android 实时流媒体监控的关键技术研究;周晶晶 等;《微型机与应用》;20131010;第32卷(第19期);第4-7页 * |
Also Published As
Publication number | Publication date |
---|---|
CN105912306A (en) | 2016-08-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN105912306B (en) | A kind of method of the data processing of high concurrent Platform Server | |
Sanvito et al. | Can the network be the AI accelerator? | |
JP5859017B2 (en) | Control node for processing cluster | |
US5752078A (en) | System for minimizing latency data reception and handling data packet error if detected while transferring data packet from adapter memory to host memory | |
US20030188054A1 (en) | Data transfer apparatus and method | |
US20160140006A1 (en) | Testbench builder, system, device and method having agent loopback functionality | |
US20050038918A1 (en) | Method and apparatus for implementing work request lists | |
CN101894092B (en) | Multi-core CPU and inter-core communication method thereof | |
CN113835902B (en) | Data processing method, device, computer equipment and storage medium | |
US20040047361A1 (en) | Method and system for TCP/IP using generic buffers for non-posting TCP applications | |
WO2010084091A1 (en) | System and method of invoking multiple remote operations | |
CN114138707B (en) | Data transmission system based on FPGA | |
US8135851B2 (en) | Object request broker for accelerating object-oriented communications and method | |
CN109358820B (en) | Data access method and device, electronic equipment and computer readable storage medium | |
US9703739B2 (en) | Return available PPI credits command | |
Hsieh et al. | Design and implementation of a generic 5G user plane function development framework | |
CN109861967A (en) | Remote direct memory based on Spark Shuffle accesses system | |
CN113507530B (en) | Data forwarding method, related system and device and storage medium | |
US8275921B2 (en) | Accessing data | |
Chanson et al. | Design and implementation of a Ferry Clip test system | |
US20050138130A1 (en) | Accelerator for object-oriented communications and method | |
JPH09224066A (en) | Communication protocol parallel processor | |
CN113138711B (en) | Storage management device and chip | |
JPS59176952A (en) | Communication control system | |
CN110247740B (en) | Data transmission method, device, equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180518 Termination date: 20210412 |
|
CF01 | Termination of patent right due to non-payment of annual fee |