CN105912306B - A kind of method of the data processing of high concurrent Platform Server - Google Patents

A kind of method of the data processing of high concurrent Platform Server Download PDF

Info

Publication number
CN105912306B
CN105912306B CN201610225491.9A CN201610225491A CN105912306B CN 105912306 B CN105912306 B CN 105912306B CN 201610225491 A CN201610225491 A CN 201610225491A CN 105912306 B CN105912306 B CN 105912306B
Authority
CN
China
Prior art keywords
data
processing
server
internet
result
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201610225491.9A
Other languages
Chinese (zh)
Other versions
CN105912306A (en
Inventor
鲁晓军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Electronic Science and Technology of China
Original Assignee
University of Electronic Science and Technology of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Electronic Science and Technology of China filed Critical University of Electronic Science and Technology of China
Priority to CN201610225491.9A priority Critical patent/CN105912306B/en
Publication of CN105912306A publication Critical patent/CN105912306A/en
Application granted granted Critical
Publication of CN105912306B publication Critical patent/CN105912306B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/38Concurrent instruction execution, e.g. pipeline or look ahead
    • G06F9/3836Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multi Processors (AREA)
  • Computer And Data Communications (AREA)

Abstract

The present invention relates to the high concurrent processing of the data to Internet of Things background server.The present invention will receive byte type data by JNI interfaces and be sent to the Transmit-Receive Unit of dynamic link libraries built in server, and current data is stored in idle data buffering area when fill up after by it, is marked as saturation and idle processing unit transmission processing notification;Alignment processing unit reads data simultaneously locks current data buffering area and by oneself state labeled as busy;Then parallel instruction is called to carry out data processing and handling result is stored in result buffer, after being disposed, by current data buffer tag for the free time and by oneself state labeled as the free time;Transmit-Receive Unit reading process is as a result, and pass through the physical quantity database that JNI interfaces send it to server.The method can significantly improve the data-handling capacities of the Internet of Things background server of SSH frameworks, have not only used the good network ability of java language, but also make use of the processing capacities of server CPU in itself.

Description

A kind of method of the data processing of high concurrent Platform Server
Technical field
The present invention relates to internet of things field, and in particular at the high concurrent of the data of Internet of Things background server Reason.
Background technology
At present, for internet of things equipment, people's general concern be terminal acquisition and sending function, it is micro- due to terminal Controller processing capacity is weaker, generally only does a little simple analysis, background server due to there is great amount of terminals to communicate, it It stores after simply handling data, and is shown in a suitable manner to user.
For the framework of Internet of Things background server, generally using SSH (struts2+spring+hibernate) framework, I.e. using the good network performance of java language, realize and carry out high concurrent processing to largely connecting, that is, pass through java language institute After the network data interface write receives the data that Internet of Things collection terminal is sent, by transferring the JVM of java language, (java is virtual Machine) correlation function high concurrent processing is carried out to the byte type data of reception and is stored in the predetermined physical of Internet of Things background server It measures in database, obtains that the handling result that user shows can be used for.And the data of Internet of Things collection terminal it is general the characteristics of be use Single byte, double byte or integer represent physical quantity, when the Internet of Things background server of SSH frameworks is connect by network data interface Receive Internet of Things collection terminal transmission data after, it is necessary to by the byte type data conversion received into floating number, then do floating number Co-efficient multiplication, be finally based on the data processings such as the add operation of offset and can just become real physical quantity and (can be directly used for Shown to user), by taking double byte as an example, when being realized based on java language to receiving data progress data processing, pass through transfer The correlation function of the JVM of java language realizes that conversion, multiplication and offset handle to obtain physical quantity, i.e.,:
For java language, its advantage is cross-platform and network performance, but data processing, especially at high concurrent data Reason is its weakness, and when being related to high concurrent data, when big data quantity arrives, the processing of data is even more the bottleneck of system.
The content of the invention
The goal of the invention of the present invention is:In order to overcome the Internet of Things background server for using SSH frameworks based on java Language carries out data processing, when obtaining corresponding actual physical amount, the insufficient skill of processing capacity of java language to receiving data Art problem provides a kind of method of the data processing of high concurrent Platform Server, can use the good network energy of java language Power, and the processing capacity of Platform Server can be improved.
A kind of method of the data processing of high concurrent Platform Server of the present invention, comprises the following steps:
When the Internet of Things background server of SSH frameworks is received by network data interface (being realized by java Programming with Pascal Language) After the data that Internet of Things collection terminal is sent, the byte type data for representing physical quantity are passed through into JNI interfaces (Java Native Interface, java local interface) it is sent to the dynamic link library built in Internet of Things background server, it is preferred that dynamic link Storehouse is based on C language programming and realizes;Wherein, dynamic link library includes Transmit-Receive Unit, data buffer zone, processing unit and result buffering Area, and the original state of processing unit and data buffer zone is the free time;
Transmit-Receive Unit selects state to be connect for idle data buffer zone deposit by JNI interface byte type data Byte type data of receipts, while the data buffer zone of current accessed is locked, after data are filled up, Transmit-Receive Unit is by current accessed Buffer tag for saturation, and send data processing notice to state for idle processing unit;
The processing unit for receiving data processing notice reads data from state for the data buffer zone of saturation, and will currently visit The data buffer zone asked locks and by oneself state labeled as busy;Processing unit calls parallel instruction to carry out data processing simultaneously Handling result is stored in result buffer;After data processing, by the data buffer zone of current accessed labeled as idle and By oneself state labeled as the free time;Transmit-Receive Unit reads the handling result of result buffer, and passes through JNI interfaces by handling result It is sent to the physical quantity database of Internet of Things background server.
Main idea is that data are become batch processing from single processing, it will be using the Internet of Things of SSH frameworks The data processing of background server by call parallel instruction carry out data processing (such as by byte type data conversion into floating number, To the multiplication of floating number, the add operation based on offset etc.), such as call SIMD (single-instruction multiple-data in compilation) It instructs and data processing is carried out to the byte type data of reception.X86 or X64 frameworks are a powerful processors, Intel or The mainstream processor of AMD all supports the instruction set of SSE (Streaming SIMD Extension, Streaming SIMD Extension), therefore Data processing of the SSE instruction completions to byte type data may be employed, so as to significantly improve the data of Internet of Things background server Processing capacity.By taking SSE is instructed as an example, the parallel instruction sequence for the data processing being related to is as shown in the table:
Instruction name Instruct purposes
Movd The data of buffering area are moved in the xmm registers of SSE instructions
Punpcklbw Byte is expanded into word
Punpcklwd Word is expanded into 32 integers
Cvtdq2ps 4 integers are changed into 4 floating numbers
Mulps 4 floating numbers are carried out at the same time multiplication
AddPs 4 floating numbers are carried out at the same time addition
movq Result is stored in result buffer
In conclusion by adopting the above-described technical solution, the beneficial effects of the invention are as follows:Significantly improve SSH frameworks The data-handling capacity of Internet of Things background server, had not only used the good network ability of java language, but also make use of server The processing capacities of CPU in itself.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, with reference to embodiment, the present invention is made into One step it is described in detail.
First, in the Internet of Things background server using SSH frameworks, (server uses the platform of X86 or X64, can incite somebody to action Substantial amounts of Internet of Things data (byte type data) deposit data buffer zone) in the built-in dynamic link library realized by C language programming. The dynamic link library includes 1 Transmit-Receive Unit, n data buffer zone (can be based on currently processed object real-time aligned data to buffer The number in area), m processing unit (can be adjusted according to processor number) and 1 result buffer (form of queue, to place Manage result and carry out first in first out), and n > m > 1.During specific implementation, Transmit-Receive Unit, processing unit are realized in a manner of thread, are Convenient for subsequent descriptions, Transmit-Receive Unit, processing unit are stated with receiving or sending thread, processing thread, original state is set to It is idle.
In the present invention, after network data interface (being realized based on java programmings) receives byte type data, do not handle directly, But the receiving or sending thread of dynamic link library is issued by JNI interfaces, receiving or sending thread in state is idle data buffer zone with Machine selects one as current accessed data buffer zone, by currently received byte type data deposit current accessed data buffer zone It is at this time saturation by the status indication of current accessed data buffer zone until being saturated, while to the processing thread of idle state Send data processing notification message.When receiving or sending thread accesses a data buffer zone, need to be locked, i.e., except current operation Outside thread, access of other threads to current accessed data buffer zone is limited.
The processing thread of data processing notification message is received, then the data buffer zone of saturation state is locked first, forbidden Then other thread accesses are called parallel instruction processing, for the server of X86 or x64 platforms, can be referred to using MMX or SSE Order carries out data processing parallel, after the completion of data processing, handling result is stored in result buffer, and is buffered to current data Area unlocks, while marks its state as the free time.
Finally, reading order of the data of result buffer based on first in first out is read by receiving or sending thread and passes through JNI Interface is stored in the physical quantity database of Internet of Things background server.
When the SSE of X86 or 64 bit platforms (Internet of Things background server) instructions is used to carry out data processing, due to java Cross-platform characteristic, SSH high concurrents framework can be that windows platforms can also be linux platforms, and that C language is write is dynamic State chained library is dll forms under windows, is so forms under linux, can be by JNI interfaces for java layers of calling. SSE instructions can be that independent assembling file can also be embedded among C language.The form of instruction can be intel forms Can be AT&T forms, depending on used system and compiler.
Embodiment
By taking linux platforms as an example, selection is 64 Ubuntu, and version 12.04, developing instrument is eclipse 4.3, C CompilerTools is gcc (the C compiler under linux), version 4.8.4.Data processing object is with the CAN of car networking Exemplified by (Controller Area Network) bus, vehicle-mounted OBD terminals (hereinafter referred to as terminal) believe original CAN message Breath is dealt into Internet of Things background server (hereinafter referred to as server), and message content meets SAE J1939-71 agreements, and following table provides With two message examples of engine temperature and oil pressure.
In two above message, each message has 8 bytes, and byte 1 and 2 is respectively cooling agent temperature in first message Degree and fuel temperature, the physical quantity all represented for single byte, byte 3 and 4 are the engine temperature of double-byte representation.Second report Text is all the oil pressure that single byte represents.The two messages have 8 bytes to need to handle.
MinaServer (common Network Communication Framework in java) in java first passes through NioSocketAcceptor () method creates the socket at the Internet of Things background server end of a non-obstruction, and bind () method is called to monitor 8011 ends Mouth receives the socket requests of car-mounted terminal.
After Socket connections foundation, the service logic thread (the existing processing thread of server) of server will receive Message be saved in obstruction queue BlockingQueue<Byte>In.
Consumer thread INFODeal (the existing processing thread of server) in data processing threads pond in server Case-based system obstruction queue in message data and pass through JNI interfaces call dynamic link library carry out data parsing.JNI modules In interactive class DataInteract in 4 nation methods:
(1) data transfer:Array buf is passed into bottom c language (for storing the byte type data for representing physical quantity) Byte type data transfer is given the dynamic link library realized based on C language programming by processing;
(2) initialize:Number (size) is connected according to terminal and returns to buffer length, the length of buffer is adjustable;
(3) result is read:From result buffer reading process result;
(4) exit:The buffer space of bottom distribution is discharged during server closing.
In the present embodiment, 8 data buffer zones of dynamic link lab setting, 6 processing threads, each data buffer zone has 8 A cell queue, each cell queue correspond to corresponding byte in CAN message, and the length of each unit queue adaptive can should determine that, The length of cell queue is arranged to 4096 in the present embodiment, i.e. data in each unit queue are made of 4096 bytes, often Terminal corresponding to a byte is recorded in preset group.By JNI interfaces, to server by network data interface be transmitted through come Byte type data, be sequentially stored by the receiving thread of dynamic link library in each corresponding cell queue, work as unit number During according to reaching 4096, receiving thread marks the state of the data buffer zone as saturation, while notifies that state is idle after locking Thread is handled to handle.By taking the byte for handling engine oil temperature as an example, by the corresponding cell queue of the 3rd and 4 bytes of message 1 (being referred to as canEE_3 and canEE_4 queues) is read in the XMM register of SSE instructions successively, and coefficient and offset are also read into In XMM register, then start SSE instructions and carry out data processing.In data handling procedure, by the 4 of canEE_3 and canEF_4 A byte is read in respectively in register cell xmm0 and xmm1, is then spliced into 4 shorts with punpcklbw instructions Number, then 4 integers are broadened into punpcklwd instructions, floating number is then converted by cvtdq2ps instructions, it is most laggard Row floating number multiplication and add operation.
After the data processing for completing 4 byte type data, it is recycled and is read in into xmm0 and xmm1, until unit team Byte type data processing in row finishes.
For single byte physical quantity, xmm1 can be emptied, then mode of operation is consistent with aforesaid way;For integer Physical quantity after must high 16 data be spliced in the manner described above, reuses punpcklwd instructions and low 16 splicings, remaining Operation is just as.
Handling result after data processing is stored in result buffer, and java functions is waited to take, i.e., will by JNI interfaces Handling result is read in physical database.
Performance compares:
Server is configured to:CPU (4 cores of AMD Athon), memory (8G), operating system (Ubuntu 12.04), application Software (tomcat7.0).When carrying out data processing, 15-20 CAN message is chosen altogether, and average each terminal is per second to send 132 The data of a byte, when simulating multiple terminal tests, performance comparison is as follows:
As can be known from the above table, after number of terminals is more than 2000,20% is maintained using the utilization rate of the CPU of pure java modes Left and right, but packet loss is serious at this time, many data are not come and processing.If therefore being handled only with java modes, work as terminal Number more than 2000 when, system begins to packet drop occur, but the utilization rate of CPU only rests on 20% or so, this be because It to be done for the floating point arithmetic in java by java virtual machines, to call a large amount of other resources, cause the long period of CPU Wait, and the function of multinuclear is not brought into play.And in the scheme invented, java is only responsible for network and receives, data processing It is instructed and completed by C+SSE, while be provided with multiple data buffer zones, and be assigned with multiprocessing thread to perform to byte type data Data processing, can fully use CPU function, even if terminal number reaches 10000, system can still be handled.Meanwhile The solution of the present invention also has stronger extended capability, if processing does not come terminal too much, can increase the check figure or a of CPU Number increases data buffer zone to complete, and multiple processing threads can be distributed according to check figure and carry out data processings, make full use of system Performance.
The above description is merely a specific embodiment, any feature disclosed in this specification, except non-specifically Narration, can be replaced by other alternative features that are equivalent or have similar purpose;Disclosed all features or all sides Method or in the process the step of, in addition to mutually exclusive feature and/or step, can be combined in any way.

Claims (2)

  1. A kind of 1. method of the data processing of high concurrent Platform Server, which is characterized in that comprise the following steps:
    When the Internet of Things background server of SSH frameworks receives Internet of Things by the network data interface that Java language programming is realized After the data that collection terminal is sent, the byte type data for representing physical quantity are sent to Internet of Things background server by JNI interfaces Built-in dynamic link library;
    The dynamic link library is based on C language programming and realizes, including 1 Transmit-Receive Unit, n data buffer zone, m processing unit With 1 result buffer, and n > m > 1;Wherein, the original state of processing unit and data buffer zone is the free time;
    Transmit-Receive Unit selects what state received for idle data buffer zone deposit by JNI interface byte type data Byte type data, while the data buffer zone of current accessed is locked, after data are filled up, Transmit-Receive Unit delays current accessed Area is rushed labeled as saturation, and data processing notice is sent for idle processing unit to state;
    The processing unit for receiving data processing notice reads data from state for the data buffer zone of saturation, and by current accessed Data buffer zone locks and by oneself state labeled as busy;Processing unit calls parallel instruction to carry out data processing and will locate Manage result deposit result buffer;After data processing, oneself labeled as the free time and is incited somebody to action in the data buffer zone of current accessed Body status indication is the free time;
    Transmit-Receive Unit reads the handling result of result buffer, and passes through JNI interfaces and handling result is sent to Internet of Things backstage The physical quantity database of server.
  2. 2. the method as described in claim 1, which is characterized in that the processing unit of dynamic link library is by calling the list in collecting Instruction multiple performs data processing according to SIMD instruction.
CN201610225491.9A 2016-04-12 2016-04-12 A kind of method of the data processing of high concurrent Platform Server Expired - Fee Related CN105912306B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610225491.9A CN105912306B (en) 2016-04-12 2016-04-12 A kind of method of the data processing of high concurrent Platform Server

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610225491.9A CN105912306B (en) 2016-04-12 2016-04-12 A kind of method of the data processing of high concurrent Platform Server

Publications (2)

Publication Number Publication Date
CN105912306A CN105912306A (en) 2016-08-31
CN105912306B true CN105912306B (en) 2018-05-18

Family

ID=56746042

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610225491.9A Expired - Fee Related CN105912306B (en) 2016-04-12 2016-04-12 A kind of method of the data processing of high concurrent Platform Server

Country Status (1)

Country Link
CN (1) CN105912306B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109714337B (en) * 2018-12-26 2021-08-10 网宿科技股份有限公司 Data encryption transmission method and equipment
CN112394675B (en) * 2019-08-12 2023-02-10 比亚迪股份有限公司 Monitoring system, method and server for rail transit
CN111212085B (en) * 2020-01-16 2022-11-22 厦门网宿有限公司 Method for synchronously calling Internet of things platform, internet of things system and network equipment
CN111405015B (en) * 2020-03-09 2022-09-30 中国建设银行股份有限公司 Data processing method, device, equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102073547A (en) * 2010-12-17 2011-05-25 国家计算机网络与信息安全管理中心 Performance optimizing method for multipath server multi-buffer-zone parallel packet receiving
CN103036959A (en) * 2012-12-07 2013-04-10 武汉邮电科学研究院 Realization method and realization system of distributed deployment application program based on input/output (IO) decoupling
CN103279360A (en) * 2013-06-09 2013-09-04 浪潮电子信息产业股份有限公司 Method for obtaining memory information through Linux command based program
CN105278922A (en) * 2015-09-29 2016-01-27 新浪网技术(中国)有限公司 Data distribution method, system and apparatus

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8863159B2 (en) * 2006-07-11 2014-10-14 Mcafee, Inc. System, method and computer program product for inserting an emulation layer in association with a COM server DLL
US20080244080A1 (en) * 2007-03-29 2008-10-02 James Thomas H Prefetching Based on Streaming Hints

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102073547A (en) * 2010-12-17 2011-05-25 国家计算机网络与信息安全管理中心 Performance optimizing method for multipath server multi-buffer-zone parallel packet receiving
CN103036959A (en) * 2012-12-07 2013-04-10 武汉邮电科学研究院 Realization method and realization system of distributed deployment application program based on input/output (IO) decoupling
CN103279360A (en) * 2013-06-09 2013-09-04 浪潮电子信息产业股份有限公司 Method for obtaining memory information through Linux command based program
CN105278922A (en) * 2015-09-29 2016-01-27 新浪网技术(中国)有限公司 Data distribution method, system and apparatus

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Android 实时流媒体监控的关键技术研究;周晶晶 等;《微型机与应用》;20131010;第32卷(第19期);第4-7页 *

Also Published As

Publication number Publication date
CN105912306A (en) 2016-08-31

Similar Documents

Publication Publication Date Title
CN105912306B (en) A kind of method of the data processing of high concurrent Platform Server
Sanvito et al. Can the network be the AI accelerator?
JP5859017B2 (en) Control node for processing cluster
US5752078A (en) System for minimizing latency data reception and handling data packet error if detected while transferring data packet from adapter memory to host memory
US20030188054A1 (en) Data transfer apparatus and method
US20160140006A1 (en) Testbench builder, system, device and method having agent loopback functionality
US20050038918A1 (en) Method and apparatus for implementing work request lists
CN101894092B (en) Multi-core CPU and inter-core communication method thereof
CN113835902B (en) Data processing method, device, computer equipment and storage medium
US20040047361A1 (en) Method and system for TCP/IP using generic buffers for non-posting TCP applications
WO2010084091A1 (en) System and method of invoking multiple remote operations
CN114138707B (en) Data transmission system based on FPGA
US8135851B2 (en) Object request broker for accelerating object-oriented communications and method
CN109358820B (en) Data access method and device, electronic equipment and computer readable storage medium
US9703739B2 (en) Return available PPI credits command
Hsieh et al. Design and implementation of a generic 5G user plane function development framework
CN109861967A (en) Remote direct memory based on Spark Shuffle accesses system
CN113507530B (en) Data forwarding method, related system and device and storage medium
US8275921B2 (en) Accessing data
Chanson et al. Design and implementation of a Ferry Clip test system
US20050138130A1 (en) Accelerator for object-oriented communications and method
JPH09224066A (en) Communication protocol parallel processor
CN113138711B (en) Storage management device and chip
JPS59176952A (en) Communication control system
CN110247740B (en) Data transmission method, device, equipment and storage medium

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20180518

Termination date: 20210412

CF01 Termination of patent right due to non-payment of annual fee