CN103023809A - Information system synchronous data processing method utilizing secondary buffer technology - Google Patents
Information system synchronous data processing method utilizing secondary buffer technology Download PDFInfo
- Publication number
- CN103023809A CN103023809A CN2012105822639A CN201210582263A CN103023809A CN 103023809 A CN103023809 A CN 103023809A CN 2012105822639 A CN2012105822639 A CN 2012105822639A CN 201210582263 A CN201210582263 A CN 201210582263A CN 103023809 A CN103023809 A CN 103023809A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- channel
- encapsulation
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 42
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000000034 method Methods 0.000 claims abstract description 52
- 238000012545 processing Methods 0.000 claims abstract description 37
- 238000013507 mapping Methods 0.000 claims abstract description 8
- 230000007246 mechanism Effects 0.000 claims abstract description 7
- 238000004806 packaging method and process Methods 0.000 claims abstract description 4
- 238000005538 encapsulation Methods 0.000 claims description 43
- 230000003139 buffering effect Effects 0.000 claims description 40
- 230000015572 biosynthetic process Effects 0.000 claims description 30
- 238000005755 formation reaction Methods 0.000 claims description 30
- 230000008569 process Effects 0.000 claims description 21
- 239000003550 marker Substances 0.000 claims description 9
- 230000011218 segmentation Effects 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 6
- 230000010365 information processing Effects 0.000 claims description 5
- 238000013461 design Methods 0.000 claims description 4
- 230000004069 differentiation Effects 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 3
- 230000001186 cumulative effect Effects 0.000 claims description 2
- 238000002224 dissection Methods 0.000 claims description 2
- 238000012856 packing Methods 0.000 claims description 2
- 238000005192 partition Methods 0.000 claims description 2
- 230000009897 systematic effect Effects 0.000 claims description 2
- 230000008901 benefit Effects 0.000 abstract description 8
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000013518 transcription Methods 0.000 description 1
- 230000035897 transcription Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
- Synchronisation In Digital Transmission Systems (AREA)
Abstract
An information system synchronous data processing method utilizing secondary buffer technology. The method includes the steps of firstly, using uniform data format to package data in information channels, and processing frame header packaging information to realize mapping of synchronous data with corresponding information channels of the synchronous data; secondly, using multithread data synchronization method, which realizes data synchronization of high-speed multi-channel information, of a secondary buffer processing mechanism; and using and processing data after information data synchronization. The method has the advantages that the method is fine in data synchronization adaptability, better in synchronization performance, more reliable and stable, high in data synchronization precision, fine in data synchronization independence, high in reducibility, and transparent to users.
Description
Technical field
The present invention relates to technical field of information processing, specifically a kind of data of information system synchronization processing method that adopts the secondary buffer technology.
Background technology
In having the integrated information system of a plurality of subsystems, information type and the path of each subsystem are more, and each system and transmission means, processing method and the processing speed of channel information in each automatic network are different.When each channel information when processing and represent, if the relevant relation of channel information, or in information processing strict sequential requirement there is opportunity, then integrated information system must be carried out accurate information synchronization to these channel informations, thereby avoid when the unified use of information terminal, drift appears in the data message of different passages.Especially in the integrated information system, the subsystem channel data that has need to carry out complicated system conversion and when obtaining, the synchronism of multi-channel data in time, the fix information sequential relationship particular importance that seems.
Have the problem of information precision deficiency to a certain extent through the synchronous information system of data, make mistakes thereby produce easily thus data, services, accuracy, consistency, reliability aspect such as can't ensure at the problem.Such as: large-scale security protection video surveillance network information system, there are a plurality of monitor video passages, information channel is many, and data volume is large; And may to have the part camera be old model, install video frequency collection card (transition card) additional behind original use simulation monitoring network, replenished again afterwards new digital camera or other digital sensor systems of a plurality of nodes, it is inconsistent that these data are sent out the data rate that the survivor sends.When these data are stored in database or are write as log file, because data volume is large, each channel data rate is different, and the data of certain information channel might be lost because of seizing less than system call, and may have the transmission information drift because thread scheduling is hesitated.We just find to recur the phenomenon that packet loss can appear in data when playback of data, and information (situation, the story of a play or opera) and original some deviation of actual conditions.Problem may go out when each road information data is received by different network interface cards, because the reason of system call and treatment effeciency has just been lost here at network interface card; During perhaps because of storage data (or write record file), the I/O operating characteristics is not enough, causes that the front end worker thread blocks, data from overflow, and information system that Here it is does not adopt high-performance data synchronously and the problem of generation.
In data of information system simultaneous techniques field; current popular processing method is main or buffered carries out round data are carried out; after that is to say that data receive through network; at once stamped markers; synchrodata adopted wheel is sought, preempt-mode is pressed into buffer queue and protects, carry out again the unified of synchrodata in the exit and use.The advantage of this method is that Design and implementation is simply clear and definite, shortcoming is that performance still is short of to some extent, when only adopting a buffered data simultaneous techniques, if the synchrodata of rear end also has other tasks when using hard disk I/O to process, its performance has than macrorelief, only adopts a data buffering to process, and buffering can be overflowed very soon, cause and block or loss of data that information flow " stifled point " will appear at the exit of data synchronization processing.
Summary of the invention
The objective of the invention is to study a kind of parallel information synchronization processing method that adopts twice buffered mechanism and use information format encapsulation intermediate data, the method can with need in the Complex Information System synchronously, record, the different rates of recurring and the data in source carry out synchronously, solving information system intersection data can not high efficiency, the problem of high-performance, reliable synchronization.
A kind of data of information system synchronization processing method that adopts the secondary buffer technology of the present invention comprises following mode:
(1) adopts unified data format that the data in the information channel are encapsulated, realize the mapping of synchrodata and its corresponding informance passage by the information processing method of frame head encapsulation;
(2) multithreaded data synchronization method of use secondary buffer treatment mechanism, the method realizes the data of two-forty, multi-channel information are carried out synchronously;
(3) adopt the inventive method to carry out information data synchronous afterwards use, the processing method of data;
The unified encapsulation of data form of described synchronizing information and data of information system passage mapping method comprise the steps:
The purpose of the data encapsulation form that the present invention adopts mainly is for the information after synchronous through data and information source (passage) being carried out the channel information mapping with relevant, so that the data synchronously in use, by data solution form, the data of different passages are turned back to its information source passage before carrying out synchronously, and reduce original synchrodata sequential relationship, use so that carry out subsequent treatment and the storage of data.
Step 1: receive the information source datagram from network, what obtain is head pointer and the message length data of primary reception data.
Step 2: after obtaining primary data information (pdi), before these application heap data, at first define and add self-defining information unit head.
Step 3: during the frame head of process information unit, information channel attribute when recording information receive in the texture field of frame head definition, the particular content of attribute comprises: data locks in time, channel link type, data message source sign, gap marker, raw information length, reserved field (data sectional sign).
Step 4: before primary data information (pdi), add the information unit frame head of having handled well, form complete Information encapsulation frame, the information of each field that comprises in this packaging information frame, but respectively mapping (enum) data come source channels and the length of initial data and content.
Step 5: because encapsulated frame has not only recorded the channel information of synchrodata, writing time when also comprising initial data and processing synchronously, therefore also be easy to reduce by the time interval controls that sends data and receive the original sequential relationship of each channel data.
Step 6: after finishing Information encapsulation, the data receiver processing threads is prepared the encapsulation of data frame is pressed into the formation of first order buffered again, waits for subsequent treatment.
The field of described data encapsulation form comprises: data locks in time, channel link type, data message source sign, gap marker, raw information length, reserved field (data sectional sign), original synchrodata.
Described data lock in time: carry out precise time when synchronous with double categorical data record and identification data, represented that decimal point is accurate to later on the 3rd (millisecond) with the UTC time (number of seconds since 0 o'clock sharp of morning on January 1st, 1970).
Described channel link type: with the employed network ip address of int categorical data identification information data channel, the value of each byte of four bytes of this field, each represents four address values in the IPV4 address, and which this field area divided data information source is.
Described data message source sign: with the multicast address at int categorical data identification information data channel place, method for expressing is identical with " channel link type ".
Described gap marker: with the port numbers of int categorical data identification information data channel use, so that differentiation is from the different port data of same multicast.
Described message length: this field represents the length of the raw information received, and unit is byte.
Described reserved field: it is reserved field that this int type field is set, and offers Information System Design person and carries out further form expansion.
Described initial data: this field contents is the original data content of synchronizing information, namely---received and distinguished the initial data of data channel from network.Data are random length, and its length represents in " message length " field.
The synchronous method of data of described use secondary buffer treatment mechanism comprises: following steps:
Step 1: the information such as the time when the data frame head of Information encapsulation form is recorded the arrival of initial data message, network, multicast, port, sign, the information after the encapsulation is carried out for the first time data buffering: will receiving also, packaged data message is pressed into first larger heap target buffer formation.
Step 2: processor arranges a plurality of first-level buffer formations and processing threads when a plurality of information channel data are processed, and the processing of multi-channel data adopts parallel mode to carry out.For the first time the purpose of buffering is to carry out synchronization properties record and buffer memory for the initial data that network receives, and avoids under the too high condition of data transfer rate, and data just go out the situation of active in this stage of network processes.
Step 3: carry out for the first time end stage of buffered in data, carry out the cumulative data segment processing according to the data transfer rate of this road information channel.When carrying out data sectional, the size of data sectional is set, and packing data is carried out accumulated counts.
Step 4: the user can judge the data partition strategy according to the systematic function requirement, can only judge the segment data size, also can judge the accumulated counts value of encapsulated frame, perhaps both judges.When the segmentation condition satisfies, then prepare to carry out the data buffering second time.The main purpose of data sectional is to consider suitably to reduce the thread scheduling frequency under the condition of higher data rate, improves data synchronization efficiency.
Step 5: finish the data segment after cushioning for the first time, in the process that is pressed into second level buffer queue, operating system receives sub-thread to data and carries out polling dispatching, and parallel multichannel data is pressed into the level 2 buffering formation, realizes the synchronizing sequence of parallel data.Data are carried out the buffering second time; when exporting after synchronously for guarantee information; can performance be not enough at short notice because of the use side of synchrodata; overflow and cause data segment to pile up; therefore, the capacity that the level 2 buffering formation arranges is larger, effectively protected data; promote the synchronous performance of data, reach the purpose that data are steadily exported.
Step 6: at place, the data buffering processing end second time, segment data is ejected, carry out the segment data resolution process.Through resolution process, it can be reduced into a plurality of self-defined encapsulation of data frames.The initial data of encapsulation before this Frame is.
Described employing the inventive method is carried out information data synchronous afterwards use, the processing method of data, comprises the steps:
Step 1: the data in level 2 buffering formation exit, through the Frame that obtains after decomposing of data sectional, because its packed user-defined format, also do not obtain its corresponding primary data information (pdi), source channels attribute information and lock in time information, can't directly be used, therefore, need to carry out dissection process to Frame.
Step 2: because the given data encapsulation format, by this form Frame is resolved, obtain lock in time, channel link type, data message source sign, gap marker, raw information length, the reserved field of initial data, and primary data information (pdi) itself.
Step 3: by parsing, the calculating to encapsulation of data, obtain the information such as target data channel attributes, time, reserved field, initial data, the blanking time of data before and after calculating, reduce its sequential relationship, these data are carried out serializing output, and carry out subsequent treatment and the use of synchrodata information.
Step 4: the output information of synchrodata, can be stored into database, write as log file, turn other forms that is stored as, recur on suitable opportunity, also can directly carry out data demonstration and use.This data synchronization processing process does not affect the subsequent applications of data.
A kind of advantage of the data of information system synchronization processing method of secondary buffer technology that adopts of the present invention is:
Advantage 1: the synchronous adaptability of data is good.The first order and the second level buffer capacity of aftertreatment that data synchronizing processing method of the present invention adopts can be set and adjust, be used for adapting to the synchronous requirement of data of information system of different performance, make the outlet design of synchrodata unrestricted, otherwise, the information channel data of different rates, in the information synchronization exit, the easy short time because of a certain circuit-switched data subsequent treatment blocks, and affects the reception of front end synchrodata.Data synchronizing processing method of the present invention adapts to the different integrated information system of information processing capability (performance).
Advantage 2: use the synchronous handling property of the method better.Because the inventive method is used the secondary data buffering, and buffer queue is larger, has good data buffering ability.Information data way that can be synchronous is many, data-handling capacity is high, and along with increase (raising) of information system synchrodata quantity (speed), the hydraulic performance decline of data synchronization processing is not obvious.
After tested, take 350 times/second, single frames length 1K byte as a circuit-switched data, system is under 2.4G single core processor, 2G internal memory condition, and method of data synchronization is processed 10 paths data same (or recurrence) during the step (back-end processing that does not contain synchrodata) simultaneously, and CPU usage is lower than 25%.In the data that comprise the I/O disk operating synchronously and when recording 5 paths data, CPU usage is lower than 30%, performance is splendid.
Advantage 3: this method of data synchronization is more reliable, more stable.With regard to data are synchronous, because it provides the first-level buffer of subchannel thread, can guarantee that channel data do not lose, and the level 2 buffering of larger self-defined length, can guarantee in the data exit, can not operate because of slower warehousing operation, magnetic disc i/o, and cause the synchronous processing threads obstruction of core and data from overflow to lose.And, in the level 2 buffering porch, be pressed into algorithm by the counting segmentation of multichannel being processed minute thread, further reduce data and process the frequency, improve performance.Thereby information packet loss (frame) probability is low, and stability is stronger.
Advantage 4: the precision height that data are synchronous.Under the High Data Rate condition, digital information method for synchronous provided by the invention relies on powerful synchronous handling property, synchronization accuracy under multichannel, the two-forty condition is higher, the information synchronization precision of centisecond level is provided, and less demanding to system hardware configuration, can adapt to most demands of using.
Advantage 5: the synchronous independence of data is good, and reducibility is strong, to user transparent.Because synchrodata encapsulates according to the designed information format of synchronization processing method, be easy to be reduced use through the information of processing synchronously, this form is simple and practical, can directly use through the data of synchronously processing, also reduction.
Description of drawings
Fig. 1 is that synchrodata is unified the encapsulation format schematic diagram
Fig. 2 is the synchronous handling process schematic diagram of data of information system
Fig. 3 is synchrodata secondary buffer work for the treatment of schematic flow sheet
Fig. 4 is that synchrodata is recurred the processing procedure schematic diagram
Embodiment
Root a tree name Fig. 1 one a kind of data of information system synchronization processing method that adopts the secondary buffer technology shown in Figure 4, embodiment comprises the steps:
1, the environmental preparation work of information synchronization processor comprises following 2 points:
(1) node at definite information data synchronous processor place in the information system core network, processor can be disposed many network interface cards, covers all subsystem place network segments, and receiving from network interface card needs synchronous data message.
(2) processor operation system time sheet is set to 1000, makes the minimum timeslice of data synchronization processing machine be set to 1 millisecond, considers the error that the data subpackage is processed, and its worst error is no more than at 10 milliseconds.
2, the synchronous data buffering processing procedure first time of data comprises following 4 steps:
(1) in information synchronization processor task, need synchronous information to divide according to different processing threads, during actual treatment, receive the data message of different passages from the socket socket of information reception network interface card definition, this receiving thread will be judged network interface card IP address, multicast address, port, the information such as datagram identification, separating needs synchronous single channel data, this information is carried out the Information encapsulation of initial data.
(2) specific practice of initial data encapsulation is: the data synchronization processing machine before being pressed into first order buffer queue, carries out the encapsulation of self-defining data form after receiving and needing synchronous network message to receive data.Worker thread adds the data frame head in the initial data front of receiving, the encapsulated frame form has the customizing messages form, frame head is recorded the information such as the data encapsulation moment, data link sign, gap marker, message identification, field length, reserved field, information package is encapsulated as the synchrodata frame, carries out again the data buffering first time after the encapsulation.
(3) synchronous processor is assigned first order receiving cache queue for receiving worker thread, this formation is the formation of a heap target buffer, the default size of buffer queue is 16M, the size of first order buffer queue, data transfer rate according to each data channel of information system arranges and changes, and the strategy of first in first out is adopted in this formation.This first-level buffer formation is used for the Frame that buffering has been packaged into user-defined format, and in a single day formation is in non-NULL (data are arranged), just with its ejection.So, larger first-level buffer formation will guarantee that network data can be not too fast because of the data that receive, and cause network interface card socket loss of data, improve performance.Have what information channel data to need synchronously what first-level buffer formations are just arranged, just exist what data to receive worker thread simultaneously, the work of these threads walks abreast, and can not block and affect each other.
(4) after buffered data is pressed into for the first time, exit in this first-level buffer, data are ejected, these ejection data are about to prepare to carry out the data buffering second time, and the data rate according to receiving initial data will eject Frame and add up, operated by rotary motion segmentation memory size is that 16K(can adjust according to the speed of Data Source), when adding up the segment data size greater than 16K, automatically perform the action that segment data is pressed into the level 2 buffering formation, the zero clearing of fragmentation count device.The purpose of doing like this is, avoids when carrying out that the second time, data buffering was processed, and the data synchronization processing thread switches the frequency between the sub-thread of data receiver too high, thereby suitably reduce the meaningless performance cost of synchronous processing threads.
3, the synchronous data buffering processing procedure second time of data comprises following 2 steps:
(1) at the data buffering end first time, packed and carried out data sectional processing one-level formation ejection data, be pressed into the level 2 buffering formation with preempt-mode, the level 2 buffering queue size is traditionally arranged to be 128M, it equally also is the strategy that adopts first in first out, it is individual no matter how many first-level buffer formations has, and the level 2 buffering formation only has one, is used for storing the data that eject from the first-level buffer formation of each initial data receiving thread.
(2) synchronous working of data mainly realizes by the data preemptive type being pressed into the level 2 buffering formation.The data transcription of all sub-threads (the first-level buffer formation is pressed into level 2 buffering with data) control all is the operation of block type Infinite Cyclic, operating system is according to the performance of thread busy degree, execution action, automatically determine data are received the poll of sub-thread, thereby the data of finishing each receiving thread are pressed into the level 2 buffering formation, and the data synchronous working just can be finished substantially.
4, the unpacking of synchrodata, decapsulation, data reduction and use before preprocessing process, comprise following 4 steps:
(1) being pressed into the data of level 2 buffering, is to have finished the data synchronizing sequence, has encapsulated the self-defining data frame head, and has carried out the internal storage data section of accumulative total segmentation, and these data segments carried out serializing according to sequential when being pressed into.Therefore, in the exit of level 2 buffering, first according to the data segment that takes out in turn that is pressed into level 2 buffering, these data segments have been confirmed size when being pressed into, and know the self-defined encapsulation format of data in each segmentation, can untie these data segments, obtain each self-defining data frame.
(2) unpack the Frame that obtains afterwards, only need to read packaging information according to the data encapsulation form, remove again the encapsulation frame head, take out initial data, obtain simultaneously the information such as length, data source IP address, multicast address, port, initial data message sign of these initial data.
(3) initial data that obtains by above two steps reduction, when using, be recorded in the field information in the frame format head during according to data encapsulation, comprise: the information such as the data encapsulation moment, data link sign, gap marker, message identification, field length, reserved field, carry out the sequence output of restoring data, carry out again further other processing.
When (4) data were finished laggard line output synchronously, according to the data of information system user demand, it can be to be stored into database, formation local disk log file or to adopt other forms to store that follow-up data is processed.Recur broadcast from record data when needing later on, also can directly carry out data playback, demonstration, use.
(5) synchrodata is when reality is used, and only needs to calculate two initial data and carries out front and back blanking time when synchronous, selects corresponding initial data passage again, it sent according to the time interval of calculating get final product.
The service efficiency of synchrodata will dynamically determine the accumulation situation of synchrodata section in the level 2 buffering formation.Larger level 2 buffering, well the protection synchrodata is can be because of the service efficiency of synchrodata rear end not high and cause data from overflow, lose.And the major function of the first-level buffer formation of less is to guarantee initial data after network receives, and can not cause thread block because each channel data rate is different, causes the data packet loss.
When the channel data of certain sub-systems in the information system is not digital Ethernet information, such as: analogue video signal, should at first this information be carried out information conversion, screening, process, such as: the operations such as filtering, extraction, collection, A/D conversion, obtain the digital information of this circuit-switched data, under the parallel condition of information, carry out uniform data and process synchronously again.
The secondary buffer mechanism that the inventive method adopts, consider data receiver, the operating position of the synchronous rear and front end of data complexity, well promoted data processing performance, improved the synchronous reliability of data, reduce the packet loss possibility, enlarged the synchronous scope of application of data.
The inventive method is applicable to, and take digital Ethernet as the basis, the complex digital information system of data synchronisation requirement is arranged.Particularly, when this information system has a plurality of subsystems, and these subsystems have a plurality of data channel, data transfer rate is higher, data have intersects and incidence relation, and when having synchronously superior performance to require to data, the present invention will provide a kind of high-performance, reliably, the data synchronization implementing method easily realizing, be convenient to dispose.
Claims (5)
1. data of information system synchronization processing method that adopts the secondary buffer technology is characterized in that: comprise following mode:
(1) adopts unified data format that the data in the information channel are encapsulated, realize the mapping of synchrodata and its corresponding informance passage by the information processing method of frame head encapsulation;
(2) multithreaded data synchronization method of use secondary buffer treatment mechanism, the method realizes synchronous to the data of two-forty, multi-channel information;
(3) carry out information data synchronous afterwards use, the processing method of data.
2. a kind of data of information system synchronization processing method that adopts the secondary buffer technology as claimed in claim 1, it is characterized in that: described self-defined synchronizing information is unified encapsulation of data form and data of information system passage mapping method, comprises the steps:
Step 1: receive the information source datagram from network, what obtain is head pointer and the message length data of primary reception data;
Step 2: after obtaining primary data information (pdi), before these application heap data, at first define and add self-defining information unit head;
Step 3: during the frame head of process information unit, information channel attribute when in the texture field of frame head definition, recording information and receive, the particular content of attribute comprises: data locks in time, channel link type, data message source sign, gap marker, raw information length, reserved field;
Step 4: before primary data information (pdi), add the information unit frame head of having handled well, form complete Information encapsulation frame, the information of each field that comprises in this packaging information frame, but respectively mapping (enum) data come source channels and the length of initial data and content;
Step 5: because encapsulated frame has not only recorded the channel information of synchrodata, writing time when also comprising initial data and processing synchronously, therefore also be easy to reduce by the time interval controls that sends data and receive the original sequential relationship of each channel data;
Step 6: after finishing Information encapsulation, the data receiver processing threads is prepared the encapsulation of data frame is pressed into the formation of first order buffered again, waits for subsequent treatment.
3. a kind of data of information system synchronization processing method that adopts the secondary buffer technology as claimed in claim 1 is characterized in that: the synchronous method of data of described use secondary buffer treatment mechanism,, comprising: following steps:
Step 1: the information such as the time when the data frame head of Information encapsulation form is recorded the arrival of initial data message, network, multicast, port, sign, the information after the encapsulation is carried out for the first time data buffering: will receiving also, packaged data message is pressed into first larger heap target buffer formation;
Step 2: processor is when processing a plurality of information channel data, a plurality of first-level buffer formations and processing threads are set, the processing of multi-channel data adopts parallel mode to carry out, the purpose of buffering is to carry out synchronization properties record and buffer memory for the initial data that network receives for the first time, avoid under the too high condition of data transfer rate, data just go out the situation of active in this stage of network processes;
Step 3: carry out for the first time end stage of buffered in data, carry out the cumulative data segment processing according to the data transfer rate of this road information channel, during data sectional, the size of data sectional is set, and packing data is carried out accumulated counts;
Step 4: the user can judge the data partition strategy according to the systematic function requirement, can only judge the segment data size, also can judge the accumulated counts value of encapsulated frame, perhaps both judges; When the segmentation condition satisfies, then prepare to carry out the data buffering second time, the main purpose of data sectional is to consider suitably to reduce the thread scheduling frequency under the condition of higher data rate, improves data synchronization efficiency;
Step 5: finish the data segment after cushioning for the first time, in the process that is pressed into second level buffer queue, operating system receives sub-thread to data and carries out polling dispatching, parallel multichannel data is pressed into the level 2 buffering formation, realize the synchronizing sequence of parallel data, data are carried out the buffering second time, when exporting after synchronously for guarantee information, can performance be not enough at short notice because of the use side of synchrodata, overflow and cause data segment to pile up, therefore, the capacity that the level 2 buffering formation arranges is larger, effectively protected data promotes the synchronous performance of data, reaches the purpose that data are steadily exported;
Step 6: at place, the data buffering processing end second time, segment data ejects, and carries out the segment data resolution process, through resolution process, it can be reduced into a plurality of self-defined encapsulation of data frames, the initial data that encapsulates before this Frame is.
4. a kind of data of information system synchronization processing method that adopts the secondary buffer technology as claimed in claim 1 is characterized in that: describedly carry out that the data of information data after are synchronously used, processing method, comprise the steps:
Step 1: the data of carrying out the buffer queue exit of secondary buffer, through the Frame that obtains after decomposing of data sectional, because its packed user-defined format, also do not obtain its corresponding primary data information (pdi), source channels attribute information and lock in time information, can't directly be used, therefore, need to carry out dissection process to Frame;
Step 2: because the given data encapsulation format, by this form Frame is resolved, obtain lock in time, channel link type, data message source sign, gap marker, raw information length, the reserved field of initial data, and primary data information (pdi) itself;
Step 3: by parsing, the calculating to encapsulation of data, obtain the information such as target data channel attributes, time, reserved field, initial data, the blanking time of data before and after calculating, reduce its sequential relationship, these data are carried out serializing output, and carry out subsequent treatment and the use of synchrodata information;
Step 4: the output information of synchrodata, can be stored into database, write as log file, turn other forms that is stored as, recur on suitable opportunity, can directly carry out data demonstration and use yet, this data synchronization processing process does not affect the subsequent applications of data.
5. a kind of data of information system synchronization processing method that adopts the secondary buffer technology as claimed in claim 2, it is characterized in that: described self-defined synchronizing information is unified the encapsulation of data form, comprises following main define field:
Encapsulation format field 1: data locks in time: carry out precise time when synchronous with double categorical data record and identification data, with UTC time (number of seconds since 0 o'clock sharp of morning on January 1st, 1970) expression, decimal point is accurate to later on the 3rd (millisecond);
Encapsulation format field 2: channel link type: with the employed network ip address of int categorical data identification information data channel, the value of each byte of four bytes of this field, each represents four address values in the IPV4 address, and which this field area divided data information source is;
Encapsulation format field 3: data message source sign: with the multicast address at int categorical data identification information data channel place, method for expressing is identical with " channel link type ";
Encapsulation format field 4: gap marker: with the port numbers of int categorical data identification information data channel use, so that differentiation is from the different port data of same multicast;
Encapsulation format field 5: message length: this field represents the length of the raw information received, and unit is byte;
Encapsulation format field 6: reserved field: it is reserved field that this int type field is set, and offers Information System Design person and carries out further form expansion;
Encapsulation format field 7: initial data: this field contents is the original data content of synchronizing information, namely---the differentiation that receives from network the initial data of data channel, data are random length, its length represents in " message length " field.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210582263.9A CN103023809B (en) | 2012-12-28 | 2012-12-28 | Information system synchronous data processing method utilizing secondary buffer technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210582263.9A CN103023809B (en) | 2012-12-28 | 2012-12-28 | Information system synchronous data processing method utilizing secondary buffer technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103023809A true CN103023809A (en) | 2013-04-03 |
| CN103023809B CN103023809B (en) | 2015-07-22 |
Family
ID=47971952
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210582263.9A Expired - Fee Related CN103023809B (en) | 2012-12-28 | 2012-12-28 | Information system synchronous data processing method utilizing secondary buffer technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103023809B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105593832A (en) * | 2013-10-18 | 2016-05-18 | 意法半导体(格勒诺布尔2)公司 | Method and apparatus for supporting the use of interleaved memory regions |
| CN106095721A (en) * | 2016-06-27 | 2016-11-09 | 深圳市金溢科技股份有限公司 | A kind of method for synchronizing time, system and board units |
| CN106254924A (en) * | 2016-10-08 | 2016-12-21 | 广东欧珀移动通信有限公司 | The player method of a kind of multi-medium data, system and relevant device |
| CN106375239A (en) * | 2016-08-25 | 2017-02-01 | 北京智芯微电子科技有限公司 | Network data reception processing method and apparatus |
| CN106850426A (en) * | 2017-01-10 | 2017-06-13 | 北京交通大学 | Based on multi-path data transmission method and device that partial data is overlapped |
| CN106878196A (en) * | 2017-01-16 | 2017-06-20 | 许继集团有限公司 | A kind of data processing method and device of ADC plug-in units |
| CN107172037A (en) * | 2017-05-11 | 2017-09-15 | 华东师范大学 | A kind of real-time subpackage analytic method of multichannel multi-channel high-speed data stream |
| CN108092942A (en) * | 2016-11-21 | 2018-05-29 | 深圳市中兴微电子技术有限公司 | A kind of message processing method and device |
| CN108563557A (en) * | 2018-02-08 | 2018-09-21 | 北京东土科技股份有限公司 | A kind of Channel Synchronous method and device of multichannel computer |
| CN109510857A (en) * | 2018-07-10 | 2019-03-22 | 中国联合网络通信集团有限公司 | Date storage method, device, server and system |
| CN110196684A (en) * | 2018-02-27 | 2019-09-03 | 爱思开海力士有限公司 | Data storage device, its operating method and the storage system with it |
| CN111258742A (en) * | 2020-02-17 | 2020-06-09 | 杭州依图医疗技术有限公司 | Data synchronization method, system, computing device and storage medium |
| CN113395489A (en) * | 2021-06-09 | 2021-09-14 | 哈尔滨低空探索智能科技有限责任公司 | Video switching method and video switching system suitable for network camera |
| CN113704212A (en) * | 2020-05-22 | 2021-11-26 | 深信服科技股份有限公司 | Data synchronization method, device and equipment of server and computer storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1946007A (en) * | 2006-09-06 | 2007-04-11 | 中兴通讯股份有限公司 | Device and its method for mixing and converging data service and SDH service |
| CN101266561A (en) * | 2008-04-29 | 2008-09-17 | 中兴通讯股份有限公司 | Inter-core message communication method for multi-core multithread processor |
| CN101299649A (en) * | 2008-06-19 | 2008-11-05 | 中兴通讯股份有限公司 | Method and device for mixing and concourse of multi-service based on all-purpose framing regulations |
-
2012
- 2012-12-28 CN CN201210582263.9A patent/CN103023809B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1946007A (en) * | 2006-09-06 | 2007-04-11 | 中兴通讯股份有限公司 | Device and its method for mixing and converging data service and SDH service |
| CN101266561A (en) * | 2008-04-29 | 2008-09-17 | 中兴通讯股份有限公司 | Inter-core message communication method for multi-core multithread processor |
| CN101299649A (en) * | 2008-06-19 | 2008-11-05 | 中兴通讯股份有限公司 | Method and device for mixing and concourse of multi-service based on all-purpose framing regulations |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105593832A (en) * | 2013-10-18 | 2016-05-18 | 意法半导体(格勒诺布尔2)公司 | Method and apparatus for supporting the use of interleaved memory regions |
| CN106095721A (en) * | 2016-06-27 | 2016-11-09 | 深圳市金溢科技股份有限公司 | A kind of method for synchronizing time, system and board units |
| CN106375239B (en) * | 2016-08-25 | 2019-02-12 | 北京智芯微电子科技有限公司 | A kind of received processing method and processing device of network data |
| CN106375239A (en) * | 2016-08-25 | 2017-02-01 | 北京智芯微电子科技有限公司 | Network data reception processing method and apparatus |
| CN106254924A (en) * | 2016-10-08 | 2016-12-21 | 广东欧珀移动通信有限公司 | The player method of a kind of multi-medium data, system and relevant device |
| CN108092942A (en) * | 2016-11-21 | 2018-05-29 | 深圳市中兴微电子技术有限公司 | A kind of message processing method and device |
| CN108092942B (en) * | 2016-11-21 | 2020-04-10 | 深圳市中兴微电子技术有限公司 | Message processing method and device |
| CN106850426A (en) * | 2017-01-10 | 2017-06-13 | 北京交通大学 | Based on multi-path data transmission method and device that partial data is overlapped |
| CN106850426B (en) * | 2017-01-10 | 2020-09-29 | 北京交通大学 | Multi-path data transmission method and device based on partial data overlapping |
| CN106878196A (en) * | 2017-01-16 | 2017-06-20 | 许继集团有限公司 | A kind of data processing method and device of ADC plug-in units |
| CN107172037A (en) * | 2017-05-11 | 2017-09-15 | 华东师范大学 | A kind of real-time subpackage analytic method of multichannel multi-channel high-speed data stream |
| CN107172037B (en) * | 2017-05-11 | 2020-01-07 | 华东师范大学 | Real-time subpackage analysis method for multi-path multi-channel high-speed data stream |
| CN108563557A (en) * | 2018-02-08 | 2018-09-21 | 北京东土科技股份有限公司 | A kind of Channel Synchronous method and device of multichannel computer |
| CN108563557B (en) * | 2018-02-08 | 2020-01-21 | 北京东土科技股份有限公司 | Channel synchronization method and device of multi-channel computer |
| CN110196684A (en) * | 2018-02-27 | 2019-09-03 | 爱思开海力士有限公司 | Data storage device, its operating method and the storage system with it |
| CN109510857A (en) * | 2018-07-10 | 2019-03-22 | 中国联合网络通信集团有限公司 | Date storage method, device, server and system |
| CN109510857B (en) * | 2018-07-10 | 2021-06-01 | 中国联合网络通信集团有限公司 | Data storage method, device, server and system |
| CN111258742A (en) * | 2020-02-17 | 2020-06-09 | 杭州依图医疗技术有限公司 | Data synchronization method, system, computing device and storage medium |
| CN111258742B (en) * | 2020-02-17 | 2023-08-04 | 杭州依图医疗技术有限公司 | Data synchronization method, system, computing device and storage medium |
| CN113704212A (en) * | 2020-05-22 | 2021-11-26 | 深信服科技股份有限公司 | Data synchronization method, device and equipment of server and computer storage medium |
| CN113395489A (en) * | 2021-06-09 | 2021-09-14 | 哈尔滨低空探索智能科技有限责任公司 | Video switching method and video switching system suitable for network camera |
| CN113395489B (en) * | 2021-06-09 | 2023-03-24 | 哈尔滨低空探索智能科技有限责任公司 | Video switching method and video switching system suitable for network camera |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103023809B (en) | 2015-07-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103023809B (en) | Information system synchronous data processing method utilizing secondary buffer technology | |
| CN103731631B (en) | The method, apparatus and system of a kind of transmitting video image | |
| CN111163018B (en) | Network equipment and method for reducing transmission delay thereof | |
| US20220343047A1 (en) | Method and system for processing simulation data | |
| CN112650448B (en) | FPGA-based large-data-volume storage file management method | |
| CN111680016A (en) | Distributed server cluster log data processing method, device and system | |
| CN104065588A (en) | Device for scheduling and buffering data packets and method thereof | |
| CN102480335A (en) | Method and system for transmitting business data | |
| CN113542043B (en) | Data sampling method, device, equipment and medium of network equipment | |
| CN111723115A (en) | Data statistics method, device, terminal and storage medium based on multi-level cache | |
| CN107430496A (en) | Frame is abandoned for long-range display | |
| CN108833195B (en) | Process-based network data flow analysis method | |
| CN102629235A (en) | Method for increasing read-write speed of double data rate (DDR) memory | |
| CN101854259B (en) | Method and system for counting data packets | |
| WO2016127953A1 (en) | Debugging method specifically for fpga of high-end fault-tolerant computer based on software-hardware architecture, and device thereof | |
| WO2024109317A1 (en) | Method and device for transmitting video frames and camera parameter information | |
| CN106896956B (en) | The implementation method of multi-point touch under one kind " road " system | |
| CN103442091A (en) | Data transmission method and device | |
| CN112104403A (en) | Message queue-based multithreading remote sensing satellite baseband data processing method and device | |
| CN110297785A (en) | A kind of finance data flow control apparatus and flow control method based on FPGA | |
| CN101674146B (en) | Method and system for aligning all dispatching services in optical transmission network | |
| CN1984031A (en) | Method and device for converting data-packet-mode into element-frame-mode | |
| CN114363304B (en) | RTP video stream storage and playing method and device | |
| CN114281725A (en) | High-speed data processing system and high-speed data processing method | |
| CN103490995A (en) | Message sending method and message sending device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20150722 |