CN105791399A - Multi-relay internet big data pushing method and system - Google Patents
Multi-relay internet big data pushing method and system Download PDFInfo
- Publication number
- CN105791399A CN105791399A CN201610111050.6A CN201610111050A CN105791399A CN 105791399 A CN105791399 A CN 105791399A CN 201610111050 A CN201610111050 A CN 201610111050A CN 105791399 A CN105791399 A CN 105791399A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- pushing
- module
- push
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
- H04L65/1073—Registration or de-registration
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Abstract
The invention discloses a multi-relay internet big data pushing method and system. The method comprises following steps of interface management, task management, name register, data preprocessing, data pushing, data relay, data reception, data verification and management and cache data management. Application instances finish big data pushing and precise delivering tasks through name registering and logging in, and the instances include methods and processes of big data pushing task reception, processing, pushing, relaying, reception, rechecking, delivering and a full-duplex working mode thereof. After the system is reasonably deployed based on the method, huge data can be pushed from a general internet access point to another remote internet access point completely, reliably, efficiently and automatically; under the conditions such as one-to-many pushing, offline pushing and access terminals belonging to different internet access operators, the system provided by the invention selects a suitable relay route, therefore data channel bottlenecks among the internet access operators are avoided; and the asymmetry of an ADSL (Asymmetric Digital Subscriber Line) is utilized effectively.
Description
Technical field
The invention belongs to the big data-pushing technology in the Internet, be specifically related to the big data push method of many trunk interconnection net and system.
Background technology
Some applications typically require and from a place, substantial amounts of data are pushed to remote another place, such as tens million of medical image files (at 500KB 25MB, minority file arrives greatly 1GB 5GB to usual each file size) are pushed to strange land as disaster-tolerant backup.The domestic medical image data having tens thousand of medical institutions is much larger than above-mentioned quantity.Our live and work marches toward big data age gradually, and similar application demand gets more and more.Data-pushing needs possess completely reliable, high efficiency, three basic features of automatization: data have the loss of only a few or damage to be unallowed in the overwhelming majority applies；Data-pushing will raise application cost slowly, or affects user's experience, affects work efficiency；The propelling movement of mass data obviously can not operate by hand and must reduce all possible fault to greatest extent, also to reduce manual intervention when fault occurs.If adopting Leased line to connect to push, because track stability is high, speed is relatively fast, not only efficiency solves substantially, out of order probability also also reduces significantly, even can not do special process, the work needing automatic business processing also reduces relatively, but Leased line link cost is high, does not possess and is widely applied to basic unit, condition to family, community.The data link of the Internet has bottleneck problem between different linking Internet operators, if the two ends of data-pushing use the incoming end of different linking Internet operators, the speed of mutual propelling data compares the application using same linking Internet operator incoming end, and speed is much lower.It addition, linking Internet is main with ADSL (AsymmetricDigitalSubscriberLine, Asymmetrical Digital Subscriber Line) technology at present, it is characterized in unsymmetry, namely downloads fast more than uploading.
The method that data-pushing can realize at present has a lot.Email be everybody use the earliest, file delivery method the most widely；FTP (file transfer protocol (FTP)) is again the standard file transfer protocols of another kind of international endorsement, programmed further by FTP, SMTP, POP3, IMAP4 and substantially can realize that data are substantial amounts of, the transmission of automatization, but being used for processing the problems referred to above, difficulty is a lot, and some even can not be broken through.The design of Email itself is not intended to transmit big data.FTP is owing to setting up the Different Period of this standard, and background and context is different, and during for big data-pushing, efficiency and safety are inadequate, and its reliability needs substantial amounts of program to process with realizing automatization, and exploitation amount is big.The complexity of system is usual and reliability negative correlation, structure and program are more complicated, is more easily reduced reliability.QQ off-line files sends, wechat file sends and can send out one file of propelling movement or a limited batch file.Some video website provides data-pushing support, pushes oneself video interested or music phone to home in office, is likely to download when go home in the evening, will not often wait for downloading because of network problem during viewing.The each service provider providing cloud disk service also all supports that data are uploaded and downloaded.
Realize at present and mode that everybody commonly uses, there is series of problems when the demand: service provider does not guarantee that the data that user pushes are bound to accurately arrive at, in fact we it is frequently necessary to again send by hand sent out but unsuccessful file；Manual operations pushes every time quantity of documents and size are restricted, far do not reach the requirement that mass data pushes；The application problem of the bottleneck problem between linking Internet operator, the asymmetric feature of ADSL could not be efficiently solved.
Although the application for a patent for invention " data push method based on condition managing that publication No. is CN104601668A, device and system ", publication No. is the application for a patent for invention " data push method and system " of CN104408132A, publication No. is the application for a patent for invention " incremental data supplying system and method " of CN104516989A, publication No. is application for a patent for invention " a kind of data push method of CN104378399A, Platform Server, client and system " etc. file, and some papers delivered all mention data push method, but do not have to solve series of problems recited above.
Summary of the invention
It is an object of the invention to overcome the deficiencies in the prior art, propelling movement demand for reliable, efficient, the automatization of the big data of satisfying magnanimity, and solve can the bottleneck problem of data channel between popularization and different linking Internet operator, it is provided that a kind of big data push method of many trunk interconnection net and system.
The big data push method of the present invention many trunk interconnection net includes:
Sequencing or manual operations accept big data-pushing task, task is organized and pretreatment, pretreatment treated that propelling data is managed, next reception example is pushed data into by data-pushing thread under task management coordination, after eventually arriving at propelling movement destination address, data are carried out consistency checking, confirm to push successfully, reception example by the data received by the interface person that submits to market demand；
According to the propelling movement demand in application and applied environment, push one to one or one-to-many pushes, by receiving whether object instance often connects, push example linking Internet end with whether object instance linking Internet end is that the condition criterions such as same linking Internet operator select by 0 or multiple relaying example propelling data；
Described example is by name registration and logs in the title and IP address that are mutually distinguishable propelling movement source, relaying and object instance；
When automation data pushes, processing procedure occurring, the fault that generality can temporarily be skipped sends exception reporting respectively with when there is expendable abnormal failure；
Each example full bi-directional mode of operation, can push simultaneously and receive.
The present invention also provides for a kind of big data delivery system of many trunk interconnection net, including:
Data-pushing module, for from an example propelling movement target data of system to another example；
Data reception module, in reception system, an example pushes the data come；
Task management module, centered by the goal task of propelling data, carries out United Dispatching to the related process module around task, and management role record and execution status of task；
Data preprocessing module, for the data being about to push are checked, and generates task description data, carries out the compression of target data as required；
Data buffering memory management module, for pretreated data and the data that receive are managed, disk storage space deficiency alarm also starts relevant treatment program；
Data Verification module, compares with pushing initial data for the data received, and the result determines that entrance task completes or enter again to perform flow process；
Interface administration module, for accepting data-pushing task, the data received are submitted to destination application, the information that sends in intermediate treatment process and accept manual intervention and control information；
Name registration service module, for providing the name registration of example, registration and address lookup；
Network detection module, is used for detecting computer network interface card duty, network connection state between the Internet connection status and example, network connect disconnect after automatically recover to connect.
Key operations " data-pushing " in the inventive method and system, is different from usually said data and uploads, downloads.Propelling movement be directly content to user, reception need not confirm.Pushing sender and have whip hand, recipient is passive；And after uploading data, need recipient to be actively downloaded operation.
The above-mentioned big data push method of many trunk interconnection net and system can run on as example one by one and be there is a need to push and receive on the computer of big data by infinite copy copy, all processes are based on automatic business processing, target data and task are compressed coding and push, push front and back end and carry out highly reliable comparison of coherence, achieve completely reliable, high efficiency on the internet, automatically push big data；Pushed by one or more levels relaying, avoid the data channel bottleneck between linking Internet operator, the propelling movement of one-to-many is effectively utilized ADSL asymmetry, also it is different from and at present uploads and downloads pattern with data center for core, simplify and dispose, reducing cost, obtain the raising in performance and stability while reducing complexity, network resource utilization is higher simultaneously.
Before present system stable operation, do not find big data way of stable transmission or propelling data on the Internet that common wide accesses always, lettergram mode, FTP mode, each cloud storage are uploaded the modes such as download and all can not be met demand, under above-mentioned condition and mode push or transmission data reach about 10GB begin to meet difficulty.Present inventor has performed and push or the experiment of transmission data connecting either directly through TCP/IP of a period of time, run into a lot of problem, such as can not connect between interconnection access carrier, data channel bottleneck is only small, network normal off line etc..After present system stable operation, being complete automatization in a particular application and push the medical image class of 1,000,000 256KB 5GB and relevant file, data are completely the same with propelling movement source data after arriving target, it is possible to normally use.According to operational monitoring, some the Internet pushing the machine that example runs connect average 2-3 minute can short interruption once；But present system still can for a long time, stable, automatically propelling data data.
Beneficial effects of the present invention: achieve completely reliable, high efficiency on the internet, automatically push big data；Disposing simplifying, reduce cost, also improve performance and stability while reducing complexity, network resource utilization is also higher.
Accompanying drawing explanation
Fig. 1 is the basic flow sheet of the big data push method of many trunk interconnection net of embodiment；
Fig. 2 is the task management flow chart of the big data push method of many trunk interconnection net of embodiment；
Fig. 3 is task record structure and the flow chart of the big data push method of many trunk interconnection net of embodiment；
Fig. 4 is that the pushing away for a pair two of the big data push method of many trunk interconnection net of embodiment, tape relay push application flow schematic diagram；" example " represents a copy of present system；
The relaying that Fig. 5 is the big data push method of many trunk interconnection net of embodiment pushes schematic flow sheet；
Fig. 6 is the data verification schematic flow sheet of the big data push method of many trunk interconnection net of embodiment；
Fig. 7 is the population structure schematic diagram of the big data delivery system of many trunk interconnection net of embodiment；
Fig. 8 is the interface administration module internal cell structure schematic diagram of the big data delivery system of many trunk interconnection net of embodiment；
Fig. 9 is the name registration inside modules cellular construction schematic diagram of the big data delivery system of many trunk interconnection net of embodiment；In one embodiment, present system is deployed in each place accessing the Internet, needs propelling data according to user each other, because linking Internet is many without fixing IP, to be identified by a unique title each other.This title is registered before using, and is called instance name or site name in elucidated hereinafter；
Figure 10 is the task management module internal cell structure schematic diagram of the big data delivery system of many trunk interconnection net of embodiment；
Figure 11 is the data buffering memory management module internal cell structure schematic diagram of the big data delivery system of many trunk interconnection net of embodiment；
Figure 12 is the Data Verification module internal cell structure schematic diagram of the big data delivery system of many trunk interconnection net of embodiment；
Figure 13 is the sequential logic schematic diagram of the embodiment big data delivery system of many trunk interconnection net.
Detailed description of the invention
Following example are further elaborated in conjunction with the enforcement technical scheme of 1～13 couple of present invention of accompanying drawing.
The big data push method of many trunk interconnection net includes:
See Fig. 1, for the basic flow sheet of the big data push method of many trunk interconnection net.
Step S101, receives propelling movement task, and gives interface management task manager propelling movement task；Basic task unit is a file or a catalogue；
The gateway of step S110, interface management, task data and message.Accept from the step S101 propelling data task entered, the data received are submitted to by step S191, the big data push method handling process of whole many trunk interconnection net occurs need the message reminded or warn by exporting from step S105 here, the control message of manual intervention is input to here from S105 equally, and is submitted to following task management S130.
In an embodiment data to be pushed are put into the propelling movement buffer directory specified, such as " D: SendingBuffer " catalogue.It is possible to further data directory title to be pushed is put in the file specified, such as: " D: SendingTask PacketTasks_IncludeAllSubDirs.txt ", below such assigned catalogue, all of subdirectory all can pushed be gone out.
In an embodiment, the destination address title pushed can wait to push appointment in buffer directory, such as: " D: SendingBuffer medical image backup center ", wherein " medical image backup center " is a destination address title, is placed on the content below this catalogue and can be pushed on the computer that " medical image backup center " this example runs.
Step S120, instance address title manages, and one of them has the example run on the computer of fixed ip address or the sensing of fixing domain name and accepts registration, logs in and inquire about, referred to as name registration Service Instance；Other instance names log in, IP address updates and inquire about address name and the IP address of propelling movement task relevant party.
In an embodiment, the safety pushing and receiving example is controlled by name registration Service Instance, namely carries out title examination & verification, login password authentication.
Step S130, is managed pushing and receive of task, and wherein concrete management process is as follows:
See Fig. 2, for the task management flow chart of the big data push method of many trunk interconnection net of embodiment.
Step S210, accepts propelling movement task, and information that task is described substantially is stored in task entity record table, enters and waits to push task queue.Task record flow process and structure are shown in Fig. 3.
Step S220, takes out ancestral task in waiting list and gives process of data preprocessing, monitoring and process after management pretreatment, to after pretreatment further standardization and refinement treat that propelling movement task data is registered and more new state.This step generates task record file, with further reference to S330；
Task status comprises in an embodiment: waits, be ready to, push and carry out, receive and carry out, finish receiving, pushed, hang up, stop, retry；
In an embodiment standardization and refinement be treat the number of files that comprises in propelling movement task, file size, document time are analyzed and record, generate assignment file, the process that pushes performs suitable method for pushing according to assignment file；
Step S230, gives data-pushing processing procedure ready assignment file and propelling movement task data, and monitoring task data pushes progress, and updates progress status；
Step S240, Monitoring Data receives process, receiving of task is joined task master record table, and updates data receiving state；
Step S250, calls data validation process, and the data pushed and receive are verified, and is proved to be successful entrance and completes flow process or relaying pushes flow process, and authentication failed is gone further data-pushing flow process again；
Step S260, propelling movement completes, and pushes the state after instance processes task completes and data process break, receives example logger task state also by interface management S260: process submits the data received to.
The structure and the flow process that push task record describe as follows:
See Fig. 3, for task record structure and the flow chart of embodiment method.
Step S320, accepts propelling movement task from S101 and S210, and task mainly describes information and recorded in main task log；Information write main task log too task is mainly described when receiving propelling movement task from S240.
In an embodiment, main task describes information and includes: mission number, propelling movement person, recipient, current location, progress status, retry state, priority, task type, task size, task time, task data original position, buffered data position, task customer number, task correlation number.
Step S330, generates assignment file, the concrete exercisable pushed information of one task of record at S140 and S220.
Assignment file includes but not limited to following information in an embodiment: describes the file header of task main information and treats pushing files list information, wherein comprising: file name, size and time attribute, subtask ID, duty, compression code, MD5 coding.
One main task generally comprises multiple subtask, and namely main task is a catalogue file collection, and subtask is a file.MD5 (MessageDigestAlgorithm, Message Digest Algorithm 5) is for data consistency checking.
In an embodiment, compression code 0 represents uncompressed, and 1 represents standardized ZIP compression, and other compression code defines with recipient's application program by pushing.
In step S340, S160 and S170 execution process, record pushes sheet segment information.
In an embodiment, S160 often pushes a successfully packet, records information in fragment log file.Whenever pushing midway Network Abnormal disconnection or other problems causes interrupting, again recovering first to attempt recovering to push from interruption position, if recipient's accreditation, continuation pushes, and otherwise starts anew.Although processing mode is different, but this process has same effect with current " breakpoint transmission " technology.
The data carrying out overcompression or other codings are recovered by step S140 after propelling data pretreatment and reception propelling data.Working in the example pushing role, extract at propelling movement preprocessing process simultaneously and treat that propelling data summary info adds to main task log, generate assignment file, task status enters and gets out state, task management process S130 scheduling enter following propelling movement flow process.
Step S150, is managed through the pretreated ephemeral data treating that propelling data, assignment file, propelling movement fragment log file, the data received and program process produce.Management operation therein includes storage space management, stale data cleaning, file and data storage, reads.
Such as in an embodiment, the buffered data storage dish of the computer of one example operation is 100GB, ADSL broadband access propelling data 8GB-10GB about every day, mass data periods of months or longer propelling movement and relaying produce ephemeral data and are easy to buffered data storage to take inventory completely, during memory space inadequate, each link is likely to produce extremely, including computer operating system.Step S150 effectively controls buffered data storage dish and excessively uses.
Step S160, is dispatched by S130, extracts data to be pushed from S150, by TCP/IP communication, finds propelling movement destination address or relay address IP from S120, and assignment file and target data are pushed to next destination address.
In an embodiment, the primitive of each propelling movement is 4KB, if remaining data is less than 4KB, pushes by real data size, but is not less than 48byte；If remaining data is less than 48byte, after mend " 0 ".
Further, comparison step S160 processing procedure can increase primitive size, attempts increasing 512Byte every time, carries out the comparison under comparable conditions with before increasing after increasing 3-5 days, before being better than increasing, can using the size after increasing as primitive size.Find most suitable primitive size by this principle, but scope is at 2KB 32KB.
Step S165, the concordance of the data after verifying pushed source beginning data and arriving at, checking is divided into checking step by step from small to large and overall checking, and wherein concrete to introduce flow process as follows:
Fig. 6 is the data verification schematic flow sheet of embodiment.
Step S610, extract propelling movement task summary info, including assignment file quantity, the total size of task, Data Position, file name list, the size of each file, file MD5 code；
Step S620, propelling data fragment consistency checking, and breakpoint can succession checkings after recovering.Often push a data slot, instant its concordance of checking；When recovering again after leading often interruption propelling movement to push, the data slot that detection both sides have successfully pushed, finds unanimously to put and recovers to push；
Step S630, pushing files consistency checking: pushed a file, verify source beginning file data and the concordance of the file data arriving target ground by MD5 code；
Step S640, propelling movement task is totally verified, after each file subtask is proved to be successful, main task confirms to be proved to be successful；
Step S650, batch is reaffirmed: carries out once batch after propelling movement completes a collection of task and reaffirms flow process.Basic process is the file pushing example one XML format of task summarization generation needing to confirm, it is pushed to data-pushing object instance, object instance requires after receiving that the application program using target data is to confirm that whether data are normal, then returns to propelling movement example result.This flow process is filled in a form as the taskpad in routine work confirmation.
After S650 verifies that flow process avoids data to be correctly pushed to destination address further, the loss of data caused in a short time due to market demand person's miscarriage or damage.Such as: target data application person causes loss of data in unloading data procedures, or maloperation have modified initial data when not backup.
In an embodiment, push batch of data, when not accepting new task in 15 minutes, automatically initiated once batch and reaffirm；Automatically initiate once batch every night 12 to reaffirm；Batch reaffirms that the generation after successfully is reported, by S105, notifies the management personnel specified with the form such as mail, wechat.If buffer-stored disk space is less, the relevant ephemeral data of the task after successfully that pushes completely is eliminated immediately.
Step S170, keeps one normal be connected with data-pushing example or a upper relaying example, wait and push the data come and receive data.
In an embodiment, after the reception data pushing task complete and pass through checking, next step flow process will be entered by task management S130.Logic flow, relaying pushes and is again pushed to next object instance after finishing receiving, and is this process below:
See Fig. 4, push application flow schematic diagram for the pushing away for a pair two of the embodiment big data push method of many trunk interconnection net, tape relay.
In an embodiment:
S410 represents the execution example that source data is pushed, and when a propelling movement task occurs, first object is example 3 (S430), pushes assignment file and target data respectively；Second target is example 5 (S450), only pushes assignment file, and target data does not repeat to push；
S420 is a relaying flow process, and this flow process receives the laggard row buffering storage of data, and one of them assignment file and data trnascription are pushed to first aim S430；When receiving second and pushing assignment file, because target data is in buffering, directly assignment file and target data are pushed to next target S440；
S430 is a data receiver flow process；
S440 is a relaying flow process, and similar S420 turns propelling movement the difference is that S440 belongs to direct repeating system.
S450 is a data receiver flow process, identical with S430.
In an embodiment, solve the data channel bottleneck problem between different linking Internet operator, be implementing method below:
Seeing Fig. 5, the relaying for embodiment pushes schematic flow sheet.
In an embodiment:
S530 is a relaying instance processes flow process；S531 represents the propelling data line attachment from a linking Internet operator, and S532 represents the line attachment propelling data from another linking Internet operator of family.It is that same access carrier, pushing speed and stability have substantive lifting that linking Internet between the pairing example of so each propelling movement and reception connects.
In an embodiment, in multilevel relay propelling movement, the method for selecting paths is one-level relaying is a fixing instance name address, and the propelling movement destination address of propelling movement person is the instance names of two grades of relayings, and two grades of relayings arrange next stage and turn propelling movement object instance, by that analogy, target location finally it is pushed to.
Such as: it is Zhang San that source data pushes instance name, push object instance name and be called Li Si.The actual route that pushes is instance name=Zhang San → instance name=service → instance name=mobile UNICOM's relaying → instance name=Li Si.Wherein one-level relaying is as a fixing instance name, and two grades of relayings are specified in name registration and pushed by another Instance Name relaying.
Step S180, detection the Internet connection status, whether the example of detection pairing work can find, the connection status of the example of detection pairing work.
Step S191, feeds back to management personnel normal operation information and abnormal failure information, and accepts the management code that management personnel send.
In an embodiment, according to name registration and cell-phone number reserved when auditing, micro-signal, after wechat pays close attention to the public service number that embodiment is specified, S191 calls the wechat message of company of Tengxun and pushes interface, and data-pushing progress msg is issued management personnel.Same mode, S191 can accept control information that management personnel send and transfer to task management flow process.The information when identity of management personnel is by name registration confirms, it is prevented that system is hacked.
Such as: " on January 11st, 2016 Monday, you received and pushed, from XXX hospital, the batch of data of coming today: general assignment number: 480, total files: 3680, total size: 8.65GB, failure quantity: 0 ".
For another example: " XXX: have a collection of propelling data to you, but your website is not connected with, it may be possible to network failure or computer shut down, please check.”
In an embodiment, control routine:
！11: submit Top Line Report to；
！12: submit memory space situation to；
！13: submit to and push Task Progress；
！14: submit to and receive Task Progress；
！77=[customer number] > [instance name]: the task of customer number client specified turns and is pushed to the instance name specified.
Customer number is generated by application person, is likely in an embodiment repeat, but pushes example for one, uses and arranges a number oneself easily identified during present system propelling data, does not generally repeat.
The not special restriction of control routine and implication, be only a kind of agreement.
Further, controlling message and comprise multiple parameter, middle symbol " ^ " separates, the type of message in first symbol section of expression of each section, symbol "！" represent execution, " P " represents control password.In true running example, control message all with password；Only control password and normally just can carry out corresponding action.
The big data delivery system of many trunk interconnection net based on said method, as it is shown in fig. 7, comprises: interface administration module 710, network detection 714, name registration service module 718, task management module 722, data preprocessing module 726, data-pushing module 730, Data Verification module 734, data buffering memory management module 738, data reception module 742.In an embodiment, the rarest three copies of system, hereinafter referred to as example, run on the computer of more than three that access the Internet, a computer runs an example, and wherein a computer has fixing IP or fixed field name analysis to point to.
In practical application, having the application of big data-pushing demand can install an example, registration instance name also just can push after logging in or receive data.So can there is unlimited number of example in an Application Example.
The effect of above-mentioned each module is as follows:
Interface administration module 710 accepts data-pushing task, the data received are submitted to destination application, send in intermediate treatment process message and accepts manual intervention to control information, wherein comprises with lower unit:
See Fig. 8, for the interface administration module internal cell structure schematic diagram of embodiment supplying system, wherein:
Interface management unit 7101, manages unit in module for management interface, and interface shape can have more change, increases according to demand；
Windows messaging task receives and submits unit 7103 to, accepts propelling movement task with windows message mechanism sendmessage (windows send out pass message basis function) function, and is given notice by windows message mechanism after receiving data.Message code is WM_COPYDATA=$ 004A, data directory to be pushed or file parameters, and the transmission of the data directory finished receiving, file PCopyDataStruct structure；
WindowsAPI task receives submits unit 7105 to, receives propelling movement task by API (application programming interface) mode and submits the data after finishing receiving to.The parameter pushing interface function includes object instance title, task type, pushes target data catalogue or file, whether includes the option of subdirectory；The parameter receiving data notification function includes task type, propelling movement person, data directory or file；
Fixed position task extraction unit 7107, timing automatic inspection is specified and is waited to push catalogue, if it find that there are data, just extract and submits propelling movement to；Manage submitted but also data sent to unconfirmed；
Message pushes and message control unit 7109, the message needing prompting or warning occurs by sending management, receive the control message of manual intervention simultaneously, be submitted to task management module after parsing in whole system handling process；
In an embodiment, the message of management type pushes and sends with the form such as mail, wechat.Such as: pushed batch of data, distant place propelling data sends, data arrive, relay the report related example administrative man such as the information that turns propelling data amount；Necessary example is not keyed up running, occur the fault etc. that cannot automatically recover in data-pushing and processing procedure gives a warning cell-phone number, micro-signal and the e-mail address that class message registers when name registration to related example administrative man；
In an embodiment, unit 7109 increases communication module, it is possible to receive SMS, and example manual intervention controls message and sends with mobile phone short messages.
Name registration service module 718, it is provided that the name registration of example, registration and address lookup；System has important advantages in that between ADSL broadband internet incoming end to push big data, and ADSL broadband great majority do not have fixed ip address, therefore this module not only admin instance title, also the IP address of example dynamically to be updated.This module includes with lower unit:
See Fig. 9, for the name registration inside modules cellular construction schematic diagram of the embodiment big data delivery system of many trunk interconnection net.
Name registration unit 7181, registers instance name, arranges password, detects whether title repeats；One that example is native system is run copy, close but incomplete same with the user of a system in our custom, is the user of a kind of specific type；
Further, when increasing Subscriber Management System in embodiment, the instance name of registration can also associate user name, mutually binds, and can notify association user during the message that is associated；User can also manage associated instance title；
Title examination & verification unit 7183, is a selectable unit, whether is responsible for the example of examination & verification registration normally and whether allow to add；
Title applying unit 7185, has three functions: the more IP address after new change, by the instance name registered and login password login authentication when example brings into operation, searches and pushes object instance title and address.The safety pushing and receiving example thus controls, and namely propelling data must arrive real object instance, and is not intercepted.
Task management module 722 centered by the goal task of propelling data, the United Dispatching that carries out to the related process module around task, and management role record and execution status of task；Including with lower unit:
See Figure 10, for the task management module internal cell structure schematic diagram of the embodiment big data delivery system of many trunk interconnection net.
Task record table administrative unit 7221, the main task record that management pushes, main task record represents the basic description information pushing task, such as Data Position, form, size, task state of progress etc..In an embodiment, master record table adopts relevant database to be managed, type of database is not limit, can be such as ACCESS, SQLServer etc. of Microsoft, but have important advantages in that data-pushing between linking Internet end widely due to the present invention, example deployment is extremely simple, direct copying or download decompression just can run, duty just can be entered after login, when therefore adopting data base administration main task record, prioritizing selection ACCESS mono-class installation process does not need the data base of manual intervention；
Further, in order to improve versatility and reduction installation difficulty, unit 7221 provides main task log operating capability, including the function increasing, deleting, inquire about, extract, move main task record；
Thread-management unit 7223, the execution state of progress according to task, the execution thread in scheduling modules, the logic of the cross-thread that coordination concurrently performs and data flow, complete propelling movement task.Ratio is if any substantial amounts of wait task, data prediction thread in data preprocessing module carries out pretreatment, there is the ready data-pushing thread started when propelling data in data-pushing module, in data reception module network connect and etc. propelling data thread to be received be in listening state, data call data verification thread after preparing and finishing receiving, multithreading is collaborative, concurrent efforts, to improve efficiency.
Task status administrative unit 7225, management pushes the duty of tasks carrying process.Propelling movement task status is set according to according to progress, such as waiting state, gets out propelling movement state, push and carry out state, receive and carry out state, finish receiving state；Further, according to current state, and the return result of Current transaction, determine next state value, as in waiting state, first time data prediction have failed, it is probably file just occupied, through three trial and successs, then next state is to retry state；If have attempted to ten times also unsuccessful, next state is fault abort state；
Task status is likely to different in various embodiments, be not limited only in the present embodiment occur state description；
Affair managing unit 7227, the generation of the summing-up work report in management course of normal operation, push process and correlated process occur general message, warning class message, failure classes message generation, and management personnel's manual intervention message perform scheduling.
The data carrying out overcompression or other codings are recovered after propelling data pretreatment and reception propelling data, are extracted and also generate the summary info treating propelling data, generate main task record description data, generate assignment file by data preprocessing module 726；Comprising a data prediction thread in this module, above-mentioned process performs in thread, can perform with other thread parallels；
Data-pushing module 730, comprises data-pushing and processes thread, extracts and treat propelling data from the ready relief area of pretreatment, by ICP/IP protocol, connects the example of next present system, and with ICP/IP protocol to its propelling data.The function that module 730 records propelling data bag summary, breakpoint recovery pushes；
Data Verification module 734, it is provided that data slot or be the data verification function of All Files and a collection of task in packet, single file, a catalogue, including with lower unit:
See Figure 12, for the Data Verification module structural representation of the embodiment big data delivery system of many trunk interconnection net.
Metadata digest information extraction unit 7341, it is provided that search catalogue file, acquisition assignment file quantity, total size, listed files, calculation document MD5 code, CRC32 code；
Data verification units 7343, it is provided that the concordance comparison based on basic file attributes, MD5 code and CRC32 of data slot, file and file set；
Breakpoint recovers authentication unit 7345, pushes instance data and pushes the consistency checking of location records and data receiver record and the verification of correctness of reception part of a file；
Batch authentication unit 7347, batch tasks is provided to reaffirm function: the batch of one XML format of mission bit stream summarization generation of a collection of task is verified application form file, it is pushed to propelling movement object instance, object instance receive after by batch proof listing requirement record, respectively one by one to whether the user checking data of target data correctly can use, result inserts application form, then result is returned to propelling movement example, receive the application form after checking, read checking record, understand and phase inversion answers processing module.
Data buffering memory management module 738, pretreated data, the data received and work process data are managed, cache disk management, the data management of buffer-stored, buffered data is provided to access and relevant issued transaction, including with lower unit:
See Figure 11, for the data buffering memory management module internal cell structure schematic diagram of the embodiment big data delivery system of many trunk interconnection net.
Buffer-stored disk management unit 7381, the state of monitoring disk storage space, clear up stale data, when memory space is less, little, start relevant treatment program；
Access visit administrative unit 7383, tissue treats the storage organization of propelling data, assignment file, intermediate buffering storage file, it is provided that directory creating, file read-write, propelling movement fragment information record function.
Share and access unit 7385, washability processing unit, to super large file, the file of such as more than 500MB, it is provided that creates in ablation process, is read the ability again relaying propelling movement simultaneously by another thread.
Data reception module 742, receives an example on native system and pushes the data come, wherein comprise a thread and TCP/IP network connection management；When working in repeater mode, being again pushed to next example flow process after optionally passing through data verification, modular structure, subsequent treatment transfers to task management module decision-making.
Network detection module 714, is used for detecting computer network interface card duty, network connection state between the Internet connection status and example, and network connects to disconnect and automatically recovers when postcondition possesses to connect.
The basic execution process of embodiment is: a self-extracting bag of (one) download system, size is about 2.0MB, self-extracting, automatically runs；(2) registration instance name, ID, for the ease of understanding also referred to as site name, login system；(3) selecting one of which to increase the mode of task, such as in task menu, choosing increases propelling movement task operating, and selects a data directory, selects to comprise subdirectory option, and type is packet, clicks and starts to push；(4) data-pushing relays example to first, and this is one and has installed and properly functioning system example, it is possible to be called services sites, although all examples are the same, but this services sites requires and has separate tP address, reliable network condition；Rent four domestic Cloud Servers at present and an external Cloud Server takes this role；After (five) first relaying examples complete a task reception, search object instance or whether two grades of relaying examples are movable, if activity, push the task data finished receiving to next stage example, otherwise notify that subordinate instance management people starts shooting reception with mail or wechat mode；(6) two grades of relayings are if it exists, perform the process the same with the 5th step；(7) after object instance finishes receiving task data, the concordance of checking data, circularize data consumer or use the application program of data；(8) data-pushing side carries out propelling movement task after completing a collection of task and reaffirms flow process, and person confirms after successfully to obtain market demand, removes buffering ephemeral data and operational data；(9) when breaking down in all processes, limited number of time is attempted automatically recovering.
See Figure 13, for the sequential logic schematic diagram of the big data delivery system of many trunk interconnection net that the present embodiment provides, in order to further illustrate example operation principle and flow process.800 represent task entrance；The 810 propelling movement work roles representing system example；812 represent system example have fixed ip address main relaying roles, generally this relaying as substantial amounts of propelling movement application example " " center ", each application example finds existence and the address of object instance by this relaying；The 814 inter-network relaying work roles representing system example, the computer running present example has two or more network interface cards, connects the network of diverse access point respectively；816 is the data receiver work role of system example, is a data-pushing destination pushing flow process, but flow process does not also terminate；821 represent taskings and data prediction flow processs, and taskings comprises generation main task log, assignment file, task description are submitted to next relaying, and data prediction flow process is shown in noted earlier；822 store management process for data buffering；823 is data-pushing flow process；824 is data receiver flow process；825 recover and data verification flow process for data, and data are recovered the data comprised compressing or carried out recompiling and returned to state；826 finally send to flow process for data, see S110 and S191；828 for attempting repairing the handling process pushed, and pushes flow process from pushing to start to attempt repairing when target returns and pushes failed message, checks the data pushing front and back end, search discrepancy；829 complete follow-up processing flow for propelling movement, comprise message informing push people, recipient, delete not in use by ephemeral data；830 represent that reparation pushes execution process, are equivalent to again push flow process, but have pushed successful data and be not repeated to push, actually push flow process and comprised this function.
One workflow comprises 2-4 example, is 810,812,814,816 respectively, wherein in most cases 814 can omit, and actual comprises three examples.Operation principle easy to understand from sequential logic schematic diagram, supplementing the flow process set forth here is: after data arrive at, confirm success or failure according to data verification flow process, reverse for assignment file propelling movement is returned propelling movement side；If data verification success, pushing main task and terminate, but flow process does not terminate, namely assignment file also to return to propelling movement side.
The foregoing is only one embodiment of the present of invention, be only used for explaining the present invention, the present invention can not be limited with this, within the scope of all thought in the present invention and method, any amendment of making, improvement, should be included within the scope of protection of the invention.Module in the present invention and unit, the form that quantity both can be single exists, it is also possible to multiple forms exists, and this is not defined by the present invention.Step numbers in the present invention and module, element numerals do not represent the sequencing of flow process.
Claims (2)
1. the big data push method of trunk interconnection net more than a kind, it is characterised in that including:
Sequencing or manual operations accept big data-pushing task, task is organized and pretreatment, pretreatment treated that propelling data is managed, next reception example is pushed data into by data-pushing thread under task management coordination, after eventually arriving at propelling movement destination address, data are carried out consistency checking, confirm to push successfully, reception example by the data received by the interface person that submits to market demand；
According to the propelling movement demand in application and applied environment, push one to one or one-to-many pushes, by receiving whether object instance often connects, push example linking Internet end with whether object instance linking Internet end is that the condition criterions such as same linking Internet operator select by 0 or multiple relaying example propelling data；
Described example is by name registration and logs in the title and IP address that are mutually distinguishable propelling movement source, relaying and object instance；
When automation data pushes, processing procedure occurring, the fault that generality can temporarily be skipped sends exception reporting respectively with when there is expendable abnormal failure；
Each example full bi-directional mode of operation, can push simultaneously and receive.
2. the big data delivery system of the net of trunk interconnection more than, including:
Data-pushing module, for from an example propelling movement target data of system to another example；
Data reception module, in reception system, an example pushes the data come；
Task management module, centered by the goal task of propelling data, carries out United Dispatching to the related process module around task, and management role record and execution status of task；
Data preprocessing module, for the data being about to push are checked, and generates task description data, carries out the compression of target data as required；
Data buffering memory management module, for pretreated data and the data that receive are managed, disk storage space deficiency alarm also starts relevant treatment program；
Data Verification module, compares with pushing initial data for the data received, and the result determines that entrance task completes or enter again to perform flow process；
Interface administration module, for accepting data-pushing task, the data received are submitted to destination application, the information that sends in intermediate treatment process and accept manual intervention and control information；
Name registration service module, for providing the name registration of example, registration and address lookup；
Network detection module, is used for detecting computer network interface card duty, network connection state between the Internet connection status and example, network connect disconnect after automatically recover to connect.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610111050.6A CN105791399B (en) | 2016-02-29 | 2016-02-29 | More trunk interconnection net big data method for pushing and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610111050.6A CN105791399B (en) | 2016-02-29 | 2016-02-29 | More trunk interconnection net big data method for pushing and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105791399A true CN105791399A (en) | 2016-07-20 |
| CN105791399B CN105791399B (en) | 2018-02-09 |
Family
ID=56403886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610111050.6A Active CN105791399B (en) | 2016-02-29 | 2016-02-29 | More trunk interconnection net big data method for pushing and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105791399B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107707328A (en) * | 2016-08-08 | 2018-02-16 | 北京京东尚科信息技术有限公司 | Summary info transmission method and device |
| CN107844575A (en) * | 2017-11-06 | 2018-03-27 | 国家计算机网络与信息安全管理中心 | A kind of personal data shared system and method for internet financial platform |
| CN108241616A (en) * | 2016-12-23 | 2018-07-03 | 阿里巴巴集团控股有限公司 | Information push method and device |
| CN109067690A (en) * | 2018-08-07 | 2018-12-21 | 腾讯科技（深圳）有限公司 | The method for pushing and device of off-line calculation result data |
| CN112752353A (en) * | 2019-10-31 | 2021-05-04 | 中移物联网有限公司 | Connection method and terminal equipment |
| US11500702B1 (en) * | 2021-04-26 | 2022-11-15 | Visa International Service Association | System and method for timed data transmission |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1771701A (en) * | 2003-04-11 | 2006-05-10 | 艾利森电话股份有限公司 | Multi-user diversity forwarding |
| US20080181148A1 (en) * | 2007-01-08 | 2008-07-31 | Freesystems Pte., Ltd. | Multi-node media content relay system |
| CN102571565A (en) * | 2010-12-24 | 2012-07-11 | 株式会社日立制作所 | Packet transport node |
| CN103209435A (en) * | 2013-03-15 | 2013-07-17 | 河海大学 | Congestion control based wireless multi-hop network multicast method |
-
2016
- 2016-02-29 CN CN201610111050.6A patent/CN105791399B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1771701A (en) * | 2003-04-11 | 2006-05-10 | 艾利森电话股份有限公司 | Multi-user diversity forwarding |
| US20080181148A1 (en) * | 2007-01-08 | 2008-07-31 | Freesystems Pte., Ltd. | Multi-node media content relay system |
| CN102571565A (en) * | 2010-12-24 | 2012-07-11 | 株式会社日立制作所 | Packet transport node |
| CN103209435A (en) * | 2013-03-15 | 2013-07-17 | 河海大学 | Congestion control based wireless multi-hop network multicast method |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107707328A (en) * | 2016-08-08 | 2018-02-16 | 北京京东尚科信息技术有限公司 | Summary info transmission method and device |
| CN107707328B (en) * | 2016-08-08 | 2020-11-24 | 北京京东尚科信息技术有限公司 | Abstract information transmission method and device |
| CN108241616A (en) * | 2016-12-23 | 2018-07-03 | 阿里巴巴集团控股有限公司 | Information push method and device |
| CN108241616B (en) * | 2016-12-23 | 2023-07-25 | 阿里巴巴集团控股有限公司 | Message pushing method and device |
| CN107844575A (en) * | 2017-11-06 | 2018-03-27 | 国家计算机网络与信息安全管理中心 | A kind of personal data shared system and method for internet financial platform |
| CN107844575B (en) * | 2017-11-06 | 2021-10-15 | 国家计算机网络与信息安全管理中心 | Personal data sharing system and method for Internet financial platform |
| CN109067690A (en) * | 2018-08-07 | 2018-12-21 | 腾讯科技（深圳）有限公司 | The method for pushing and device of off-line calculation result data |
| CN112752353A (en) * | 2019-10-31 | 2021-05-04 | 中移物联网有限公司 | Connection method and terminal equipment |
| CN112752353B (en) * | 2019-10-31 | 2022-06-10 | 中移物联网有限公司 | Connection method and terminal equipment |
| US11500702B1 (en) * | 2021-04-26 | 2022-11-15 | Visa International Service Association | System and method for timed data transmission |
| US20230016188A1 (en) * | 2021-04-26 | 2023-01-19 | Visa International Service Association | System and method for timed data transmission |
| US11755391B2 (en) * | 2021-04-26 | 2023-09-12 | Visa International Service Association | System and method for timed data transmission |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105791399B (en) | 2018-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105791399B (en) | More trunk interconnection net big data method for pushing and system | |
| CN101711386B (en) | Synchronizing email messages between an external and/or local email server and/or a wireless device | |
| CN102193899B (en) | System and method for the data of the multiple environment of management spanning | |
| US6779022B1 (en) | Server that obtains information from multiple sources, filters using client identities, and dispatches to both hardwired and wireless clients | |
| US9009265B2 (en) | System and method for automatic transfer of data from one device to another | |
| US7698392B2 (en) | Method and system for establishing a user-friendly data transfer service application executing within a heterogeneous distributed service application execution environment | |
| CN101142567B (en) | System and method for disaster recovery and management of an email system | |
| JP5246332B2 (en) | Enhanced messaging platform | |
| KR100677497B1 (en) | Duplicated notification processing method in a terminal | |
| JP5537181B2 (en) | Message system | |
| EP1655915B1 (en) | Method for managing duplicated arrival notification messages in multimedia messaging services | |
| CN101702943A (en) | Apparatus and method for caching email messages within a wireless data service | |
| CN101043519A (en) | Network storage system | |
| US20060086798A1 (en) | Deferred email message system and service | |
| JP5607461B2 (en) | System and gateway | |
| CN101635908A (en) | Ticket transmission processing method and system of billing system | |
| CN101529399A (en) | Proxy server | |
| CN101001213B (en) | Short message transmission system and method | |
| CN103684837B (en) | Backup processing method, system and the back-up processing server of communication information | |
| CN100558188C (en) | Message processing device, system and method | |
| US8751583B2 (en) | System and method for providing business continuity through secure e-mail | |
| CN106850980B (en) | The method for cleaning and mobile terminal of Email attachment in a kind of mobile terminal | |
| CN106817680B (en) | Method for sending push short message, short message center and system | |
| US20120278409A1 (en) | Mail system | |
| CN103716228A (en) | Mail transmission method, mail gateway and mail transmission system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20180428 Address after: 650000 B007 2, 11 building, Shilin street, Chenggong District, Kunming, Yunnan. Patentee after: Yunnan open secret science and Technology Co., Ltd. Address before: 650000 701 unit 1, 36 building, 343 Longquan Road, Kunming, Yunnan. Patentee before: Zheng Jialiang |
|
| TR01 | Transfer of patent right |