Between the streamline multi-process based on the mass data transfers method of message queue
Technical field
The invention belongs to the extensive accurately data handling utility technology category of telecom operators, relate in particular between the multi-process mass data transfers method based on message queue.
Background technology
The data that charge system is handled all are mass datas, and operator of China Telecom generally is that unit builds charge system with the province at present, and the charge system of common 1,000 ten thousand customer volumes was handled ticket in every month and reached hundred million orders of magnitude.How improving system handles efficient, is vital problem.Solving on the efficiency, Linkage's charging product has the aspect of a lot of autonomous innovations, stresses message queue innovation on the data transmission between process in charging product here.
The telecom charging product in order to improve system performance, makes full use of the characteristic of the many CPU of main frame handling under the mass tickets data conditions both at home and abroad, adopts multi-process pipeline processes technology, realizes the efficient processing of ticket.Pipelining is mainly drawn concurrent operation overlappingly and processing by allowing a plurality of processes submit at one time, and is temporal parallel to realize.
Application number is the patent of CN00808157.3: be used for monitoring the just foundation of the subprocess in the application that computer system is carried out and the method and apparatus of releasing, wherein each subprocess has the subprocess database structure of a correspondence, this is just being carried out and is using the application process database structure with a correspondence, and this method comprises: provide one to be used for storing and this is just carrying out the table of the existing subprocess database structure of the existing subprocess correspondence in using; Judge when this application sets up new process in carrying out; The new process data library structure of retrieval and this new process correspondence; Utilize the parent process database structure of the parent process correspondence of this new process data library structure retrieval and this new process; Relatively this parent process database structure and this application process database structure; If this parent process database structure is identical with this application process database structure, be this new process identification (PID) an existing subprocess; And, this new process data library structure is added in this existing subprocess database structure table as if being this new process identification (PID) existing subprocess.
At least be divided into a plurality of processes such as format, regular (perhaps being called letter sorting), row's weight, wholesale price, warehouse-in in the general charging product business processing flow, article one, ticket will be in proper order by processes such as the format in the flow process, regular, row's weight, wholesale price, warehouse-ins, article 1, after ticket finished through whole flow process, the 2nd ticket just entered this flow process.Suppose to handle in the T time ticket, format, regular, heavy, the wholesale price of row, putting each process in storage, to handle the time of ticket be T time of 1/5, and system adopts after the pipelining, under many CPU hosted environment so, taking into account system IO expense not, conversation list processing speed will improve 5 times.Yet this is very desirable situation, because the IO of system expense can cause processing speed can not improve 5 times.If can avoid the IO expense, does not processing speed just improve so? but present charging product is considered the consistance of affairs, considers the rollback of ticket, still adopts the method for file transfer to realize the transmission of ticket between process.
In the charging product, generally use multi-process pipeline parallel method treatment technology to improve system handles efficient at home and abroad.Actual operating position, it also is apparent in view that efficient improves.Charging flow generally comprises a plurality of processes such as format, regular, row's weight, wholesale price, warehouse-in, is equivalent to a plurality of posies in the streamline.Ticket in the original bill files is placed on the input directory of charge system, after each process of experience streamline, finally is loaded in the database through the warehouse-in process.
Why use file transfer between each process in the charging product of other producers? why system does not handle a ticket, is just ticket given next process?
This is because the stability requirement of charge system is very high, in case make mistakes, needs rollback to handle, and rollback is very complicated in charge system.So many billing company can't be accomplished rollback,, whole file is submitted to so wait a file to dispose fully.If a ticket has been handled well, just submit to next process, the machine just in case system exception is delayed, different tickets are handled the very difficult judgement of what stage in the file, and this file can't be handled again, has destroyed data integrity.If submit to whole file, a file processing finishes and just result is submitted to, if delay machine so in processing procedure, this file does not all have to submit to.
Write the file the inside IO expense is arranged, why not call bill data treated in the file is put into the internal memory the inside, the intact submission again of file processing? because memory size is limited, but the CDR file size is unlimited, so a process is not handled before the file, and the ticket of having handled is kept in the temporary file, waits to dispose, change this temporary file into official document again, give next process.
This thought is very simple, but has brought the IO expense to system, directly has influence on the treatment effeciency of system.
Summary of the invention
The present invention seeks to: under the prerequisite of stability that guarantees system and data processing integrality, adopt the message queue mode to realize the transmission of ticket between process, reduce the IO of system expense, speed up processing.
The technology of the present invention solution is: between the streamline multi-process based on the mass data transfers method of message queue, a ticket order is by being divided into format, regular (perhaps being called letter sorting), row's weight, wholesale price, a plurality of processes of warehouse-in at least in the charging product business processing flow, and charging method adopts single step and the whole mechanism that combines of submitting to:
Format is as the source of charging flow, ticket in the file is distributed to the regular process the inside of different follow-up flow processs according to distribution rules, when finishing a document formatting, formatting process sends this end of file signal to each follow-up regular process; Each regular process receives an end of file signal, just carries out the single step of this process and submits to, and disposed all tickets of this file of being transported to this flow process of this process are described; The heavy process of row that regular then process is given next step with end signal, the same treatment mechanism of the heavy process of row is carried out single step and is submitted to; Heavy process of row and wholesale price process also are same processing modes; The wholesale price process is given end signal the warehouse-in process again, and the warehouse-in process is carried out single step and submitted to; Have only the warehouse-in process that ought format all flow processs that ending message is carried that this file is submitted to and finish, system just carries out the integral body of this file and submits to; If system in processing procedure, needs to interrupt, this time, a file did not dispose in whole streamline, and charging process so adopts out of true/fuzzy breakpoint; Out of true/fuzzy breakpoint is meant in taking place and has no progeny, the computing machine interruption that do not make an immediate response, but wait for that all tickets in the file finish, and just respond this interruption;
If the machine of delaying unusually takes place in system, so system restart the time, system finds to have the preceding process of which file warehouse-in process to submit to automatically, but the final process warehouse-in of streamline process is not submitted to this file, illustrate that so this file has problems, and just alarms in processing procedure; The maintainer checks, finds reason, carries out the file-level rollback of this file and handles
The file-level rollback need be cleared up the vestige that whole file stays in charge system everywhere, is the contrary operation of charging process, returns to initial conditions.
Charging method of the present invention takes into full account the rollback of system, and strict simultaneously submission mechanism is carried out assurance for adopting message queue.
Charging method of the present invention adopts single step and the whole mechanism that combines of submitting to.
Have only the warehouse-in process that ought format all flow processs that ending message is carried that this file is submitted to and finish, system just carries out the integral body of this file and submits to.
The method of pipelining handling interrupt has two kinds of inexact breaking method and exact breaking methods.Inexact breaking method is meant in taking place and has no progeny, the computing machine interruption that do not make an immediate response, but wait for that all tickets in the file finish, and just respond this interruption.
Affairs control technology and rollback technology have guaranteed the stable of system, and the introducing that also is message queue is as technique guarantee.
The present invention can be implemented in the message queue type that adopts under the varying environment by configuration; Message queue is the chained list of a message; In charge system, message just can be equal to ticket, is the record with specific format; There is the process of write permission can be to message queue to wherein adding new information according to certain rule; There is the process of read right then can from message queue, read away message to message queue; Adopting unified message queue data structure, is exactly the structure by the standard ticket, can set up message queue, the transmission of ticket between such each process of flexible realization by configuration between the process arbitrarily; The transmission of the call bill data of charge system magnanimity between process all by message queue, realizes in the internal memory the inside.
The present invention is by automatic allocating task, and the load balancing management is distributed to the message queue together of looking younger with ticket, and according to service logic, self-defined mode is disposed.
Characteristics of the present invention are: adopt the transmission of call bill data between process of the charge system magnanimity of the inventive method realization, all by message queue, processing procedure can realize there is not the expense of the IO of system in the internal memory the inside, and speed improves greatly.Employing obviously improves based on the system handles efficient of the pipeline parallel method treatment technology scheme of message queue.Processing speed is at home and abroad chargeed and is come out at the top in the producer.
Description of drawings
Fig. 1 is that different modes is finished the time course synoptic diagram that the ticket task is handled.
Fig. 2 is automatic allocating task treatment progress figure.
Fig. 3 is a treatment scheme synoptic diagram in the charge system of the present invention.
Embodiment
As shown in Figure 2, by automatic allocating task among Fig. 2, the load balancing management is distributed to different message queues with ticket.
Message queue is exactly the chained list of a message.In charge system, message just can be equal to ticket, is the record with specific format.There is the process of write permission can be to message queue to wherein adding new information according to certain rule; There is the process of read right then can from message queue, read away message to message queue.
Mainly contain two types message queue at present: POSIX message queue and the V of system message queue, system V message queue is used at present in a large number.Consider the portability of program, charging method of the present invention is implemented in to adopt which kind of message queue type under the varying environment by configuration.
Charging flow of the present invention is can be according to service logic, and self-defined mode is disposed, and for example some flow process does not need the heavy process through row, so directly sets up message queue between regular process and wholesale price process; Linkage adopts unified message queue data structure, is exactly the structure of standard ticket, can set up message queue, the transmission of ticket between such each process of flexible realization by configuration between the process arbitrarily.
The transmission of the call bill data of charge system magnanimity between process all by message queue, realizes there is not the expense of the IO of system in the internal memory the inside, and speed strengthens greatly.
It below is the test case of charge system treatment effeciency.
Test environment: IBM P650 16*1.5G CPU, 32G RAM
Tested object: the charging bill of a province, number of files to be put in storage is: 101724, have 628779011 records.
Method of testing: file is positioned over system input directory; System adopts single process (single current waterline) to test, and the difference of 2 kinds of tests is: first kind of method of testing ticket transmission between process is based on file mode, and second kind of method of testing ticket transmission between process is based on the message queue mode; Start charge system, the start-of-record time; The file number of monitoring input directory is until being zero, the end of record (EOR) time.
Test result: transmit between the process of ticket based on message queue: system is consuming time: 36557 seconds
Transmit between the process of ticket based on file: system is consuming time: 66362 seconds.
Test result: ticket is based under the transmission mode between the process of message queue, and system handles efficient improves a lot.As shown in Figure 3:
1, at first in charge system, disposes treatment scheme, also can be called and dispose a streamline, dispose the process that this flow process relates on stream; The foundation of message queue is self-defined mode, according to service logic, connects with black line between 2 associated process, and message queue is set up on the system backstage automatically;
2, the condition that the last process of configuration is distributed to subsequent processes in the message queue attribute;
3, process initiations all in the streamline is got up;
4, file first process of sending into streamline is a formatting process, this process is with File Open, read in a ticket, dispose, according to condition, formatting process passes through message queue, transmit telephone bill is to its follow-up some regular process, this regular process is constantly scanning message queue, if find the message queue the inside between this process and its related formatting process ticket is arranged, so just handle immediately, dispose, ticket is passed to the heavy process of the row who satisfies condition by message queue; Row weighs process and wholesale price process, wholesale price are advanced and the warehouse-in process also is same principle; Final warehouse-in process is delivered to ticket in the database;
If 5 a certain processes are handled slowly, cause last process at full capacity with the message queue space of self setting up, will stop up so, last process can't transmit ticket by message queue, last process is just carried out sleep, has living space up to this message queue, just can continue to handle;
When 6, formatting process is handled a file,, send document end message, carry out single step simultaneously and submit to follow-up regular process just by message queue; Processes such as regular, row weight, wholesale price, warehouse-in are received end of file message as a same reason, just carry out presents in this process single step submission;
7, receive the ending message of certain file when all warehouse-in processes, system carries out this document and submits in the integral body of native system.
The mode of adding new information is:
Create message queue in the time of system start-up, the kernel continuation of message queue requires all corresponding unique key assignments in system scope of each message queue, so, obtain the describing word of a message queue, only need provide the key assignments of this message queue to get final product; The message queue describing word is to be generated by key assignments unique in system scope, and key assignments can be regarded a corresponding intrasystem road warp as.
In the system handles process, there is a large amount of read-write operations in message queue, send message for last process, at first preset a msgbuf buffer zone and check whether message queue is full, if full, so with regard to first sleep, have vacantly up to message queue, call corresponding transmission function and get final product, adopt the msgsnd system function at the V of system message queue, in the POSIX message queue, use the mq_send system function, message content is sent in the message queue; Read message for back one process, at first distribute such msgbuf buffer zone, check in the message queue whether message is arranged, if there is not message, just enter the sleep state, message is arranged in message queue, adopt the msgrcv system function at the V of system message queue, use the mq_receive system function in the POSIX message queue, the message that will enter message queue from message queue at first reads out.
Be implemented in the mode (message queue is the chained list of a message) that adopts different messages formation (POSIX message queue and the V of system message queue, system V message queue) type under the varying environment by configuration among Fig. 3;
Because operating system definition, some system call meeting differences, at first in CONFIG.SYS, need the defining operation system type, the operating system of main flow comprises HP UNIX, SOLARIS, LINUX and AIX at present, the standard that system supports is influential to message queue, has POSIX or SYSTEM_V standard
At the compiling defined file the inside of system configuration operation system type, adopting the POSIX message queue by OS Type control still is the SYSTEM_V message queue.
Compiling defined file form is as follows:
# defining operation system type
#OS_DEF=DEF_HP
#OS_DEF=DEF_SOLARIS
OS_DEF=DEF_LINUX
#OS_DEF=DEF_AIX
The standard that the # system supports, this is influential to message queue
#POSIX?or?SYSTEM_V
The message queue of the machine of # Hewlett-Packard uses POSIX, other machine SYSTEM_V
:ifeq($(OS_DEF),DEF_HP)
SYS_TYPE=POSIX
:else
SYS_TYPE=SYSTEM_V
:endif
System satisfies the realization of the message queue under the different definition by conditional compilation.
Adopting unified message queue data structure, is exactly the structure by the standard ticket, can set up message queue, the transmission of ticket between such each process of flexible realization by configuration between the process arbitrarily; The transmission of the call bill data of charge system magnanimity between process all realizes in the internal memory the inside by message queue.
Adopting unified message queue data structure, is exactly the structure of standard ticket, and general bill structures comprises following content at least:
Struct?StdEvent
{
// original caller rs number
char?m_sOrgCallingNBR[40];
// original called number
char?m_sOrgCalledNBR[40];
// incident zero-time
char?m_sStartDate[15];
// incident duration
int?m_iDuration;
}
Can set up message queue, the transmission of ticket between such each process of flexible realization by configuration between the process arbitrarily.For example there is last process A, back one process B, the attribute of the message queue Q of configuration process B in allocation list of system will be set up the message queue Q of corresponding size according to definition system start-up the time so, and the owner is process B.Simultaneity factor is set up the distribution rules allocation list, is distribution conditions in the allocation list, and direction is that A is sent to B.Process A handles a ticket so, will judge whether this ticket satisfies distribution conditions, if satisfy distribution conditions, just ticket is sent to the message queue of B process.