CN105046378A - Operation scheduling method based on seismic data - Google Patents

Abstract

The invention discloses an operation scheduling method based on seismic data, comprising the following steps: S1, acquiring resource information of cluster nodes; S2, calculating the complexity of seismic operation; and S3, scheduling seismic operation through an operation scheduling strategy based on a chaos algorithm. The method of the invention is based on a seismic operation scheduling strategy under the conditions of optimal total processing time and load balancing. Seismic operation scheduling is finally implemented through the seismic operation scheduling strategy based on the chaos algorithm according to the acquired resource information of the cluster nodes and the complexity of seismic operation.

Description

A kind of job scheduling method based on geological data

Technical field

The invention belongs to Clustering field, be specifically related to a kind of design of the job scheduling method based on geological data.

Background technology

Along with the development of technology, although the performance of single computing machine is more and more higher, when the process for large-scale datas such as geological datas, the ability only by constantly strengthening single computing machine can not meet the growing demand of application.General large-scale geological data can up to hundreds of GB, and very complicated to its calculating carried out, and the working time of one earthquake data processing is the longest reaches tens days, therefore adopts Clustering for we providing a solution.

Cluster (cluster) technology is a kind of newer technology, pass through Clustering, can obtain the relatively high income in performance, reliability, dirigibility when paying lower cost, wherein task scheduling is then the core technology in group system.

In cluster resource management software, job scheduling method have impact on the efficiency of whole system to a great extent, good job scheduling method can reduce the interstitial content needed for running job within the regular hour, can run more operation, improve resource utilization ratio.For different applied environments and target, dispatching method also can be very different, and the principal element affecting task scheduling has homework type and structure, computing power and system mechanism, network service performance etc.So can not there is a kind of job scheduling system be suitable under any application of any environment, current many main flow job management systems both provide dispatching algorithm interface flexibly, to reach better dispatching effect.We need according to the software and hardware demand in actual items, general-purpose scheduler algorithm basis is designed the dispatching method of applicable native system, can realize real practicality and efficient scheduling scheme.

The evaluation of dispatching algorithm quality depends on multiple standards, the simplicity of dispatching system, dirigibility, stability should be considered from the angle used, the dispatching efficiency, resource utilization ratio, system throughput, system load balancing etc. of dispatching system should be considered from the angle of performance.

The dispatching efficiency of system and dispatching algorithm are designed with and contact closely, will shorten the stand-by period of operation as much as possible.The operation stand-by period refers to that in cluster, each operation is from being submitted to the waited for time span that brings into operation, and reduces the system resource that dispatching algorithm itself is spent, improves the response speed of system.

System throughput refers to the operation quantity that in certain hour section, system completes, and system throughput not only depends on the software and hardware resources in system, also has very large relation with dispatching method.In general the raising of resource utilization ratio can make system within the unit interval, run more operation, improves the throughput of system.

The target that load balancing will reach makes the job task amount of each node assumes in cluster identical as much as possible.The load balancing of consideration system is exactly be assigned on the lower node of load according to the resource utilization of cluster interior joint by operation, some hereinafter involved dispatching algorithms are mainly dispatched operation successively according to the submission time of operation and job size, so need the consideration of load balancing to join in the design of dispatching algorithm, the dispatching method that just can obtain.

Due to the characteristic of Linux system open source software, most system information is all that the form by reading configuration file obtains.Therefore in order to obtain the information such as cpu busy percentage, network I/O utilization power of node, need to read proc file system.

Proc file system is a pseudo file system, and it only exists in the middle of internal memory, and does not take external space.It in the mode of file system for the operation of access system kernel data provides interface.User and application program can obtain the information of system by proc, and can change some parameter of kernel.Due to the information of system, as process, be dynamically change, so during file in user or application program readings/proc catalogue, proc file system dynamically reads information needed and submission from system kernel.

Chaos phenomenon is the performance of a kind of inherent stochastic process in non-linear deterministic system, is prevalent in occurring in nature, and it takes the form of one " unordered is orderly ".Determinacy is its " order ", and the unpredictability of net result is then its " randomness ".But strictly speaking, the character for chaos judges, the definition that mathematically neither one is unified.The relatively conventional three kinds of chaos had under Li-Yorke, Devaney, Matotto meaning, in fact often use Lyapunov exponential sum entropy to portray chaotic dynamics character.

The behavior of a chaos system is the set of many ordering behaviors, but each ordering behavior is not occupied an leading position under normal circumstances.If upset a chaos system in some way, this system just can be made to work with in its many ordering behavior.Because chaos system can be changed between many different behaviors, so seem flexible especially.The research of chaos starts from mathematics and physics in history, then expands to engineering field, and chaology is applied to engineering design field by recent people gradually.

According to the difference of chaos applications method, stability, comprehensive and analyze several aspect can be divided into.Stability is exactly the susceptibility utilizing initial value, adds small sample perturbations to system, makes it the state entering a certain hope, as chaos controlling; Comprehensive is exactly utilize the artificial chaos generated to obtain the possible function of chaotic dynamics, as avoided Local Minimum; Analysis analyzes observable chaotic signal from natural and artificial complication system to hide rule wherein to find, as seasonal effect in time series nonlinear deterministic is predicted.

Summary of the invention

The object of the invention is only to achieve simple FCFS dispatching algorithm to solve executive control system in prior art, not considering the problem of Seismic Operation total processing time optimum, load balancing to propose a kind of job scheduling method based on geological data.

Technical scheme of the present invention is: a kind of job scheduling method based on geological data, comprises the following steps:

The resource information of S1, acquisition clustered node;

The complexity of S2, calculating Seismic Operation;

S3, by the job scheduling strategy based on chaos algorithm, Seismic Operation to be dispatched.

Further, step S1 comprises step by step following:

S11, computing cluster node;

S12, reading configuration file;

S13, carry out data processing.

Further, the configuration file in step S12 comprises/proc/stat system file ,/proc/net/dev system file and/etc/mtab system file.

Further, in step S13, data processing specifically comprises: the performance index of computing node cpu busy percentage, computing node I/O utilization factor, computing node disk utilization and computing cluster node.

Further, the type of the processing module that the complexity of Seismic Operation adopts based on the size of Seismic Operation and Seismic Operation in step S2.

Further, step S3 comprises step by step following:

S31, algorithm initialization;

S32, first time carrier wave is carried out to Chaos Variable;

S33, with first time carrier wave after Chaos Variable carry out iterative search;

S34, judge that whether Chaos Variable remains unchanged after the iterative search of step S33, if then enter step S35, otherwise returns step S33;

S35, second time carrier wave is carried out to Chaos Variable;

S36, with second time carrier wave after Chaos Variable proceed iterative search;

S37, judge that whether Chaos Variable remains unchanged after the iterative search of step S36, if then enter step S38, otherwise returns step S36;

S38, output optimum solution.

The invention has the beneficial effects as follows: the Seismic Operation scheduling strategy that the present invention is based on total condition such as processing time optimum, load balancing, by the resource information of clustered node that gets and the complexity of Seismic Operation, utilize the Seismic Operation scheduling strategy of chaos algorithm, following beneficial effect can be reached:

(1) resource utilization of the understanding clustered node that the resource information obtaining clustered node in the present invention can help user detailed;

(2) Seismic Operation amount of complexity is turned to the complexity of the job run based on the size of Seismic Operation and the type of seismic module by the present invention;

(3) by the Seismic Operation scheduling model that the present invention sets up, make use of the optimisation strategy of chaos algorithm, consider the load balance of node and job run time minimum optimization aim, finally complete the scheduling to Seismic Operation.

Accompanying drawing explanation

Fig. 1 is a kind of job scheduling method process flow diagram based on geological data provided by the invention.

Fig. 2 is the process flow diagram step by step of step S1 of the present invention.

Fig. 3 is the process flow diagram step by step of step S3 of the present invention.

Embodiment

Below in conjunction with accompanying drawing, embodiments of the invention are further described.

The invention provides a kind of job scheduling method based on geological data, as shown in Figure 1, comprise the following steps:

The resource information of S1, acquisition clustered node.

As shown in Figure 2, this step comprises step by step following:

S11, computing cluster node.

S12, reading configuration file, comprise/proc/stat system file ,/proc/net/dev system file and/etc/mtab system file.

Wherein ,/proc/stat system file has contained the many statistical informations about kernel and system since system starts, and comprising CPU ruuning situation, interrupts the information such as statistics, start-up time, contextual processing number of times, operating process.

/ proc/net/dev system file contains system network interface card statistical information.

What/etc/mtab system file was recorded is the file system that present system has been loaded, and comprises the virtual file etc. that operating system is set up.

S13, carry out data processing, specifically comprise: the performance index of computing node cpu busy percentage, computing node I/O utilization factor, computing node disk utilization and computing cluster node.

In order to computing node cpu busy percentage, need to extract four data from/proc/stat system file: the user model (nice) of user model (user), low priority, kernel mode (system) and the processor time of free time (idle).They are all positioned at/the first row of proc/stat system file.Then node cpu utilization factor is as shown in formula (1):

cpu_usage＝100*(user+nice+system)/(user+nice+system+idle)(1)

In order to obtain the related data of node I/O utilization factor, needing to obtain two data from/proc/net/dev system file: the number-of-packet exported from the machine, flows into the number-of-packet of the machine, they are all positioned at/fourth line of proc/net/dev system file.Performance collection program starts the initial value recording these two data, all deducts this initial value later and be the packet passed through from this node from cluster starts after each this value of acquisition.Utilize above-mentioned data can calculate node I/O utilization factor, as shown in formula (2):

network_load＝(output_packet+input_packet)/2(2)

Wherein network_load is node I/O utilization factor, and output_packet is the packet exported the unit time, and input_packet is the packet flowed into the unit time.

Utilizing information to obtain linux system disk space, needing to obtain data from/etc/mtab system file: the subregion name of current carry, it is positioned at/secondary series of often going of etc/mtab system file.By the zone name obtained, then by system function intfstatfs (intfd, structstatfs*buf) function, just can obtain the disk space information of this subregion.Therefore the disk utilization of node can be expressed as:

used＝(fs_uesed/fs_size)(3)

Wherein fs_uesed represents the disk space used, and fs_size represents the disk space that node is total.

Then the performance index of clustered node can be expressed as:

$perfor\max ce=\frac{1}{used+network\_load+cpu\_usage}---\left(4\right)$

The complexity of S2, calculating Seismic Operation;

The type of the processing module that the complexity of Seismic Operation adopts based on size and the Seismic Operation of Seismic Operation.Therefore, during each schedule job, all should resolve the size of data of operation and total evaluation is carried out to the module type wherein used, obtaining the complexity of module, being calculated the complexity of Seismic Operation by the complexity of module and the size of operation:

task_complex＝u*task_size+(1-u)*module_complex(5)

Wherein task_size is the size of Seismic Operation, u is a weighted value and 0 < u < 1, in the embodiment of the present invention, u=0.5, module_complex are the complexity of module, calculate by formula (6):

$\mathrm{mod}ule\_complex=\underset{i=1}{\overset{n}{\&Sigma;}}\frac{1}{n}*\mathrm{mod}ule\_{\mathrm{complex}}_i---\left(6\right)$

Wherein n is total number of modules of Seismic Operation, module_complex _iit is the complexity of i-th module.

S3, by the job scheduling strategy based on chaos algorithm, Seismic Operation to be dispatched.

In this step, first select Chaos Variable, select the Logistic shown in formula (7) to map:

x _n+1＝μx _n(1-x _n)(7)

Wherein μ is controling parameter, in the embodiment of the present invention, gets μ=4.If 0≤x _i≤ 1i=0,1,2 ..., n, can prove: when μ=4, system is in chaos state completely.Utilize chaos to the feature of initial value sensitivity, the initial value being assigned to (7) formula i fine difference can obtain i Chaos Variable.Then chaos state variable is incorporated in optimized variable, and the span of the traversal scope " amplification " of chaotic motion to optimized variable, recycling Chaos Variable is searched for.If total total clustered node M, Seismic Operation has N number of, therefore arranges f (x _i) formula be:

$f\left(x_i\right)=\underset{i,j}{\&Sigma;}\frac{\mathrm{mod}ule\_{\mathrm{complex}}_i}{{\mathrm{performance}}_j}---\left(8\right)$

Wherein i and j represents respectively to be assigned to by Seismic Operation i on clustered node j and runs, with the formula (8) total processing time can be expressed as.

As shown in Figure 3, this step comprises step by step following:

S31, algorithm initialization.

Put parameter k=1, k'=1, obtain the complexity of each Seismic Operation, generate one dimension character string, be expressed as and at random a resource distributed in subtask, composition i group sequence x _i, as (2143) represent, Seismic Operation 1 is distributed to node 2, Seismic Operation 2 is distributed to node 1, Seismic Operation 3 is distributed to node 4, Seismic Operation 4 is distributed to node 3.They are mapped in [0,1] coordinate system, make 0≤x _i≤ 1.By formula (7), then can obtain the different Chaos Variable x of i track _{i, n+1}.

S32, first time carrier wave is carried out to Chaos Variable.

By formula (7) by i selected Chaos Variable x _{i, n+1}be incorporated into respectively in i optimized variable in formula (9), make it change into Chaos Variable x' _{i, n+1}, and the variation range of Chaos Variable is amplified to respectively the span of corresponding optimized variable.

x' _i,n+1＝c _i+d _i*x _i,n+1(9)

Wherein c _iand d _ifor constant, be equivalent to enlargement factor.C is established in the embodiment of the present invention _i=x _min, d _i=x _max; And x _maxand x _minmaximal value M and the minimum value 0 of node number respectively.

S33, with first time carrier wave after Chaos Variable carry out iterative search.

Make x _i(k)=x' _{i, n+1}, calculate corresponding performance index f _i(k).

First the initialization operation of iterative search duration is carried out, order if f _i(k)≤f ^*, represent that the performance index of current iteration gained are more excellent, so order carry out next iteration; Otherwise the performance index representing current iteration gained are not more excellent values, abandon x _i(k), and make k=k+1, then carry out next iteration.

S34, judge whether Chaos Variable remains unchanged after the iterative search of step S33, i.e. f after the some steps search in step S33 ^*whether remain unchanged, if then enter step S35, otherwise return step S33.

S35, by formula (10), second time carrier wave is carried out to Chaos Variable:

$x_{i,n+1}^{*}=x_i^{*}+a_i{x}_{i,n+1}---\left(10\right)$

Wherein a _ix _{i, n+1}for traveling through interval very little Chaos Variable, a _ifor regulating constant, its value can be less than 1, for current optimum solution.

S36, with second time carrier wave after Chaos Variable proceed iterative search.

First carry out the initialization operation of iterative search duration, make x _i(k')=x' _{i, n+1}, calculate corresponding performance index f _i(k'), if f _i(k')≤f ^*, represent that the performance index of current iteration gained are more excellent, then make carry out next iteration; Otherwise the performance index representing current iteration gained are not more excellent values, abandon x _i(k ^'), and make k ^'=k ^'+ 1, then carry out next iteration.

S37, judge whether Chaos Variable remains unchanged after the iterative search of step S36, i.e. f after the some steps search in step S36 ^*whether remain unchanged, if then enter step S38, otherwise return step S36.

S38, output optimum solution.

Here the total processing time f (x tried to achieve in optimum solution and formula (8) _i).

Although chaotic motion has ergodicity in certain scope, some state may need the long period just can reach, then search time is longer, and therefore this algorithm introduces second carrier wave.After one section of search, an approximate optimal solution is found out, then second carrier wave on the basis of approximate optimal solution with first time carrier wave.The scope of second carrier wave is very little, is equivalent to carry out finecomb in the field of approximate optimal solution, can find globally optimal solution very soon like this, greatly improve search efficiency.Use chaos algorithm can carry out optimizing operation expeditiously.In optimizing operating process, chaos operator travels through whole solution interval, only utilizes objective function to judge.That is, chaos algorithm is restricted less than traditional optimizing algorithm institute, the most important thing is that it does not need an obvious objective function expression formula.

Those of ordinary skill in the art will appreciate that, embodiment described here is to help reader understanding's principle of the present invention, should be understood to that protection scope of the present invention is not limited to so special statement and embodiment.Those of ordinary skill in the art can make various other various concrete distortion and combination of not departing from essence of the present invention according to these technology enlightenment disclosed by the invention, and these distortion and combination are still in protection scope of the present invention.

Claims (6)

1. based on a job scheduling method for geological data, it is characterized in that, comprise the following steps:

The resource information of S1, acquisition clustered node;

The complexity of S2, calculating Seismic Operation;

S3, by the job scheduling strategy based on chaos algorithm, Seismic Operation to be dispatched.

2. job scheduling method according to claim 1, is characterized in that, described step S1 comprises step by step following:

S11, computing cluster node;

S12, reading configuration file;

S13, carry out data processing.

3. job scheduling method according to claim 2, is characterized in that, the configuration file in described step S12 comprises/proc/stat system file ,/proc/net/dev system file and/etc/mtab system file.

4. job scheduling method according to claim 2, it is characterized in that, in described step S13, data processing specifically comprises: the performance index of computing node cpu busy percentage, computing node I/O utilization factor, computing node disk utilization and computing cluster node.

5. job scheduling method according to claim 1, is characterized in that, the type of the processing module that the complexity of Seismic Operation adopts based on the size of Seismic Operation and Seismic Operation in described step S2.

6. job scheduling method according to claim 1, is characterized in that, described step S3 comprises step by step following:

S31, algorithm initialization;

S32, first time carrier wave is carried out to Chaos Variable;

S33, with first time carrier wave after Chaos Variable carry out iterative search;

S34, judge that whether Chaos Variable remains unchanged after the iterative search of step S33, if then enter step S35, otherwise returns step S33;

S35, second time carrier wave is carried out to Chaos Variable;

S36, with second time carrier wave after Chaos Variable proceed iterative search;

S37, judge that whether Chaos Variable remains unchanged after the iterative search of step S36, if then enter step S38, otherwise returns step S36;

S38, output optimum solution.