CN107766927A - Universal parallel method of the intelligent optimization algorithm based on individual population on Spark - Google Patents
Universal parallel method of the intelligent optimization algorithm based on individual population on Spark Download PDFInfo
- Publication number
- CN107766927A CN107766927A CN201711072393.7A CN201711072393A CN107766927A CN 107766927 A CN107766927 A CN 107766927A CN 201711072393 A CN201711072393 A CN 201711072393A CN 107766927 A CN107766927 A CN 107766927A
- Authority
- CN
- China
- Prior art keywords
- population
- individual
- class
- fitness
- individuals
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005457 optimization Methods 0.000 title claims abstract description 31
- 238000000034 method Methods 0.000 title claims abstract description 16
- 238000004806 packaging method and process Methods 0.000 claims description 13
- 238000011156 evaluation Methods 0.000 claims description 10
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000005538 encapsulation Methods 0.000 abstract description 2
- 108090000623 proteins and genes Proteins 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 230000035772 mutation Effects 0.000 description 5
- 238000005192 partition Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000002068 genetic effect Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000004791 biological behavior Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses universal parallel method of the intelligent optimization algorithm based on individual population on Spark, includes the individual abstract base class of encapsulation, the abstract base class and intermediate data format class of population；Setting individual realize class and population realize class；Initialize population, individual and global optimum's individual；Initialization context context, generate broadcasting amount and the broadcast of constant information；Parallelization generates Species structure formula data set；Fitness optimum individual is returned after each population is by evolving alone, population's fitness is assessed；Find out contemporary optimum individual and renewal global optimum individual；Data distribution formula data set is exchanged among parallelization generation；Carry out the steps such as the individual exchange between population.The beneficial effects of the present invention are, for the intelligent optimization algorithm based on individual population, a kind of universal parallel method is proposed on Spark platforms, script is operated in the algorithm on unit and is distributed on different computers while runs, to reduce run time, raising efficiency.
Description
Technical Field
The invention relates to the technical field of distributed computing, in particular to a general parallelization method of an individual-population-based intelligent optimization algorithm on Spark.
Background
There is a class of intelligent optimization algorithms that can give a better quality of feasible solution to the problem to be solved at an acceptable cost (including time and space) by mimicking the biological behavior of nature or by mimicking physical phenomena. The intelligent optimization algorithm is also called meta-heuristic algorithm, and more commonly used are genetic algorithm, particle swarm algorithm, ant colony algorithm, artificial bee colony algorithm and the like. Compared with the traditional optimization algorithm, the algorithm is not perfect theoretically, the optimal solution of the problem cannot be found, but the algorithm is wide in adaptability, the solution searching cost can be controlled within an acceptable range, and the method is widely applied to the actual engineering field. The intelligent optimization algorithm has the obvious and common characteristics and has individual and population concepts. The population is a set of individuals, the individuals represent a solution of the problem, and the whole population is promoted to develop and evolve in a better direction through some behaviors of the individuals or mutual cooperation among the individuals, so that a better solution of the problem is found.
Along with the increase of the complexity and the scale of the problem, the operation time for solving the problem by using the intelligent optimization algorithm in a single machine environment is increased. In order to improve the execution efficiency of the algorithm, reduce the running time and meet the actual application requirements, the algorithm can be modified to improve the performance, but the modification modes of different algorithms are different, and even if the same algorithm is used for different problems, the algorithm has various varieties and is difficult to realize. The other method is to carry out parallelization processing on the algorithm, so that the algorithm originally running on a single computer is distributed to different computers to run simultaneously, the running time is reduced, and the efficiency is improved.
With the rapid development of cloud computing and big data, various distributed computing frameworks are proposed, and a Spark platform is one of the distributed computing frameworks, so that a bottom foundation is laid for parallelization.
Disclosure of Invention
The invention aims to provide a general parallelization method of an individual-population-based intelligent optimization algorithm on Spark.
The technical scheme for realizing the purpose of the invention is as follows:
a general parallelization method of an intelligent optimization algorithm based on an individual-population on Spark comprises the following steps:
(1) packaging an abstract base class BaseIndigual of an individual, packaging an abstract base class BasePopulation of a population, and packaging an intermediate data format class exchange data used for individual exchange among the populations;
(2) setting an implementation class SomeInitial of an individual of the intelligent optimization algorithm, and setting an implementation class SomePopulation of a population of the intelligent optimization algorithm; the SomeIndIvidual is inherited from BaseIndIvidual, and SomePopulation is inherited from BasePopulation;
(3) initializing M populations, wherein M is more than or equal to 2; initializing N individuals in each population, wherein N is more than or equal to 100; initializing global optimal individual gbest, initializing maximum iteration times MAX _ T, and setting the current iteration times T as 1;
(4) initializing a sparkgontext, calling a sparkgontext.
(5) A SparkContext. parallelize () interface is called to parallelize the M population objects and generate a corresponding population distributed data set PopulationRDD;
(6) calling a PoputationRDD.map () interface to execute parallel computation; in the PopulationRDD.map () interface, each population returns the first K individuals with the optimal fitness in the population after independent evolution and population fitness evaluation in sequence, wherein K is more than or equal to 2;
(7) calling a PopulationRDD.collectint () interface, collecting K individuals returned by each population, and finding out the current generation optimal individual cbest; if the fitness of cbest is better than that of gbest, updating the gbest by using cbest;
(8) packaging K individuals returned by each group to generate M intermediate data for individual exchange, performing disordering and shuffling on the M intermediate data, and then calling a spark context.
(9) Invoking PopulationRDD.zipPartitions () interface and transmitting into exchange RDD to perform individual exchange among populations;
(10) and (4) if T is T +1, if T is less than MAX _ T, the step (6) is carried out, otherwise, the loop is exited, and the gbest is output.
The invention has the beneficial effects that aiming at the intelligent optimization algorithm based on individual-population, a general parallelization method is provided on a Spark platform, so that the algorithm originally running on a single computer is distributed to different computers to run simultaneously, the running time is reduced, and the efficiency is improved.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a diagram of the BaseIndIvidual class in an embodiment of the invention;
FIG. 3 is a BasePoction class diagram in an embodiment of the invention;
FIG. 4 is an exchange data class diagram in an embodiment of the invention;
FIG. 5 is a GAIndIviral class diagram in an embodiment of the invention;
FIG. 6 is a GAPopulation class diagram in an embodiment of the present invention.
Detailed Description
The invention will be further explained with reference to the drawings.
As shown in fig. 1, the general parallelization method of the individual-population based intelligent optimization algorithm on Spark includes the steps of:
(1) packaging an abstract base class BaseIndigual of an individual, packaging an abstract base class BasePopulation of a population, and packaging an intermediate data format class exchange data used for individual exchange among the populations;
(2) setting an implementation class SomeInitial of an individual of the intelligent optimization algorithm, and setting an implementation class SomePopulation of a population of the intelligent optimization algorithm; the SomeIndIvidual is inherited from BaseIndIvidual, and SomePopulation is inherited from BasePopulation;
(3) initializing M populations, wherein M is more than or equal to 2; initializing N individuals in each population, wherein N is more than or equal to 100; initializing global optimal individual gbest, initializing maximum iteration times MAX _ T, and setting the current iteration times T as 1;
(4) initializing a sparkgontext, calling a sparkgontext.
(5) A SparkContext. parallelize () interface is called to parallelize the M population objects and generate a corresponding population distributed data set PopulationRDD;
(6) calling a PoputationRDD.map () interface to execute parallel computation; in the PopulationRDD.map () interface, each population returns the first K individuals with the optimal fitness in the population after independent evolution and population fitness evaluation in sequence, wherein K is more than or equal to 2;
(7) calling a PopulationRDD.collectint () interface, collecting K individuals returned by each population, and finding out the current generation optimal individual cbest; if the fitness of cbest is better than that of gbest, updating the gbest by using cbest;
(8) packaging K individuals returned by each group to generate M intermediate data for individual exchange, performing disordering and shuffling on the M intermediate data, and then calling a spark context.
(9) Invoking PopulationRDD.zipPartitions () interface and transmitting into exchange RDD to perform individual exchange among populations;
(10) and (4) if T is T +1, if T is less than MAX _ T, the step (6) is carried out, otherwise, the loop is exited, and the gbest is output.
In the above-mentioned steps, the step of,
in the step (1), the class baselndigial has 4 attributes, which are respectively: the FITNESS maximum MAX _ FITNESS is 999999.0, the FITNESS minimum MIN _ FITNESS is-999999.0, the current FITNESS value of the individual is FITNESS, and the question type is isminproplem. If isminpublem is true, it means that the minimization problem is solved, otherwise, the maximization problem is solved. When the individual initializes, if isminpublem is true, then FITNESS is MAX _ FITNESS, otherwise, FITNESS is MIN _ FITNESS.
In step (1), the class basepapulation has 3 attributes, which are respectively: number of individuals individual _ num, individual subject containers individuals, question type flag isMinProblem. The class basepapulation has 1 abstract interface, exchange indexes of (exchange data), the incoming data of the interface is in a middle data format, the main function of the interface is to realize individual exchange among groups, and the interface is rewritten and realized by a group realization class of a derived specific optimization algorithm.
In step (1), the class exchange data has 2 attributes, which are respectively: using a flag used _ flag, the individual object container data. The data stores a series of individual objects that are passed to a population and used in an exchange. If the used _ flag is true, it indicates that the intermediate data has been used by a certain group, and the intermediate data is skipped and not used, otherwise, the intermediate data can be used.
In the step (2), the individual realization class somee _ lnndividual of a specific optimization algorithm is inherited from baselnndividual, and somee _ lnndividual needs to design a reasonable coding mode according to a specific problem to be solved and provide a fitness evaluation interface according to the relevant information and the coding mode of the problem. The SomeInitial also needs to provide a proper initialization interface according to the problem information and the coding mode.
In step (2), the population implementation class someeposition of a specific optimization algorithm is inherited from basepropagation, and someeposition needs to provide an initialization interface according to specific problem information to be solved, and design specific member attributes and operator interfaces according to specific characteristics of the optimization algorithm and core operators thereof. SomePopulation provides a fitness evaluation interface of the population according to the specific question information, and calls the fitness evaluation interface of SomeIndaividual in the interface to calculate the fitness value of each individual in the population. In someposition, the exchange indexes (exchange data) interface of baseexpanding must be rewritten to realize a specific inter-population individual exchange mode.
In the step (3), the initialized population refers to a population implementation class SomePopulation of a specific optimization algorithm. The initialized individuals in the population refer to the individual implementation class SomeIndIvidual of a specific optimization algorithm. The global optimal individual gbest is also an object of the SomeInitial class.
In the step (4), the SparkContext is an environment context provided by the spare platform, and is an essential component for running a program on the spare platform, and specifically to a code level, the SparkContext can be understood as a class provided by the spare platform.
In the step (5), the initialized M population objects are stored in an array, and when a SparkContext.
In the step (5), the full name of the RDD is an elastic distributed data set, which is a distributed data set provided by the Spark platform. The RDD can perform distributed encapsulation on the localized data and perform some parallelization operations on the RDD. Specifically to the code level, RDD is also a class provided by Spark. Invoking the sparkgontext.parallelize () interface can create an RDD, named postlationrdd, that is generated by invoking the sparkgontext.parallelize () interface for M population objects.
In the step (6), each population independently evolves, which means that a population implementation SomePopulation-like correlation operator interface of a specific optimization algorithm is called, and the whole population and each individual in the population are updated. And (3) carrying out fitness evaluation on each population, namely calling a fitness evaluation interface of SomePopulation to calculate the fitness value of each individual in the population. Returning K individuals with better fitness can be adjusted according to specific algorithms and problems, for example, returning the first 20% of the individuals sorted according to the fitness value or the first 30% of the individuals sorted according to the fitness value.
In the step (7), each population returns K individuals, and M populations return M × K individuals in total. All returned individuals are stored in a two-dimensional array, the length of the two-dimensional array is M, each element is a one-dimensional array and represents all returned individuals of a population, and the length of the one-dimensional array is K, each element represents an individual.
In step (8), the intermediate data refers to class exchangeable data. Each intermediate data encapsulates K individuals returned by one population, and M populations generate M intermediate data. M pieces of intermediate data are stored in an array, each element represents one piece of intermediate data, after the sequence of the elements in the array is disordered and shuffled, when a SparkContext.
In step (9), individual exchange among the populations, that is, an exchange index (exchange data) interface after the rewriting of the population realization class SomePopulation called a specific optimization algorithm is called, wherein the exchange data contains K individuals returned by another population, and the K individuals returned by another population interact with the individuals owned by the current population according to a certain strategy or are screened and reserved, so that the individual exchange among the populations is realized.
The following is a specific example.
The specific implementation method of the general parallelization is described by taking the genetic algorithm (GA for short) to solve the multicast routing optimization problem as an example.
Multicast routing problem model description: simulating the communication subnet into a undirected weighted connectivity graph G ═ (V, E), V ═ V1，v2，...，vNDenotes a node set, E ═ E1，e2，...，eMAnd N and M respectively represent the number of nodes and edges. s e V is the multicast source node,is a multicast destination node set, and K represents the number of destination nodes. For each edge E ∈ E, cost (E) is defined as the cost of the edge E, i.e., the weight on the edge E. Taking a source node s as a root node, and containing a tree T ═ V of all destination node sets DT，ETT is the multicast tree, wherein VTIs the set of all nodes in T, ETIs the set of all the edges in T,the multicast tree T is a sub-graph of G. The problem of multicast routing is to find a multicast tree T ═ V in GT，ET) Make the overhead of the multicast tree TAnd minimum. The multicast routing problem is the minimum overhead multicast tree from the source node to the destination node set, i.e. the smaller the overhead, the better the solution of the problem, and the minimum problem.
The scenario assumption of the multicast routing problem is as follows: a connectivity graph communication sub-network having 200 nodes, 446 edges, where the overhead value on each edge takes a random value within (0, 50). The multicast source node is node number 5 and the set of multicast destination nodes is [13,25,36,46,55,70,90,108,127,167,192,101,3,66 ].
First, an abstract base class BaseIndigual of an individual, an abstract base class BasePopulation of a population, and an intermediate data format class exchange data are explained.
The baselndivual class is defined as follows in figure 2.
Wherein,
MAX _ fitess: fitness maximum, initialized to 999999.0.
MIN _ fitess: fitness minimum, initialized to-999999.0.
isminproplem: and taking true to represent the minimization problem, and otherwise, maximizing the problem.
fitness: the current FITNESS value of an individual is initialized to MAX _ FITNESS if the minimization problem is not present, or to MIN _ FITNESS otherwise.
The definition of BasePopulation class is shown in FIG. 3.
Wherein,
indivisual _ num: the number of individuals in the population.
individuals: containers of individual subjects, storing all individuals in the population.
isminproplem: the types of the problems to be solved have the same values.
exchangeinduviduals (exchangedata): and the abstract interface is used for transmitting parameters to an exchange data class object, providing an individual exchange function between the populations, and rewriting and realizing the abstract interface by a derived population realization class of a concrete optimization algorithm.
The ExchangeData class is defined as in fig. 4.
Wherein,
used _ flag: the flag is used, true, indicating that the intermediate data has been used by a certain population.
data: an individual object container storing a series of individual objects to be transferred to a certain population and used for exchange.
Next, according to a specific multicast routing problem scenario and the algorithm characteristics of the GA, the individual implementation class and the population implementation class of the GA are encapsulated. Let the individual implementation class of GA be defined as GAIndIvidual (corresponding to SomeIndIvidual in the previous flow), and the population implementation class of GA be defined as GAPopulation (corresponding to SomePopulation in the previous flow), and detailed descriptions of GAIndIvidual and GAPopulation are given below.
The GAIndividual class is defined as in FIG. 5.
GAIndidual is inherited from BaseIndidual, possessing all public member attributes of BaseIndidual, wherein,
gene: and expressing the gene codes of individuals, and adopting a binary coding mode of coding according to edges according to the assumption of the multicast routing problem scene. Specifically, if there is 446 side in the problem scene, gene is an array with a length of 446, where each element corresponds to one side, the value of each element is true or false, where true indicates that the side is selected and usable, and false indicates that the side is not selected and unusable.
init (): and the individual initialization interface initializes each member attribute of the individual object according to the input information of the multicast routing problem.
calcfiltness (): and the individual fitness calculation interface calculates the current fitness value of the individual object according to the input information of the multicast routing problem and the gene coding gene of the individual. Specifically, the original input topology of the problem scenario is G ═ (V, E), where V is a node set of 200 nodes, E is an edge set of 446 edges, and the individual gene is ═ l1,l2,...,l446]Where each bit takes either true or false, if lkIf "true" k ∈ {1, 2.., 446}, then l will be equal tokAdding the corresponding edge to the new edge set EnNewly generated edge setOn top of that, a new topology G is generatedn＝(V,En) In the new topology GnThe multicast source node is 5, and the multicast destination node set is [13,25,36,46,55,70,90,108,127,167,192,101,3,66]Is 5 ofSource node, EnThe cost value of each edge is the weight value of the edge, Dijkstra algorithm is operated, if the source node 5 is communicated with any destination node, the Dijkstra algorithm is operated successfully, the obtained result contains 14 shortest paths from the source node 5 to all the destination nodes, and all the edges contained in the 14 shortest paths are removed to form a new edge set EmCalculating the fitness value of the individual according to a formula:where cost (e) represents the cost value on an edge e. If the source node 5 is not communicated with a certain destination node, the Dijkstra algorithm fails to operate, the individual is an illegal individual, and the FITNESS of the individual is made to be MAX _ FITNESS.
The GAPopulation class is defined as in FIG. 6.
GAPopulation inherits from basepapulation, possesses all public member attributes and interfaces of basepapulation, wherein,
exchangeinduviduals (exchangedata): the interface of the abstract parent class BasePopulation is rewritten, and the function of individual exchange among populations is realized concretely. Specifically, it is assumed that the exchange data includes 20% of individuals transmitted from another group, and since the multicast routing problem is a minimization problem, the smaller the fitness value of the individuals is, the better the fitness value of the individuals is, all the individuals of the current group are sorted from small to large according to the fitness value, the last 20% of the individuals with the largest fitness value are discarded, and then all the individuals of the exchange data are merged into the current group to complete the individual exchange.
populioninit (): and the population initialization interface is used for initializing each member attribute of the population and initializing all individuals in the population.
fitnessEvaluate (): and the population fitness evaluation interface sequentially calls the individual fitness calculation interfaces to calculate the fitness value of each individual in the population.
croscover (): the method comprises the steps of traversing a population, randomly extracting two different individuals from the population each time, and interchanging gene codes of the two individuals according to a certain crossover probability and a certain crossover mode to generate a new individual. Available crossing modes include single-point crossing, multi-point crossing and uniform crossing.
variation (): the specific method comprises traversing the population and carrying out mutation operation on each gene position in each individual according to a certain mutation probability. For example, a true is taken for a bit in an individual gene and a false is taken for the bit after the mutation operation.
select (): selecting an operator interface and one of the core operators of the GA, and specifically, selecting individuals with better fitness in a population by adopting a championship competition mode and transmitting the individuals to the next generation.
The general parallelization method for solving the multicast routing problem scene by the GA comprises the following steps:
(1) and inputting relevant information of the multicast routing problem scene, wherein the relevant information comprises node, edge, source node and destination node set information.
(2) 6 GAPopulation population objects are initialized, and 100 GAIndaidual individual objects are initialized in each population. Initializing a global optimal individual gbest, wherein the maximum iteration time MAX _ T is 200, and the current iteration time T is 1.
(3) Initializing the SparkContext, setting information related to the multicast routing problem, such as node and edge information, source node and destination node set information, calling SparkContext.
(4) Sparkcontext. parallelize () is called to parallelize the 6 GAPopulation population objects, generating the corresponding populiationrdd. When parallelizing, the number of parallelized partitions is set to 6, namely one partition corresponds to one population.
(5) Map () interface is called to execute parallel computation, in the interface, each population independently evolves, fitness evaluates and returns part of individuals with better fitness, specifically, the sub-steps executed in the interface are as follows:
1. cross over operation of gapdisclosure
2. Mutation () mutation operation
3. Population fitness evaluation of gapopopulation
4. Recording the first 20 individuals with the best fitness
5. Select () operation, GAPopulation
6. Returning to the individuals recorded in substep 4
(6) Collect 20 individuals returned by each population (120 individuals are returned by 6 populations), find out the best individual cbest of the current generation, if the fitness value of cbest is smaller than the gbest, the cbest is better, and update the gbest with cbest.
(7) Packaging 120 individuals returned by 6 populations to generate 6 exchange data class objects (namely intermediate data), wherein each exchange data comprises 20 individuals, then disordering the sequence of the 6 exchange data, and then calling sparklcontext. And (4) during parallelization, similarly setting the number of parallelized partitions to be 6, and contrasting the number in the step (4), namely one partition corresponds to one population and corresponds to one intermediate data.
(8) Zippartitions () interface is called and into ExchangeRDD, where individual exchanges between populations are implemented, specifically the sub-steps performed in this interface are as follows:
1. exchange of individuals (exchange data) was performed
(9) And (5) if T is T +1, if T is less than MAX _ T, turning to the step (5), otherwise, exiting the loop, outputting the gbest, and solving the multicast tree under the gbest.
Claims (1)
1. The general parallelization method of the individual-population-based intelligent optimization algorithm on Spark is characterized by comprising the following steps of:
(1) packaging an abstract base class BaseIndigual of an individual, packaging an abstract base class BasePopulation of a population, and packaging an intermediate data format class exchange data used for individual exchange among the populations;
(2) setting an implementation class SomeInitial of an individual of the intelligent optimization algorithm, and setting an implementation class SomePopulation of a population of the intelligent optimization algorithm; the SomeIndIvidual is inherited from BaseIndIvidual, and SomePopulation is inherited from BasePopulation;
(3) initializing M populations, wherein M is more than or equal to 2; initializing N individuals in each population, wherein N is more than or equal to 100; initializing global optimal individual gbest, initializing maximum iteration times MAX _ T, and setting the current iteration times T as 1;
(4) initializing a sparkgontext, calling a sparkgontext.
(5) A SparkContext. parallelize () interface is called to parallelize the M population objects and generate a corresponding population distributed data set PopulationRDD;
(6) calling a PoputationRDD.map () interface to execute parallel computation; in the PopulationRDD.map () interface, each population returns the first K individuals with the optimal fitness in the population after independent evolution and population fitness evaluation in sequence, wherein K is more than or equal to 2;
(7) calling a PopulationRDD.collectint () interface, collecting K individuals returned by each population, and finding out the current generation optimal individual cbest; if the fitness of cbest is better than that of gbest, updating the gbest by using cbest;
(8) packaging K individuals returned by each group to generate M intermediate data for individual exchange, performing disordering and shuffling on the M intermediate data, and then calling a spark context.
(9) Invoking PopulationRDD.zipPartitions () interface and transmitting into exchange RDD to perform individual exchange among populations;
(10) and (4) if T is T +1, if T is less than MAX _ T, the step (6) is carried out, otherwise, the loop is exited, and the gbest is output.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711072393.7A CN107766927A (en) | 2017-11-03 | 2017-11-03 | Universal parallel method of the intelligent optimization algorithm based on individual population on Spark |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711072393.7A CN107766927A (en) | 2017-11-03 | 2017-11-03 | Universal parallel method of the intelligent optimization algorithm based on individual population on Spark |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107766927A true CN107766927A (en) | 2018-03-06 |
Family
ID=61272173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711072393.7A Pending CN107766927A (en) | 2017-11-03 | 2017-11-03 | Universal parallel method of the intelligent optimization algorithm based on individual population on Spark |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107766927A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109871934A (en) * | 2019-01-16 | 2019-06-11 | 湖北工业大学 | Feature selection approach based on the distributed parallel binary of Spark a flying moth darts into the fire algorithm |
| CN110086855A (en) * | 2019-03-29 | 2019-08-02 | 江苏开拓信息与系统有限公司 | Spark task Intellisense dispatching method based on ant group algorithm |
-
2017
- 2017-11-03 CN CN201711072393.7A patent/CN107766927A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109871934A (en) * | 2019-01-16 | 2019-06-11 | 湖北工业大学 | Feature selection approach based on the distributed parallel binary of Spark a flying moth darts into the fire algorithm |
| CN110086855A (en) * | 2019-03-29 | 2019-08-02 | 江苏开拓信息与系统有限公司 | Spark task Intellisense dispatching method based on ant group algorithm |
| CN110086855B (en) * | 2019-03-29 | 2021-11-05 | 江苏开拓信息与系统有限公司 | Intelligent Spark task perception scheduling method based on ant colony algorithm |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Baykasoglu et al. | Artificial bee colony algorithm and its application to generalized assignment problem | |
| Deb | Introduction to evolutionary multiobjective optimization | |
| CN107911300B (en) | Multicast routing optimization method based on whale algorithm and application of multicast routing optimization method on Spark platform | |
| CN107430704A (en) | Neural network algorithm is realized in nerve synapse substrate based on the metadata associated with neural network algorithm | |
| Bohm et al. | MABE (modular agent based evolver): A framework for digital evolution research | |
| CN101482876B (en) | Weight-based link multi-attribute entity recognition method | |
| CN114386694A (en) | Drug molecule property prediction method, device and equipment based on comparative learning | |
| Mavrovouniotis et al. | A benchmark generator for dynamic permutation-encoded problems | |
| Chen et al. | Imperial competitive algorithm with policy learning for the traveling salesman problem | |
| Park et al. | Genetic programming for order acceptance and scheduling | |
| CN107766927A (en) | Universal parallel method of the intelligent optimization algorithm based on individual population on Spark | |
| Almeida et al. | An experimental analysis of evolutionary heuristics for the biobjective traveling purchaser problem | |
| Bonnet et al. | Jumanji: a diverse suite of scalable reinforcement learning environments in jax | |
| Monteiro et al. | Concave minimum cost network flow problems solved with a colony of ants | |
| Asgari et al. | Dyfen: Agent-based fee setting in payment channel networks | |
| Alba et al. | Evolutionary algorithms | |
| Xiang et al. | Inference in multiply sectioned bayesian networks: Methods and performance comparison | |
| CN107257307A (en) | A kind of parallelization genetic algorithm for solving multiple terminals collaboration network access method based on Spark | |
| CN109977998A (en) | Information processing method and device, storage medium and electronic device | |
| CN114840326A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| Krzywicki et al. | Generation-free agent-based evolutionary computing | |
| Ekmekci | A pheromonal artificial bee colony-pABC-algorithm for optimization problems | |
| Kimovski et al. | Multi-objective optimization framework for vmi distribution in federated cloud repositories | |
| CN109981368A (en) | A kind of overlapping community discovery method based on interaction force between node | |
| Tran et al. | Solving fuzzy job-shop scheduling problems with a multiobjective optimizer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180306 |
|
| WD01 | Invention patent application deemed withdrawn after publication |