KR20200094577A - Realtime Accelerator Controlling System using Artificial Neural Network Simulator and Reinforcement Learning Controller - Google Patents

Abstract

The present invention relates to an accelerator control system for accelerator control devices based on an artificial neural network. The accelerator control system for accelerator control devices according to the present invention comprises: a plurality of device simulators corresponding to a plurality of the accelerator control devices, respectively, and performing learning and simulation based on an artificial neural network; and a reinforcement learning controller adjusting at least one control parameter corresponding to the plurality of device simulators, managing final accelerator output quality according thereto, generating a plurality of sets of learning data in which an output of an injector for the accelerator, a control parameter value of each device simulator and a final accelerator output quality value are matched, and performing machine learning by using the corresponding plurality of sets of learning data to calculate optimal control parameter values for maximizing the final accelerator output quality. Each of the plurality of device simulators includes: a primary simulator; a noise processing unit receiving the output of the primary simulator and additionally processing noises to output the processing result; and a feedback processing unit generating a feedback signal based on the output of the noise processing unit, and providing the feedback signal to the primary simulator. According to the present invention, it is possible to rapidly determine the optimal value of control parameters for increasing the final accelerator output quality.

Description

Realtime Accelerator Controlling System using Artificial Neural Network Simulator and Reinforcement Learning Controller

The present invention relates to an accelerator control system, and more specifically, an artificial neural network simulator that determines a parameter value for each accelerator control device through learning and simulation after constructing a simulator corresponding to the accelerator control devices based on an artificial neural network. And a real-time accelerator control system using a reinforcement learning controller.

A particle accelerator is a device for accelerating an atomic nucleus or a basic particle, but ultimately a device for observing and determining the microstructure of a material through collision of particles and observation thereof.

These particle accelerators are cationic accelerators, anion accelerators, heavy ion accelerators, and electron accelerators (radiation accelerators) depending on the object to be accelerated, and there are various types, such as linear accelerators and circular accelerators depending on the acceleration method.

However, in this particle accelerator, in order to accelerate the particle to a speed close to the speed of light, it is composed of a large number of devices, and the scale is also large.

For example, in the case of the 4th generation Pohang accelerator, an electron gun that strikes a powerful ultraviolet laser to copper to protrude electrons, a linear accelerator that greatly compresses the length of the electron beam by the electron gun, and the compressed accelerated electron beam pass between permanent magnets. An accelerator system consists of an undulator that generates X-ray emission light that is considerably brighter than light, and an X-ray experiment device (beam line) that outputs X-ray emission light to identify the structure and phenomena of a substance to a molecular structure. In this acceleration system, various control devices related to electron gun control, particle acceleration control, and radiation control are gathered to form one accelerator control system.

However, such an accelerator control system spatially covers a region from nm to Km, and temporally handles a data region of femto-sec, and handles hundreds of thousands of control variables. For various devices included in the accelerator control system, Optimization of the control parameters for the system is quite difficult.

That is, in the accelerator control system, as described above, various control devices (a sensor may be provided inside) exist in a small number of dozens to as many as hundreds over the entire section of the acceleration system, and each of these control devices There are various parameters (control variables) that determine the operation method, and the optimization of the parameters of each of these control devices is conventionally dependent on the experiences of researchers.

In order to reduce the trial and error of setting actual control parameters, physical simulation is performed in advance using Matlab, etc., but simulation-based analysis of the entire system has many problems in terms of accuracy and calculation time, and control parameters in Pohang Accelerator Research Center, etc. Is using control parameter setting according to expert judgment based on long operating experience.

The physical model simulation using the current Matlab or the optimization of each equipment according to the actual operating experience can achieve partial optimization, but the final output quality of the accelerator (for example, Q-BPM total) depending on the mutual influence of various control parameters. Value), there is a limit to searching/determining the optimal control parameter to maximize.

Particularly, there are attempts to implement a virtual accelerator based on Elegant, an accelerator simulator, and to implement optimization of control parameters offline, but there is a problem that a real-time accelerator control system cannot be optimized due to the speed problem of the conventional accelerator simulator. have.

Patent Publication No. 10-2007-0054457

The present invention has been devised to solve the above-mentioned conventional problems, and its purpose is to provide a system for calculating/searching an optimum value for maximizing the final output quality of the accelerator for various control parameters included in the accelerator control system. will be.

In order to achieve the above object, the accelerator control system for the accelerator control devices according to the present invention includes a plurality of device simulators corresponding to each of the plurality of accelerator control devices and performing learning and simulation based on an artificial neural network; After performing adjustment for at least one control parameter corresponding to the plurality of device simulators and accelerator final output quality, the output of the injector for the accelerator, the control parameter value of each device simulator, and the final accelerator output quality value And a reinforcement learning controller for generating a plurality of matched learning data, and performing machine learning using the plurality of learning data to calculate values of optimal control parameters such that the final output quality of the accelerator is the highest. Each of the plurality of device simulators includes a primary simulator; A noise processing unit receiving the output of the primary simulator and performing noise addition processing to output the noise; And a feedback processor that generates a feedback signal based on the output of the noise processor and provides it to the primary simulator.

Here, the primary simulator, the noise processor, and the feedback processor are all made of artificial neural networks, and may be formed by machine learning based on output values according to input values of an accelerator control device corresponding to each device simulator.

Here, the reinforcement learning controller designates a set of control parameter sets corresponding to a collection of at least one control parameter provided in the plurality of device simulators, changes to values of control parameters included in the set of control parameters, and the same After learning the accelerator final output quality through an artificial neural network-based learning process, it may be to calculate values of optimal control parameters such that the final accelerator output quality is highest.

Here, the reinforcement learning controller may calculate optimal control parameter values one by one in the order included in the corresponding control parameter set among control parameters included in the control parameter set.

As described above, according to the present invention, by performing artificial neural network-based machine learning on the control parameters of each accelerator control devices, it is possible to quickly determine values of optimal control parameters that increase the final output quality of the accelerator.

In addition, after replacing the actual accelerator control device with an artificial neural network-based device simulator, by collecting learning data, it is possible to prevent the accelerator from being interrupted to derive optimal control parameters, as well as through proper learning. It is possible to significantly increase the judgment accuracy for optimal control parameters that increase the accelerator final output quality.

1 is a functional block diagram of an accelerator control system according to an embodiment of the present invention,
2 is an example of the structure of an artificial neural network of the device simulator of FIG. 1,
3 is an example of a specific configuration of each device simulator,
4 is an example of data used by the reinforcement learning controller of FIG. 1 for supervised learning,
5 is an example of the configuration of a conventional EPICS-based accelerator control system,
6 is a view showing a combination between an accelerator control system and a conventional EPIC system according to an embodiment of the present invention compared to FIG. 5.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Hereinafter, each embodiment according to the present invention is only one example to help understanding of the present invention, and the present invention is not limited to these embodiments. In particular, the present invention may be configured in a combination of at least one or more of individual components and individual functions included in each embodiment.

1 is a functional block diagram of a real-time accelerator control system 100 (hereinafter referred to as an'accelerator control system 100') for accelerator control devices according to an embodiment of the present invention, such accelerator control system 100 May correspond to a parameter determination system for optimizing each parameter of the device simulator 110 as described below.

As shown in the figure, the accelerator control system 100 includes a plurality of artificial neural network based device simulators 110 (hereinafter, referred to as'device simulators 110') and a reinforcement learning controller 120.

Here, each device simulator 110 corresponds to each accelerator control device, and as described in the background art, dozens to hundreds of accelerator control devices are required to operate one accelerator, if necessary. Each device simulator 110 of FIG. 1 implements each of these accelerator control devices.

In particular, the device simulator 110 may be configured to perform learning or simulation based on an artificial neural network.

For example, the device simulator 110 may configure an artificial neural network as shown in FIG. 2 and determine an optimal internal parameter value to show the same characteristics as a real accelerator control device through machine learning.

For example, the device simulator 110 may be an artificial neural network that simulates an elegant, Genesis simulator that is conventionally used as an off target in an accelerator system.

In particular, each of the plurality of device simulators 110 may include a primary simulator 101, a noise processing unit 102, and a feedback processing unit 103, as shown in FIG. 3.

Here, the primary simulator is to simulate the function of each accelerator control device, and may correspond to, for example, an elegant LLRF (Low Level RF) simulator or a Genesis Undultator simulator.

The noise processing unit performs a function of receiving the output of the primary simulator and performing noise addition processing to output the noise.

Here, the noise processing unit may artificially generate noise due to external force, vibration, etc., which may or may occur in an actual situation during the operation of the accelerator.

The feedback processing unit performs a function of generating a feedback signal based on the output of the noise processing unit and providing it to the primary simulator.

The primary simulator, the noise processing unit, and the feedback processing unit are all formed of an artificial neural network, and may be formed by machine learning the characteristics (input and output) of the existing simulator.

The input and output beam position monitor (BPM) in FIG. 3 may be a cavity BPM (cBPM) or a stripline BPM (sBPM).

For reference, since the form shown in FIG. 2 is only a form widely known in the field of constructing an artificial neural network for machine learning, a description will be given of a known technique for making the weight in each layer an optimal value by machine learning. Omitted.

However, as shown in FIG. 2, each device simulator 110 receives the result of the device simulator 110 located at the front end as a sensor parameter, and also receives a control parameter, and then an internal hidden layer (of the hidden layer After passing through the shape or weight, etc., depending on the function of each accelerator control device corresponding to the device simulator 110, finally, the sensor parameters may be transferred from the output layer to the device simulator 110 located at the next stage.

That is, in this case, each device simulator 110 is configured to receive a result value of the device simulator 110 located at the front end and output a result according to an internal control parameter value.

Here, the fact that each device simulator 110 is located in the'front stage' or'next stage' means that the accelerator control devices for actual accelerator operation are arranged in such order.

That is, the plurality of device simulators 110 are matched to a specific placement order, and the placement order matched to the device simulator 110 is to match the placement order of the accelerator control devices for operating the actual accelerator.

After performing the machine learning based on the artificial neural network, the weight value of the internal variable of each layer (artificial neural network layer) of the device simulator 110 is determined, and after that, each accelerator control for the best accelerator final output quality Devices, that is, the control parameters of each device simulator 110 are calculated or determined by the reinforcement learning controller 120. To this end, the reinforcement learning controller 120 may also be configured as an artificial neural network as shown in FIG.

Hereinafter, this process will be described in detail.

First, the reinforcement learning controller 120 adjusts at least one control parameter corresponding to the plurality of device simulators 110 and performs final output quality of the accelerator, and then outputs the injector for the accelerator and each device simulator ( A plurality of training data matching the control parameter value of 110) and the final output quality value of the accelerator is generated.

That is, in the prior art, data generated or applied during the operation of the accelerator was used, but in the present invention, a plurality of such training data are generated and used using the device simulator 110 made of an artificial neural network.

As such, when each device simulator 110 is composed of an artificial neural network, the time for acquiring training data may be significantly shortened. That is, as described above, when the device simulator 110 is only an elegant-based virtualization device, the processing speed is significantly reduced, and thus, learning data cannot be obtained quickly.

On the other hand, when the device simulator 110 is composed of an artificial neural network and implements the characteristics of each accelerator control device through machine learning, it is possible to apply control parameter changes, etc. that could not be applied during actual accelerator operation, and for learning accordingly. Data can be easily acquired.

Subsequently, the reinforcement learning controller 120 performs machine learning using the generated plurality of learning data to calculate values of optimal control parameters such that the final accelerator output quality is highest.

That is, in order for the reinforcement learning controller 120 to calculate the control parameters of the respective accelerator control devices in which the final accelerator output quality according to the situation is highest, machine learning for the above-described learning data must be preceded. While the data that was applied during accelerator operation is collected and used as learning data, in the present embodiment, since the device simulator 110 corresponding to each accelerator control device is composed of an artificial neural network, various control parameters (which could not be applied to actual accelerators) After creating the learning data to which the control parameter) is applied and using it, the advantage of the artificial neural network can be maximized.

In other words, in the case of machine learning using an artificial neural network, proper learning can be achieved only when the learning data is diverse. However, the learning data collected during the actual operation of the accelerator must be limited data due to the stability of the accelerator. There is no, after all, if this is not used, proper learning cannot be achieved, but since the learning data according to the present embodiment is data collected by applying control parameters that cannot be applied in the prior art, the reinforcement learning controller 120 has a proper artificial neural network ( It is possible to perform machine learning for constructing an artificial neural network) so as to calculate values of optimal control parameters that have the highest final output quality for each situation.

Hereinafter, a specific process in which the reinforcement learning controller 120 progresses learning using a set of control parameter sets.

The reinforcement learning controller 120 designates a set of control parameter sets SET corresponding to a collection of at least one control parameter provided in the plurality of device simulators 110 and sets values of parameters included in the set of control parameter sets. After the change and the resulting accelerator final output quality are learned through an artificial neural network-based learning process, a function of calculating the values of optimal control parameters for the highest accelerator final output quality is performed.

For example, if the parameters included in each of the plurality of device simulators 110 are C0_1, C1_1, C2_1, and C3_1, respectively, the reinforcement learning controller 120 sets them as one parameter collection set'{C0_1, C1_1, C2_1, C3_1 }', the control parameters that make the final output quality the highest while changing the value of each parameter (ie, C0_1, C1_1, C2_1, C3_1) included in the parameter set through machine learning Is to calculate the value.

At this time, the reinforcement learning controller 120 may calculate an optimal parameter value one by one in the order included in the corresponding control parameter set among the parameters included in the control parameter set.

That is, when the vowel set is'{C0_1, C1_1, C2_1, C3_1}' as in the above-described example, the reinforcement learning controller 120 designates a value for the first parameter C0_1 and sets the value to a fixed value. In one state, machine learning yields C1_1 that satisfies the optimal final output quality.

Subsequently, the reinforcement learning controller 120 calculates C2_1 that satisfies the optimal final output quality through machine learning in a state where the predetermined or calculated values of C0_1 and C1_1 are fixed, and similarly, of C0_1, C1_1, and C2_1. C3_1 that satisfies the optimal final output quality can be calculated through machine learning while the value is a fixed value.

Here, the set of control parameters may include control parameters included in the plurality of device simulators 110 in the same order as the arrangement order of the corresponding device simulator 110.

That is, as described above, the plurality of device simulators 110 may each have a specific arrangement order according to the corresponding accelerator control device, and the control parameter set includes each control according to the arrangement order of each of the device simulators 110. The parameters are included, and the reinforcement learning controller 120 determines/calculates the value of each control parameter in the order included in the set of control parameter sets.

For example, in order to operate an accelerator, a first accelerator control device, a second accelerator control device, a third accelerator control device, and a fourth accelerator control device must be arranged in that order, and the first accelerator control device includes a first control parameter (C0_1). ), a second control parameter (C1_1) for the second accelerator control device, a third control parameter (C2_1) for the third accelerator control device, and a fourth control parameter (C3_1) for the fourth accelerator control device. Assuming that, the set of control parameter sets may be configured as'{C0_1, C1_1, C2_1, C3_1}', and the reinforcement learning controller 120 performs a process of determining each parameter value in that order.

In this embodiment, although each accelerator control apparatus has one parameter, each accelerator control apparatus may have a plurality of parameters, and there may be priorities among the plurality of parameters.

As described above, the reinforcement learning controller 120 may use a Monte Carlo Tree Search (MCTS) algorithm based on the reinforcement learning model to calculate the optimal control parameter value of each device simulator 110 (accelerator control device). .

Here, the MCTS algorithm is a term that refers to an algorithm for probabilistically calculating a function value using a random number, and is used to approximate the value to be calculated when it is not expressed as a closed value or complex.

For example, if the area of a circle is obtained by applying the Monte-Carlo algorithm, the area of the circle is approximated by calculating the probability that the point is placed inside the circle by drawing a square inscribed in the circle and the circle and a large number of points in the square. The accuracy of the Monte-Carlo algorithm increases as the number of random trials increases.

Monte-Carlo Tree Search (MCTS) is a method for finding the optimal decision. It is mainly used when it is not easy to find a solution by formulating with an experiential search algorithm for decision making. It is used as a way to find the number of people.

For example, if MCTS is applied to a game, each number placed in the game is a node, and the entire process of the game is represented by a tree of each number. The win rate is recorded in each node, and the process of finding the best number in the game can be approximated by finding the node with the highest win rate, and MCTS calculates the win rate for each node and refers to the process of finding the node with the highest win rate. have.

The problem with tree search is that the search takes a lot of time when there are many child nodes. MCTS does not search all the possibilities, but corresponds to an algorithm that obtains game results through a number of random simulations and applies them to the node's odds. do.

The MCTS algorithm consists of four steps: Selection, Expansion, Simulation, and Back propagation.

(1) Selection: Start at the root R and follow the child nodes in succession and select Node L.

(2) Expansion: If the game has not ended in Node L, create a new child Node C or select one of the existing child nodes as Node C.

(3)Simulation: Perform random playout for node C.

(4) Back propagation: Update the result of playout from node C to root R.

Since the MCTS algorithm itself is a known technique, a more detailed description is omitted.

The reinforcement learning controller 120 is provided with a plurality of learning data matching the output of the injector for the accelerator, the control parameter value of each device simulator 110, and the final output quality value of the accelerator through the above-described process. After learning through supervised learning (SL) using a preset number of learning data among the learning data, reinforcement learning (RL) can be performed to calculate optimal control parameter values. have.

Here,'supervised learning' refers to learning in a state in which a label for data-an explicit correct answer-is given, and previously calculated big data (i.e., a device simulator composed of artificial neural networks as described above) (110) refers to learning based on artificial neural networks using input, control parameters, and final output quality values obtained while changing control parameters, and reinforcement learning means that an agent acts on a given state. It means taking a (action) and learning while getting some reward from it.

The theoretical content of the supervised learning or reinforcement learning also corresponds to a well-known technique, and the feature of the present invention is that the learning method is used to calculate optimal parameters of the accelerator control devices.

4 is an example of data acquired in advance for'supervised learning' of the reinforcement learning controller 120.

That is, each row of FIG. 4 corresponds to the aforementioned learning data, and the reinforcement learning controller 120 extracts the learning data corresponding to several cases in a predetermined order or in a random manner among the plurality of learning data collected in this way. Then, it is possible to perform'supervised learning' in an artificial neural network using the extracted learning data.

For reference, in each learning data of FIG. 4,'I' is the output value of the injector, Q-BPM is the final output quality value of the accelerator, and the rest corresponds to control parameters of the respective accelerator control devices.

The accelerator control system 100 described in the above-described embodiment may be operated in combination with, for example, an EPICS-based accelerator control system.

5 shows a conventional EPICS-based accelerator control system, and FIG. 6 shows that the control devices are replaced by the device simulator 110 according to the present invention and the reinforcement learning controller 120 is added and operated in the EPICS-based accelerator control system. It shows the form.

Referring to FIG. 5, accelerator control devices such as LLRF and BPM are arranged in dozens of sections in the entire accelerator section to perform optimization in each section. At this time, the accelerator control devices are configured for the Pv data for a given time period for each device in EPICS IOC. Generates an event, or generates an event for changed values, and the EPICS IOC (Input Output Controller) server collects the events of each device and sends data to display on the CSS screen used by the operator, or sends data later. Data can be saved by sending it to an AA (Archiver Appliance) server for use, and if a value outside a specific setting value is entered, an alarm message is output to the CSS system through the Alarm Server so that the operator can take action.

FIG. 6 illustrates a state in which the accelerator control device is replaced with an artificial neural network based device simulator 110, and a reinforcement learning controller 120 that communicates with the device simulator 110 is added.

By replacing the accelerator control device that is actually operated as shown in FIG. 6 with an artificial neural network-based device simulator 110, it is possible to prevent the accelerator operation for optimal control parameter derivation, as well as the device simulator 110, the artificial neural network By having a form based on learning, it is possible to approach the characteristics of the actual accelerator control device considerably.

In FIG. 6, for convenience, an accelerator control device is additionally included in the conventional EPICS, but the accelerator control device may be configured as a separate system or may be configured as one system together with the device simulator 110 described above.

In the above-described embodiment, the feedback processing unit is included in each device simulator as an example, but the feedback processing unit itself may be included in the reinforcement learning controller.

Meanwhile, of course, the process of performing each of the above-described embodiments may be performed by a program or an application stored in a predetermined recording medium (for example, a computer readable). Here, the recording medium includes both an electronic recording medium such as a random access memory (RAM), a magnetic recording medium such as a hard disk, and an optical recording medium such as a compact disk (CD).

At this time, the program stored in the recording medium may be executed on hardware such as a computer or a smartphone to perform each of the above-described embodiments. In particular, at least one of the functional blocks according to the present invention described above may be implemented by such a program or application.

In addition, the present invention is not limited to the specific embodiments described above, but can be implemented by modifying and modifying in various ways without departing from the gist of the present invention. It will be apparent that such variations and modifications are included in the present invention if they fall within the scope of the appended claims.

100: accelerator control system 110: device simulator
120: reinforcement learning controller

Claims (4)

A plurality of device simulators corresponding to each of the plurality of accelerator control devices and performing learning and simulation based on an artificial neural network;
After adjusting the at least one control parameter corresponding to the plurality of device simulators and performing final accelerator output quality, the output of the injector for the accelerator, the control parameter value of each device simulator, and the final accelerator output quality value It includes a reinforcement learning controller that generates a plurality of matched learning data, and performs machine learning using the plurality of learning data to calculate values of optimal control parameters such that the final output quality of the accelerator is the highest.
Each of the plurality of device simulators includes a primary simulator; A noise processing unit receiving the output of the primary simulator and performing noise addition processing to output the noise; And a feedback processor that generates a feedback signal based on the output of the noise processor and provides it to the primary simulator. According to claim 1,
The primary simulator, the noise processing unit, and the feedback processing unit are all made of an artificial neural network, and accelerator control characterized in that it is formed by machine learning based on an output value according to an input value of an accelerator control device corresponding to each device simulator. Accelerator control system for devices. According to claim 1,
The reinforcement learning controller designates a set of control parameter sets corresponding to a collection of at least one control parameter provided in the plurality of device simulators, changes to values of control parameters included in the set of control parameters, and accelerators accordingly. After learning the final output quality through the artificial neural network-based learning process, the accelerator control system for the accelerator control devices, characterized in that for calculating the value of the optimal control parameters to the highest accelerator final output quality. According to claim 3,
The reinforcement learning controller is an accelerator control system for accelerator control devices, characterized in that for calculating the optimum control parameter value one by one in the order included in the corresponding control parameter set among the control parameters included in the set of control parameters.

Images