KR20200055495A - Reinforcement learning model creation method for automatic control of PTZ camera, and the reinforcement learning model - Google Patents

Abstract

Disclosed in the present invention is a method for generating a reinforcement learning model which can improve image control efficiency of an intelligence security control system by generating an efficient reinforcement learning model for the automatic control of a PTZ camera used in the intelligence security control system. To this end, the present method comprises: (1) a learning image data acquisition and analysis step in which learning image data including object information are acquired to perform reinforcement learning, and information of the position and the size of an object in an image is analyzed; (2) a step in which the action value of a PTZ camera is selected for controlling the PTZ camera; (3) a step in which a reward is calculated by estimating the control direction of the PTZ camera; (4) a step in which a data set including the state before and after the action of the camera and a reward is stored; (5) a step in which it is determined whether the number of the stored data sets is equal to or greater than the predetermined number; and (6) a step in which a reinforcement learning model is generated based on the stored data sets.

Description

Reinforcement learning model creation method for automatic control of PTZ camera, and the reinforcement learning model}

The present invention relates to a method for generating a reinforcement learning model for automatic control of a PTZ camera to improve image control efficiency in an environment such as a general situation room or CCTV integrated control center to which an intelligent security control system is applied, and a reinforcement learning model generated thereby. It is about.

The present invention is an information communication and broadcasting research and development project of the Ministry of Science and ICT and the Information and Communication Technology Promotion Center in 2017. It was carried out as a result of research on the development of core technologies for intelligent defense surveillance systems.

Currently, many CCTV integrated control centers have been built and operated, and many cameras are linked to it (as of 2018, more than 190 CCTV integrated control centers are in operation in Korea and about 409,000 cameras are in use).

Due to the continued increase in CCTV demand, local governments and industrial sites are complaining of difficulties due to lack of control personnel after establishing a CCTV integrated control center. For this reason, there is a problem of controlling tens to hundreds of cameras per person, and this is a major cause of a decrease in control efficiency. In addition, there is a problem that a lot of budget needs to be secured when additionally deploying control personnel.

Recently, smart control has been introduced to alleviate these problems and increase the efficiency of the control system of the CCTV integrated control center. Various technologies are applied to smart control, such as video analysis service, reduction in the number of control channels of control personnel, and priority determination of cameras. However, it is necessary to manually control the PTZ (pan, tilt, zoom) of the camera when detecting an event such as a dangerous situation.

Conventional PTZ camera control technology controls the camera based on the difference between the object and the center point of the camera in order to position the object in the center of the camera screen. However, this method is a simple camera movement method that does not consider the object size, and has a disadvantage that can be applied only to the object tracking field. In addition, the PTZ control method using homography considers the object size, but the zoom control is not accurate because the zoom is controlled in consideration of the physical distance between the object and the camera.

In order to overcome the above problems, the present invention proposes a method for generating a reinforced learning model that improves the image control efficiency of the intelligent security control system by generating an efficient reinforcement learning model for automatic control of the PTZ camera used in the intelligent security control system. .

According to the present invention, for automatic control of a PTZ camera using reinforcement learning, it is possible to efficiently perform video control of an intelligent security control system by generating a reinforcement learning model that predicts camera movement and adaptively rewards a situation. .

Reinforcement learning (RL) is a new methodology for machine learning that solves probabilistic decision-making problems. Unlike supervised learning, reward or reward functions are given to maximize the average of reward values to be obtained in the future. It is a machine learning technique to find the policy function of. That is, unlike supervised learning, the target value is a reward and the predicted value is a policy or an action.

According to one aspect for solving the above problems, a method for generating a reinforcement learning model for automatic control of a PTZ camera is provided. The method includes: 1) acquiring and analyzing learning image data to acquire learning image data including object information, and analyzing location and size information of objects in the image to perform reinforcement learning; 2) selecting an action value of the PTZ camera to control the PTZ camera; 3) Calculate the Reward by estimating the control direction of the PTZ camera including Pan Left, Pan Right, Tilt Up, Tilt Down, and Zoom, and use the selected action value to move the PTZ camera to the estimated camera control direction. Moving, and adaptively granting a Reward to a change in the position of the object in the image after moving the PTZ camera; 4) storing the data set including the state before and after the camera action and the reward; 5) determining whether the number of stored data sets is greater than or equal to a predetermined number; 6) creating a reinforcement learning model based on the stored dataset.

In addition, according to another aspect for solving the above problem, a reinforcement learning model including an artificial neural network including an input layer, an output layer, and a hidden layer is provided. This reinforcement learning model includes means for setting a current learning target for PTZ camera control by calculating a deep Q-learning function using the weight of the artificial neural network; Means for setting a next step learning target for PTZ camera control by performing an actual PTZ camera action according to the set current learning target for PTZ camera control; And means for performing reinforcement learning on the PTZ camera control by updating weights to reduce errors in the current learning target and the next learning target for the PTZ camera control.

The configuration and operation of the present invention introduced above will be further clarified through specific embodiments described with reference to the drawings.

According to the present invention, it is possible to control the PTZ camera optimized for the position and size of the object by using reinforcement learning, and the automatic control of the PTZ camera based on the reinforcement learning improves the video control efficiency of the intelligent security control system and increases the workload. Can be reduced.

1 is an artificial neural network structure of the reinforcement learning model of the present invention
Figure 2 is a parameter applied to the artificial neural network of Figure 1
3 is a flowchart of a method for generating a reinforcement learning model for automatic control of a PTZ camera of an intelligent video control system proposed in the present invention.

Advantages and features of the present invention and methods for achieving them will be made clear by referring to embodiments described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete so that ordinary knowledge in the technical field to which the present invention pertains is provided. It is provided to fully inform the possessor of the scope of the invention. The technical scope of the invention is defined by the claims.

On the other hand, the terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, "comprises" or "comprising" means the presence of one or more other components, steps, operations and / or elements other than the components, steps, operations and / or elements mentioned, or Addition is not excluded.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In adding reference numerals to the components of each drawing, the same components are assigned the same reference numerals as possible, even though they are shown on different drawings, and in describing the present invention, detailed descriptions of related well-known components or functions When the subject matter of the present invention can be obscured, the detailed description thereof will be omitted.

1 shows an artificial neural network structure used in the reinforcement learning model of the present invention. The artificial neural network shown in FIG. 1 has four hidden layers 30-1, 30-2, 30-3, and 30-4 between the input layer 10 and the output layer 20. ).

FIG. 2 shows parameters applied to the artificial neural network of FIG. 1. Each parameter is schematically described in FIG. 2.

-Adam was used as the optimization technique. The optimization is to find the value of the hyperparameter in the direction of minimizing the value of the loss function according to the training of the neural network model and the result, and the Adam technique, one of various techniques for this optimization, was used.

-The loss function (Loss functioin) is a function that calculates the difference between the result value generated through the model and the value that actually wanted to occur, and there can be various types of functions depending on the purpose. square error) was also used in the present invention. mse measures the mean of the squares of errors or deviations to measure the difference between the estimated and expected values. Since the squared value is used, there is an advantage in that the accuracy of the guess is increased because the value change due to the error is large.

-Activation function is a function that receives a signal and outputs it through appropriate processing. It determines whether the outputted signal is activated in the next step. As an activation function, a ReLU (Rectified Linear Unit) function was used for the hidden layer and a linear function was used for the output layer.

-Learning rate means the amount of learning how much to learn when learning once, and the weighting parameter is updated after learning with one learning amount. The learning rate value should be determined in advance, such as 0.01 or 0.001. In general, if this value is too large or too small, it is difficult to find a suitable point. In neural network learning, we usually check if we are learning correctly by changing this learning rate value. If the learning rate is too large, a large value is output. If it is too small, the learning is ended without being updated. In the present invention, 0.001 was set as the learning rate.

-Batch size (Batch size) means the number of data used in one learning, was set to 128 here.

-The number of households (Epochs) means one learning number (including forward and backward) for the entire data, and was set to 35 in the present invention.

-The episode (Episode), which is the unit section in which the reward is returned, is set to 1000.

The reinforcement learning model including such an artificial neural network calculates a deep Q-learning function using the weight of the artificial neural network, sets a current learning target for PTZ camera control, and then sets an actual PTZ camera. Perform control actions to set next-level learning goals for PTZ camera control. Reinforcement learning is performed on the PTZ camera control by updating the weights to reduce errors in the current learning target for the PTZ camera control and the next learning target. The reinforcement learning model will be described again later.

3 is a flowchart of a method for generating a reinforced learning model for automatic control of a PTZ camera of an intelligent video control system proposed in the present invention. A method of generating a reinforcement learning model for automatic control of a PTZ camera according to the present invention will be described with reference to FIG. 3.

110: Acquiring learning image data including object information to perform reinforcement learning, and acquiring learning image data analyzing the location and size [x, y, w (width), h (height)] of the object in the image And analysis steps.

After performing steps 120 and 110, selecting an action value of the PTZ camera to control the PTZ camera.

는 수학식 1에서

값에 의해 무작위로 선택되거나, 학습되지 않은 강화학습 모델로 액션값을 선택한다. 여기에서

는 1이며, 반복횟수

가 증가할수록 수학식 2의 제약 조건에 따라

값이 감소한다. 즉, 감소된

값이 임계값인

와 같거나 이보다 클 경우에만 액션값

를 무작위로 선택하며, 그렇지 않은 경우에는 현재 상태

와 학습되지 않은 강화학습 모델로 액션값을 선택한다. Here, the action value of the camera is defined as (0.02, 0.06, 0.1, 0.14, 0.18, 0.22, 0.26, 0.3), and the defined action value is selected through Equation 1. Action value of PTZ camera

In Equation 1

It is randomly selected by value, or an action value is selected by an untrained reinforcement learning model. From here

Is 1 and the number of repetitions

According to the constraints of equation (2)

Value decreases. That is, reduced

With a threshold value

Action value only if equal to or greater than

Randomly selects, otherwise the current state

And the action value is selected as a reinforcement learning model that is not learned.

는

를 감소시키는 값이고,

은

값이 마이너스로 떨어지지 않게 하는 양수 값이다.Equation 2 is a constraint,

The

Is a value that decreases

silver

A positive value that prevents the value from falling negatively.

130: After performing the previous 120 steps, calculating a reward by estimating the automatic control direction of the PTZ camera

Equation 3 analyzes the center coordinates of the screen and the current coordinates of the object to estimate the control direction of the PTZ camera (ie, Pan Left, Pan Right, Tilt Up, Tilt Down, Zoom).

는 현재 객체의 가로 위치와 취득한 영상의 가로 중심 거리의 차이를 나타내고,

는 현재 객체의 세로 위치와 취득한 영상의 세로 중심 거리의 차이를 말한다. 또한

는 임계값 상수이고,

와

가

보다 작을 때 Zoom In을 카메라 제어 방향으로 추정한다. 그 외에는 Pan, Tilt를 수행하는데

가 양수일 경우 Pan Right, 음수일 경우 Pan Left를 이동 방향으로 추정하고,

가 양수일 경우 Tilt Down, 음수일 경우 Tilt UP으로 카메라 제어 방향을 추정한다.here

Indicates the difference between the horizontal position of the current object and the horizontal center distance of the acquired image,

Is the difference between the vertical position of the current object and the vertical center distance of the acquired image. In addition

Is the threshold constant,

Wow

end

When smaller, Zoom In is estimated as the camera control direction. Other than that, Pan, Tilt is performed.

If is positive, Pan Right, if negative, Pan Left is estimated as the moving direction,

If is positive, Tilt Down, and if negative, Tilt UP to estimate the camera control direction.

In this step, the PTZ camera is moved in the estimated camera control direction using the action value selected in step 120 above. Reward is given adaptively to the change of the position of the object in the image after moving the PTZ camera.

는 카메라 움직임 이전 상태에서 객체의 위치와 화면 중심의 거리 차이고,

은 카메라 이동 후 변한 객체의 위치와 화면 중심의 거리 차이다.

은 정규화 상수이고, τ ₁ 은 Pan과 Tilt의 목표 크기이다.To this end, when PTZ camera movements related to Pan and Tilt occur, Reward r _t corresponding to the movement direction and the action value is calculated using Equations 4, 5, and 6. here

Is the distance between the object's position and the center of the screen in the state before the camera movement,

Is the difference between the position of the object changed after the camera moves and the center of the screen.

Is the normalization constant, and τ ₁ is the target size of Pan and Tilt.

와

는 각각 카메라 이동 전,후에 대한 객체 크기 정보이고, 앞의

과 수학식 5, 6의

는 정규화 상수로 예컨대 각각 100, 10으로 설정가능하다. 그리고

는

를 반영하는 상수로 예컨대 1.2로 설정가능하며,

는 Zoom의 목표 크기로 예컨대 70으로 설정할 수 있다. Next, Reward r _t for Zoom is calculated using Equations 7 and 8. here

Wow

Is object size information for before and after camera movement, respectively.

And Equations 5 and 6

Can be set to 100 and 10, respectively, as normalization constants. And

The

It can be set to 1.2 as a constant reflecting,

Can be set to 70 as the target size of Zoom.

As described above, using the equations 4 to 8, Pan, Tilt, and Zoom movements have + Reward when they are close to the horizontal and vertical centers and target sizes, and -Reward in the opposite direction, so they are adaptive to the movement values and directions. Reward can be granted.

140: After performing the previous 130 steps, before and after the action and storing the state and Reward.

The data set including the state before and after the action to be saved and the reward is composed of 11 detailed data, and each detailed data item is as follows.

-Object status information before moving the camera (x, y, width, height)

-Action value estimation information (0 ~ 7)

-Reward

-Object status information after moving the camera (x, y, width, height)

-Exit Flag

150: After performing the previous 140 steps, determining whether the number of stored datasets is a predetermined number or more.

In one embodiment, for the purpose of use in reinforcement learning, it is checked whether the number of datasets composed of before and after actions and rewards is 3000 or more.

160: After performing the previous 150 steps, generating a reinforcement learning model based on the stored dataset.

Equation (9) is used to create the reinforcement learning model.

Equation 9 above is a formula for setting a learning goal in reinforcement learning, and the definition of each variable is as follows.

: 반복 횟수,

: 총 반복 횟수

: Number of repetitions,

: Total number of iterations

: 현재 단계의 상태,

: 다음 단계의 상태. 좀 더 구체적으로,

는 객체의 현재 좌표 및 크기이고,

은 120과 130 단계에서 추정된 액션값과 방향으로 PTZ 카메라를 이동시켰을 때의 객체의 좌표 및 크기이다(객체의 좌표 및 크기

임).

: Status of the current stage,

: Status of the next step. More specifically,

Is the current coordinate and size of the object,

Is the coordinates and size of the object when the PTZ camera is moved in the direction and action values estimated in steps 120 and 130 (the coordinates and size of the object)

being).

: 현재 단계의 액션값,

: 다음 단계의 액션값 (본 발명에서 액션값은 (0.02, 0.06, 0.1, 0.14, 0.18, 0.22, 0.26, 0.3) 중 선택된 값)

: Action value of the current step,

: Action value of the next step (in the present invention, the action value is a value selected from (0.02, 0.06, 0.1, 0.14, 0.18, 0.22, 0.26, 0.3))

: 무작위성을 조절하는 가중치(Weight)(E-greedy 방법 사용)

: Weight to control randomness (using E-greedy method)

: 현재 단계의 Reward

: Reward at the current stage

: 인공신경망의 Weight,

: 복사된 인공신경망의 Weight

: Weight of artificial neural network,

: Weight of the copied artificial neural network

: Deep Q-learning 함수 (Q값을 산출한다)

: Deep Q-learning function (calculates Q value)

를 이용해 PTZ 카메라의 현재 상태

와 현재 액션값

로 Q값(딥 Q-러닝 함수값)을 계산한다(PTZ 카메라에 대한 현재 학습 목표 설정). ②번 항은 ①번 항에서 선택한

값으로 실제 PTZ 카메라 액션을 수행한 후 변경된 다음 단계의 PTZ 카메라 상태

와 새롭게 선택된 액션값

를 이용하여 복사된 인공신경망

을 통해 최대가 되는 Q값을 계산한다. 산출된 Q값은 가중치

로 조절하며, ①번 항 수행 후에 습득한 Reward를 결합한다(PTZ 카메라에 대한 다음 단계 학습 목표 설정). ① 항과 ② 항의 에러를 줄일 수 있는 인공신경망

를 학습함으로써 PTZ 카메라에 대한 강화학습이 수행된다.Paragraph ① in Equation 9 is an artificial neural network.

Current status of the PTZ camera using

And current action value

Calculate the Q value (dip Q-learning function value) with (Set current learning target for PTZ camera). Item ② is selected in the item ①

After performing the actual PTZ camera action with the value, the status of the next PTZ camera changed

And the newly selected action value

Artificial neural network copied using

Calculate the maximum Q value. The calculated Q value is the weight

Adjust to, and combine the Rewards acquired after performing item ① (setting the next step learning target for PTZ camera). Artificial neural network that can reduce the errors in terms ① and ②

Reinforcement learning for the PTZ camera is performed by learning.

In the above, the configuration of the present invention has been described in detail through a preferred embodiment of the present invention, but those skilled in the art to which the present invention pertains are disclosed herein without changing the technical spirit or essential features of the present invention. It will be understood that it may be implemented in a specific form different from the content. It should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention is defined by the claims below, rather than the above detailed description, and all changes or modifications derived from the claims and equivalent concepts should be interpreted to be included in the technical scope of the present invention. .

Artificial neural network input layer 10, output layer 20, hidden layer (30-1, 30-2, 30-3, 30-4)

Claims (11)

As a method of creating reinforcement learning model for automatic control of PTZ camera,
1) acquiring and analyzing learning image data for acquiring learning image data including object information and analyzing position and size information of objects in the image to perform reinforcement learning;
2) selecting an action value of the PTZ camera to control the PTZ camera;
3) Calculate the Reward by estimating the control direction of the PTZ camera including Pan Left, Pan Right, Tilt Up, Tilt Down, and Zoom, and use the selected action value to move the PTZ camera to the estimated camera control direction. Moving, and adaptively rewarding a change in the position of the object in the image after moving the PTZ camera;
4) storing the data set including the state before and after the camera action and the reward;
5) determining whether the number of stored data sets is greater than or equal to a predetermined number;
6) A method of generating a reinforcement learning model for automatic control of a PTZ camera, comprising the step of generating a reinforcement learning model based on the stored dataset. In claim 1, In the step of selecting the action value of the PTZ camera for the 2) PTZ camera control, the action value

The

Is selected through the equation of
Constant

The value is the threshold that was passed in advance

If equal to or greater than, the action value is randomly selected, otherwise the current state

And a reinforcement learning model generation method for automatic control of a PTZ camera in which an action value is selected as a reinforcement learning model that is not learned. According to claim 1, In the step of calculating a Reward by estimating the control direction of the PTZ camera, the control direction of the PTZ camera is

Estimated by the equation of,

Wow

end

If it is smaller, Zoom In is estimated as the camera control direction. Otherwise, Pan and Tilt are estimated as the camera control direction.
here

Is the difference between the horizontal position of the current object and the horizontal center distance of the acquired image,

Is the difference between the vertical position of the current object and the vertical center distance of the acquired image,

Is a method of generating a reinforcement learning model for automatic control of a PTZ camera, which means a predetermined threshold constant. In claim 3, the

If is positive, Pan Right, if negative, Pan Left is estimated as the camera control direction,

A method of generating a reinforcement learning model for automatic control of a PTZ camera that estimates Tilt Down when is positive and Tilt UP when it is negative in the camera control direction. In claim 3, in relation to the Pan and Tilt of the PTZ camera

,

,

Reward r _t is calculated using the equation of
here

Is the distance between the object's position and the center of the screen in the state before the camera movement,

Is the difference between the position of the object changed after the camera moves and the center of the screen,

Is a normalization constant, and τ ₁ is a method of generating a reinforcement learning model for automatic control of a PTZ camera, which is a target size of Pan and Tilt. In claim 3, with respect to the zoom of the PTZ camera

And

Reward r _t is calculated using the equation of
here

Wow

Is object size information for before and after camera movement,

Is the normalization constant,

The

Is a constant reflecting

Is a method of creating a reinforcement learning model for automatic control of a zoom target PTZ camera. In the method of claim 1, 4) In the step of saving the data set including the before and after states of the camera action and the reward, the stored data set is the object state information before the camera movement, the action value estimation information, the reward, the object after the camera movement Method for creating reinforcement learning model for automatic control of PTZ camera including status information. In the method of claim 1, 5) In the step of determining whether the number of stored datasets is greater than or equal to a predetermined number, a method of creating a reinforcement learning model for automatic control of a PTZ camera having 3000 predetermined datasets. In step 1, 6) generating a reinforcement learning model based on the stored dataset,
Means for setting a current learning target for PTZ camera control by calculating a deep Q-learning function with a current state and a current action value using an artificial neural network;
Obtaining a reward after performing the current learning goal setting step,
After performing the actual action with the current action value, the maximum deep Q-learning function value is calculated through the artificial neural network copied using the changed state of the next step and the newly selected action value, and the calculated deep Q-learning function A method of generating a reinforced learning model for automatic control of a PTZ camera, comprising adjusting a value with a weight and combining the rewards acquired after performing the current learning target setting step to set a next learning target. A reinforcement learning model used for automatic control of the PTZ camera according to any one of claims 1 to 9,
An artificial neural network including an input layer, an output layer, and a hidden layer;
Means for setting a current learning target for PTZ camera control by calculating a deep Q-learning function using the weight of the artificial neural network;
Means for setting a next step learning target for PTZ camera control by performing an actual PTZ camera action according to the set current learning target for PTZ camera control;
A reinforcement learning model for automatic control of a PTZ camera, comprising means for performing reinforcement learning on a PTZ camera control by updating weights to reduce errors in the current learning target and the next learning target for PTZ camera control. In claim 10, In the artificial neural network
Loss functioin uses mse (mean square error),
The Activation function uses the ReLU (Rectified Linear Unit) function for the hidden layer and the Linear function for the output layer.
Learning Rate is set to 0.001,
Batch size is reinforced learning model for automatic control of PTZ camera set to 128.

Images