KR102645254B1 - Deblurring network learning method, motion blur reduction method and robot - Google Patents

Abstract

One embodiment includes a motion blur image captured by a robot in a moving situation, a target image with reduced motion blur in the motion blur image, and inertial sensor measurement values of the robot in the situation in which the motion blur image was captured. A step in which learning data is acquired; Quantizing the inertial sensor measurement value; Extracting an inertial reflection parameter by inputting the quantized inertial sensor measurement value into a condition parameter extraction module; preparing a deblurring network that uses the inertial reflection parameter as a condition parameter, receives the motion blur image as an input, and generates a deblurring image with reduced motion blur for the motion blur image; and training the condition parameter extraction module and the deblurring network using the learning data so that the deblurring image follows the target image.

Description

Deblurring network learning method, motion blur reduction method and robot {DEBLURRING NETWORK LEARNING METHOD, MOTION BLUR REDUCTION METHOD AND ROBOT}

This embodiment relates to technology for reducing motion blur in images.

VSLAM (Visual Simultaneous Localization And Mapping) technology, which estimates location using various sensors in an unknown environment and creates a 3D environment map, is receiving attention. Using this VSLAM technology, the robot can autonomously navigate to its destination by estimating its own location using a camera and identifying the structure of the surrounding environment.

In the VSLAM field, technology that analyzes images acquired through cameras, estimates one's own location, and determines the structure of the surrounding environment to create a map is mainly addressed. However, in order to implement this technology, it must first be assumed that an accurate external image can be obtained. If the image captured by the camera contains a lot of noise, the result of the processing performed at a later stage may not be accurate.

In particular, when shooting the outside in a moving situation such as a robot, there is a high possibility that the image contains motion blur, and if this is not appropriately reduced, errors are likely to appear in the VSLAM results.

Against this background, the purpose of this embodiment is, in one aspect, to provide a technology for reducing motion blur in images captured by a robot. In another aspect, the purpose of this embodiment is to provide a technology to increase the accuracy of VSLAM by removing noise from the image.

In order to achieve the above-mentioned purpose, one embodiment includes a motion blur image captured by a robot in a moving situation, a target image in which motion blur is reduced from the motion blur image, and the motion blur image in a situation in which the motion blur image is captured. Obtaining learning data including inertial sensor measurements of the robot; Quantizing the inertial sensor measurement value; Extracting an inertial reflection parameter by inputting the quantized inertial sensor measurement value into a condition parameter extraction module; preparing a deblurring network that uses the inertial reflection parameter as a condition parameter, receives the motion blur image as an input, and generates a deblurring image with reduced motion blur for the motion blur image; and training the condition parameter extraction module and the deblurring network using the learning data so that the deblurring image follows the target image.

In the deblurring network learning method, the deblurring network is implemented in the form of a U-Net, and the condition parameter may be a parameter applied to Feature-wise Linear Modulation (FiLM) of the U-Net.

In the deblurring network learning method, the condition parameter extraction module may include a non-linear function.

In the deblurring network learning method, the inertial sensor measurement value may include angular acceleration values for each axis of space.

In the deblurring network learning method, in the step of learning the deblurring network, the deblurring network can be trained so that a loss calculated as a difference between the deblurring image and the target image is minimized.

Another embodiment includes obtaining an original image captured by a robot and an inertial sensor measurement value of the robot in the situation in which the original image was captured; Quantizing the inertial sensor measurement value; Extracting an inertial reflection parameter by inputting the quantized inertial sensor measurement value into a pre-learned condition parameter extraction module; Applying the inertial reflection parameter as a condition parameter of a pre-learned deblurring network; A motion blur reduction method is provided including the step of inputting the original image into the deblurring network and then generating a deblurring image with reduced motion blur.

In the motion blur reduction method, the deblurring network is implemented in the form of a U-Net, and the condition parameter may be a parameter applied to Feature-wise Linear Modulation (FiLM) of the U-Net.

The motion blur reduction method further includes calculating an angular acceleration value for each axis using the robot's acceleration sensor measurement value, and the inertial sensor measurement value may include the angular acceleration value.

In the motion blur reduction method, the condition parameter extraction module can extract the inertial reflection parameter through linear combination of quantized inertial sensor measurement values and nonlinear function calculation.

In the motion blur reduction method, the deblurring network includes an encoder block and a decoder block, and the inertial reflection parameter can be applied to the condition parameter of the encoder block.

Another embodiment includes a camera that photographs the outside of the robot body; an inertial sensor that measures the inertia of the body; And quantize the measured value of the inertial sensor, input the quantized measured value into a pre-learned condition parameter extraction module to extract an inertial reflection parameter, and apply the inertial reflection parameter as a condition parameter of a pre-learned deblurring network. and inputting the original image captured by the camera to the deblurring network and then generating a deblurring image with reduced motion blur.

The robot further includes a mobile device that controls movement of the body, and can determine whether to apply the original image to the deblurring network according to a control signal from the mobile device.

The processor may estimate the position of the robot in space through analysis of the deblurring image.

The processor may implement Visual Simultaneous Localization And Mapping (VSLAM) using the deblurring image.

The inertial sensor may include an acceleration sensor, and the processor may calculate an angular acceleration value for each axis using the measured value of the acceleration sensor, and extract the inertial reflection parameter using the angular acceleration value.

As described above, according to this embodiment, motion blur can be reduced in images captured by a robot, and through this, the accuracy of VSLAM can be improved.

1 is a configuration diagram of a robot according to one embodiment.
Figure 2 is a diagram showing the flow of data and signals for reducing motion blur in a robot according to an embodiment.
Figure 3 is a configuration diagram of a processor according to one embodiment.
Figure 4 is a configuration diagram of a deblurring network according to an embodiment.
Figure 5 is a configuration diagram of an encoder block according to one embodiment.
Figure 6 is a flowchart of a deblurring network learning method according to an embodiment.
Figure 7 is a flowchart of a motion blur reduction method according to an embodiment.

Hereinafter, some embodiments of the present invention will be described in detail through illustrative drawings. When adding reference numerals to components in each drawing, it should be noted that identical components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

Additionally, when describing the components of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, or order of the component is not limited by the term. When a component is described as being “connected,” “coupled,” or “connected” to another component, that component may be directly connected or connected to that other component, but there is another component between each component. It will be understood that elements may be “connected,” “combined,” or “connected.”

1 is a configuration diagram of a robot according to one embodiment.

Referring to FIG. 1, the robot 100 may include a body 110, a mobile device 120, an inertial sensor 130, a camera 140, and a processor 150.

The body 110 can be implemented in various forms, and various mechanical and electrical components can be built into it.

The moving device 120 may include a device that moves the entire body 110. For example, the moving device 120 includes wheels in contact with the ground and can move the body 110 on the ground through rotation of the wheels.

The moving device 120 may include a device that moves a portion of the body 110. For example, the moving device 120 may include a device that moves the arms and legs of the robot constituting the body 110, and may include a device that moves the head of the robot. By driving this device, the body 110 can not only move parallel to the ground but also move in a direction perpendicular to the ground - for example, the direction of gravity.

The inertial sensor 130 can measure the inertia of the body 110. Inertia can be measured in the form of acceleration or in the form of angular acceleration.

The inertial sensor 130 may include an acceleration sensor or an angular acceleration sensor. If the robot 100 does not include an angular acceleration sensor, the robot 100 can calculate the angular acceleration value using the measured value of the acceleration sensor.

The inertial sensor 130 may be placed at a location where movement occurs in the body 110. For example, the inertial sensor 130 may be placed at the joints of the arms and legs of the body 110, It may also be placed in the neck portion at (110).

The camera 140 can photograph the exterior 10 of the body 110. The robot 100 can analyze captured images of the outside 10 to form a map of the surrounding space and estimate the position of the body 110 within the map.

The robot 100 may include only one camera 140 or may include a plurality of cameras 140.

The processor 150 can reduce motion blur in the image captured by the camera 140 using the measured value of the inertial sensor 130.

Figure 2 is a diagram showing the flow of data and signals for reducing motion blur in a robot according to an embodiment.

Referring to FIG. 2, the processor 150 may obtain an original image (oIMG) from the camera 140 and reduce motion blur in the original image (oIMG).

The processor 150 can reduce motion blur of the original image (oIMG) using the inertial sensor measurement value (sDT) obtained from the inertial sensor 130. The processor 150 may include a deblurring network implemented in the form of a neural network, and motion blur of the original image (oIMG) can be reduced by reflecting the inertial sensor measurement value (sDT) in the deblurring network.

The processor 150 can control the movement of the robot by transmitting a control signal (CTR) to the mobile device 120. Depending on the control signal (CTR), it determines whether to apply the original image (oIMG) to the deblurring network. You can decide. For example, if the control signal (CTR) is a control signal that prevents the robot from moving, the processor 150 can use the original image (oIMG) as is without applying it to the deblurring network. As another example, when the control signal (CTR) is a control signal that causes the robot to move, the processor 150 applies the original image (oIMG) to the deblurring network and uses the resulting image (deblurring image). VSLAM, etc. can be implemented. The processor 150 can estimate the position of the robot in space through analysis of the deblurring image and implement VSLAM using the deblurring image.

Figure 3 is a configuration diagram of a processor according to one embodiment.

Referring to FIG. 3, the processor 150 may include a data mapping module 310, a condition generation module 320, and a deblurring network 330.

The data mapping module 310 may quantize the inertial sensor measurement value (sDT) to generate a quantized measurement value (QDT). Inertial sensor measurements (sDT) can have various resolutions depending on the sensor. The higher the resolution, the more the measurement can have values at various levels, and these various levels of values can complicate calculations.

The data mapping module 310 can quantize the inertial sensor measurement value (sDT) and generate a quantized measurement value (QDT) in order to simplify calculations in subsequent steps and standardize it to prevent sensor-dependent deviations. .

The quantized measurement value (QDT) can be divided into sections with less identical range than the inertial sensor measurement value (sDT). For example, when inertia can be measured in the range of 0 to 100 (U: basic unit), the inertial sensor measurement value (sDT) can be divided into sections of 0.001 (U) - the resolution is 0.001 ( U) -, the quantized measurement value (QDT) is divided into intervals of 10 (U), 0, 10, 20, … , can have a value of 100.

The data mapping module 310 may further perform the function of converting some measurement values. For example, the data mapping module 310 can convert the measured values of the accelerometer into angular acceleration values for each axis.

The condition generation module 320 can generate an inertial reflection parameter (PAR) using a quantized measurement value (QDT).

The condition creation module 320 can be learned as a module that receives inertial sensor measurement values and generates parameters. The condition generation module 320 may include a calculation function including a linear combination of input values and a non-linear function. This calculation function may be learned in advance to generate parameters by receiving inertial sensor measurement values.

The inertial reflection parameter (PAR) generated in the condition generation module 320 can be applied as a condition parameter of the deblurring network 330.

The deblurring network 330 includes blocks in which conditional parameters are used, and inertia reflection parameters (PAR) can be applied to these conditional parameters.

The deblurring network 330 may be trained in advance to receive an original image (oIMG) and generate a deblurring image (pIMG) with reduced motion blur from the original image (oIMG). At this time, the deblurring network 330 can be learned together with the condition generation module 320 to detect the effect of motion blur due to inertia and remove it from the original image (oIMG).

Figure 4 is a configuration diagram of a deblurring network according to an embodiment.

Referring to FIG. 4, the deblurring network 330 may include a plurality of encoder blocks 410a and 410b and a plurality of decoder blocks 420a and 420b.

The original image (oIMG) input to the deblurring network 330 is encoded in a plurality of encoder blocks 410a and 410b and then decoded in a plurality of decoder blocks 420a and 420b to form a deblurring image. (pIMG).

The deblurring network 330 may be implemented in the form of U-Net. U-Net is a network mainly used in image processing and can have a U-shaped network structure.

In U-Net, the encoder blocks 410a and 410b may include Feature-wise Linear Modulation (FiLM), and the inertial reflection parameter (PAR) may be applied to the condition parameter of FiLM.

Figure 5 is a configuration diagram of an encoder block according to one embodiment.

Referring to FIG. 5, the encoder block 410 may include a 2D convolution block 510, a network calculation block 520, a FiLM 530, and a Recified Linear Unit (ReLU) block 540.

The 2D convolution block 510 is a block that performs two-dimensional convolution, and the network calculation block 520 can perform calculations as shown in FIG. 5. ReLU (540) is a rectification linear block that rectifies the input and outputs the rectified value linearly.

And, FiLM 530 can create a linear transformation of the input value as shown in [Equation 1] below, where an inertial reflection parameter can be applied to the condition parameters γ and β.

[Equation 1]

FiLM(x) = γ * x + β

x: input value

γ: Coefficient of linear function (condition parameter)

β: Constant of linear function (condition parameter)

The deblurring network can improve deblurring performance by reflecting inertia in the FiLM part.

Figure 6 is a flowchart of a deblurring network learning method according to an embodiment.

Referring to FIG. 6, the learning device or robot measures a motion blur image captured by the robot in a moving situation, a target image with reduced motion blur in the motion blur image, and the inertial sensor measurement of the robot in a situation in which the motion blur image was captured. Learning data including values can be obtained (S600).

Learning about the deblurring network may be performed by a processor built into the robot, or may be performed by a separate learning device. If the robot itself includes a processor with sufficient computational performance, the deblurring network can be learned using its own processor. Alternatively, if the robot itself does not have a processor with sufficient computational performance, learning about the deblurring network can be performed by a separate learning device.

Even if learning about the deblurring network is performed by a separate learning device, the learning device is directly or indirectly linked to the robot and can acquire motion blur images and inertial sensor measurements generated by the robot. Below, for convenience of explanation, the deblurring network is explained as being learned by a separate learning device.

The target image with reduced motion blur may be created by a separate imaging device or captured by a robot. Here, the motion blur image and the target image may be images taken from the same scene. Additionally, the motion blur image may be an image in which motion blur appears as noise for the scene, and the target image may be an image in which motion blur does not appear because the scene is captured while the scene is still.

The learning device can data map the inertial sensor measurements from the learning data (S602).

Data mapping may include quantization of measurements. The learning device can quantize inertial sensor measurements. The learning device can quantize inertial sensor measurements and generate quantized measurements to simplify calculations in later stages and standardize them to prevent sensor-dependent deviations.

Data mapping may include transformation of measurements. The learning device can convert the measured values of the accelerometer into angular acceleration values for each axis, and allow the angular acceleration values for each axis to be used in a later step.

The learning device can extract the inertial reflection parameter by inputting the quantized inertial sensor measurement value into the condition parameter extraction module (S604). Here, the condition parameter extraction module may be a module for which learning has not been completed, and accordingly, the inertia reflection parameter may not be an optimized parameter.

The learning device uses the inertial reflection parameter as a condition parameter, takes a motion blur image as input, and can prepare a deblurring network that generates a deblurring image with reduced motion blur for the motion blur image (S606). .

And, the learning device can learn the condition parameter extraction module and deblurring network using the learning data so that the deblurring image follows the target image (S608).

For example, when learning of the conditional parameter extraction module has not been completed, the inertia reflection parameter generated from the conditional parameter extraction module may not be an optimized parameter. Additionally, the result (deblurring image) of the deblurring network to which these inertial reflection parameters are applied may not have a high sensitivity to motion blur reduction. Accordingly, differences may occur between the deblurring image and the target image, and the learning device may update the internal parameters of the condition parameter extraction module and/or the internal parameters of the deblurring network to minimize these differences.

The learning device can define a loss calculated as the difference between the deblurring image and the target image, and learn the deblurring network and/or condition parameter module to minimize this loss.

Figure 7 is a flowchart of a motion blur reduction method according to an embodiment.

Referring to Figure 7, the robot can acquire the captured original image and the inertial sensor measurement value of the robot in the situation in which the original image was captured (S700).

And, the robot can process the inertial sensor measurements into data mapping (S702).

The data mapping process may be the same as the data mapping process of the learning device described with reference to FIG. 6. The robot can quantize the inertial sensor measurements and convert the inertial sensor measurements as needed.

Then, the robot can extract the inertial reflection parameter by inputting the quantized inertial sensor measurement value into the pre-learned condition parameter extraction module (S704).

And, the robot can apply the inertial reflection parameter as a condition parameter of the deblurring network learned in advance (S706).

Here, the deblurring network may be implemented in the form of U-Net, and the condition parameter may be a parameter applied to Feature-wise Linear Modulation (FiLM) of U-Net.

U-Net may include an encoder block and a decoder block, and the inertia reflection parameter can be applied to the condition parameter of the encoder block.

Additionally, the robot can generate a deblurring image with reduced motion blur after inputting the original image into the deblurring network (S708).

As described above, according to this embodiment, motion blur can be reduced in images captured by a robot, and through this, the accuracy of VSLAM can be improved.

Terms such as “include,” “comprise,” or “have,” as used above, mean that the corresponding component may be included, unless specifically stated to the contrary, and do not exclude other components. It should be interpreted that it may further include other components. All terms, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains. Commonly used terms, such as terms defined in a dictionary, should be interpreted as consistent with the meaning in the context of the related technology, and should not be interpreted in an idealized or overly formal sense unless explicitly defined in the present invention.

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications and variations will be possible to those skilled in the art without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

Claims (15)

In the deblurring network learning method for generating a deblurring image with reduced motion blur for the motion blur image by deblurring a motion blur image taken in a situation where a robot is moving,
Obtaining learning data including the motion blur image, the target image, and an inertial sensor measurement value of the robot in a situation in which the motion blur image was captured;
Quantizing the inertial sensor measurement value;
Extracting an inertial reflection parameter by inputting the quantized inertial sensor measurement value into a condition parameter extraction module;
preparing a deblurring network that uses the inertial reflection parameter as a condition parameter, inputs the motion blur image, and generates the deblurring image; and
Comprising: training the condition parameter extraction module and the deblurring network using the learning data so that the deblurring image follows the target image,
The target image is an image in which motion blur has been reduced in the motion blur image, and is an image in which motion blur is not displayed by shooting the scene of the motion blur image while still,
The deblurring network is implemented in the form of a U-Net, and the condition parameter is a parameter applied to the U-Net. A deblurring network learning method. According to paragraph 1,
The condition parameter is a deblurring network learning method that is a parameter applied to FiLM (Feature-wise Linear Modulation) of the unit. According to paragraph 1,
The condition parameter extraction module is a deblurring network learning method including a non-linear function. According to paragraph 1,
A deblurring network learning method in which the inertial sensor measurement values include angular acceleration values for each axis of space. According to paragraph 1,
In the step of learning the deblurring network,
A deblurring network learning method for learning the deblurring network so that the loss calculated as the difference between the deblurring image and the target image is minimized. In the motion blur reduction method of deblurring an original image taken in a situation where a robot is moving and generating a deblurring image with reduced motion blur for the original image,
Obtaining the original image, the target image, and an inertial sensor measurement value of the robot in a situation in which the original image was captured;
Quantizing the inertial sensor measurement value;
Extracting an inertial reflection parameter by inputting the quantized inertial sensor measurement value into a pre-learned condition parameter extraction module;
Applying the inertial reflection parameter as a condition parameter of a pre-learned deblurring network;
Including the step of inputting the original image into the deblurring network and then generating a deblurring image with reduced motion blur,
The target image is an image in which motion blur has been reduced from the original image, and is an image in which motion blur does not appear because the corresponding scene of the original image is captured while still,
The deblurring network is implemented in the form of a U-Net, and the condition parameter is a parameter applied to the U-Net. According to clause 6,
The condition parameter is a motion blur reduction method that is a parameter applied to FiLM (Feature-wise Linear Modulation) of the unit. According to clause 6,
Further comprising calculating the angular acceleration value for each axis using the acceleration sensor measurement value of the robot,
A method of reducing motion blur wherein the inertial sensor measurement value includes the angular acceleration value. According to clause 6,
The condition parameter extraction module is,
A motion blur reduction method that extracts the inertial reflection parameters through linear combination of quantized inertial sensor measurements and non-linear function calculation. According to clause 6,
The deblurring network includes an encoder block and a decoder block,
A motion blur reduction method in which the inertial reflection parameter is applied to the condition parameter of the encoder block. In a robot that deblurs a motion-blurred original image taken in a moving situation,
A camera that photographs the outside of the robot's body;
an inertial sensor that measures the inertia of the body;
A moving device that controls movement of the body; and
Quantize the measured value of the inertial sensor, input the quantized measured value into a pre-learned condition parameter extraction module to extract an inertial reflection parameter, and apply the inertial reflection parameter as a condition parameter of a pre-learned deblurring network. , a processor that generates a deblurring image with reduced motion blur after inputting the original image captured by the camera to the deblurring network,
The processor transmits a control signal to the mobile device, and when the control signal is a control signal to move the robot, motion blur is reduced after inputting the original image captured by the camera to the deblurring network. Create a deblurred image,
The inertial sensor is disposed at a location where movement occurs in the body of the robot, including the joints and neck of the arms and legs of the robot body,
The deblurring network is implemented in the form of a U-Net, and the condition parameter is a parameter applied to the U-Net. delete According to clause 11,
A robot wherein the processor estimates the position of the robot in space through analysis of the deblurring image. According to clause 11,
The processor is a robot that implements VSLAM (Visual Simultaneous Localization And Mapping) using the deblurring image. According to clause 11,
The inertial sensor includes an acceleration sensor,
The processor calculates an angular acceleration value for each axis using the measured value of the acceleration sensor, and extracts the inertial reflection parameter using the angular acceleration value.

Images