KR20220114264A - System and method for deep learning based image correction for non-face-to-face training and non-face-to-face rehabilitation - Google Patents

Abstract

Provided is a method for correcting a user training image collected by a user image collection device by an image correction system operated by at least one processor. According to the present invention, feature information of an input image is extracted, and rotation angle learning data, in which a rotation angle of an image is mapped to the extracted feature information, is generated to allow a convolutional neural network to learn. In addition, noise is inserted into the image, and motion blur learning data is generated by mapping the image to the image into the noise inserted. Then, a denoising autoencoder is learned using the motion blur learning data. Moreover, the user training image is received from the user image collection device, and the user training image is corrected using the learned convolutional neural network and the denoising autoencoder.

Description

System and method for deep learning based image correction for non-face-to-face training and non-face-to-face rehabilitation

The present invention relates to a deep learning-based image correction system and method for non-face-to-face training and non-face-to-face rehabilitation.

In order to prevent the spread of the recent epidemic in the community, social distancing is being implemented. Due to social distancing, people tend to avoid working from home, group events, or gatherings.

However, it is difficult to provide a non-face-to-face training service because the rehabilitation training to improve the patient's health condition or health management ability or the training conducted with a trainer in a fitness center is customized to the patient or center member. Nevertheless, due to social distancing, some local governments are providing services for non-face-to-face training as part of community-oriented rehabilitation projects or to promote exercise.

In order to provide non-face-to-face training, the patient or user performs training by viewing the image provided through the device, and by photographing the training patient or user, it is determined whether training is being performed correctly with the patient or user's image at a remote center. . In this way, in order to check whether a patient or a user performs training with an image in a remote center, an image recognition technique must be used.

However, in the current image recognition technique, the motion recognition rate of the patient or user is often lowered due to the movement of the subject, that is, the patient or user and the rotation of the camera.

To solve this problem, attempts to recognize the motion of a patient or user by grafting an artificial intelligence model to the image recognition technique are increasing. The artificial intelligence model drives the image recognition algorithm based on the image and the pixel values of the image, but the recognition rate decreases for the rotated image and the blurred image.

Accordingly, the present invention provides a deep learning-based image correction system and method for non-face-to-face training and non-face-to-face rehabilitation capable of correcting image rotation and motion blur based on deep learning in order to provide a non-face-to-face rehabilitation service.

As a method of correcting a user training image or rehabilitation image collected by a user image collection device by an image correction system operated by at least one processor, which is one feature of the present invention for achieving the technical problem of the present invention,

extracting feature information on the input image, generating rotation angle learning data by mapping the rotation angle of the image to the extracted feature information, learning a convolutional neural network using the rotation angle learning data, the image generating motion blur learning data by inserting noise into the image and mapping the image to the noise-inserted image, learning the denoising autoencoder using the motion blur learning data, and from the user image collecting device and receiving a user training image or rehabilitation image, and correcting the user training image or rehabilitation image using a learned convolutional neural network and a denoising autoencoder.

The training of the convolutional neural network may include extracting RGB pixel values from the image, extracting feature information of the image for each RGB pixel value, and when feature information of the image is input, the mapped rotation angle It may include training the convolutional neural network to output this.

The step of learning the denoising autoencoder may include extracting RGB pixel values from the noise-inserted image, and compressing feature information of the image for each RGB pixel value so that an image with motion blur correction is output. It may include the step of learning the denoising autoencoder.

The step of correcting the user training image or rehabilitation image may include extracting a plurality of images from the received user training image or rehabilitation image, estimating a rotation angle from each image, and rotating by rotating in the reverse direction by the estimated rotation angle Correcting the direction, inserting noise into the image in which the rotation direction is corrected, inputting the noise to the denoising autoencoder, and generating a corrected user training image from each image from which the noise is removed in the denoising autoencoder may include the step of

As a system that operates by at least one processor, which is another feature of the present invention for achieving the technical problem of the present invention, and corrects a training image or a rehabilitation image,

A rotation angle correction unit that receives the training image or rehabilitation image collected by the image collection device, and corrects the rotation angle of a plurality of images extracted from the training image or the rehabilitation image to generate an image with the rotation angle corrected, and the rotation angle and a motion blur correcting unit that receives the image with the rotation angle corrected from the correcting unit, corrects the motion blur of the image with the rotation angle corrected, and generates a training image or a rehabilitation image, wherein the rotation angle correcting unit includes an image and When the image is input based on the rotation angle learning data to which the rotation angle is mapped, it is learned to output the mapped rotation angle, and the motion blur compensator is a motion blur in which the original image and the noise-inserted image are mapped. It is learned to output the original image when the image into which the noise is inserted is input using the training data.

The rotation angle correcting unit may be implemented as a plurality of deep convolutional neural networks, and the motion blur correcting unit may be implemented as a denoising autoencoder.

The rotation angle corrector may extract RGB pixel values from the plurality of images, extract image characteristic information for each RGB pixel, and estimate rotation angles of the plurality of images based on the image characteristic information.

The rotation angle corrector may correct the rotation angle by rotating the plurality of images in a direction opposite to the estimated rotation angle.

The motion blur compensating unit extracts RGB pixel values from the image in which the rotation angle is corrected, inserts noise for each extracted RGB pixel, and reduces the noise-inserted RGB pixels to extract features of the noise-inserted image can do.

The motion blur compensator may restore a characteristic of the image in which the noise is inserted, and remove the added noise component by comparing the restored image with the image in which the rotation angle is corrected.

According to the present invention, by adding an image rotation correction algorithm based on a convolutional neural network to the image recognition system, it is possible to alleviate a decrease in recognition rate due to rotation of a subject according to rotation of a camera.

According to an embodiment, motion blur caused by a user's movement may be mitigated through the denoising autoencoder technique.

According to an embodiment, the recognition rate of the image recognition system may be improved through two types of recognition rate improvement algorithms.

1 is an exemplary diagram of an environment to which an image correction system according to an embodiment of the present invention is applied. 2 is a structural diagram of an image correction system according to an embodiment of the present invention. 3 is an exemplary diagram of a denoising auto encoder according to an embodiment of the present invention. 4 is a flowchart of an image correction method according to an embodiment of the present invention. 5 is a structural diagram of a computing device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Hereinafter, a deep learning-based image correction system and method for non-face-to-face training and non-face-to-face rehabilitation according to an embodiment of the present invention will be described in detail with reference to the drawings. Here, the non-face-to-face training may include home training or non-face-to-face rehabilitation exercise, and the embodiment of the present invention is not limited to any one training.

1 is an exemplary diagram of an environment to which an image correction system according to an embodiment of the present invention is applied.

As shown in FIG. 1 , when the user 100 in the house trains according to the training image provided through the user device (eg, IPTV, monitor, etc.) 200 , the user image collecting apparatus (eg, A camera, CCTV, etc.) 300 collects images of users in training.

Here, the user image collecting apparatus 300 may be implemented to be included in the user device 200 or may be implemented separately. If implemented separately, the user device 200 and the user image collecting apparatus 300 may be interoperable with each other.

The user image collection device 300 captures a training or rehabilitation state of the user as an image, and transmits the captured image to the image correction system 400 in a remote location. The image correction system 400 corrects the image by estimating the rotation angle of the user image collection device 400 based on the deep convolutional neural network, corrects the motion blur based on the denoising autoencoder, and corrects the image create

In the embodiment of the present invention, the image correction system 400 is described as an example that is included in the remote trainer device 500, but is not necessarily limited thereto. Based on the corrected image generated by the image correction system 400, the trainer 600 guiding the training of the user 100 checks whether the user 100 follows the training image with the correct posture, and according to the confirmed result, the user The training posture of (100) can be taught. In addition, in the embodiment of the present invention, for convenience of explanation, a training image among a training image and a rehabilitation image will be described as an example.

The structure of the image correction system 400 for correcting an image in the environment described above will be described with reference to FIG. 2 .

2 is a structural diagram of an image correction system according to an embodiment of the present invention.

As shown in FIG. 2 , the image correction system 400 includes a rotation angle correcting unit 410 and a motion blur correcting unit 420 .

When receiving the original image from the user image collecting device 300 , the rotation angle corrector 410 extracts a plurality of images for each frame constituting the original image. In this case, the original image includes a motion blur element according to the user's inclination on the screen or the user's movement.

Since the rotation angle corrector 410 extracts images from the original image can be performed in various ways, the embodiment of the present invention is not limited to any one method. Also, in the embodiment of the present invention, for convenience of description, extracting images from the original image is described as an example, but the original image itself may be used.

The rotation angle corrector 410 checks identification information mapped to the user image collecting device 300 that has transmitted the original image. Then, coordinate information on which the user image collecting device 300 is installed is checked. Identification information and coordinate information mapped to the user image collecting device 300 are registered in a database (not shown) of the image correction system 400 or registered in a management server that manages the user image collecting device 300 from the outside. may have been

The rotation angle corrector 410 checks RGB pixel values for each of the plurality of extracted images. A method for the rotation angle corrector 410 to check RGB pixel values in an image is already known, and a detailed description thereof will be omitted in the exemplary embodiment of the present invention.

Also, the rotation angle correcting unit 410 extracts image characteristic information according to the size and image characteristics of each of the plurality of extracted images. Here, the image feature will be described as an example including coordinate information indicating the location of objects including the user and surrounding objects, and motion information taken by the user, but is not necessarily limited thereto.

The rotation angle corrector 410 is configured with a plurality of deep convolutional neural networks to extract image feature information from each of a plurality of images. To this end, the rotation angle correction unit 410 composed of a plurality of deep convolutional neural networks is implemented with deep convolutional neural networks based on training data (hereinafter, referred to as 'rotation angle learning data' for convenience of explanation). Train feature information extractors.

Here, the rotation angle learning data includes an image and feature information of the image, and when the image is input to the feature information extractor, that is, the deep convolutional neural network, the deep convolutional neural network is trained so that the feature information of the image is output. A method of learning a deep convolutional neural network using rotation angle learning data is already known, and detailed description thereof will be omitted in the embodiment of the present invention.

Here, a method for extracting image feature information from an image input by a deep convolutional neural network in various ways can be performed in various ways, and thus, the embodiment of the present invention will not be limited to any one method.

Then, the rotation angle corrector 410 outputs a rotation angle of the image based on the image characteristic information. Since the rotation angle correction unit 410 outputs the rotation angle of the image by using the image characteristic information can be executed in various ways, the embodiment of the present invention is not limited to any one method.

The rotation angle correcting unit 410 corrects the rotation direction by rotating the image in the reverse direction based on the output rotation angle. In the embodiment of the present invention, after multiplying the output rotation angle by -1, rotating the image in the reverse direction will be described as an example.

The rotation angle correction unit 410 divides the rotation-corrected image into RGB pixels again, and transmits it to the motion blur correction unit 420 . In this case, the RGB pixels of the rotation-corrected image transmitted to the motion blur compensator 420 are in a state in which rotation correction is performed, but the effect on motion blur remains.

The motion blur compensator 420 extracts features of the image by compressing RGB pixel values of the rotation-corrected image received from the rotation angle compensator 410 . Then, the motion blur compensating unit 420 outputs an image in which motion blur is reduced based on the extracted features. In this case, the motion blur compensator 420 is configured with denoising auto encoders that encode R pixels, G pixels, and B pixels of the image, respectively, in order to output an image with reduced motion blur.

The denoising autoencoder is an encoder that enhances the learning effect by removing noise. It is an autoencoder that takes an image mixed with Gaussian noise as an input and outputs a normal image with Gaussian noise removed. In an embodiment of the present invention, the denoising autoencoder is learned using learning data (hereinafter, referred to as 'motion blur learning data' for convenience of explanation).

Here, the motion blur learning data includes an image affected by motion blur (hereinafter, referred to as a 'motion blur effect image' for convenience of explanation) and an image with reduced motion blur (hereinafter, for convenience of explanation, the 'original image') images') are included in pairs. The denoising autoencoder is trained in advance so that the original image is output when the motion blur effect image is input to the denoising autoencoder.

The motion blur compensator 420 converts the motion blur effect image for each RGB pixel in the pre-trained denoising autoencoder, that is, the image in which the motion blur is alleviated from the image whose rotation angle is corrected by the rotation angle compensator 410, for each RGB pixel. extract each. Then, the motion blur-relieved image is compressed for each extracted RGB pixel, and the motion blur-relieved original image is output.

At this time, the denoising autoencoder in the motion blur compensator 420 will be described with reference to FIG. 3 .

3 is an exemplary diagram of a denoising autoencoder according to an embodiment of the present invention.

As shown in FIG. 3 , the denoising autoencoder of the present invention may be composed of one single artificial neural network or a plurality of artificial neural networks connected in series. And it is designed so that the input size and output size of the denoising autoencoder are the same.

By using the encoder 422 , an image that is multidimensional input data is reduced to a low-dimensional code in the hidden layer 423 to extract features of the input data. Then, using the decoder 424, the low-dimensional code is restored to the first input multidimensional data.

In this case, the network volume of the encoder 422 , that is, the number of parameters is greater than the number of parameters of the hidden layer 423 , and the number of parameters of the decoder 423 is also greater than the number of parameters of the hidden layer 423 . In the embodiment of the present invention, the network volume, ie, the difference in the number of parameters, is expressed by the block length of each component, but it is not necessarily implemented in this way.

In this case, the image input to the encoder 422 means an image in which noise is inserted into the original image by the noise insertion module 421 . A method for the noise insertion module 421 to insert noise into original data is a known technique, and the embodiment of the present invention is not limited to any one method.

By comparing the output image reconstructed by the decoder 423 with the original image to which noise is not added, the denoising autoencoder is learned. Since the dimension of information is reduced compared to the input data in this way, the noise component can be removed. In addition, by tuning the input and output values to be the same as possible in the process, the motion blur compensator 420 can extract the features of the original image well.

A method of correcting a user image collected by the user image collecting apparatus 300 using the above-described image correction system 400 will be described with reference to FIG. 4 .

4 is a flowchart of an image correction method according to an embodiment of the present invention.

As shown in FIG. 4 , the image correction system 400 trains the deep convolutional neural network constituting the rotation angle correction unit 410 with the rotation angle learning data, and learns the denoising autoencoder with the motion blur learning data, respectively. make it (S100).

That is, the image correction system 400 uses the deep convolutional neural network of the rotation angle correction unit 410 so that, when an image included in the rotation angle learning data is input, feature information of the corresponding image included in the rotation angle learning data is extracted. learn

And, when the image correction system 400 is input to the motion blur compensating unit 420 when the noise-inserted image in which the noise is inserted into the original image included in the motion blur learning data is input to the motion-blur compensating unit 420, the image is restored by correcting the motion blur of the noise-inserted image. print out The output image is compared with the original image, and the motion blur compensating unit 420 is trained until the noise-inserted image becomes the same as the original image.

When the image correction system 400 learns the rotation angle correcting unit 410 and the motion blur correcting unit 420, respectively, the remotely installed user image collecting device 300 is a user training image that the user follows while watching the training image. is collected (S110). The user training image collected by the user image collecting device 300 may include not only an image but also identification information of the user image collecting device 300 .

The user image collecting device 300 transmits the user training image collected in step S110 to the image correction system 400 (S120). The image correction system 400 extracts a plurality of images from the user training image received in step S120 (S130). In the embodiment of the present invention, it is described as an example of extracting a plurality of images constituting the user training image and using it to correct the image, but the image itself may be corrected.

The image correction system 400 extracts RGB pixel values from each image extracted in step S130 . Since the method of extracting the pixel values of R, G, and B from the image may be performed in various ways, the embodiment of the present invention is not limited to any one method.

The image correction system 400 extracts the characteristic information of each image extracted in step S130, and estimates the rotation angle of the image based on the characteristic information of the extracted image (S150). In the embodiment of the present invention, estimating the rotation angle of the image based on the degree of inclination of the user image collecting apparatus 300 and the degree of inclination of the user appearing in the image will be described as an example. However, since the method of estimating the rotation angle can be estimated in various ways, the embodiment of the present invention is not limited to any one method.

The image correction system 400 corrects the rotation angle of the image extracted in step S130 based on the rotation angle of the image estimated in step S150 ( S160 ). That is, the image correction system 400 rotates the image in the reverse direction by multiplying the value of the rotation angle estimated in step S150 by -1 to correct the rotational reflection.

When the rotation angle is corrected in this way, the image correction system 400 re-extracts RGB pixel values from the image for which the rotation angle is corrected ( S170 ). This is because the rotation of the image has been corrected, but the effect of motion blur may remain on the image.

Accordingly, the image correction system 400 extracts image features by compressing the pixel values of the R field, G field, and B field extracted in step S170 ( S180 ). To this end, the image features extracted from the RGB pixel values extracted in step S170 are input to the denoising autoencoder learned in step S100, and an image in which motion blur is corrected, that is, noise is removed, is output (S190).

When the motion blur-corrected original image is output as described above, the image correction system 400 may generate an original user training image from the plurality of corrected original images ( S200 ). In the embodiment of the present invention, only the procedure for correcting the image is shown for convenience of explanation, but by providing the original user training image generated in step S200 to the trainer through the trainer's device 200, training that the user takes non-face-to-face You can control the action.

5 is a structural diagram of a computing device according to an embodiment of the present invention.

As shown in FIG. 5 , the image correction system 400 executes a program including instructions described to execute the operations of the present invention in the computing device 700 operated by at least one processor. do.

The hardware of the computing device 700 may include at least one processor 710 , a memory 720 , a storage 730 , and a communication interface 740 , and may be connected through a bus. In addition, hardware such as an input device and an output device may be included. The computing device 700 may be loaded with various software including an operating system capable of driving a program.

The processor 710 is a device for controlling the operation of the computing device 700, and may be a processor 710 of various types that processes instructions included in a program, for example, a central processing unit (CPU), an MPU (Central Processing Unit) It may be a micro processor unit), a micro controller unit (MCU), a graphic processing unit (GPU), or the like. The memory 720 loads the corresponding program so that the instructions described to execute the operation of the present invention are processed by the processor 710 . The memory 720 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 730 stores various data, programs, etc. required for executing the operation of the present invention. The communication interface 740 may be a wired/wireless communication module.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto. is within the scope of the right.

Claims (10)

A method for calibrating a user training image collected by a user image collection device by an image correction system operated by at least one processor, wherein feature information for an input image is extracted, and a rotation angle of the image is mapped to the extracted feature information Generating one rotation angle learning data, Training a convolutional neural network using the rotation angle learning data, inserting noise into the image, and mapping the image to the image into which the noise is inserted to obtain motion blur learning data generating, learning a denoising autoencoder using the motion blur learning data, and receiving a user training image or rehabilitation image from the user image collection device, and using the learned convolutional neural network and denoising autoencoder Comprising the step of correcting the user training image or the rehabilitation image, the image correction method. The method of claim 1 , wherein the training of the convolutional neural network comprises: extracting RGB pixel values from the image; extracting feature information of the image for each RGB pixel value; and inputting feature information of the image. Including the step of training the convolutional neural network to output the mapped rotation angle when the image correction method. The method of claim 2, wherein the learning of the denoising autoencoder comprises: extracting RGB pixel values from the noise-inserted image; Comprising the step of learning the denoising autoencoder so that the image is output, image correction method. According to claim 3, Correcting the user training image or rehabilitation image, Extracting a plurality of images from the received user training image or rehabilitation image, Estimating a rotation angle from each image, and the estimated rotation angle correcting the rotation direction by rotating it in the reverse direction as much as possible, inserting noise into the image whose rotation direction is corrected and inputting it into the denoising autoencoder, and correcting from each image from which the noise is removed in the denoising autoencoder An image correction method comprising the step of generating a user training image or a rehabilitation image. A system that operates by at least one processor and corrects a user's training or rehabilitation image, Receives a training image or a rehabilitation image collected by an image collection device, and rotates angles of a plurality of images extracted from the training image or the rehabilitation image A rotation angle correction unit generating an image with a corrected rotation angle by correcting for, and receiving the image with the rotation angle corrected from the rotation angle correction unit, correcting the motion blur of the rotation angle corrected image, and training Comprising a motion blur compensator for generating an image or a rehabilitation image, The rotation angle correction unit is learned to output the mapped rotation angle when the image is input based on the rotation angle learning data to which the image and the rotation angle are mapped. and wherein the motion blur compensator is trained to output the original image when the noise-inserted image is input using motion blur learning data in which the original image and the noise-inserted image are mapped. The image correction system of claim 5 , wherein the rotation angle correction unit is implemented as a plurality of deep convolutional neural networks, and the motion blur correction unit is implemented as a denoising autoencoder. The rotation angle of the plurality of images according to claim 6 , wherein the rotation angle correcting unit extracts RGB pixel values from the plurality of images, extracts image characteristic information for each RGB pixel, and rotates angles of the plurality of images based on the image characteristic information. Estimating, image correction system. The image correction system of claim 7 , wherein the rotation angle correcting unit corrects the rotation angle by rotating the plurality of images in a direction opposite to the estimated rotation angle. The method of claim 8, wherein the motion blur compensator extracts RGB pixel values from the image in which the rotation angle is corrected, inserts noise for each extracted RGB pixel, and reduces the noise-inserted RGB pixels to insert the noise. An image correction system that extracts the features of the image. The image correction system of claim 9 , wherein the motion blur compensator restores features of the image in which the noise is inserted, and removes the added noise component by comparing the restored image with the image in which the rotation angle is corrected. .

Images