KR20190021009A - Image correction system and method - Google Patents

Abstract

An objective of the present invention is to realize both the convenience and performance by using inertial measurement unit (IMU) sensor data in the existing application. To this end, a method for correcting a video photographed by a user terminal comprises the steps of: measuring and storing displacement data in x-axis, y-axis, and z-axis directions of a user terminal and roll, pitch, and yaw values of the user terminal by using an IMU sensor in photographing a video by the user terminal; transmitting the stored roll, pitch, and yaw values, the displacement data, and video data to a server from the user terminal; and correcting the video data transmitted from the server based on the roll, pitch, and yaw values and the displacement data to be retransmitted to the user terminal. The IMU sensor can be detachably coupled to the user terminal.

Description

[0001] IMAGE CORRECTION SYSTEM AND METHOD [0002]

The present invention relates to a moving image correction system and method, and more particularly, to a system and method for compensating a moving image using an external IMU sensor by minimizing blurring and capturing a moving image after a moving image is captured.

As smart phones become more popular and the performance of built-in cameras in smart phones has improved, more people are using smart phones mainly when taking pictures and videos. In the past, tools such as digital cameras were often used separately, but smartphones replaced them.

As the frequency of use of smartphones increases during photography and movie shooting, many techniques are being developed to compensate for the blur, which is the biggest problem that can occur during shooting. Many smartphone users who are interested in shooting need to improve their shake. The latest mobile phones such as the Galaxy S7 have a built-in anti-shake function that compensates for shaking when taking pictures. As such, techniques for correcting the shaking that occur in photographing are being implemented at a high level. However, the technology that compensates the shake when shooting a smartphone is not yet fully implemented.

Currently, there are two technologies and devices that compensate the shake when taking a video of a smartphone. The first is to use the gimbal unit. When shooting with a smartphone attached to the gimbal unit, 3-axis control technology is used to compensate for the tremors. However, the survey responded that many people do not use it because it is hard to carry because of the large volume and heavy weight of the gimbals. The second is the use of video correction applications. Applications complement the gimbals' disadvantage of being hard to carry. However, when I actually used it, there was a problem with the shake correction performance itself.

Also, referring to FIG. 1, it can be seen from a survey result that, among the people who enjoy moving pictures using a mobile phone at normal times, many people want to minimize the shaking at the time of shooting. Users want a technology that is easy to carry and yet has good anti-shake performance. One of the inconveniences that may be felt when using the camera for the user is that there is a lot of data consumption since the video must be transmitted and received to the server when correcting the video after shooting.

There are two major technologies and products that can correct the video shake correction of smartphone. The first is the gimbal unit and the second is the application type. The gimbal device is good at correcting the video shake, but it is difficult to carry around in everyday life. In addition, when a commercially available application is used, the shake correction performance is lacking in various aspects such as a depth direction shake or a sudden motion. The survey confirms that the present invention is more competitive than existing products.

To investigate the overall camera usage environment, we surveyed 275 online and 59 offline users. The questionnaires were conducted on 58% of males, 42% of males, 16% of teenagers, 56% of females, 14% of females and 14% of females. As a result of the survey, referring to FIG. 1, users who use a smartphone camera consider the core value of the product as simple and convenient. Thus, despite the considerable inconvenience to the shaking, it was confirmed that the use of additional hardware devices was cumbersome and much preferred to software solutions that did not have to carry around separately. That is, referring to FIG. 2, the gimbals device has a size and weight that are burdensome to carry, so that it takes a lot of burden in terms of portability, and the gimbets that implement the proper performance are expensive, The movement of the arm is limited.

In addition, the shake correction through the application only affects the picture quality deterioration because the screen shakes a lot, the angle of view decreases much, and blurring occurs a lot. The stabilization feature added to Google Photos at the end of April 2017 also conforms to the yaw motion correction of the mobile phone, but the roll motion and pitch motion The correction is insufficient, and the angle of view becomes narrow.

We proposed the first idea as a gimbal method that directly controls the camera movement. However, it did not satisfy the convenience and simplicity of the core value of the smartphone, and revised the idea. Next, we proposed a post-processing method using software. Previously, there were applications that corrected images by software. However, there was a clear limit to the way software processing using smartphones only. There is a disadvantage that the application using the calibration method using the built-in IMU sensor of the smartphone has a very high angle of view, and the post-processing method is slow due to the limit of using the smartphone GPU (Graphics Processing Unit) , There is a problem in that the correction is not perfect when viewed visually. In April 2017, Google Photos software from Google, which was relatively well-calibrated, was good at compensating for small fluctuations, but lacking in correction for rolls and pitch.

In other words, although the gimbal method that directly controls the camera movement was the first method proposed to compensate for image blurring during the video shooting of a smartphone, it lacked marketability because it undermines the core value of the smartphone camera, which is convenience and simplicity. In addition, although some applications provide image stabilization by software post-processing, there are limitations in the nature of software post-processing. Accordingly, the present invention has developed a method having both advantages by merely using the IMU sensor data used in the gimbal and fusing the software solution.

There is also an IMU sensor built into the smartphone. However, the built-in IMU sensor uses sensors that focus on reducing the size rather than removing resolution or noise, so the performance varies sensitively with temperature. An external IMU sensor can be used to overcome temperature-sensitive drawbacks.

The present invention aims at realizing both convenience and performance by using IMU sensor data for existing applications in order to solve problems of gimbals and existing applications. To obtain IMU sensor data, an external IMU sensor can be attached when necessary.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention which are not mentioned can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. It will also be readily apparent that the objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

According to another aspect of the present invention, there is provided a method of correcting a motion picture photographed by a user terminal, the method comprising the steps of: measuring a roll, a pitch, a yaw rate, yaw and displacement data of the user terminal in the x, y and z axis directions, and transmitting the stored roll, pitch, yaw and displacement data and the moving image data to the server from the user terminal And correcting the motion picture data transmitted based on the roll, pitch, yaw and displacement data in the server and transmitting the corrected moving picture data back to the user terminal, wherein the IMU sensor is detachably mounted on the user terminal Can be combined.

In the present invention, a method having both merits has been developed by merely using IMU (Inertial Measurement Unit) sensor data used for gimbals and software solution method. Using a computer CPU, a more complex calibration method can be used and calibration time can also be shortened to compensate with a computer CPU rather than a mobile phone. When you take a video from a mobile phone and transfer it to a computer server, it is calibrated on your computer. On the CPU server, as mentioned above, a correction program based on Opencv (Open Source Computer Version) is activated. The calibration result is sent back to the mobile phone and the user can view the corrected image.

Video Stabilizer Google Photo Horizon Camera Our developed product Angle of view usually It's a little wide. narrow Very wide. Degree of correction Inadequate Very good good Very good money Application installation cost 2800 won free Ad O
free Ad O
free Availability possible possible impossible possible convenience Can be calibrated immediately It is somewhat complicated. Can be calibrated immediately Chip type external IMU attachment required

Referring to Table 2, the present invention is corrected using the Ransac Method, and the angle of view is much wider than other existing applications. For a small shake, you can get a similar image with a wider angle of view than Google Photo. In order to compensate for the existing application method which does not compensate for the roll and pitch, we used the method of changing the image ratio and correcting it. By changing the image ratio to the rotation angle, the image can be corrected as if it is not shaken, and the possibility for the pitch is confirmed.

FIG. 1 is a diagram showing a result of a questionnaire survey related to shake correction at the time of photographing a smartphone.
FIG. 2 is a diagram comparing shake correction using a gimbal unit and shake correction using software.
3 is a diagram illustrating a construction of a pitch correction algorithm in a moving image correction method according to an embodiment of the present invention.
FIG. 4 is a graph showing the orthogonality between the MPU 6050 and the Galaxy S6.
FIG. 5 is a diagram illustrating a connection of an external IMU sensor of a moving image correction system according to an exemplary embodiment of the present invention. Referring to FIG.
6 is a flowchart illustrating a moving image correction method according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above objects, features and advantages will become more apparent through the following examples in conjunction with the accompanying drawings.

It is to be understood that the specific structure or functional description is illustrative only for the purpose of describing an embodiment according to the concept of the present invention and that the embodiments according to the concept of the present invention may be embodied in various forms, Should not be construed as limited to these.

The embodiments according to the concept of the present invention can make various changes and have various forms, so that specific embodiments are illustrated in the drawings and described in detail in the specification of the present application. However, it should be understood that the embodiments according to the concept of the present invention are not intended to limit the present invention to specific modes of operation, but include all changes, equivalents and alternatives included in the spirit and scope of the present invention.

Whenever an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but there may be other elements in between It should be understood. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Other expressions for describing the relationship between components, such as "between" and "between" or "adjacent to" and "directly adjacent to" should also be interpreted.

The terminology used in the specification of the present application is used only to describe a specific embodiment and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. It is to be understood that the terms such as " comprises " or " having " in this specification are intended to specify the presence of stated features, integers, But do not preclude the presence or addition of steps, operations, elements, parts, or combinations thereof.

Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

Hereinafter, the present invention will be described in detail with reference to the preferred embodiments of the present invention with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

DJI OSMO MOBILE SNOPPA M1 NEXT BT22GB

3-axis control 3-axis control Stabilization does not work well with 2-axis control. Price of 400,000 won late About 200,000 won 50,000 won middle and later I need some attention and study. Application, and so on. Only 2 hours for one charge Except for mobile phone, it is over 500g. Except for mobile phones, more than 450g. It is very bulky to use in everyday life. It is somewhat bulky for use in everyday life. It is relatively easy to carry around compared to other gym bees.

First of all, referring to Table 2, when a gimbal device which is sold among the gimbal devices that are sold in the market is mainly characterized, the gimbal device that is sold in the market is mainly characterized by a three-axis control, which corrects the shaking well, There is a product group that has little shake correction using only 2-axis control. The first product was excellent in performance, but the price was too expensive for the average user to use, and the gimbals were heavy and bulky, making it difficult to carry around in everyday life. The second product can be purchased at a low price of less than 100,000 won, but since it uses only 2-axis control, it can be seen that the shake correction is performed only in a limited situation.

IMU is a device that measures the speed, direction, gravity, and acceleration of a moving object. It is a sensor-based method. The IMU-based position estimation is a method of recognizing the motion situation of a pedestrian and a moving object by using an accelerometer, a speedometer, a geomachine and an altimeter. The IMU inertial measurement device inserted in the smartphone generally has a built-in three-axis accelerometer and three-axis angular velocity meter, and it can detect the acceleration in the direction of movement, the direction of the lateral direction and the height, and the roll, pitch, yaw angular velocity And it is possible to calculate the velocity and the attitude angle of the moving object (pedestrian) by integrating the acceleration and the angular velocity obtained from the IMU. Here, the roll is a rotation about an axis in a horizontal plane parallel to the moving direction, the pitch is a rotation about an axis in a horizontal plane perpendicular to the moving direction, Lt; RTI ID = 0.0 > vertical < / RTI >

In addition, a specific correction point (Waypoint) can be obtained by using various technologies for more precisely positioning a position in an indoor space, such as a wireless LAN infrastructure, pseudolites, a laser, a sound wave, RFID (Radio Frequency Identification) Can be specified to correct the cumulative error of the IMU positioning.

3 is a diagram illustrating a construction of a pitch correction algorithm in a moving image correction method according to an embodiment of the present invention. Referring to FIG. 3, the moving image correction method using an external IMU sensor according to an embodiment of the present invention divides a moving image into frames, and finds points to be characteristic of each frame. After that, each feature point is matched so that the image is corrected as if it does not shake, and the roll, pitch, and yaw values of the camera are detected using the IMU sensor. Particularly, the part related to yaw is corrected well by finding the above minutia points, so that the motion in the pitch direction is corrected. At this time, the motion of the pitch direction is analyzed, and the rectangle image is transformed into a trapezoid shape and corrected by the Ransac method.

MPU 6050 Galaxy S7 Range Accelerometer: 2g Accelerometer: 2g Gyroscope: ± 250˚ / sec Gyroscope: ± 245˚ / sec Sensitivity Accelerometer: 16384LSB / g Accelerometer: 16384LSB / g Gyroscope: 131LSB / (° / s) Gyroscope: 114LSB / (° / s) Sensitivity change vs
Temperature ± 0.02% / ° C ± 1% / ° C Rate noise density
(Angular random walk) Accelerometer: 0.4 mg / √Hz Accelerometer: 2.0 mg / √Hz Gyroscope: 0.05˚ / s / √Hz Gyroscope: 0.14˚ / s / √Hz

Table 3 shows a table comparing the performance of MPU 6050 and Galaxy S7. Referring to Table 3, in order to verify the necessity of an external IMU sensor according to an embodiment of the present invention, the built-in IMU sensor and the external IMU sensor of the Galaxy s6 and s7 were compared and analyzed. The Galaxy s6 and s7 models are suitable for comparing the performance of the Android OS smartphone with the sensor for the Android shake correction application, which was launched in the next two years. The external IMU sensor was developed and used by InvenSense's MPU 6050 model, which is an inexpensive IMU sensor with a consumer price of about 2000 won. It can be seen that the sensitivity of the built-in IMU sensor is not stable due to the large change in temperature, and that both the accelerometer and the gyroscope have larger noise than the external IMU sensor. The gyroscope's sensitivity appears to be slightly better on the built-in IMU sensor, but it is not a noticeable difference. We also confirmed that there are a variety of external IMU sensors available, and that there are products with better sensitivity and other performance at a price that is not significantly different.

FIG. 4 is a graph showing the orthogonality between the MPU 6050 and the Galaxy S6. In FIG. 4, the orthogonality means that the axes of the accelerometer form 90 degrees with each other. The orthogonality was confirmed by drawing an external IMU sensor and a Galaxy S6 horizontally on the floor and taking an acceleration value for 10 seconds.

The MPU 6050 has a constant acceleration in the Z-axis direction and zero acceleration in the X and Y axes. The fact that the Z axis has a constant acceleration is due to the action of the earth's gravity, and thus the three axes of the MPU 6050 accelerometer are orthogonal to each other.

Galaxy S6 also has a constant acceleration in the Z axis. However, the accelerations in the X and Y axes have non-zero values. This is because the axes are not perpendicular to each other. From the above results, it is confirmed that it is difficult to obtain accurate acceleration information by using the built-in IMU sensor of Galaxy S6. In addition to comparing the performance of the built-in IMU sensors, an Android OS-based smartphone that does not have a built-in gyroscope has been found and found that a significant number of smartphones do not have a built-in sensor. Such smart phones may include, for example, Samsung Galaxy A series, Samsung Galaxy J series, Samsung Galaxy On series, Samsung Galaxy Grand series, LG X series, LG K10, LG Stylus series, LG Class and LG Wine Smart . As a result, the angular velocity obtained through the gyroscope is information necessary to refine the acceleration data and obtain roll, pitch, and yaw, which are the rotation angles of the mobile phone, so that the absence of the gyroscope can not compensate for the shake I make it. Therefore, there is a need for an external IMU sensor.

FIG. 5 is a diagram illustrating a connection of an external IMU sensor of a moving image correction system according to an exemplary embodiment of the present invention. Referring to FIG. In FIG. 5, an external IMU sensor according to an embodiment of the present invention may use an MPU 6050. In order to process the MPU 6050 sensor, it can be connected to the Arduino Mega via I2C communication. When using an external IMU sensor, one thing to note is to remove noise and bias first. Due to the IMU characteristic, there is a problem that the accelerometer is severely noisy and the gyroscope accumulates errors over time. To solve this problem, Digital Motion Processor (DMP) was used. In order to remove the noise, there are complementary filter and Karman filter which are commonly used, but DMP is more precise and there is no need to process separately in MCU.

We also implemented an algorithm to find the offset along with noise and bias elimination, run it for a few minutes with the best horizontal, then calculate the average value to find the offset and collect the corrected data.

Roll, pitch, and yaw also indicate whether the IMU has rotated about each axis. Accelerometers were used together with the gyroscope to obtain accurate roll, pitch and yaw.

In addition, since displacement requires two integrations, it is difficult to obtain accurate data even if a lot of noise is reduced. Therefore, only the displacement at a very short time (0.001 s) was continuously brought in and the position of the screen shot by the camera was predicted using this.

In the moving image correction method using the external IMU sensor according to an embodiment of the present invention, an OTG cable can be used for communication between the external IMU sensor and Android. In the prototype, IMU is connected to Arduino and Arduino is connected via Android and OTG cable. Roll, pitch, yaw and displacement data in x, y, and z axis directions calculated using Arduino are transmitted to Android.

In the moving image correction method using an external IMU sensor according to an embodiment of the present invention, the moving image correction can be implemented in an Android application. It is to store the roll, pitch, yaw and displacement values of x, y, z axis direction of the mobile phone with the moving picture with time. In order to make a motion picture and to compensate for the blurring, it transmits data values received from the video and IMU to the server and receives the corrected image again from the server.

You can use Android studio to make an Android application. Android Studio is a Java based program. In order to implement an application, you need to write a LayOut xml file that composes the actual screen and a Java file that actually gives the application function. do. Depending on the function you are using, you may need to grant permission, which allows you to modify the xml file in the manifest folder.

The anti-shake application can be made up of two activities. This is the main activity (MainAcivitivy) and the album activity (AlbumActivity). In the main activity, there is a code to activate the video recording app. In addition, it contains code to store the text file that the mobile phone moves and angles according to time when it records the video and the video to the Android phone. The album activity contains the video saved in the main activity, the code for sending the corresponding text file to the server, and the code for receiving the video sent from the server. Manifest files can include permissions to use the Internet and the network, and permissions to read and write external storage.

The most important code for implementing an application is as follows.

- Code to send the original video and txt to the server before the compensation in the album activity

 public static void sendFile (Socket sock, File myFile)

         try {

            //Send

            int fileSize = (int) myFile.length ();

            byte [] myByteArray = new byte [fileSize];

            FileInputStream fis = new FileInputStream (myFile);

            BufferedInputStream bis = new BufferedInputStream (fis);

            bis.read (myByteArray, 0, fileSize);

            OutputStream os = sock.getOutputStream ();

            System.out.println ("Sending ...");

            os.write (fileSize &255);

            os.write ((fileSize >> 8) &255);

            os.write ((fileSize >> 16) &255);

            os.write ((fileSize >> 24) &255);

            System.out.println ("sending fileSize:" + fileSize);

            os.write (myByteArray, 0, fileSize);

            os.flush ();

         } catch (UnknownHostException e) {

            // TODO Auto-generated catch block

            e.printStackTrace ();

        } catch (IOException e) {

            // TODO Auto-generated catch block

            e.printStackTrace ();

  }

}

- Code that receives the calibrated video from the server in the album activity

public static void sendFile (Socket sock, File newFile) {

        try {

            newFile.createNewFile ();

            newFile.setWritable (true);

            // Receive

            InputStream is = sock.getInputStream ();

            // newFile: write file

            FileOutputStream fos = new FileOutputStream (newFile);

            BufferedOutputStream bos = new BufferedOutputStream (fos);

            byte [] aByte = new byte [1024];

            int bytesRead;

            while ((bytesRead = is.read (aByte))! = -1) {

                bos.write (aByte, 0, bytesRead);

            }

            is.close ();

            bos.flush ();

         } catch (UnknownHostException e) {

            // TODO Auto-generated catch block

            e.printStackTrace ();

         } catch (IOException e) {

            // TODO Auto-generated catch block

            e.printStackTrace ();

        }

        System.out.println ("receive complete");

    }

- A code that creates a txt file that records your smartphone movements with time so that they match the videos in the main activity.

private class AccSensorListener implements SensorEventListener {

        // Record the sensor value

        public void onSensorChanged (SensorEvent event) {

            if (! videoStarted) {

                videoStarted = true;

                videoStartTime = event.timestamp;

}

     String line = (event.timestamp - videoStartTime) /1000000.0 + "" + event.values [0] + "" + event.values [1] + "" + event.values [2] + "\ n";

            try {

                sensorStream.write (line.getBytes ());

            } catch (IOException e) {

                e.printStackTrace ();

            }

            if (! once) {

                once = true;

                System.out.println (event.timestamp);

            }

         }

The server implementation of the moving image correction method using the external IMU sensor according to an embodiment of the present invention can be constructed using Python in Windows. The role of the server is to receive the video, correct it and send it back to the Android app. The transmission-reception method can use TCP-IP communication with video and txt file sent from the Android app. Correct the video through the video correction program that we implemented the text and video we received. Then, the corrected image is sent back to the Android smartphone. The code for this is as follows.

import socket

import threading

import struct

import os

def recvN (clientSocket, n):

data = clientSocket.recv (n)

while len (data) <n:

data + = clientSocket.recv (n - len (data))

return data

// function to receive file from window

def recvFile (clientSocket, fileName):

f = open ('C: \ Users \ JeongHwan \ Videos \ FileName,' wb ')

sz = recvN (clientSocket, 4)

size = struct.unpack ('i', sz) [0]

print fileName, "size:", size

f.write (recvN (clientSocket, size))

f.close ()

// function to send file from window

def sendFile (clientSocket, fileName):

f = open ('C: \ Users \ JeongHwan \ Videos \ FileName,' rb ')

clientSocket.sendall (f.read ())

f.close ()

// The process of calibrating video received from Android through video program

def serveClient (clientSocket, clientId):

recvFile (clientSocket, "video_% 03d.mp4"% clientId)

recvFile (clientSocket, "imu_% 03d.txt"% clientId)

#TODO receive external imu sensor file

#TODO video correction

os.system ("C: \ Users \ JeongHwan \ Videos \ videostab.exe C: \ Users \ JeongHwan \ Videos \ video_% 03d.mp4-o = C: \ Users \ JeongHwan \ Videos \ corrected_% 03d.mp4" (clientId, clientId, clientId))

sendFile (clientSocket, "corrected_% 03d.mp4"% clientId)

#sendFile (clientSocket, "video_% 03d.mp4"% clientId)

clientSocket.close ()

serverSocket = socket.socket (socket.AF_INET, socket.SOCK_STREAM)

serverSocket.bind (("", 5000))

serverSocket.listen (5)

clientId = 1

while True:

print "waiting ..."

clientSocket, addr = serverSocket.accept ()

print "connected from:", addr

th = threading.Thread (target = serveClient, args = (clientSocket, clientId))

clientId + = 1

th.start ()

Also, data actually observed in real life always includes noise, and it is very important to capture noise. In this case, the Ransac method is used to exclude the noise as much as possible, and the inlier and the outlier must be known in order to use the Ransack method. An inlier refers to data that is contained within a normal distribution, while an outlier refers to data that does not fall into the normal category. The algorithm that determines the distribution line with the greatest number of inliers is called the lush method. The Random method is often used in object matching. There are various applications such as searching the feature points of an object, finding the distribution of each feature point, searching for a specific object in the background through it, or tracking the movement of a specific object.

In addition, it is possible to correct the pitch motion for better image stabilization (video stabilization). It is a method to help find the feature points when correcting by changing the image to the trapezoidal shape by the angle corresponding to the pitch motion.

Image stabilization using the Ransac Method is a method of finding and correcting feature points using the Ransac Method. This is a method used to stabilize the current image by matching the feature points to each other so as not to shake the image.

Also, looking at the additional code for implementing an APPENDIX application,

-Pitch Motion analyze

int trans (float theta, Mat frame)

{

if (frame.empty ())

return -1;

Mat mapX, mapY;

mapX.create (frame.size (), CV_32FC1);

mapY.create (frame.size (), CV_32FC1);

float seta;

seta = (0.62 * theta * theta + 1.39 * theta + 0.62) / (2.77 * theta);

printf ("% f \ n", seta);

y (frame.rows / 2); y ++)

for (int x = 0; x <frame.cols; x ++)

{

mapX.at <float> (y, x) = x;

mapY.at <float> (y, x) = y - ((frame.rows / 2 - y) * x) / (seta * frame.cols - x);

}

y (frame.rows); y ++) < / RTI &gt;

for (int x = 0; x <frame.cols; x ++)

{

mapX.at <float> (y, x) = x;

mapY.at <float> (y, x) = y + ((y - frame.rows / 2) * x) / (seta * frame.cols - x);

}

Mat dstImage (frame.size (), frame.type ());

remap (frame, dstImage, mapX, mapY, INTER_LINEAR);

// imshow ("frame", frame);

imshow ("dstImage", dstImage);

waitKey ();

return 0;

}

string lines [5000];

float y [3000] [3];

float z [3000];

struct float3

{

float a;

float b;

float c;

};

float3 getfloat (string line)

{

string a [3];

int j = 16;

int k = -1;

while (j <line.length ())

{

while (line [j]! = 9)

{

a [k] + = line [j];

j ++;

if (j> = line.length ())

{

break;

}

}

if (line [j] == 9)

k ++;

j ++;

}

float3 f;

f.a = stof (a [0]);

f.b = stof (a [1]);

f.c = stof (a [2]);

return f;

}

int main ()

{

string filePath = "text.txt";

Mat dStImage;

// read File

ifstream openFile (filePath.data ());

int i = 0;

if (openFile.is_open ()) {

while (getline (openFile, lines [i])) {

// cout << lines [i] << endl;

i ++;

}

openFile.close ();

}

int n = 0;

(int l = 5; l <i-1; l ++)

{

if (lines [l] .find ("ypr")! = string :: npos)

{

// printf ("% d", lines [l] .find ("ypr"));

float3 t = getfloat (lines [l]);

y [n] [0] = t.a;

y [n] [1] = t.b;

y [n] [2] = t.c;

z [n] = (y [n] [1] - y [n + 1] [1]) /0.01745;

n ++;

}

}

CvCapture * m_Capture;

m_Capture = cvCreateFileCapture ("play.mp4");

IplImage * scene;

VideoWriter writer;

Mat stabilizedFrame;

int nframes = 0;

// writer.open ("trans.avi", CV_FOURCC ('D', 'I', 'V', 'X'), 120, cvSize (1200, 1600), 1);

for (int k = 0; k <3000; k + +)

{

cvSetCaptureProperty (m_Capture, CV_CAP_PROP_POS_FRAMES, k);

cvGrabFrame (m_Capture);

Mat frame = cvarrToMat (cvRetrieveFrame (m_Capture));

if (frame.empty ())

break;

else

{

trans (z [k], frame);

//writer.write(dstImage);

}

}

return 0;

}

- Ransac Method

#include <string>

#include <vector>

#include <cstdlib>

#include <iostream>

#include <fstream>

#include <sstream>

#include <stdexcept>

#include <opencv2 / core.hpp>

#include <opencv2 / core / utility.hpp>

#include "opencv2 / video.hpp"

#include "opencv2 / imgproc.hpp"

#include "opencv2 / videoio.hpp"

#include "opencv2 / highgui.hpp"

#include "opencv2 / videostab.hpp"

#include "opencv2 / opencv_modules.hpp"

#define arg (name) cmd.get <string> (name)

#define argb (name) cmd.get <bool> (name)

#define argi (name) cmd.get <int> (name)

#define argf (name) cmd.get <float> (name)

#define argd (name) cmd.get <double> (name)

using namespace std;

using namespace cv;

using namespace cv :: videostab;

Ptr <IFrameSource> stabilizedFrames;

string saveMotionsPath;

double outputFps;

string outputPath;

bool quietMode;

void run ();

// void saveMotionsIfNecessary ();

void printHelp ();

MotionModel motionModel (const string &str);

void run ()

{

VideoWriter writer;

Mat stabilizedFrame;

int nframes = 0;

// for each stabilized frame

while (! (stabilizedFrame = stabilizedFrames-> nextFrame ()). empty ())

{

nframes ++;

// init writer (once) and save

if (! outputPath.empty ())

{

if (! writer.isOpened ())

writer.open (outputPath, VideoWriter :: fourcc ('X', 'V', 'I', 'D'),

outputFps, stabilizedFrame.size ());

writer << stabilizedFrame;

}

// show stabilized frame

if (! quietMode)

{

imshow ("stabilizedFrame", stabilizedFrame);

char key = static_cast <char> (waitKey (3));

if (key == 27) cout << endl; break;

}

}

cout << "processed frames:" << nframes << endl

<< "finished \ n";

}

void printHelp ()

{

cout << "OpenCV video stabilizer.nn"

"Usage: videostab <file_path> [arguments] \ n \ n"

"Arguments: \ n"

"-m =, --model = (transl | transl_and_scale | rigid | similarity | affine | homography) \ n"

"Set motion model." The default is affine.nn

"-lp =, --lin-prog-motion-est = (yes | no) \ n"

"Turn on / off LP based motion estimation. The default is no. \ N"

"--subset = (<int_number> | auto) \ n"

"Number of random samples per motion hypothesis. The default is auto.\n"

"--thresh = (<float_number> | auto) \ n"

"Default error to classify match as inlier." The default is auto.\n "

"--outlier-ratio = <float_number> \ n"

"The motion estimation outlier ratio hypothesis. The default is 0.5. \ n"

"--min-inlier-ratio = <float_number> \ n"

"Minimum inlier ratio to decide if estimated motion is OK." The default is 0.1. \ n "

"--nkps = <int_number> \ n"

"Number of keypoints to find in each frame." The default is 1000. \ n "

"--local-outlier-rejection = (yes | no) \ n"

"Perform local outlier rejection. The default is no.nnn"

"-sm =, --save-motions = (<file_path> | no) \ n"

"Save estimated motions into file." The default is no.

"-lm =, --load-motions = (<file_path> | no) \ n"

"Load motions from file. The default is no.nnn"

"-r =, --radius = <int_number> \ n"

"Set sliding window radius." The default is 15. \ n "

"--stdev = (<float_number> | auto) \ n"

"Set smoothing weights standard deviation." The default is auto \ n "

"(ie sqrt (radius)). \ n"

"-lps =, --lin-prog-stab = (yes | no) \ n"

"Turn on / off linear programming based stabilization method. \ N"

"--lps-trim-ratio = (<float_number> | auto) \ n"

&Quot; Trimming ratio used in linear programming based method &

"--lps-w1 = (<float_number> | 1) \ n"

"1st derivative weight. The default is 1. \ n"

"--lps-w2 = (<float_number> | 10) \ n"

"2nd derivative weight. The default is 10. \ n"

"--lps-w3 = (<float_number> | 100) \ n"

"3rd derivative weight. The default is 100. \ n"

"--lps-w4 = (<float_number> | 100) \ n"

"Non-translation motion components weight. The default is 100. \ n \ n"

"--deblur = (yes | no) \ n"

"Do deblurring.\n"

"--deblur-sens = <float_number> \ n"

"Set deblurring sensitivity (from 0 to + inf). The default is 0.1. \ n \ n"

"-t =, --trim-ratio = <float_number> \ n"

"Set trimming ratio (from 0 to 0.5). The default is 0.1. \ n"

"-et =, --est-trim = (yes | no) \ n"

"The default is yes. \ N"

"-ic =, --incl-constr = (yes | no) \ n"

"Ensure the inclusion constraint is always satisfied. The default is no.nnn"

"-bm =, --border-mode = (replicate | reflect | const) \ n"

"Set border extrapolation mode." The default is replicate.\n\n "

"--mosaic = (yes | no) \ n"

"Do consistent mosaicing. The default is no. \ N"

"--mosaic-stdev = <float_number> \ n"

"Consistent mosaicing stdev threshold. The default is 10.0. \ n \ n"

"-mi =, --motion-inpaint = (yes | no) \ n"

"Do motion inpainting (requires CUDA support). The default is no. \ N"

"- mi-dist-thresh = <float_number> \ n"

"Estimated flow distance threshold for motion inpainting. The default is 5.0. \ n \ n"

"-ci =, --color-inpaint = (no | average | ns | telea) \ n"

&Quot; Do color inpainting. The defailt is no.

"--ci-radius = <float_number> \ n"

"Set color inpainting radius (for ns and telea options only). \ N"

"The default is 2.0 \ n \ n"

"-ws =, --wobble-suppress = (yes | no) \ n"

"Perform wobble suppression. The default is no. \ N"

"--ws-lp = (yes | no) \ n"

"Turn on / off LP based motion estimation. The default is no. \ N"

"--ws-period = <int_number> \ n"

"Set wobble suppression period." The default is 30. \ n "

"--ws-model = (transl | transl_and_scale | rigid | similarity | affine | homography) \ n"

"Set wobble suppression motion model (must have more DOF than motion \ n"

"estimation model). The default is homography.\n"

"--ws-subset = (<int_number> | auto) \ n"

"Number of random samples per motion hypothesis. The default is auto.\n"

"--ws-thresh = (<float_number> | auto) \ n"

"Default error to classify match as inlier." The default is auto.\n "

"--ws-outlier-ratio = <float_number> \ n"

"The motion estimation outlier ratio hypothesis. The default is 0.5. \ n"

"--ws-min-inlier-ratio = <float_number> \ n"

"Minimum inlier ratio to decide if estimated motion is OK." The default is 0.1. \ n "

"--ws-nkps = <int_number> \ n"

"Number of keypoints to find in each frame." The default is 1000. \ n "

"--ws-local-outlier-rejection = (yes | no) \ n"

"Perform local outlier rejection. The default is no.nnn"

"-sm2 =, --save-motions2 = (<file_path> | no) \ n"

"Save motions estimated for wobble suppression. The default is no. \ N"

"-lm2 =, --load-motions2 = (<file_path> | no) \ n"

"Load motions for wobble suppression from file. The default is no.nnn"

"-gpu = (yes | no) \ n"

"Use CUDA optimization whenever possible." The default is no.nnn "

"-o =, --output = (no | <file_path>) \ n"

"Set output file path explicitely. The default is stabilized.avi.\n"

"--fps = (<float_number> | auto) \ n"

"Set output video FPS explicitly. By default the source FPS is used (auto). \ N"

"-q, --quiet \ n"

"Do not show output video frames. \ N \ n"

"-h, --help \ n"

"Print help.\n\n"

"Note: some argument configurations lead to two passes, some to single pass. \ N \ n";

}

// motion estimator builders are for concise creation of motion estimators

class IMotionEstimatorBuilder

{

public:

virtual ~ IMotionEstimatorBuilder ()

virtual Ptr <ImageMotionEstimatorBase> build () = 0;

protected:

IMotionEstimatorBuilder (CommandLineParser & command): cmd (command)

CommandLineParser cmd;

};

class MotionEstimatorRansacL2Builder: public IMotionEstimatorBuilder

{

public:

MotionEstimatorRansacL2Builder (CommandLineParser & command, bool use_gpu, const string & _prefix = "")

: IMotionEstimatorBuilder (command), gpu (use_gpu), prefix (_prefix) {}

virtual Ptr <ImageMotionEstimatorBase> build ()

{

Ptr <MotionEstimatorRansacL2> est = makePtr <MotionEstimatorRansacL2> (motionModel (arg (prefix + "model")));

RansacParams ransac = est-> ransacParams ();

if (arg (prefix + "subset")! = "auto")

ransac.size = argi (prefix + "subset");

if (arg (prefix + "thresh")! = "auto")

ransac.thresh = argf (prefix + "thresh");

ransac.eps = argf (prefix + "outlier-ratio");

est-> setRansacParams (ransac);

est-> setMinInlierRatio (argf (prefix + "min-inlier-ratio"));

Ptr <IOutlierRejector> outlierRejector = makePtr <NullOutlierRejector> ();

if (arg (prefix + "local-outlier-rejection") == "yes")

{

Ptr <TranslationBasedLocalOutlierRejector> tblor = makePtr <TranslationBasedLocalOutlierRejector> ();

RansacParams ransacParams = tblor-> ransacParams ();

if (arg (prefix + "thresh")! = "auto")

ransacParams.thresh = argf (prefix + "thresh");

tblor-> setRansacParams (ransacParams);

outlierRejector = tblor;

}

#if defined (HAVE_OPENCV_CUDAIMGPROC) && defined (HAVE_OPENCV_CUDAOPTFLOW)

if (gpu)

{

Ptr <KeypointBasedMotionEstimatorGpu> kbest = makePtr <KeypointBasedMotionEstimatorGpu> (est);

kbest-> setOutlierRejector (outlierRejector);

return kbest;

}

#else

CV_Assert (gpu == false && "CUDA modules are not available");

#endif

Ptr <KeypointBasedMotionEstimator> kbest = makePtr <KeypointBasedMotionEstimator> (est);

kbest-> setDetector (GFTTDetector :: create (argi (prefix + "nkps")));

kbest-> setOutlierRejector (outlierRejector);

return kbest;

}

private:

bool gpu;

string prefix;

};

class MotionEstimatorL1Builder: public IMotionEstimatorBuilder

{

public:

MotionEstimatorL1Builder (CommandLineParser & command, bool use_gpu, const string & _prefix = "")

: IMotionEstimatorBuilder (command), gpu (use_gpu), prefix (_prefix)

virtual Ptr <ImageMotionEstimatorBase> build ()

{

Ptr <MotionEstimatorL1> est = makePtr <MotionEstimatorL1> (motionModel (arg (prefix + "model")));

Ptr <IOutlierRejector> outlierRejector = makePtr <NullOutlierRejector> ();

if (arg (prefix + "local-outlier-rejection") == "yes")

{

Ptr <TranslationBasedLocalOutlierRejector> tblor = makePtr <TranslationBasedLocalOutlierRejector> ();

RansacParams ransacParams = tblor-> ransacParams ();

if (arg (prefix + "thresh")! = "auto")

ransacParams.thresh = argf (prefix + "thresh");

tblor-> setRansacParams (ransacParams);

outlierRejector = tblor;

}

if defined (HAVE_OPENCV_CUDAIMGPROC) && defined (HAVE_OPENCV_CUDAOPTFLOW)

if (gpu)

{

Ptr <KeypointBasedMotionEstimatorGpu> kbest = makePtr <KeypointBasedMotionEstimatorGpu> (est);

kbest-> setOutlierRejector (outlierRejector);

return kbest;

}

#else

CV_Assert (gpu == false && "CUDA modules are not available");

#endif

Ptr <KeypointBasedMotionEstimator> kbest = makePtr <KeypointBasedMotionEstimator> (est);

kbest-> setDetector (GFTTDetector :: create (argi (prefix + "nkps")));

kbest-> setOutlierRejector (outlierRejector);

return kbest;

}

private:

bool gpu;

string prefix;

There are two kinds of software and gimbal device, as mentioned above. In each case, there is a chapter and a disadvantage of each. Software can shoot only with smartphone, but it has a disadvantage that shake correction performance is not very good, and gimbal device is excellent in shake correction performance, but has a drawback in that it is not portable.

In contrast to the above two types of products, in the case of the moving image correction method using the external IMU sensor according to the embodiment of the present invention, not only has excellent portability in an application format but also is corrected using the IMU sensor, Performance.

Particularly, in the case of the present finished product, the problem that the angle of view, which is a problem of existing applications, narrows. It also has better performance by not only stabilizing the yaw direction but also compensating the pitch in the pitch direction.

For the present invention, a business model was constructed based on the 9 Block Business Model Canvas. The customer of the present invention is a smartphone user who is interested in photographic movie shooting, and has a wide range of customers because it is a structure that earns profit through advertisement in an application.

The provided value of the present invention can solve the inconvenience of shaking during shooting of a smartphone, and is easy to use because an external device other than the IMU sensor is not needed, and the performance of the existing products is improved by using the IMU sensor . Concretely, it is possible not only to correct the yaw direction but also to correct the shake in the roll and pitch directions, to minimize the blurring phenomenon, and to perform the stable correction even in the sudden fluctuation. Do.

In addition, existing companies are providing services through ordering, manufacturing and supplying hardware devices such as gimbals, or applications that do not utilize IMU data. Therefore, since the service is provided to the customer through the App Store and the customer is more accessible than the hardware channel, the external IMU is sold through the Internet homepage. We can maintain competitiveness through continuous application update through customer feedback. In addition to shake correction, IMU data is used to develop various functions. 3D images, panoramas, 3D emoticons, and more. And because the update is based on software development, it does not cost a lot.

In addition, the present invention enables revenue generation by allowing an advertisement of an external company to be watched while a moving image is being corrected in a server, selling an IMU sensor near a cost, and providing free functions such as a shake correction, To attract investors. In order to maintain the good calibration performance of the present invention, it is important to secure human resources with programming ability. In order to generate profits, it is important to establish a friendly relationship with the IMU sensor manufacturing company and advertisers.

In addition, IMU sensor data and feature point search are used to develop and provide shake correction algorithms, to sell IMU sensors over the Internet, to deal with companies that want to advertise, and to negotiate with IMU sensor suppliers It is important. From the business viewpoint of the present invention, it is expected that the costs of developing IMU sensors and distributing them to consumers, and the development, maintenance, and repairing of applications are expected.

The motion compensation method using an external IMU sensor according to an embodiment of the present invention uses an IMU sensor to solve the problem that an existing application does not correct the roll and the pitch. It is possible to perform post-processing by correcting the image ratio using the angle data obtained from the IMU sensor, and by changing the image ratio captured for the image that is staggered in the pitch direction, The image can be corrected by a method of trimming. To apply this method to an image, a clear relationship between angle and aspect ratio change must be found and applied to the image. It is also necessary to implement a method of correcting rolls in the same way.

You can use your computer graphics card in the calibration process because you are using a computer CPU. CUDA method can be added to server coding to improve image correction performance and speed.

It is also possible to adjust the values of the trim ratio, mosaic, deblurring sensitivity, smoothing, sliding window inlier ratio, outlier ratio, etc. The image correction is different. Even if the same value was used, the image correction performance was changed according to the shake degree and the shake direction. Therefore, if the optimum value of the above values can be found depending on the shaking speed, angle, and direction of the image, a more clearly corrected image can be obtained.

In addition to video stabilization, you can create a panoramic picture by matching the feature points with the angle of the smartphone and the captured image. Then, 3D reconstruction can be performed by photographing the shape of an object as an image. 3D imaging is possible through smartphone location and image feature point matching.

6 is a flowchart illustrating a moving image correction method according to an embodiment of the present invention. 6, first, at step S100, a motion picture is taken to a user terminal, and at the same time, a roll, a pitch, a yaw value of a user terminal, and stores the displacement data in the x, y, and z axis directions together with the moving image data. Next, in step S200, the displacement data and the moving image data stored in the user terminal are transmitted to the server. In step S300, the transmitted moving image data is corrected using the displacement data transmitted to the server. At this time, the correction of the moving image data is performed by searching the characteristic points of the objects photographed in the transmitted moving image data, finding the distribution of each characteristic point, locating a specific object within the background of the transmitted moving image data, In addition, it is possible to more easily find the feature points of the objects by transforming the image into a trapezoidal shape by the corresponding angle with respect to the pitch motion in the transmitted moving image data. Then, in step S400, the server transmits the corrected moving image data to the user terminal.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the scope of the invention as defined by the appended claims. will be. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (6)

A method for correcting a moving image captured by a user terminal,
Pitch, and yaw values of the user terminal and displacement data in the x, y, and z-axis directions of the user terminal using the IMU sensor when capturing a moving image with the user terminal ;
Transmitting the stored roll, pitch, yaw and displacement data and the moving image data to the server at the user terminal; And
And correcting the moving picture data transmitted based on the roll, pitch, yaw and displacement data at the server, and transmitting the corrected moving picture data back to the user terminal,
The IMU sensor is detachably coupled to the user terminal
How to calibrate your video.
The method according to claim 1,
The correction of the motion picture data may be performed,
Searching the minutiae points of the objects photographed in the transmitted moving image data, searching the distribution of each of the minutiae points, locating a specific object within the background of the transmitted moving image data, or tracking the movement of a specific object, And correcting the moving image.
3. The method of claim 2,
The feature points of the photographed objects include,
Wherein the moving image is searched by transforming the image into a trapezoidal shape by a corresponding angle with respect to a pitch motion in the transmitted moving image data.
A user terminal having a moving image capturing section for capturing a moving image;
A pitch and a yaw value of the user terminal and a displacement of the user terminal in x, y, and z-axis directions when capturing a moving image with the user terminal, An IMU sensor for measuring data;
And a moving picture corrector for correcting the moving picture data transmitted from the server based on the roll, pitch, yaw value and the displacement data, and transmitting the corrected moving picture back to the user terminal.
5. The method of claim 4,
Wherein the correction of the moving picture data by the moving picture correction unit
Searching the minutiae points of the objects photographed in the transmitted moving image data, searching the distribution of each of the minutiae points, locating a specific object within the background of the transmitted moving image data, or tracking the movement of a specific object, And correcting the moving image.
6. The method of claim 5,
The feature points of the photographed objects include,
Wherein the moving image is searched by transforming the image into a trapezoidal shape by a corresponding angle with respect to a pitch motion in the transmitted moving image data.

Images