KR102259401B1 - Composing Method of Augment Reality Additional Information in Drone Moving Picture - Google Patents

Abstract

The invention relates to a method of synthesizing augmented reality additional information in a drone video. The invention relates to the method for synthesizing augmented reality additional information in a drone video, which comprises the steps of: parsing drone video data and rendering the data by a video renderer; parsing an augmented reality additional information file to derive augmented reality additional information; extracting camera parameters from a drone video to extract camera movement information, and extracting 3D spatial information composed of feature points of each image; modeling augmented reality additional information in the extracted 3D space; performing time-axis-based mapping of camera movement information to properties of a viewing camera for 3D rendering; and repeating a process of, generating 3D augmented reality additional information to be displayed in the extracted 3D space by 3D spatial rendering of the augmented reality additional information, and synthesizing the 3D augmented reality additional information in an image formed by rendering the drone video, from the start to the end of the drone video. By using the present invention as described above, it is possible to conveniently edit the augmented reality additional information, and it is possible to synthesize the augmented reality additional information into the drone video.

Description

Method of synthesizing augmented reality additional information in drone video {Compositing Method of Augment Reality Additional Information in Drone Moving Picture}

The present invention relates to a method of synthesizing augmented reality additional information in a drone video.

The drone is a kind of small flight device as shown in FIG. 1 and is mainly used for broadcasting, monitoring, or surveying by mounting a camera, and has been widely used to take aerial images as shown in FIG. 2 .

In addition, recently, as drones become cheaper and smaller, there are increasing cases in which individuals purchase drones, shoot various videos using the drone, upload them to YouTube or Facebook, etc. and share the videos with friends.

So, as shown in the upper and lower left drawings of FIG. 3 , a red box or ellipse is added to a specific person on the toposurface in the drone video to keep following this specific person in the video, or a video of a golf course as shown in the right drawing of FIG. The distance from to the hole cup is continuously displayed, or a circle (white circle, red circle, green circle indicating the starting point depending on the type of golfer (male, female, elderly, etc.) The need to insert various augmented reality additional information (eg text, figure, image, etc.) into a video shot with a drone to express intention or explain information to others is emerging.

However, inserting augmented reality additional information into the drone video is to add augmented reality additional information to this target while following a specific target (eg, a specific person in FIG. 3, a specific place in FIG. 4) whose location is continuously changed on the screen. Therefore, it is not possible to apply the method of displaying the augmented reality additional information in a fixed specific position on the screen, such as adding subtitles in a video such as a movie.

Korean Patent Publication No. 10-1843025 (registered on March 22, 2018)

In view of the problems of the prior art as described above, the present invention provides a method of synthesizing augmented reality additional information in a drone video.

The present invention for solving these problems, in a method of synthesizing augmented reality additional information in a drone video,

Parsing the drone video data and rendering by a video renderer;

parsing the augmented reality additional information file to derive augmented reality additional information;

extracting camera parameters from a drone video to extract camera movement information, and extracting three-dimensional spatial information composed of feature points of each image;

modeling augmented reality additional information in the extracted three-dimensional space;

performing time-axis-based mapping of the camera movement information to properties of a viewing camera for 3D rendering;

3D augmented reality additional information to be displayed in the extracted 3D space is generated by 3D spatial rendering of the augmented reality additional information, and the 3D augmented reality additional information is synthesized with the image formed by rendering the drone video. It characterized in that it includes the step of repeating the process from the start to the end of the drone video.

By using the present invention as described above, it is possible to conveniently edit the augmented reality additional information, and it is possible to synthesize the augmented reality additional information into the drone video.

1 shows the shape of a general drone.
2 shows an image taken by a drone.
3 shows a drone image to which augmented reality additional information is added in the present invention.
Figure 4 shows a comparison of the original video and edited video taken by drones in the present invention.
Figure 5 shows the playback of the video when the augmented reality additional information is inserted into the drone video in the present invention.
6 is a diagram illustrating a process of extracting camera parameters, movement line information, and 3D spatial information from a drone video according to the present invention.
7 is a diagram illustrating point clouds, movement line information, and 3D spatial information extracted from a drone video according to the present invention.
8 shows the structure of a meta file used in the present invention.
9 is a schematic conceptual diagram of synthesizing augmented reality additional information in a drone video according to the present invention.
Figure 10 shows the mapping of the position of the drone camera and the viewing camera of the 3D model renderer according to the present invention.
11 shows a detailed process of synthesizing augmented reality additional information in a drone video according to the present invention.

The present invention will now be described in detail with reference to the drawings.

In the case of a video shot by a drone, in most cases, buildings, trees, rivers, various facilities, etc. that exist on the terrain are photographed. Therefore, when displaying information in the captured image, as shown in FIG. 3, various information is taken in consideration of the topographical surface. It is natural to insert

For example, the starting point is indicated according to the augmented reality additional information (distance to the hole cup, type of golfer (male, female, elderly, etc.) When a white circle (white tee, for general men)/red circle (red tee, for general women)/green circle (green tee, for the elderly)) is inserted, it becomes an edited drone shooting video as shown on the right side of FIG. 4, such augmented reality If the insertion method is applied to the entire drone video, augmented reality additional information is inserted into the video taken while the drone is moving as shown in FIG. 5, and when the inserted video is played, even if the screen continues in the video, a specific location (eg, red A red circle appears from the start time specified in the circle position) to the end time.

So, for example, in the existing golf course guide drone shooting video, only the drone video about the golf course appears, so it is not possible to know information about the starting point for each golfer type in the video or the distance from that point to the hole cup. , when the present invention is applied, a video in which augmented reality additional information such as the starting position and the distance to the hole cup according to the type of golfer is inserted is played, and the corresponding augmented reality additional information is displayed at each specific location for a specific time. While grasping the overall view of the golf course through the video, it is possible to easily grasp various additional information (for example, information about the starting location of each type of golfer or the distance to the hole cup) through the augmented reality additional information. .

Next, the process of extracting camera parameters, point clouds, camera movement information, and 3D spatial information from a drone video (original video) in the present invention will be described in detail with reference to FIGS. 6 and 7 .

First, an original video shot using a camera mounted on the drone is prepared (S1). Here, still photos of the original video are shown in the upper drawing of FIG. 7 .

Next, through this original video, the internal parameters (focal length, angle of view, lens curvature, etc.) of the camera mounted on the drone and the external parameters (the camera position based on the world coordinate system, the line of sight (LOS) of the camera) Up vector (Up vector), etc.) is extracted (S2).

In this internal and external parameter extraction, camera information used for shooting is included in all captured images, and when multiple images of the same object are combined, internal and external parameters of the cameras used for shooting and 3D information of the object to be shot are obtained. Existing known techniques that can be restored will be used.

Now, the movement information of the drone camera is extracted (S3). Here, the moving line of the camera is shown in the middle drawing of FIG. 7 . The position of the drone corresponding to each image is extracted based on the camera position and the line of sight (LOS) of the camera among the information extracted above, and the arrangement on the time axis corresponds to the camera movement line shown in the middle of FIG. 7 .

Next, a point cloud that is a set of feature points of the image captured by the drone camera is extracted (S4). A point cloud is shown in the middle diagram of FIG. 7 . Cloud points are extracted during multi-view image processing, and trigonometry is basically used.

That is, when an object is photographed using two cameras with a certain distance, the two cameras and the object form a triangle, and the distance between the two cameras and the crosshair for the camera to photograph the object can be calculated in advance. It is possible to find out the distance and angle from each camera to the object by the general trigonometry principle.

In addition, the feature points in the image appear in common in each image, and the points that are well distinguished and can be matched are extracted and utilized. In this case, the algorithm used for extracting the feature points uses various well-known methods.

In addition, cloud points are points formed by collecting all the feature points extracted from the image. Since the distance to the feature points is known based on the photographed camera, if all of them are collected and high, it becomes the three-dimensional spatial information formed by the object to be photographed.

Finally, 3D spatial information constituted by these feature points is extracted (S5). The 3D spatial information extracted from the original video is shown in the lower drawing of FIG. 7 .

At this time, the 3D spatial information should be described based on the world coordinate system. Since the movement information of the camera based on the world coordinate system is extracted as described above and distance information between the camera and the feature point is extracted as described above, it is possible to extract the coordinates of the feature points based on the world coordinate system. In addition, the Delaunay triangulation method is widely used to transform the spatial information composed of these feature points into surface information.

As a result, when the camera movement line, point cloud, and 3D spatial information are extracted as described above, the point cloud of the terrain and the camera movement line are displayed on the extracted 3D space as shown in the lower part of FIG. 7 .

On the other hand, when adding augmented reality additional information to a drone shot video or removing or modifying added augmented reality additional information, it is necessary to manually apply the functions of an editing tool such as Adobe After Effect and perform several hours of work. Since it is difficult to change (add, remove, modify) additional information, a more convenient method of changing augmented reality additional information is required. A meta file (management file) (SARI: Synchronized Augmented Reality Information) structure for drone video will be described.

8 is a meta file structure used in the present invention.

This meta file includes the name of the original video file (1), the GPS coordinates of the camera used for shooting (2), and the internal and external parameters of the camera used for shooting (focal length, lens principal point, asymmetry coefficient, etc.) (3), augmented reality additional information (figure, text, table, number, graph, image, etc.) (4-1, 4-2 ....), 3D coordinates where each augmented reality additional information should appear (5-1, 5-2 ....), time synchronization information (6-1, 6-2 .....) in which each augmented reality additional information should appear is included.

Here, the GPS coordinates 2 of the camera have the same information as the path traveled by the drone. This is used to provide a convenient function when playing the augmented reality video, and is mainly used to synchronize the starting point of the video with the user's location based on the user's location.

And, the time synchronization information (6-1, 6-2.....) describes at what point in time the augmented reality additional information (text, figure, image, etc.) inserted into the video should appear and disappear.

So, when the augmented reality additional information is changed (added, removed, or modified), the corresponding augmented reality additional information area (4-1, 4-2 ....) to be changed in the meta file and the augmented reality additional information appear Because it is necessary to change the three-dimensional coordinate area (5-1, 5-2 ....) that should be, and the time synchronization information area (6-1, 6-2...) where the augmented reality additional information should appear. , it becomes possible to conveniently change the augmented reality additional information.

Next, a method of synthesizing augmented reality information in a drone video in real time using the meta file according to the present invention will be described with reference to FIGS. 9, 10 and 11 .

9 shows a schematic configuration of a real-time synthesis method according to the present invention.

Drone video data is rendered by a video renderer, and augmented reality additional information (augmented reality additional information (4-1, 4-2 ...)) is extracted from the meta file shown in FIG. When rendered by a renderer and synthesized in real time based on a timeline, a drone video (augmented reality video) containing augmented reality additional information (CONFIELD video at the bottom of FIG. 9) is formed.

Now, with reference to FIG. 10, the mapping of the position of the drone camera used in the present invention and the viewing camera of the 3D renderer will be described.

As described in FIG. 6, when the movement line information (position information for each time) of the drone camera is extracted from the drone video (original video) and the 3D spatial information is extracted, the drone shooting trajectory is formed as shown in the upper diagram of FIG. The three-dimensional space of the drone video as shown in the lower drawing of 10 is formed.

In addition, as described with reference to FIG. 6, when the parameters of the drone camera are extracted, the position of the camera, the crosshair of the camera, the upvector of the camera, and the like are extracted.

And, the camera information thus extracted is the position, viewing angle, and navigation path information of the viewing camera of the 3D renderer for rendering the extracted three-dimensional space, as shown in the lower diagram of FIG. 10 . make use of

At this time, the 3D spatial information to be rendered includes 3D spatial information extracted from the image and 3D additional information modeled by the augmented reality additional information extracted from the augmented reality additional information management file as shown in FIG. will do Here, in the lower drawing of FIG. 10, a blue box indicates 3D spatial information extracted from a video, and an ellipse, a bunker display, and a cylinder indicates 3D additional information extracted from the additional information file.

Then, the two images (image generated by the video renderer and the image generated by the 3D renderer) generated on the time axis in this way can be synthesized based on the mapping time.

Next, a method of synthesizing augmented reality additional information in a drone video in real time in the present invention will be described in detail with reference to FIG. 11 .

First, a drone video file is prepared (S11).

Then, the drone video file is parsed (S12), and since the augmented reality additional information is synthesized from the beginning to the end of the video, the current time (T) = 0 and the end time (Te) = (total length of the video) is set. do (S13).

Now, by setting the first time point (T) = 0 (S14) and rendering a video for this view point by the video renderer (S15), an image for this time point (T=0) is formed.

Meanwhile, the additional information file (meta file or SARI file) is prepared (S21), parsed (S22), and after modeling the augmented reality additional information in the three-dimensional space derived in FIG. 6 (S23), the additional information file Viewing camera properties (viewing position, viewing direction), movement path (viewing position, viewing direction) for 3D rendering of camera movement information (position, aim line, up vector) stored in drone video recording navigation path)) (S24).

Now, when the viewpoint (T) = 0 is set (S25), and the 3D space rendering is performed for the augmented reality additional information for this viewpoint (S26), the augmented reality 3D additional information to be displayed in the 3D space is It is created from the viewpoint of the viewing camera set earlier.

When the augmented reality 3D additional information generated in this way is synthesized with the image formed by video rendering in step S15, the image of the time point (T = 0) formed through step S15 is formed through step S26. The augmented reality additional information at the time point (T=0) is synthesized (S30).

Next, it is determined whether the current time has reached the end time (whether the current time is less than the end time) (S31), and if the current time is not the end time, T is set to 1 to perform the above process for the next time point T. By increasing the number (S32, S33), the process returns to steps (S15) and (S25), respectively, and if this process is performed until the end time, the process of synthesizing (inserting) augmented reality additional information for the entire drone video is completed. will become

Although the embodiments of the present invention have been described in detail above, it should be noted that the present invention is not limited to these embodiments, and various modifications or changes are possible without departing from the spirit of the present invention.

Claims (4)

In a method of synthesizing augmented reality additional information in a drone video,
Parsing the drone video file and rendering by a video renderer;
parsing the augmented reality additional information file to derive augmented reality additional information;
extracting movement line information of the camera through the camera parameters extracted from the drone video, and extracting 3D spatial information constituted by the feature points of each image from the drone video;
modeling augmented reality additional information in the extracted three-dimensional space;
Time-axis-based mapping of the moving line information of the camera to the properties of the viewing camera for 3D rendering;
At each time point from the start to the end of the drone video, 3D augmented reality additional information to be displayed in the extracted 3D space from the viewpoint of the viewing camera is generated by 3D spatial rendering of the augmented reality additional information, A method of synthesizing augmented reality additional information into a drone video, comprising repeating the process of synthesizing the generated 3D augmented reality additional information with an image formed by rendering the drone video. The method of claim 1,
The property of the viewing camera is a method of synthesizing augmented reality additional information to a drone video, characterized in that it includes a viewing position, a viewing direction, and movement path information. The method of claim 1,
The method of synthesizing the augmented reality additional information in the drone video, characterized in that the camera movement information includes the camera position, the line of sight and the up vector. The method of claim 1,
The method of synthesizing augmented reality additional information in a drone video, characterized in that the feature points are displayed as a point cloud on the image.

Images