KR102260519B1 - 3D stereoscopic image conversion method - Google Patents

Abstract

The present invention relates to a 3D stereoscopic image conversion method, by making a certain mark on each object in the process of producing a stereoscopic image, the positional relationship of each object can be accurately identified based on the same, and each object has an accurate positional relationship and can be converted into a 3D stereoscopic image as the stereoscopic image is converted into a 3D image based on this.

Description

3D stereoscopic image conversion method {3D stereoscopic image conversion method}

The present invention relates to a 3D stereoscopic image conversion method, and more particularly, by marking each object in a process of producing a stereoscopic image, it is possible to accurately grasp the positional relationship of each object based on this, and based on this It relates to a 3D stereoscopic image conversion method in which each object can be converted into a 3D stereoscopic image with an accurate positional relationship by converting a stereoscopic image into a 3D image.

With the recent development of technology, as the spread of 3D displays increases, users' demands for 3D images are also increasing. Accordingly, in order to provide an insufficient 3D image, a method of converting an existing 2D image into a 3D image is widely used. However, since the 3D conversion of the 2D image as described above is complex and sophisticated, it is usually performed manually, and thus, there is a problem in that a professional manpower and a lot of work time are required. Therefore, recently, a tool for automatically converting an image is being developed.

Meanwhile, in the stereoscopic image generation, a depth map is obtained by segmentation of a background and an object and assigning a depth value to the separated object or by creating a 3D object and fitting it with a 2D image. Left and right binocular images are generated by rendering using a depth map or a 3D object. The work from object separation to generation of a stereoscopic image needs to be configured in an efficient pipeline form, but each work is currently being performed by a different tool. Therefore, when compatibility between tools is not supported or format support is not supported, there is a problem in that the efficiency and quality of production are deteriorated.

In addition, the conventional 3D stereoscopic image conversion method simply expresses the 3D stereoscopic image in consideration of the vertical position and size relationship of the human body, so there is a limitation in that many errors occur and a sense of separation from reality increases.

On the other hand, the above-mentioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the derivation process of the present invention, and it cannot be said that it is necessarily known technology disclosed to the general public before the filing of the present invention .

Korean Patent Registration No. 10-1489586

One aspect of the present invention is to mark each object in a certain way in the process of producing a stereoscopic image, so that the positional relationship of each object can be accurately identified based on this. Based on this, the stereoscopic image is converted into a 3D image, and each object is provides a 3D stereoscopic image conversion method that can be converted into a 3D stereoscopic image with an accurate positional relationship.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In the 3D stereoscopic image conversion method according to an embodiment of the present invention, a 3D stereoscopic image is generated by using a 2D image prepared in advance for producing stereoscopic image content.

The 3D stereoscopic image conversion method,

extracting a still image at every predetermined frame interval from the two-dimensional image;

extracting an object constituting the still image and displaying a feature point for each extracted object;

generating a depth map using the feature points; and

generating a stereoscopic image of a first still image among a plurality of the still images by using the depth map, and generating a stereoscopic image of a subsequent still image of the feature point;

Extracting the objects constituting the still image and displaying the feature points for each extracted object comprises:

By analyzing the movement amount of feature points for each object included in a plurality of still images generated during a predetermined time period, the object with the least movement is set as a reference object, and the remaining objects except for the reference object in the still image are set as comparison objects. characterized by,

generating a stereoscopic image for a first still image among a plurality of the still images using the depth map, and generating a stereoscopic image for a subsequent still image of the feature point,

generating a 3D image of the first still image by using the depth map;

The separation distance between the feature points of the reference object and the first comparison object is set as the first distance information, and the separation distance between the feature points of the first comparison object and the closest second comparison object is set as the second distance information. According to the amount of change of the ratio of the first distance information and the second distance information in the second still image to the ratio of the first distance information and the second distance information in the first still image, It may be characterized in that a 3D image is generated.

According to one aspect of the present invention described above, by making a certain mark on each object in the process of producing a stereoscopic image, the positional relationship of each object can be accurately identified based on this, and the stereoscopic image is converted into a 3D image based on this. Accordingly, each object can be converted into a 3D stereoscopic image with an accurate positional relationship.

1 is a flowchart illustrating a schematic flow of a 3D stereoscopic image conversion method according to an embodiment of the present invention.
2 to 4 are diagrams illustrating a specific example of a stereoscopic image conversion method using the 3D stereoscopic image conversion method of FIG. 1 .

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0023] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scope equivalents to those as claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a diagram showing a schematic flow of a 3D stereoscopic image conversion method according to an embodiment of the present invention.

The 3D stereoscopic image conversion method according to the present invention relates to a technique for converting a 2D video into a 3D stereoscopic image, and may be performed by an image conversion apparatus.

Here, the image conversion apparatus includes at least one processor and at least one memory like a PC, and software (application) for performing each step of configuring the 3D stereoscopic image conversion method according to the present invention to be described later is installed in advance. can be The image conversion device is driven by software (application) using a processor and a memory.

Specifically, the 3D stereoscopic image conversion method according to an embodiment of the present invention includes the steps of extracting a still image at every predetermined frame interval from a 2D image (S10), extracting an object constituting the still image, and extracting the extracted Displaying a feature point for each object (S20), generating a depth map using the feature point (S30), and generating a stereoscopic image for a first still image among a plurality of still images using the depth map and (S40), and generating a stereoscopic image of a subsequent still image based on the amount of change of the feature point (S50).

In the step (S10) of extracting a still image at every predetermined frame interval from the two-dimensional image, the image conversion apparatus extracts a still image from all frames constituting the two-dimensional image, or extracts a still image at a predetermined frame interval (eg, 5 frames) can be extracted.

In step S20, an object constituting the still image is extracted, and a feature point is displayed for each extracted object.

The image conversion apparatus can classify an object included in a still image by using any one of a variety of known image processing technologies, and since such image processing technology is already known and widely used, a detailed description thereof will be omitted.

Thereafter, the image conversion apparatus may display a characteristic point for each divided object. For example, the image conversion apparatus may detect an outline for each object, shift a window of a predetermined standard within a still image, and detect a feature point for each object using a change amount or a slope of the outline for each object. In addition to this, the image conversion apparatus may extract feature points for each object in the image by using various known feature point extraction algorithms.

At this time, the image conversion apparatus analyzes the movement amount of the feature point for each object included in the plurality of still images generated for a predetermined time period, sets the object with the least movement as the reference object, and the remaining objects except the reference object in the still image. is set as a comparison object.

For example, as shown in FIG. 2 , the image conversion apparatus may analyze a plurality of still images generated for 0 to 2 seconds, detect an object with the smallest movement amount of a feature point, and set it as a reference object. In the illustrated embodiment, the image conversion apparatus may set the first feature point P1 displayed on the door part of the building as a reference object.

Thereafter, the image conversion apparatus may set the remaining objects except for the object set as the reference object in the still image as the comparison object. In the illustrated embodiment, male and female are respectively set as comparison objects, and feature points P2 and P3 may be set and displayed for each of the set comparison objects.

Next, the image conversion apparatus may generate a depth map using the feature point (S30).

In this step, depth information of the extracted feature points is calculated. The image conversion apparatus may find an optimal value that minimizes a reprojection error by repeatedly adjusting 3D coordinates (3D points) and camera parameters. To this end, the image conversion apparatus obtains camera trajectory information (r, t) and three-dimensional coordinates (X) based on a cost function that minimizes the re-projection error. Here, the re-projection error means a difference between a coordinate (xij) corresponding to the feature point (j) of a specific image (i) and a value obtained by projecting a three-dimensional coordinate related to this coordinate into the normal coordinate system of the camera.

At this time, the image conversion apparatus obtains a three-dimensional structure in a small motion and represents the rotation matrix of the camera external matrix P as a small angle approximation. The image conversion apparatus converts the rotation vector into a linear one using a linear interpolation method. And the image conversion apparatus calculates a different rotation vector and position vector for each column based on the linearly transformed rotation vector and position vector.

Depth information of all pixels of the image is restored based on the depth information of the feature points obtained in this way. The image conversion apparatus may acquire depth information of the entire image by using a depth propagation method using the depth information of the extracted feature points as initial depth information.

The image conversion apparatus uses the color information of the image and the three-dimensional coordinates of the feature points to obtain depth information for the entire image. The image conversion apparatus calculates a depth vector D at which the cost function is minimized based on the cost function E(D).

In some other embodiments, the image conversion apparatus analyzes each frame and estimates depth map information using at least one or more of a screen tilt, an object shadow, a screen focus, and an object pattern. For example, the image conversion apparatus may estimate depth map information by determining that an image located at the bottom of the screen within the frame is close and an image located at the top of the frame is far by using a gradient within the frame. Also, the image conversion apparatus may estimate depth map information by determining that a dark part of an object in a frame is far away and a bright part is near by using the shadow of the object. In other words, the shadow is always behind the object and is a method using the principle.

In addition, the image conversion apparatus may estimate depth map information by determining that a clear object is in front and a blurred object is in the back by using the screen focus. In addition, when patterns of the same type are continuously generated by using an object pattern, the image conversion apparatus may determine that a larger pattern is in front of a smaller one and estimate depth map information.

Thereafter, the image conversion apparatus performs a function of moving the depth map information for each pixel in the X-axis direction. The image conversion device adds and subtracts depth map information for each pixel to render a left-view image and a right-view image, and moves each pixel to an X-axis pixel position added by depth map information for each pixel. , so that the added pixel becomes an image for the left field of view. The image conversion apparatus sequentially applies the depth map information from the pixel position, which is the end point of the X-axis coordinate, to the pixel position, which is the start point, in the frame to become an image for the left view.

In addition, the image conversion apparatus moves each pixel to an X-axis pixel position subtracted by the depth map information for each pixel, and the subtracted pixel becomes an image for right viewing. The image conversion apparatus sequentially applies the depth map information from the pixel position of the start point of the X-axis coordinate in the direction of the pixel position of the end point in the frame to the image for the right view.

On the other hand, the image conversion apparatus recognizes the object whose motion is estimated through comparison of the current frame and the reference frame that is the previous or future frame among each frame, and moves the pixel corresponding to the object in the X-axis direction by the depth map information. make it Here, the image conversion apparatus preferably applies the depth map information to both the object and the background in the frame, but in the application of an embodiment of the present invention, the depth map information may be applied only to the object by separating the background and the object in the frame. have.

In addition to this, the image conversion apparatus may generate a depth map from a two-dimensional image using various known techniques.

Thereafter, the image conversion apparatus generates a stereoscopic image of a first still image among the plurality of still images by using the depth map.

Specifically, as shown in FIG. 3 , the image conversion apparatus sets the separation distance between the feature point P1 of the reference object and the feature point P2 of the first comparison object as the first distance information D1a, and the first comparison A separation distance between the object P2 and the feature point P3 of the second comparison object closest to the object P2 is set as the second distance information D2a.

At this time, the image conversion apparatus calculates the ratio of the first distance information and the second distance information in the first still image shown in FIG. 3 , and as shown in FIG. 4 , the second extracted next to the first still image 2 A ratio of the first distance information D1b to the second distance information D2b in the still image is calculated.

Thereafter, the image conversion apparatus uses the amount of change of the ratios (D1a: D2a and D1b: D2b) calculated from each of the first still image and the second still image that is a subsequent image of the first still image in three dimensions for the second still image. create an image

That is, the image conversion apparatus converts the first still image into a three-dimensional image using the depth map, and then converts the subsequent still image into a three-dimensional image using only the change amount of the feature point based on the first still image, so that the image using the depth map Not only can the time required for conversion be reduced, but by marking each object in the process of producing a stereoscopic image, the positional relationship of each object can be accurately identified based on this, and based on this, a 2D image can be converted into a 3D image. , so that each object can be converted into a 3D stereoscopic image with an accurate positional relationship.

In some other embodiments, the image conversion apparatus includes: a collection management module for setting at least one collection criterion of a collection target data source module, a collection target keyword, and a collection target period; a data collection module for collecting big data from the corresponding data source module according to the data collection standard set by the collection management module and storing it in the database module; a morpheme analysis module that analyzes morphemes of big data stored in the database module, classifies them by morpheme, and generates morpheme analysis data; and a data analysis module for calculating statistical values by processing morpheme analysis data with a distributed parallel processing-based statistical analysis algorithm and outputting a weak signal according to the statistical value; a big data analysis-based weak signal derivation system (for convenience of explanation) including; (not shown in the drawing) may be used to extract an illegal image that violates the law from among the received two-dimensional images.

Here, the image conversion apparatus includes: a distributed file system for generating morpheme analysis data by dividing a file name of a received two-dimensional image by morphemes having a meaning, and storing the morpheme analysis data; a data processing unit providing a virtualized database interface to process morphological analysis data existing in the distributed file system based on SQL (structured query language); And by executing a statistical analysis algorithm through the virtualized database interface provided by the data processing unit, from the morpheme analysis data, the number of occurrence frequencies by year (instance frequencies) by year, the number of simultaneous appearances between words by year, the number of appearances by year and by year The word-to-words may include a statistical analysis unit for calculating at least one or more of the processed values from the number of simultaneous appearances.

In addition, the statistical analysis unit, the statistical analysis unit may calculate the rate of increase in the number of occurrences per word based on the number of occurrences per year (instance frequencies) and the number of simultaneous occurrences between words.

In addition, the statistical analysis unit divides the number of occurrences per year (instance frequencies) by the number of documents per year to standardize the number of appearances by year of the words per document to calculate a standardized increase rate of the frequency of appearance per word for the words. can

In addition, the statistical analysis unit calculates the increase rate of the degree of connection degree centrality for each word by using the word-word co-occurrence matrix, and the standardized increase rate of the frequency of appearance for each word and the increase rate of the degree of centrality of the connection degree are higher than a certain portion of the word or A word that is less than a certain lower part can be derived as a weak signal.

Meanwhile, the data collection module may collect big data including unstructured data of at least any one of image, video, voice, sensor, GPS, GIS, and M2M data.

Accordingly, the image conversion apparatus can improve the reliability of the converted stereoscopic image by detecting an image generated in an illegal form through big data analysis through the weak signal derivation system including the configuration as described above.

The technology for providing such a 3D stereoscopic image conversion method may be implemented as an application or implemented in the form of program instructions that may be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the computer-readable recording medium are specially designed and configured for the present invention, and may be known and available to those skilled in the computer software field.

Examples of the computer-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and DVD, and a magneto-optical medium such as a floppy disk. media), and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although the above has been described with reference to the embodiments, it will be understood by those skilled in the art that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the following claims. will be able

P1: Reference object
P2, P3: comparison object

Claims (2)

In the 3D stereoscopic image conversion method for generating a 3D stereoscopic image using a 2D image produced in advance for producing 3D video content, the method comprising:
The 3D stereoscopic image conversion method,
extracting a still image every predetermined frame interval from the two-dimensional image;
extracting an object constituting the still image and displaying a feature point for each extracted object;
generating a depth map using the feature points; and
A stereoscopic image is generated for a first still image that is a first still image among a plurality of the still images by using the depth map, and a stereoscopic image is generated for a second still image that is a subsequent still image based on the change amount of the feature point comprising the steps of
Extracting the objects constituting the still image and displaying the feature points for each extracted object comprises:
By analyzing the movement amount of the feature point for each object included in the plurality of still images generated for a predetermined time period, the object with the least movement is set as the reference object, and the remaining objects except the reference object in the still image are set as the comparison object. characterized by,
A stereoscopic image is generated for a first still image that is a first still image among a plurality of the still images by using the depth map, and a stereoscopic image is generated for a second still image that is a subsequent still image based on the change amount of the feature point comprising the steps of
generating a 3D image of the first still image by using the depth map;
The separation distance between the feature points of the reference object and the first comparison object is set as first distance information, and the separation distance between the feature points of the second comparison object closest to the first comparison object is set as the second distance information, According to the amount of change of the ratio of the first distance information and the second distance information in the second still image to the ratio of the first distance information and the second distance information in the first still image, A 3D stereoscopic image conversion method, characterized in that generating a 3D image.

delete

Images