KR20210027918A - Method of making 3D image FOR HMD from 2D image - Google Patents

Abstract

A method for generating a three-dimensional image for an HMD from a two-dimensional image according to the present invention includes the steps of: generating an equirectangular first panoramic image with respect to a 360-degree space; extracting a depth image from the first panoramic image; generating a left-eye image by matching the first panoramic image and depth image to a square cubemap comprising a front surface, left-side surface, rear surface, right-side surface, planar surface and bottom surface; dividing, from the first panoramic image, an area overlapping consecutive images of the front surface, left-side surface, rear surface and right-side surface of the cubemap image into two-dimensional images having predetermined uniform intervals; and generating a first stereoscopic right-eye image by means of a predetermined algorithm with respect to the divided two-dimensional images. Accordingly, simple 3D digital cartoon content can be provided without significantly consuming computing power.

Description

Method of making 3D image FOR HMD from 2D image}

The present invention relates to generating a 3D image, and more specifically, to a method of generating a 3D image from a 2D image for generating a 3D image using a single 2D image and a depth image. .

Various devices and software are being developed to provide virtual reality. Virtual reality is a distorted three-dimensional reality provided by a computer, and it focuses on reducing the sense of heterogeneity with the real space.

Virtual reality is mainly provided through a head mount device (HMD), and since the user's field of view is blocked by the HMD, the user cannot perceive other images other than the provided virtual space image. Therefore, there is an advantage of allowing the user to have a sense of immersion in a given image.

In the case of a digital cartoon using a continuous image that composes a story, it is composed of a continuous image and conveys the flow of the story to the user. In recent years, it is evolving into a service to deliver a sense of realism as if the user is located in the inner space of the story of an actual digital cartoon beyond a service in the form of a flat image. Sphere Toon can be defined as a cartoon platform that provides digital cartoon content such as webtoon in VR (Virtual Reality). Install the speartoon app on your smartphone and configure the screen so that you can enjoy it by attaching a Head Mount Device (HMD). In this case, the characters, backgrounds, objects, etc. of the webtoon provided by the speartoon application are expressed, and layers for each object are all arranged differently.

In order to implement such a spear toon, a method of arranging objects on each layer after setting each layer in advance is mainly used. The three-dimensional implementation method of this type corresponds to a method of arranging each object after drawing in consideration of the arrangement in advance. In order to display a two-dimensional image in three dimensions, the background and each object must be configured separately. In order to change the screen composition, each object must be reconfigured, so there is a limit to freely changing the screen composition.

As another method of implementing a 3D image, a method of transmitting each image to both eyes of an HMD wearer based on images obtained through two cameras is also used.

According to the above methods, it is necessary to consider the point of having to go through several stages of screen composition in order to change the fixed 2D image to 3D or the point of using a method of obtaining an image in a stereo method.

If a two-dimensional image drawn by hand can be easily applied as a three-dimensional image, digital cartoon content using HMD can be provided in a wider variety.

The method of generating a three-dimensional image from a two-dimensional image according to an embodiment of the present invention is based on the degree of three-dimensional effect to be obtained according to the principle of binocular parallax, a principle of human stereoscopic perception, from a single 2D image using a computer device. The purpose of generating a three-dimensional image by extracting the binocular parallax angle from the depth image and creating an image with a different viewing angle by rearranging the original image according to the extracted angle, providing the deformed image and the original image as a pair of stereoscopic images. do.

The object of the present invention is not limited to those mentioned above, and other objects that are not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

A method of generating a 3D image for an HMD from a 2D image according to an aspect of the present invention for solving the above technical problem includes: generating a first equirectangular panoramic image for a 360-degree space; Extracting a depth image from the first panoramic image; Generating a left-eye image by matching the first panoramic image and the depth image with a square cube map consisting of a front, left, rear, right, plane and bottom surface; Dividing a region overlapping the continuous images of the front, left, rear, and right sides of the cubemap image from the first panorama image into two-dimensional images of equal intervals set in advance; And generating a first stereoscopic right-eye image using a preset algorithm for the divided 2D image.

Here, it may include providing the left-eye image and the first stereoscopic right-eye image as left and right images for applying to the HMD, respectively.

The preset equal interval may be obtained by dividing the first equirectangular panoramic image for the 360-degree space into eight.

According to another aspect of the present invention, a method for generating a 3D image for an HMD from a 2D image includes: generating a second equirectangular panoramic image for a 360-degree space; Inputting isotropic depth image information for a 360-degree space corresponding to the second panoramic image; Separating regions of different depth values from the second panoramic image using the depth image information; Generating a left-eye image by matching the separated area to n (n≥0, integer) square cube maps consisting of a front, left, rear, right, plane, and bottom surface; And generating a second stereoscopic right-eye image using the left-eye image using a preset algorithm.

Here, it may include providing the left-eye image and the second stereoscopic right-eye image as left and right images for applying to the HMD, respectively.

The preset equal interval may be obtained by dividing the second equirectangular panoramic image for the 360-degree space into eight.

The method of generating a three-dimensional image from a two-dimensional image according to an embodiment of the present invention is based on the degree of three-dimensional effect to be obtained according to the principle of binocular parallax, a principle of human stereoscopic perception, from a single 2D image using a computer device. Computing because a 3D image can be generated by providing an image with a different viewing angle by extracting the binocular parallax angle from the depth image and rearranging the original image according to the extracted angle, providing the deformed image and the original image as a pair of stereoscopic images. Simple 3D digital cartoon content can be provided without consuming a lot of power.

The effects of the present invention are not limited to those mentioned above, and other effects that are not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 is a diagram showing a system block diagram of the present invention.
FIG. 2 is a flowchart illustrating a method of generating a 3D image for an HMD from a 2D image according to an embodiment of the present invention.
3 is a system configuration diagram according to an embodiment of the present invention.
4 is a diagram illustrating a 360 degree panoramic image and a depth image thereof.
5 is a diagram illustrating an equirectangular drawing method for a 360 degree panoramic image.
6 is a diagram illustrating a cube map for mapping a 360 degree panoramic image.
FIG. 7 is a diagram illustrating areas in which the front, left, rear, and right surfaces of a cube map are shown in an isometric diagram.
8 is a conceptual diagram of a right-eye image generated using a left-eye image and a depth image.
9 is a diagram illustrating a distance to an actual object corresponding to a depth image.
10 to 13 are exemplary diagrams of a method of generating a 3D image for an HMD from a 2D image according to another embodiment of the present invention.

Objects and effects of the present invention, and technical configurations for achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. In describing the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of donation in the present invention, which may vary according to the intention or custom of users or operators.

However, the present invention is not limited to the embodiments disclosed below, and may be implemented in various different forms. The present embodiments are provided only to make the disclosure of the present invention complete, and to fully inform the scope of the invention to those skilled in the art to which the present invention pertains, and the present invention is defined by the scope of the claims. It just becomes. Therefore, the definition should be made based on the contents throughout the present specification.

Throughout the specification, when a part "includes" or "includes" a certain element, it means that other elements may be further included rather than excluding other elements unless otherwise stated. . In addition, terms such as "... unit", "... unit" or "... module" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware and It can be implemented as a combination of software.

On the other hand, in an embodiment of the present invention, each component, functional blocks or means may be composed of one or more sub-components, and the electrical, electronic, and mechanical functions performed by each component are an electronic circuit, It may be implemented with various known devices or mechanical elements such as an integrated circuit and an application specific integrated circuit (ASIC), and may be implemented separately or two or more may be integrated into one.

Also, combinations of each block of the attached block diagram and each step of the flowchart may be performed by computer program instructions. These computer program instructions can be mounted on the processor of a general purpose computer, special purpose computer, portable notebook computer, network computer, mobile device such as a smart phone, an online game service providing server or other programmable data processing equipment. The instructions, executed by the processor of the possible data processing equipment, will create means for performing the functions described in each block of the block diagram or each step of the flow chart, which will be described below. These computer program instructions can also be stored in memory available to a computer device or computer readable memory that can be directed to a computer device or other programmable data processing equipment to implement a function in a specific way, so each block of the block diagram Alternatively, it is also possible to produce a product containing instruction means for performing the functions described in each step of the flow chart. Since computer program instructions can also be mounted on a computer device or other programmable data processing equipment, each block and flow chart of the block diagram is created by creating a process for performing a series of operational steps on the computer device or other programmable data processing equipment. It is also possible to provide steps for executing the functions described in each step of the.

In addition, each block or each step may represent a module, segment, or part of code that contains one or more executable instructions for executing the specified logical function(s). In addition, it should be noted that in some alternative embodiments, functions mentioned in blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, or the blocks or steps may sometimes be performed in the reverse order depending on the corresponding function.

In the embodiment of the present invention, the term user device means all calculation means capable of collecting, reading, processing, processing, storing, and displaying data such as a desktop computer, a notebook computer, a smart phone, a PDA, a mobile phone, and a game machine. do. In particular, the user device in the embodiment of the present invention is a device having a function of executing software written in readable code, and displaying and delivering it to the user. In addition, if necessary, the software can be stored on its own or can be read together with data from the outside.

In addition, the terminal in the embodiment of the present invention includes not only the above data processing functions but also functions such as input, output, and storage, and for this purpose, CPU, main board, graphic card, hard disk, Not only various elements such as sound card, speaker, keyboard, mouse, monitor, USB, communication modem, etc., but also CPU, main board, graphic chip, memory chip, sound engine, speaker, touch pad, USB, etc. of wireless smartphone terminals. It may include an external connection terminal, a communication antenna, a communication modem capable of implementing communication such as 3G, LTE, LTE-A, WiFi, and Bluetooth. These various elements may be implemented alone or in combination of two or more, or a part of various elements may be combined to implement one or more functions, and in an embodiment of the present invention, a device indicated by one or more blocks in the drawings or detailed description, or The parts may mean that various elements included in the user device as described above are used alone or in combination of two or more, or that parts of various elements are combined to represent one or more functions.

Hereinafter, a method of generating a 3D image for an HMD from a 2D image according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing a system block diagram of the present invention, FIG. 2 is a diagram showing a flow chart of a method for generating a 3D image for an HMD from a 2D image according to an embodiment of the present invention, and FIG. A system configuration diagram according to an embodiment of the present invention.

1 to 3, a method of generating a 3D image for an HMD from a 2D image according to an embodiment of the present invention may be performed by a user device or a service server relaying a user and a third party.

The user device or service server is provided to transmit and receive various types of information in a wired or wireless manner, such as when used in a method of generating a 3D image for an HMD from a 2D image or when transmitting a left or right eye image to the HMD. Next, the image generator may be provided to generate an image, such as a case of mapping a panoramic image or a panoramic image to a cube map for use in a method of generating a 3D image for an HMD from a 2D image. Next, the image conversion unit may be provided to extract the depth image or to divide the generated image in a set manner.

3, when the method of generating a 3D image for an HMD from a 2D image according to an embodiment of the present invention is performed by the service server 100, the service server 100 is transmitted from the user device. After processing and editing the image information using the image information, it is transmitted to the third user device so that the user of the third user device can enjoy the 3D image through the HMD.

Referring to FIG. 2, a method of generating a 3D image for an HMD from a 2D image according to an embodiment of the present invention includes 3D images for an HMD from a 2D image according to an aspect of the present invention for solving the above technical problem. The method of generating a dimensional image includes generating an equirectangular first panoramic image for a 360-degree space (S110).

The first panoramic image may be received and used from a user device or the like. Referring to FIGS. 4 and 5, the first panoramic image is an image representing a 360-degree space starting from the left as the starting point of the image, and may be provided so as to end at the right end of the image. Referring to FIGS. 4 and 5, it can be seen that a 360-degree panoramic image represented by an equirectangular diagram in which a three-dimensional space is expressed in one plane is expressed.

Next, it includes the step of extracting a depth image from the first panoramic image (S120).

In FIG. 4, it can be seen that (b) corresponding to the depth image is extracted from (a), which is an example of the first panorama image. Referring to FIG. 4, in the case of a first panoramic image, as a color image, it may be divided into pixels of regions having different depth values. It can be expressed as a depth image to apply different shades and shades to the pixels and use the degree of each shade as information on the depth value. In this case, the degree of contrast may be provided to be expressed as n integers (n≥0) according to a preset method.

Next, a step of generating a left-eye image by matching the first panorama image and the depth image with a square cube map consisting of a front, left, rear, right, plane, and bottom surface (S130).

Referring to FIG. 6, it can be seen that when the first panoramic image expressed by the isotropic drawing method is matched to the cube map, it is divided into a total of six images. The cube map can be used by matching images corresponding to the front, left, back, right, plane, and bottom surfaces.

FIG. 7 is a diagram illustrating areas in which the front, left, rear, and right surfaces of a cube map are shown in an isometric diagram. Referring to FIG. 7, it can be seen that on the first panoramic image expressed by the equirectangular drawing method, portions in which the continuous surfaces of the front, left, rear, and right surfaces of the cube map excluding the plane and the bottom are expressed are represented by shades. .

The method of generating a three-dimensional image for an HMD from a two-dimensional image according to an embodiment of the present invention is to express a three-dimensional effect of the image in the direction the user is looking at. Since the definition of the left/right eye area is changed according to the user's gaze change, it may be provided to apply the depth value only to the front, left, rear, and right areas, excluding the depth value application area. .

8 is a conceptual diagram of a right-eye image generated using a left-eye image and a depth image, and FIG. 9 is a diagram illustrating a distance to an actual object corresponding to the depth image.

8 and 9, the difference between the left and right eye images can be calculated using a baseline corresponding to the distance between the left and right eyes and the distance between the first object P1 and the second object P2. I can. That is, it can be seen that the relatively close first object P1 has a larger binocular disparity value (l1-r1> l2-r2) than the second object P2. A left eye image may be generated by correspondingly adjusting the pixel position of the input image.

Next, a step of dividing a region overlapping the continuous images of the front, left, rear and right sides of the cubemap image from the first panoramic image into a two-dimensional image of equal intervals set in advance (S140).

Rather than generating an overall right-eye image for the first panorama image while the left-eye image is fixed, the left-eye and right-eye images for each divided area by dividing each area of the front, left, back, and right side of the matched cube map into equal parts. In this case, each region may be divided into 45 degree intervals corresponding to 1/8 of the total 360 degree space. However, the present invention is not limited thereto, and the area may be divided at appropriate equal intervals if necessary.

Next, it includes generating a first stereoscopic right-eye image with a preset algorithm for the divided 2D image (S150). Specifically, a right-eye image may be generated through a distance corresponding to the distance between the left-eye image and the left-right and both eyes, and a distance between the object.

In addition, it is necessary to define the values of the distance between the binoculars (baseline) and the distance between the eyes and the image area (f), which are operating environment variables given separately from the input image. It is preferred that it is provided.

In general, VR devices such as HMD have a distance from the bodily sensation screen of about 1 to 2 m, so the farthest distance is less than about 2 m, and the latest distance is also set larger than the focal length provided by the device (the distance between the lens and the eye). It is desirable to be. Preferably, the distance between the object and the left or right eye is less than or equal to about 1.5 m, and may be set to be greater than the distance between the lens of the HMD and the left or right eye.

The method of generating a 3D image for an HMD from a 2D image according to an embodiment of the present invention further includes providing the left-eye image and the first stereoscopic right-eye image as left and right images for applying to the HMD, respectively. can do.

Each image provided as a left and right image is transmitted to a third user and transmitted to be displayed on the left and right eyes of the HMD, so that the third user can enjoy a three-dimensional image.

A method of generating a 3D image for an HMD from a 2D image according to an embodiment of the present invention extracts a depth image from one panoramic image, expresses each image using an equirectangular diagram, and then matches it to a cube map. In the state, the image is separated into the remaining images except the upper and lower images, and it is converted into a panoramic image of equirectangular diagram, and divided into equal parts to create a left-eye image and a right-eye image.Therefore, it can be played back in the HMD using one two-dimensional image. A possible three-dimensional image can be provided.

A method of generating a 3D image for an HMD from a 2D image according to another embodiment of the present invention includes generating a second equirectangular panoramic image for a 360-degree space.

The second panoramic image may correspond to a two-dimensional image drawn by a user. As a more preferable example, the depth image information may be extracted from the first panoramic image. However, the present invention is not limited thereto, and may be provided so that a user may arbitrarily create depth image information to generate a left-eye or right-eye image together with the second panoramic image.

Equilateral square depth image information for a 360-degree space extracted from the first panoramic image corresponding to the second panoramic image in order to create an image that conveys a three-dimensional effect by expressing the second panoramic image on the first panoramic image described above. It may include the step of inputting.

Next, it includes the step of separating regions of different depth values from the second panoramic image by using the depth image information.

The depth image information extracted from the first panorama image may be converted into a 3D image harmonized with the first panorama image by being provided as a depth value for the divided image expressed in the second panorama image.

And generating a left-eye image by matching the separated area to n (n≥0, integer) square cube maps consisting of a front, left, rear, right, plane, and bottom surface. The cubemap may be provided as a left eye image for delivering a three-dimensional effect as two or more images having various depth values.

Next, it includes the step of generating a second stereoscopic right-eye image by using a predetermined algorithm for the left-eye image.

A right-eye image may be generated through a proportional equation using a distance between two eyes of a user corresponding to a distance between the lenses of both eyes of the HMD and a distance (depth value) with at least one object.

Here, it may include providing the left-eye image and the second stereoscopic right-eye image as left and right images for applying to the HMD, respectively. The left-eye image and the right-eye image may be provided as a single image having a three-dimensional effect to a user wearing an HMD through a visual difference.

In addition, the preset equal interval may be obtained by dividing the first equirectangular panoramic image for the 360-degree space into eight. For example, instead of generating the entire right-eye image for the first panorama image while the left-eye image is fixed, each of the front, left, back, and right sides of the matched cube map is divided into equal parts for each divided area. Left-eye and right-eye images may be generated, in which case, each area may be divided at 45 degree intervals corresponding to 1/8 of the total 360 degree space.

10 to 13 are exemplary diagrams of a method of generating a 3D image for an HMD from a 2D image according to another embodiment of the present invention.

Referring to FIGS. 10 to 13, first, a panoramic image in which a continuous image consisting of front, left, rear and right surfaces excluding areas corresponding to the top and the bottom in one cube map is expressed by the isometric drawing method. Next, extracting the depth image from the panoramic image is illustrated in FIG. 11. The depth image is converted into depth information of a preset value for each object on the panorama image, and may be used to implement the depth level for each object. Next, a left-eye image and a right-eye image can be generated respectively through a proportional equation using the distance between the user's eyes and the distance to each object.

FIG. 12 shows a left-eye image using an algorithm using a proportional equation, and FIG. 13 shows a right-eye image using an algorithm of a proportional equation. When each image is provided to the left and right eye lenses of the HMD, a three-dimensional effect for objects of different depths This can be provided as a single image expressed.

In other words, after matching one panorama image by equirectangular projection to the cube map, excluding the images corresponding to the upper and lower parts, converting them into consecutive panoramic images of the front, left, rear and right sides on the horizontal plane, and then a preset algorithm. According to the implementation, the left-eye and right-eye images are provided, so that a VR image having a three-dimensional effect from one image can be provided through the HMD, even if excessive computing power is not required.

According to another embodiment of the present invention, after inputting one 360-degree panoramic image to a user device, a depth image for providing a depth value to each object of the panoramic image is additionally input, and the panoramic image and the depth After matching each image to the cube map, the values excluding the plane and bottom images of the cube are converted into a panorama image and used, instead of using the images of the front, left, rear, and right sides corresponding to the horizontal image of the cube map from the beginning. Depth image information corresponding to this may be input to the user device to generate a new panoramic image. The new panoramic image thus prepared may be provided to be converted into a left-eye image and a right-eye image from a proportional expression using a depth value of an object expressed on the image, a distance between the user's eyes, and a virtual line in a direction perpendicular to the user. The detailed algorithm is omitted because it is based on the example described above.

A method of generating a 3D image for an HMD from a 2D image according to another exemplary embodiment of the present invention uses a single 360-degree image that has been edited by a user to provide a left-eye image and a right-eye image to the HMD. There is an advantage that it can be easily provided as a three-dimensional content as it is deformed. Content provided in this way can be highly useful because it can be provided as an image with a three-dimensional effect through an HMD, especially when grafted to a two-dimensional digital cartoon provided by an artist.

In the present specification and drawings, preferred embodiments of the present invention have been disclosed, and although specific terms are used, these are merely used in a general meaning to easily describe the technical content of the present invention and to aid understanding of the present invention. It is not intended to limit the scope. In addition to the embodiments disclosed herein, it is apparent to those of ordinary skill in the art that other modifications based on the technical idea of the present invention may be implemented.

Claims (7)

Generating a first equirectangular panoramic image for a 360-degree space; Extracting a depth image from the first panoramic image; Generating a left-eye image by matching the first panoramic image and the depth image with a square cube map consisting of a front, left, rear, right, plane and bottom surface; Dividing a region overlapping the continuous images of the front, left, rear, and right sides of the cubemap image from the first panorama image into two-dimensional images of equal intervals set in advance; And generating a first stereoscopic right-eye image with respect to the divided 2D image by a preset algorithm.
The method of claim 1, wherein the method comprises:
And providing the left-eye image and the first stereoscopic right-eye image as left and right images for applying to the HMD, respectively.
The method of claim 1, wherein the method is
The preset equal interval is a method for generating a 3D image for an HMD from a 2D image, characterized in that the first equirectangular panoramic image for the 360-degree space is divided into 8 equal parts.
Generating a second equirectangular panoramic image for a 360-degree space; Inputting isotropic depth image information for a 360-degree space extracted from a first panorama image corresponding to the second panorama image; Separating regions of different depth values from the second panoramic image using the depth image information; Generating a left-eye image by matching the separated area to n (n≥0, integer) square cube maps consisting of a front, left, rear, right, plane, and bottom surface; And generating a second stereoscopic right-eye image using a predetermined algorithm for the left-eye image.
The method of claim 4, wherein the method comprises:
And providing the left-eye image and the first stereoscopic right-eye image as left and right images for applying to the HMD, respectively.
The method of claim 4, wherein the method
The preset equal interval is a method for generating a 3D image for an HMD from a 2D image, characterized in that the first equirectangular panoramic image for the 360-degree space is divided into 8 equal parts.
A storage medium storing a program for performing a method of generating a three-dimensional image for an HMD from the two-dimensional image according to any one of claims 1 to 6.

Images