KR101791917B1 - Method for automatically converting normal video to virtual reality video and device using the same - Google Patents

Abstract

Provided is an apparatus for automatically converting a general image into a virtual reality image. The apparatus comprises: an image source receiving part for receiving a general image other than a virtual reality (VR) image; an image source analysis part for analyzing the received general image; and a VR conversion part for converting the received general image into a VR image based on the analysis result of the image source analysis part. Therefore, simple and natural VR image is generated.

Description

TECHNICAL FIELD [0001] The present invention relates to a method of automatically converting a general image into a virtual reality image, and a device using the same. [0002]

The present invention relates to a method for automatically converting a general image into a VR image instead of a virtual reality (VR) image, and an apparatus using the same. More particularly, the present invention relates to a method for automatically converting a content including a general image or an image into a natural VR image and a device using the same.

Currently, the VR industry is not popularized because of the lack of high quality content due to high VR content production cost. There is a demand for a technique of expanding a 2D image to a natural VR image in order to show a general two-dimensional (2D) image occupying the majority in a virtual reality display device such as a VR headset.

Currently, there is a method of extending a 2D screen to a 360-degree screen, but there is a problem that the image at the upper or lower part of the image is reduced, and the enlarged part at the center is unnaturally enlarged. Further, in the current conversion method, when the close-up image is enlarged, the image quality deteriorates and the model of the object is distorted, which is not suitable for the user to view. In addition, And it is necessary to optimize the user experience. In addition, currently available methods do not provide a function of converting images such as animations, webtoons, and the like into fused VR images in addition to the 2D images actually shot.

Korean Registered Patent No. 10-1648673

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and apparatus capable of generating a simple and natural VR image by automatically converting conventional conventional images into VR images.

In addition, the present invention adaptively provides a VR conversion method suitable for a state of a current image source based on a result of analyzing a state of a video source provided in converting a conventional 2D image into a VR image, And to provide images.

It is another object of the present invention to provide a method and apparatus for providing a multi-angle based VR image display by fusing not only a 2D video but also a still picture or an image.

The present invention has been made in view of the above problems,

According to an embodiment of the present invention, there is provided an apparatus for automatically converting a general image into a virtual reality (VR) image, the apparatus comprising: an image source receiver for receiving a general image, not a virtual reality image; An image source analyzer for analyzing the received general image; And a VR converting unit for converting the received general image into a VR image based on an analysis result of the image source analyzing unit.

Here, the image source analyzing unit may include a background region identifying unit for identifying a background region of the received general image. In addition, the VR converter may include a background image generator for generating a background image, and the background image generator may enlarge the image of the background area by 360 degrees based on the identification result of the background area of the background area identification part . Also, the VR converter may include an angle-of-view determining unit that determines an angle of view of the received general image, and the angle-of-view determining unit may determine an angle of view of the object image of the received general image.

In addition, the image source analyzing unit may analyze at least one of an image quality state, a stereoscopic state, and a portrait close-up state of the received general image. In addition, the VR converter may generate a VR image so that the auxiliary image associated with the received general image may be overlaid on the background area of the received general image.

In another embodiment of the present invention, there is provided a method of automatically converting a general image into a virtual reality (VR) image, comprising: receiving an image, not a virtual reality image, Analyzing the received general image; And a VR converting step of converting the received general image into a VR image based on an analysis result of the image source.

Here, the image source analyzing step may perform background region identification of the received general image. In addition, the VR conversion step may include generating a background image for generating a background image, and the generating of the background image may include enlarging the image of the background area by 360 degrees based on the result of the background area identification . In addition, the VR conversion step may include an angle-of-view determination step of determining the angle of view of the received general image, and the angle of view determination step may determine an angle of view of the object image of the received general image.

In addition, the image source step may perform analysis of at least one of an image quality state, a stereoscopic state, and a portrait close-up state of the received general image. The VR conversion step may generate the VR image so that the auxiliary image related to the received general image may be overlaid on the background area of the received general image.

According to an embodiment of the present invention, there is provided a recording medium on which a computer program for performing each step configuration according to the above-described VR image conversion method is recorded.

According to the present invention, it is possible to provide a method and apparatus capable of generating a simple and natural VR image by automatically converting conventional general images into VR images.

According to the present invention, a VR conversion method suitable for a current image source state is adaptively provided based on a result of analyzing a state of a provided image source in converting a conventional 2D image into a VR image, It is possible to provide a high VR image.

It is another object of the present invention to provide a method and apparatus for providing a multi-angle based VR image display by fusing not only a 2D video but also a still picture or an image.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram for explaining a configuration of an apparatus for automatically converting a general image into a virtual reality (VR) image according to an embodiment of the present invention.
2 is a flowchart illustrating a method for automatically converting a general image into a virtual reality image according to an exemplary embodiment of the present invention.
3 is a flowchart illustrating a method performed in an image source analyzing step according to an exemplary embodiment of the present invention.
4 is a flowchart illustrating a method performed in the VR image conversion step according to an embodiment of the present invention.
5 illustrates a multi-angle based VR display screen according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification.

The terms " comprises, "" comprising, " " comprising, " or " comprising ", when used in this application, specify the presence or absence of one or more other components, steps, operations and / Do not exclude the addition.

Furthermore, terms including ordinals such as first, second, etc. used in the present invention can be used to describe elements, but the elements should not be limited by terms. These terms are used only for the purpose of distinguishing one component from another. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

In addition, the components shown in the embodiments of the present invention are shown independently to represent different characteristic functions, and do not mean that the components are composed of separate hardware or software constituent units. That is, each constituent unit is described by arranging each constituent unit for convenience of explanation, and at least two constituent units of each constituent unit may be combined to form one constituent unit or one constituent unit may be divided into a plurality of constituent units to perform a function. The integrated embodiments and the separate embodiments of each of these components are also included in the scope of the present invention without departing from the essence of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The configuration of the present invention and the operation and effect thereof will be clearly understood through the following detailed description.

1 is a conceptual diagram for explaining a configuration of an apparatus for automatically converting a general image into a virtual reality (VR) image according to an embodiment of the present invention. Referring to FIG. 1, the proposed apparatus includes an image source receiving unit 100, an image source analyzing unit 200, and a VR converting unit 300.

The image source receiving unit 100 may receive an image source having a general image format instead of a VR image, and the image source may be composed of various types of content sources such as a moving image, an image, and related data information for converting the image source into a VR image . The image source receiving unit 100 is capable of transmitting and receiving data through at least one of a wired communication and a wireless communication and is connected to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN) (LTE-Advanced), wideband code division multiple access (WCDMA), wireless fidelity (WiFi), and the like. , Bluetooth, and near field communications (NFC).

The image source receiving unit 100 may include an image source storage unit 110 for storing an image source, and the image source storage unit 110 may include a memory for storing information related to a plurality of received contents and contents . In addition, the image source storage unit 110 may store programs and data necessary for the VR image conversion process of the image source.

The image source analyzing unit 200 analyzes the general image received through the image source receiving unit 100 and analyzes and determines the status and the status of the received image source. The video source analyzing unit 200 classifies whether the received video source is a moving picture, a still picture, or an image, and discriminates which level of resolution quality such as SD, HD, UHD, A 2D image or a 3D image, a real image or an animation image, or a close-up of objects such as a person, in order to judge whether the 3D image is in a stereoscopic / stereoscopic state.

In addition, the image source analyzing unit 200 may include a background region identifying unit 210 for identifying the background region from the object region in the image. The object area is an area where an object interested by a user who views the image exists. The object area mainly refers to the area where the moving object exists, and corresponds to the foreground area when compared with the background. On the other hand, the background region may refer to a non-object region that has no motion, is very small, or has a patterned object, i.e., is not processed as an object region. For example, in the case of an interior, the background may be a wall, a floor, a frame, a furniture, and the like may be a natural environment such as a building, a parked car, a tree, a street, a mountain or the sea.

The background region identifying unit 210 may analyze the image source and judge the object region and the background region in consideration of the motion and the layout of the screen, thereby enlarging the background region when converting into the VR image.

In addition, it is possible to utilize artificial intelligence (AI) technology for determining resolution, image quality, three-dimensional feeling, close-up determination, and background area of the image source analysis unit 200. For example, deep learning learning technique, a convolutional neural network or a recurrent neural network technique can be used. Accordingly, the image source analyzing unit 200 automatically recognizes the resolution and automatically adjusts the degree of enlargement of the panoramic image, automatically distinguishes the background of the image and the character, and if the image is a close-up image of the person, You can automatically adjust the magnification for naturalness of the screen when switching.

The VR conversion unit 300 converts the received general image into a VR image based on the analysis result of the image source analysis unit 200.

The VR converting unit 300 includes an angle-of-view determining unit 310 that determines an angle of view at which the received general image is to be enlarged. The angle-of-view determining unit 310 determines an angle of view For example, the object area is enlarged by 90 degrees, 120 degrees, 180 degrees, 240 degrees, 300 degrees, 360 degrees, and so on. For example, in the case of a close-up image of a person, if a person who is an object image is enlarged by 80 to 120 degrees, which is a general viewing angle, a close-up screen may appear burdensome to the user on the VR screen. In this case, The angle of view of any one of 30 degrees, 60 degrees, and 90 degrees can be determined with a maximum value within a range of 120 degrees to 120 degrees

The VR conversion unit 300 includes a background image generation unit 320 for generating a background image and the background image generation unit 320 generates a background image based on the background region identification result of the background region identification unit 210 , And the background area can be enlarged to a 360-degree viewing angle screen. When the background region is detected by dividing the background region and the object region in the image source analysis unit 200, the background region is expanded to a 360-degree panoramic image to generate a realistic VR image. In addition, the enlargement angle of the background area in the background image generator 320 may range from 90 degrees to 360 degrees, and may be generally determined to be larger than the enlargement angle of the object area.

In addition, the VR converting unit 300 may include a multi-angle processing unit 330, and the multi-angle processing unit 330 may be configured to overlay the auxiliary image related to the received general image on the background area of the general image VR images can be generated. For example, a plurality of sub-screens for displaying a sub-image related to a background area on a VR screen on which a received general image is converted are generated, and a plurality of sub-screens including auxiliary information including data, images, Layered image can be generated by overlaying the content of the background image on the background area. Such sub-screens may be arranged in a main character in the main screen or in a background area in which no main action occurs, and thus each sub-screen may be overlaid and arranged in multi-angles so as to have a viewing angle different from that of the main screen . Accordingly, various contents can be effectively arranged on a 360-degree screen, thereby enhancing user convenience and experience.

In addition, the multi-angle processing unit 330 may arrange a plurality of screens in parallel while arranging the multi-layer images in an overlay manner. For example, when the main screen is enlarged and displayed at 90 degrees, By arranging additional pieces in parallel, it is possible to arrange a full 360 degree VR screen to be filled.

2 is a flowchart illustrating a method of automatically converting a general image into a virtual reality (VR) image according to an exemplary embodiment of the present invention.

In order to convert a VR image, an image source having a general image format other than a VR image is firstly received (S210). Here, the image source may be composed of various types of content sources such as a moving image, an image, It is the target that you want to convert to video.

Next, a step of analyzing the received general video source for conversion into a natural VR image suitable for the received video is performed (S220). The analysis of the video source is performed based on the resolution, image quality, And identification and division of the object area and the background area.

The VR image conversion step of converting the received general image into the VR image is performed based on the analysis result of the image source performed in this manner. (S230) In the VR image conversion step, The background region is distinguished from the object region in the image and the auxiliary image related to the received general image is overlaid on the background region of the general image so that the VR image is displayed so that the viewing angle of the sub- And a step of generating the data.

After converting the VR image into a VR image, the VR image preview function can be provided to the user (S240). The user can determine whether to convert the VR image into a VR image or perform a correction operation through the VR image preview function. The image conversion can be completed (S250)

The converted VR image can be reproduced by various display devices supporting VR image display. For example, a VR headset, a head mounted display (HMD), a wearable device such as a smart glass, a smart phone, Such as mobile devices such as mobile phones, tablets, and notebooks.

3 is a flowchart illustrating a method performed in an image source analyzing step according to an exemplary embodiment of the present invention.

Referring to FIG. 3, the video source analysis step will be described in more detail. First, an operation of discriminating whether a received video source is a video, an image, or a still image is performed (S310)

In addition, if the image source is a moving image, the image quality or the resolution level can be determined. For example, it can be determined whether the image source belongs to the SD, HD, or UHD image quality and whether the image source is a stereoscopic 3D image or a general 2D image (S320)

Next, by analyzing the image source, it is possible to determine the object area and the background area in consideration of the motion and the layout composition of the screen (S330). Thus, by separating and extracting the background area, Only the background area can be enlarged by 360 degrees.

In addition, it is possible to adjust the degree of close-up of the object region in the image source (S340). For example, it is possible to check whether the object is close-up according to the size of the person in the image source and adjust the degree of close-up. The close-up operation will not be performed. In addition, for example, when the size of a person occupies 50% or less in the general viewing angle screen, the size of the person can be appropriately enlarged through close-up.

Although each of the analysis steps has been described in order in FIG. 3, the order of each step may be changed, and at least some of the steps may be performed simultaneously.

4 is a flowchart illustrating a method performed in the VR image conversion step according to an embodiment of the present invention. When the received image source is analyzed as shown in FIG. 3, the conversion to the optimized VR image can be automatically performed based on the analysis result.

Referring to FIG. 4, an angle of view in which the object region is enlarged in the image source is determined (S410). For the 360 degree VR image, an enlarged image can be obtained by increasing the angle of view of the object region. 120 degrees, 180 degrees, 240 degrees, 300 degrees, and 360 degrees based on the image size of the image.

Next, it is possible to detect a background area separated from the object area in the image source, and enlarge the detected background area by 360 degrees of viewing angle (S420). Thus, by expanding the detected background area by 360 degrees, The object region and the background region are separated and processed. Thus, the object region is present within the basic viewing angle of, for example, 80 to 120 degrees, so that the object region is unreasonably enlarged It is possible to provide a natural screen.

In addition, by overlaying the auxiliary image related to the received general image on the background area of the general image, the multi-angle content can be arranged so that the viewing angles of the auxiliary images can be displayed in multi-angles differently (S430) The content arranged at a viewing angle different from that of the main screen may be configured in various forms, for example, another image associated with the received general image, related data, or an image. The arrangement of the multi-angle content will be described later in more detail with reference to FIG.

Finally, the conversion into the VR image can be completed through an automatic stitching operation and a correction operation if necessary (S440)

Although each of the analysis steps is described in order in Fig. 4, the order of each step may be changed, and at least some of the steps may be performed simultaneously.

The VR image conversion methods described with reference to FIGS. 2 to 4 may be configured as a computer application program for performing a flow in various types of user terminals such as a computer, a notebook, a smart phone, and a tablet. Such application programs can be recorded on various types of computer-readable recording media such as USB memory, DVD, CD, SD card, and the like.

5 illustrates a multi-angle based VR display screen according to an embodiment of the present invention.

5 shows an example of a multi-angle based VR display screen appearing within a basic viewing angle, e.g., a viewing angle of 80 degrees to 120 degrees. The VR display screen can be broadly divided into a main screen 510 and a sub screen 520 and displays a plurality of contents overlaid on the main screen 510. In the case of the sub screen 520, So as not to overlap with each other.

Referring to FIG. 5, the main screen 510 shows the front of the operating room around the position where the operation is performed. The auxiliary screen 520 displays each long term data related to the operation, Are received together and displayed as multi-angles. Here, it is possible to determine the important position in which the surgeons are located in the main screen 510 as an object area, determine a position in which no object such as a doctor is present as a background area, 520 can be disposed. The distinction between the object region and the background region can be automatically performed by a computer through image processing technology and image analysis through deep learning, and the sub screen 520 can be automatically arranged at an appropriate position on the background region . In addition, it is also possible that the worker manually proceeds to arrange the sub screen 520 in an appropriate place on the background area.

In addition, the user can select a desired screen among various content screens displayed in multi-angles through a device such as a VR headset. For example, a screen such as a surgical site scene, an operation room foreground scene, each physical information monitor, and the like can be selected. The selected screen may be enlarged to the large screen as the main screen on the front. In addition, the main screen displayed in front of the user through the display interface technology in the VR headset worn by the user is a surgical scene of the surgical site. The screen is displayed as a 360 degree VR image, The part can be displayed as a multi-display as a multi-angle and various contents as a menu.

Although FIG. 6 shows only the basic viewing angle of the VR screen, the background area can be extended to a 360-degree viewing angle to provide a 360-degree VR screen. At this time, a plurality of contents can be overlaid and displayed on the multi-angle within the background screen of the 360-degree VR screen.

The present invention solves the problem that the VR content is limited to the market due to the high cost of the VR photography at present, so that the number of VR images can be explosively increased as conventional general videos are converted into VR images. It also gives users the opportunity to see virtually every existing video with a VR headset. In addition, by providing VR image contents with high quality and high quality through the VR version image distribution of existing 2D images, it will be possible to increase the value of contents and provide an additional profitable business utilizing conventional contents.

The embodiments disclosed in the specification of the present invention are merely illustrative, and the present invention is not limited thereto. The scope of the present invention should be construed according to the following claims, and all the techniques within the scope of equivalents should be construed as being included in the scope of the present invention.

100: video source receiving unit 110: video source storage unit
200: image source analysis unit 210: background area identification unit
300: VR conversion unit 310:
320: Background image generating unit 330: Multi-angle processing unit
510: main screen 520: auxiliary screen

Claims (13)

An apparatus for automatically converting a general image into a virtual reality (VR) image,
An image source receiving unit for receiving a two-dimensional (2D) general image other than a virtual reality image;
An image source analyzer for analyzing the received general image; And
And a VR converting unit for converting the received general image into a VR image based on an analysis result of the image source analyzing unit,
Wherein the image source analyzing unit includes a background region identifying unit for identifying a background region of the received general image,
Wherein the VR conversion unit includes a background image generation unit for generating a background image from the received general image, and the background image generation unit generates an image based on the identification result of the background area of the background area identification unit, Expanding,
Wherein the VR converting unit further includes an angle-of-view determining unit that determines an angle of view at which the received general image is to be enlarged, and the angle-of-view determining unit determines an angle of view at which the object area of the received general image is to be enlarged. delete delete delete The apparatus of claim 1, wherein the image source analyzing unit analyzes at least one of an image quality state, a stereoscopic state, and a portrait close-up state of the received general image. The apparatus of claim 1, wherein the VR converter generates a VR image so that the auxiliary image associated with the received general image can be overlaid on the background area of the received general image. A method for automatically converting a general image into a virtual reality (VR)
An image source receiving step of receiving a two-dimensional (2D) general image, not a virtual reality image;
Analyzing the received general image; And
A VR conversion step of converting the received general image into a VR image based on an analysis result of the image source
Lt; / RTI >
Wherein the image source analyzing step performs background region identification of the received general image,
Wherein the step of generating the background screen includes a step of generating only the background area by 360 degrees based on the result of the background area identification, The image is enlarged,
Wherein the VR conversion step further includes an angle-of-view determining step of determining the angle of view at which the received general image is to be enlarged, and the angle-of-view determining step determines an angle of view at which the object area of the received general image is to be enlarged, Conversion method. delete delete delete 8. The method according to claim 7, wherein the analyzing of the image source performs analysis of at least one of an image quality state, a stereoscopic state, and a portrait close-up state of the received general image. 8. The method of claim 7, wherein the VR conversion step generates a VR image so that the auxiliary image related to the received general image can be overlaid on the background area of the received general image. A recording medium on which a computer program for performing the automatic image conversion method according to any one of claims 7, 11, and 12 is recorded.

Images