KR102654323B1 - Apparatus, method adn system for three-dimensionally processing two dimension image in virtual production - Google Patents

Abstract

The present invention relates to a method, device, and system for stereoscopic processing of two-dimensional images in virtual production. A device according to an embodiment of the present invention provides content in a virtual space captured by a virtual camera that is linked according to the movement and lens status of the camera that captures the first image, which is a live-action image, during virtual production-based shooting. A device for reproducing a second image containing a light-emitting diode (LED) wall so that the second image is displayed, comprising: a memory storing information about a two-dimensional flat image; and a control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image.

Description

Method, device and system for three-dimensional processing of two-dimensional images in virtual production {APPARATUS, METHOD ADN SYSTEM FOR THREE-DIMENSIONALLY PROCESSING TWO DIMENSION IMAGE IN VIRTUAL PRODUCTION}

The present invention relates to a three-dimensional processing technology for two-dimensional images in virtual production. More specifically, when shooting video based on virtual production, the two-dimensional image displayed on the LED (light-emitting diode) wall is more accurate than that of the camera. It is about technology for processing two-dimensional images so that they can be captured in three-dimensional form.

Recently, as travel between countries and gatherings of people have been restricted due to the coronavirus, interest in non-face-to-face technology based on virtual reality is growing. According to this trend, attempts to introduce virtual production technology are increasing in video production fields such as broadcasting, movies, and performances.

At this time, virtual production is based on real-time rendering CG (computer graphics) technology, which is the content of a virtual space created by a game engine, and camera tracking technology that reflects the movement of the camera and the state of the lens, and provides immediate visualization and It refers to a technology that improves the efficiency and graphics quality of the video production process through video synthesis. Recently, it has been mainly used as a concept of displaying graphics reflecting actual camera movements through an LED (light-emitting diode) wall in real time and filming at least part of the LED wall with an actor located in front of the LED wall. . Such virtual production can not only reduce constraints such as shooting space, time, and weather, but also reduce the time of post-production of video production, contributing to improving the productivity of video production.

Meanwhile, there are cases where virtual production is performed by displaying the source of a two-dimensional image on an LED wall and photographing at least a portion of the LED wall with a camera. However, in this case, due to the limitations of the two-dimensional image source, the image captured by the camera causes the display portion of the LED wall to appear as a flat image rather than a three-dimensional image, resulting in a problem that the overall image appears unnatural.

However, the above-described content merely provides background information on the present invention and does not correspond to previously disclosed technology.

In order to solve the problems of the prior art as described above, the present invention displays the source of a two-dimensional image on an LED wall and performs a virtual production in which at least a part of the LED wall is photographed with a camera, so that the image captured by the camera is more visible. The purpose is to provide technology to process the source of the two-dimensional image so that it can be viewed three-dimensionally.

However, the problem to be solved by the present invention is not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

A device according to an embodiment of the present invention for solving the above problems is a virtual camera that is linked according to the movement and lens status of the camera that captures the first image, which is a live-action image, during virtual production-based shooting. A device that reproduces the second image containing the content of the virtual space captured by the light-emitting diode (LED) wall so that the second image is displayed, a memory that stores information about the two-dimensional flat image. ; and a control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image.

The control unit arranges a flat plate for the flat image in a virtual space, and arranges the flat plate at a position where the center point of the flat plate is spaced apart from the reference position in the virtual space, and then according to the curved separation distance. A concave hemispherical curved plate relative to the flat plate is created at a position spaced apart from the flat plate using the flat plate, and the virtual camera is linked according to the movement of the camera and the state of the lens to create the curved plate. You can control capture.

The curved plate may have an area that increases as the distance between the curved surfaces increases.

The control unit may control the curved plate to be created at a position where each of the flat plate and the curved plate are identically matched when the virtual camera views the flat plate and the curved plate from the reference position.

The control unit can control the curved plate to be created at a position where each vertex forming the flat plate and each vertex forming the curved plate have the reference position as the origin.

When the control unit draws an extension line for each vertex of the flat plate at the reference position, the control unit displays an image displayed at the vertex of the flat plate where the extension line meets the flat plate, and the curved plate where the extension line meets the curved plate. The curved plate can be controlled to be created at a location such that the image displayed at the vertex is the same.

The control unit can control the arrangement of the flat plate using information about the camera setting state at the time the flat image is captured.

The control unit generates separate curved plates for each layer of each separate image in the flat image according to different curved separation distances, and the virtual camera is linked according to the movement of the camera and the state of the lens to create the curved plates. You can control capture.

The control unit determines that, for the first layer of the first segment image and the second layer of the second segment image that is farther than the first segment image, the curved surface separation distance of the first layer is shorter than the curved surface separation distance of the second layer. You can control it to be set.

The control unit determines the curved separation distance of the first layer with respect to the first layer of the first segment image that is the near-ground, the second layer of the second segment image that is the mid-view, and the third layer of the third segment image that is the distance. It is shorter than the curved surface separation distance of the second layer, and the curved surface separation distance of the second layer can be controlled to be set shorter than the curved surface separation distance of the third layer.

The control unit uses a depth map for at least one of the curved plates to control at least one vertex of the curved plate to protrude from the curved plate in the direction of the reference position, thereby providing a sense of thickness to the curved plate. You can.

The method according to an embodiment of the present invention is to capture content in a virtual space captured by a virtual camera that is linked according to the movement and lens status of the camera that captures the first image, which is a real-life image, during virtual production-based shooting. A method performed in a device for reproducing a second image such that the second image containing a light-emitting diode (LED) is displayed on a wall, comprising: performing three-dimensional image processing on a two-dimensional flat image; and processing the image resulting from the three-dimensional image processing to be included in the second image and displayed on the LED wall.

The performing step includes placing a flat plate for the flat image in a virtual space, where the center point of the flat Plate is spaced from a reference position in the virtual space; and generating a concave hemispherical curved plate with respect to the flat plate at a position spaced apart from the flat plate according to the curved surface separation distance using the flat plate. The processing step includes: It may include capturing the curved plate while the virtual camera is linked according to movement and lens status.

The performing step may include generating the curved plate at a position where each of the flat plate and the curved plate are identically matched when the virtual camera views the flat plate and the curved plate from the reference position.

The performing step may include creating the curved plate at a position where each vertex forming the flat plate and each vertex forming the curved plate correspond to the reference position as the origin.

The step of performing the above is when drawing an extension line for each vertex of the flat plate at the reference position, an image displayed at the vertex of the flat plate where the extension line meets the flat plate, and the image displayed at the vertex of the flat plate where the extension line meets the curved plate. It may include creating the curved plate at a location such that the images displayed at the vertices of the curved plate are the same.

The performing step may include arranging the planar plate using information about the camera setting state at the time the planar image is captured.

The performing step includes generating separate curved plates for each layer of each separate image in the flat image according to different curved surface separation distances, and the processing step depends on the movement of the camera and the state of the lens. Accordingly, it may include capturing the curved plates while the virtual camera is linked.

The step of performing the above is that, for the first layer of the first segment image and the second layer of the second segment image that is farther than the first segment image, the curved surface separation distance of the first layer is the curved surface separation distance of the second layer. It may include a shorter setting step.

The performing step is a curved separation distance of the first layer with respect to the first layer of the first segment image that is the near-ground, the second layer of the second segment image that is the mid-ground, and the third layer of the third segment image that is the distance. may include setting the curved surface separation distance of the second layer to be shorter than the curved surface separation distance of the second layer and the curved surface separation distance of the third layer to be shorter than the curved surface separation distance of the third layer.

The performing step uses a depth map for at least one of the curved plates to cause at least one vertex of the curved plate to protrude from the curved plate in the direction of the reference position, thereby giving a sense of thickness to the curved plate. May include steps.

A system according to an embodiment of the present invention is a system used for virtual production-based shooting, and includes a camera for shooting a first image, which is a live-action image; and reproducing the second image so that the second image containing the contents of the virtual space captured by the virtual camera linked according to the movement of the camera and the state of the lens is displayed on the light-emitting diode (LED) wall. A device comprising: a memory storing information about a two-dimensional flat image; and a control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image.

A program according to an embodiment of the present invention is a program stored on a medium that is executed by at least one processor and causes the at least one processor to execute the above-described method.

The present invention configured as described above displays the source of a two-dimensional image on an LED wall and processes the source of the two-dimensional image when performing a virtual production in which at least a portion of the LED wall is photographed with a camera so that the image captured by the camera is It has the advantage of making it look more three-dimensional.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

Figure 1 shows a schematic block diagram of a system 10 according to an embodiment of the present invention.
Figure 2 shows a conceptual diagram of virtual production shooting using the system 10 according to an embodiment of the present invention.
FIGS. 3 and 4 show various examples of movement of the virtual camera (VC) linked with the movement of the camera 200 during filming according to FIG. 2.
Figure 5 shows a block diagram of the playback device 300.
Figure 6 shows an example of a vanishing point.
Figure 7 shows an example of a fog phenomenon.
Figure 8 shows an example when a camera takes pictures from the side of an LED wall.
Figure 9 shows an example of shooting with a camera when the LED wall has a bent surface shape.
Figure 10 shows an example of the absence of the parallax effect that appears when shooting with a camera while simply displaying the source of a two-dimensional image on an LED wall.
Figures 11 and 12 show examples of conceptual diagrams for anamorphic distortion.
Figure 13 shows a conceptual diagram of creating a curved plate using a flat plate (reference plane).
Figure 14 shows an example of photographing a flat plate with a virtual camera.
Figure 15 shows an example of photographing a curved plate with a virtual camera.
FIG. 16 shows an example of generating each layer (Lyaer) for a plurality of separate images divided by distance with respect to the plane source of FIG. 7 in virtual space.
Figure 17 shows the creation of each curved plate for the layer of each separate image according to Figure 16 in virtual space.
FIG. 18 shows examples of second images captured while the virtual camera (VC) moves from left to right while facing each curved plate created according to FIG. 17 in virtual space.
FIG. 19 shows an example of the process of applying a depth map to each layer (Layer 1, Layer 2, and Layer 3) created according to FIG. 16.
FIG. 20 shows an example of applying a depth map to the third layer (Layer 3) among the layers (Layer 1, Layer 2, and Layer 3) created according to FIG. 16.
Figures 21 and 22 are the results of shooting with a virtual camera after applying a depth map to the first layer (Layer 1) among the layers (Layer 1, Layer 2, and Layer 3) created according to Figure 16. Examples of the second image are shown.
Figure 23 shows an operational flowchart of a processing method according to an embodiment of the present invention.
Figure 24 shows an example of a shooting image according to a first method using a flat plate and a second method using a layered plate.
Figure 25 shows an example of a photographing result according to the first method.
Figure 26 shows an example of a photographing result according to the second method.
Figure 27 shows examples of imaging according to the first and second methods.

The above purpose and means of the present invention and the resulting effects will become clearer through the following detailed description in conjunction with the accompanying drawings, and thus the technical idea of the present invention will be easily understood by those skilled in the art. It will be possible to implement it. Additionally, in describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

The terminology used herein is for describing embodiments and is not intended to limit the invention. In this specification, singular forms also include plural forms, as appropriate, unless specifically stated otherwise in the context. In this specification, terms such as “comprise,” “provide,” “provide,” or “have” do not exclude the presence or addition of one or more other components other than the mentioned components.

In this specification, terms such as “or” and “at least one” may represent one of words listed together, or a combination of two or more. For example, “A or B”, “at least one of A and B” may include only A or B, or both A and B.

In this specification, descriptions under "for example" and the like may not exactly match the information presented, such as the cited characteristics, variables, or values, and may be subject to tolerances, measurement errors, limits of measurement accuracy and other commonly known factors. Effects such as modifications, including, should not limit the embodiments of the invention according to various embodiments of the present invention.

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in between. It must be understood that it may be possible. On the other hand, when a component is mentioned as being 'directly connected' or 'directly connected' to another component, it should be understood that there are no other components in between.

In this specification, when a component is described as being ‘on’ or ‘in contact with’ another component, it may be in direct contact with or connected to the other component, but there may be another component in between. It must be understood that it can be done. On the other hand, if a component is described as being 'right above' or 'in direct contact' with another component, it can be understood that there is no other component in the middle. Other expressions that describe the relationship between components, such as 'between' and 'directly between', can be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various components, but the components should not be limited by the above terms. Additionally, the above term should not be interpreted as limiting the order of each component, but may be used for the purpose of distinguishing one component from another component. For example, a 'first component' may be named a 'second component', and similarly, a 'second component' may also be named a 'first component'.

Unless otherwise defined, all terms used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

First, to understand the present invention, we will look at virtual production technology from three different perspectives.

1. Video production stage perspective

Real-time graphics rendering technology, commonly referred to as game engines, has recently evolved to the point where it can achieve photo-realistic quality. This immediate, high-quality visualization technology can be used for different purposes at various points in the video production pipeline, which is divided into pre-planning (Pre) - production (production) - post-production (Post).

First, in the pre-planning stage, the game engine can be actively used for story visualization (Pre Vis) or space/equipment simulation (Tech Vis). In other words, in the planning stage, not only the development of content, but also space, camera movement, cut editing, etc. are implemented in advance and various issues (set size, movement line, lens angle of view, etc.) are checked in advance. In particular, recently, the scope of use of virtual reality technology has been increasing, focusing on movies, such as editing components in a virtual environment through VR scouting techniques or simulating shooting with a virtual camera.

Next, at the production stage, virtual production technology is used to perform CG (computer graphics) compositing, which was done in post, at the same time as filming and transmit it live, or to reduce the burden of post-compositing. For this purpose, an XR solution that synthesizes real-time video and CG in real time is used along with a tracking device that detects camera movement.

Lastly, in the post-production stage, status data such as tracking information generated during filming can be utilized. Using this, additional effects or CG elements can be created even after filming is completed, and scenes that have problems or need correction can be corrected more quickly.

2. CG compositing method perspective

The core of virtual production technology is the visualization function that synchronizes real space and virtual space and synthesizes CG and live-action images in real time or displays them on LED without any sense of heterogeneity. Therefore, depending on the type of CG synthesis, the terms VR (virtual reality), AR (augmented reality), and XR (extended reality) are sometimes used.

First, VR refers to real-time spatial synthesis based on chroma key studio. For example, VR-type virtual production technology can be used to show a person and a virtual space simultaneously in the third person. In other words, you can achieve the same effect as shooting directly in virtual reality through a camera with real-time tracking. The reason why this kind of production is possible is because the video synthesized from the virtual space is created in real time and provided to the camera director and director, allowing immediate judgment and response.

In the case of AR, as opposed to VR, it refers to a method of adding specific graphics to live video, and has recently been widely used in events such as broadcasts, performances, and product launch shows. In particular, as the CG quality of game engines gradually approaches real-life due to the development of rendering technologies such as Realtime Ray Tracing and DLSS, attempts to express scenes that are difficult to implement physically or production that costs a lot of money with AR graphics are increasing. there is. For example, by adding AR elements to a video with only one car in the actual stage space, special productions such as a new car emerging from the floor or a new building being created are possible.

In the case of XR, it has been used in earnest since 2020, focusing on non-face-to-face events. Unlike AR or VR, XR synthesizes both distant and near scenes with CG, allowing the user to express themselves as if they were completely in a virtual space. To this end, an LED wall is placed in the background to output a distant scene, AR graphics are added in front of the subject to express a sense of space, and the physical boundaries of the LED wall are expanded with CG to create an expression as if one is in an infinite space. possible.

3. Purpose of use perspective

AR, VR, and XR technologies can be used in various areas such as broadcasting, performances, and movies. Although it is a similar XR technology, there is a difference in the approach when used in event-oriented filming such as broadcasting and performances and dramatic video filming such as movies and dramas.

First, in event filming, virtual production technology is mainly used to quickly create fresh visual effects or obtain synthesized video directly on site without additional post-production work. These videos are broadcast live immediately or consumed quickly after a short editing period. Therefore, tolerance for content quality or technical limitations is relatively wide, but requirements for camera movement speed and video latency are high. On the other hand, in dramatic video production such as movies, more focus is placed on the quality of the final video along with faithful support for professional filming techniques. For example, when shooting an event such as a performance, it is acceptable to see LED elements on an LED wall or to slightly show moiré phenomenon (interference between the grid of the camera CCD and the grid of LED elements), but it is absolutely not allowed in movies. In addition, if you look closely at the expression that expands and fills the external area of the LED wall in real time, there is a slight gap and color difference, so it is mainly used only for event photography. In the case of cameras, B4 mount broadcast cameras and zoom lenses are used for the former, but large CCD cinema cameras and single lenses are mainly used for the latter.

Meanwhile, In Camera VFX technology, one of the virtual production technologies, refers to a method of mapping CG that reflects camera movements on a large LED wall space and filming various scenes by moving the shooting point or changing the spatial state. There are some similarities to XR shooting at performances, but the devices are constructed in a form that focuses more on natural spatial lighting and reflection based on larger, higher-spec LED walls.

Figure 1 shows a schematic block diagram of a system 10 according to an embodiment of the present invention.

System 10 (hereinafter referred to as “this system”) according to an embodiment of the present invention is a system used for filming the above-described virtual production, and as shown in FIG. 1, a light-emitting device wall (light- It may include an emitting diode wall (LED wall) 100, a camera 200, and a player 300.

In this system 10, various media (i.e., images) played through the playback device 300 are displayed through the display unit located on the front of the LED wall 100, and in this state, one of the display units of the LED wall 100 At least a portion of the person (A) or object (O) located in front of the LED wall 100 is photographed together with the camera 200. Of course, external environmental elements such as lighting effects, lightning effects, sound effects, wind effects, fog effects, snow effects, or rain effects may be added to the camera (if necessary during filming of 200). It can be added through a device (not shown). At this time, video shooting for virtual production can be performed more smoothly by controlling the playback device 300 and the special effect device through a control device (not shown).

Specifically, the LED wall 100 is a large display device that displays media played by the playback device 300 on the front display using an LED method, and may have a wall shape. Of course, the LED wall 100 may include walls with different angles that display media at different angles, or may also include walls that display media from the bottom to the top, but is not limited thereto.

The LED wall 100 may include a small unit display 110 divided into n zones (where n is a natural number of 2 or more). In other words, the LED wall 100 can implement one large display through small unit displays 110. At this time, the LED wall 100 can display various media depending on the type of playback device 300. That is, the media selected among the plurality of playback devices 300 under the control of a control device (not shown) is played by the playback device 300, and the media played in this way can be displayed through the LED wall 100. .

For example, at least one media among images, videos, and camera tracking images may be displayed through the LED wall 100. However, since the present invention deals with the content of a three-dimensional image displayed through the LED wall 100 by processing the source of a two-dimensional image as described later, hereinafter, media related to the image are used for image processing and image processing of the playback device 300. The explanation will be that it is displayed on the LED wall 100 according to playback. That is, in the present invention, when the screen of the image-processed image displayed through the LED wall 100 is captured by the camera 200, the image captured by the camera 200 is the original two-dimensional image according to the image processing. It can appear more three-dimensional than the source. Image processing for a three-dimensional display effect for the source of such two-dimensional images will be described later.

The camera 200 captures a first image, which is a real-life image, by optical action on an object existing in real space (physical space). That is, the camera 200 can photograph at least a portion of the display portion of the LED wall 100 and a person (A) or object (O) located in front of the LED wall 100. Accordingly, the captured first image may include a person (A) or an object (O) and media that is displayed on the display unit of the LED wall 100 and becomes a background element of the person (A) or object (O). there is. Of course, a plurality of cameras 200 may be provided as needed. For example, the first image may be a video in which various images change over time. In this way, the first image, which is a live-action image captured by the camera 200, can be transmitted to a recording device (media) and recorded.

Additionally, the movement of the camera 200 and the state of the lens can be determined by a tracking device (not shown). This tracking information can be transmitted to the playback device 300 that plays the camera tracking linked video. For example, a tracking device is a device installed (connected) to the camera 200 and tracks the movement and lens state of the camera 200 to generate tracking information, which is data for the tracking. At this time, data about the movement of the camera 200 included in the tracking information may include data related to the position and angle of the camera 200. Additionally, data on the lens state included in the tracking information may include data related to the zoom and focus of the lens of the camera 200.

For example, the tracking device can be implemented using an optical method based on real-time vision sensing or an encoder method that calculates by combining the rotation value of the machine. This tracking device has the manufacturer's own transmission protocol and can transmit tracking information including data on the position, angle, and lens status (zoom and focus) of the camera 200 using a common UDP-type protocol called FreeD.

The playback device 300 is an electronic device capable of computing, stores information for real-time rendering CG (computer graphics) (i.e., information about media to be displayed on the LED wall 100), and plays the media. You can. In particular, the playback device 300 plays the linked image based on the tracking information of the camera 200 transmitted from the tracking device (hereinafter, this may be referred to as “camera tracking linked association”) so that it is displayed on the LED wall 100. You can. At this time, the playback device 300 that plays the camera tracking linked video may also be referred to as “N Player.”

At this time, the camera tracking linked video is an video in which the contents of the media being played are changed by reflecting the movement of the camera 200 and the state of the lens. In other words, when the camera 200 moves or the lens state changes, the virtual camera (VC) that photographs (captures) the contents of the virtual space is linked within the virtual space according to the change, thereby affecting the content within the virtual space. An image whose position or angle has been changed (i.e., camera tracking-linked video) is captured by a virtual camera (VC), and the camera tracking-linked image captured in this way is played by the playback device 200 and displayed on the LED wall 100. can be displayed. To this end, a tracking device that determines the movement and lens status of the camera 200 transmits the tracking information to the playback device 300, and the playback device 300 plays a camera tracking linked video based on the received tracking information. .

For example, electronic devices include desktop personal computers, laptop personal computers, tablet personal computers, netbook computers, workstations, and smart pads. Or it may be a media playback device, but it is not limited thereto.

For example, the playback device 300 may be equipped to play each of the n small unit displays of the LED wall 100. In particular, in the case of the N player, since there is a lot of data related to camera tracking linked video, n N players may be provided to play the divided camera tracking linked video in charge of each small unit display.

In other words, the N player is a player that divides the virtual space image displayed on the LED into n zones and plays it. Accordingly, n N players run on each of the n rendering capable electronic devices, and each N player communicates with the control program in real time and exchanges control signals. However, in the case of N Player, it may not be possible to simply split and display a large video on a large screen. For example, for In Camera VFX shooting, N Player projects the movement and lens state of the real camera 200 onto the virtual camera and sets the area (frustum) contained in the lens of the real camera 200 to a higher resolution than other areas. It can be visualized, and the shape of the LED wall 100 can be reflected and distorted. Accordingly, when viewed from the perspective of the viewfinder of the camera 200 (i.e., in the first image captured by the camera 200), the virtual space according to the media displayed on the LED wall 100 appears as if it were an actual three-dimensional space. can be displayed.

Figure 2 shows a conceptual diagram of virtual production shooting using the system 10 according to an embodiment of the present invention.

That is, referring to FIG. 2, the first image V1, which is a live-action image, is captured through the camera 200. At this time, the tracking device generates tracking information about the movement of the camera 200 and the state of the lens, and the generated Tracking information is transmitted to the playback device 300. Of course, a server (not shown) that receives tracking information from the tracking device and transmits it to the playback device 300 may be included between the tracking device and the playback device 300. Afterwards, in the playback device 300, the media of the second video V2 (i.e., camera tracking linked video) containing the content B of the virtual space linked based on the received tracking information is displayed on the LED wall 100. Play it as much as possible.

At this time, the content (B) of the virtual space is real-time rendering CG (computer graphics) generated by a game engine, etc., and corresponds to content that can be changed in conjunction with tracking information that tracks the movement of the camera 200 and the state of the lens. . For example, the content of the virtual space may be media produced by Unreal Engine or Unity3d Engine, but is not limited thereto.

The content (B) of the virtual space is CG content prepared in advance in the above-mentioned pre-planning stage, and corresponds to the content of the camera tracking linked video. In a virtual space with 3D virtual content, the virtual camera (VC) It corresponds to the part that was filmed (acquired). At this time, the playback device 300 causes the virtual camera (VC) to change movement and lens status in conjunction with tracking information at a reference position in the virtual space, thereby capturing the content (B) of the virtual space, thereby capturing the second image (V2). ) can be created and played.

FIGS. 3 and 4 show various examples of movement of the virtual camera (VC) linked with the movement of the camera 200 during filming according to FIG. 2. However, FIG. 4 shows a case where the reference position (R) in FIG. 3 is changed.

Meanwhile, the second image V2 played by the playback device 300 and displayed on the LED wall 100 contains content in a virtual space that reflects the movement of the camera 200 and the state of the lens. That is, in the present invention, the second image (V2) is a virtual camera (VC) whose movement and lens state are linked based on tracking information targeting media in a virtual space in which the source of the two-dimensional image has been image-processed, which will be described later. It may be created by filming (capturing) media in the virtual space while being linked according to the tracking information.

Accordingly, the playback device 300 generates a second image (V2) containing the content (B) of the virtual space linked based on the tracking information and plays it to be displayed on the LED wall 100, and in this playback state, the camera In the first image (V1) captured by (100), there is a person (A) or object (O), and the content (B) of the second image (V2) displayed on the LED wall (100) (i.e., a two-dimensional image The content of a three-dimensional image obtained by processing the source as described later may also be included.

For example, as shown in FIGS. 3 and 4, when the camera 200 moves from the first position to the second position, the virtual camera (VC) reflects the position change based on the reference point (R). As the location changes, content B in the virtual space may be captured and included in the second image.

That is, in the case of FIG. 3, the virtual camera (VC) that is taking pictures by capturing the reference point (R) on the side of the content (B) of the car in the virtual space may be linked to the movement of the camera 200 in one direction. Accordingly, the side portion of the content B captured while the position of the content B of the car in the virtual space is changed along one direction may be included in the second image. Additionally, in the case of FIG. 4 , a virtual camera (VC) that is taking pictures by capturing a reference point (R) in front of the content (B) of a car in a virtual space may be linked to the movement of the camera 200 in one direction. Accordingly, the front portion of the content B captured while changing the position along one direction relative to the front of the content B of the car in the virtual space may be included in the second image.

Figure 5 shows a block diagram of the playback device 300.

As shown in FIG. 5, the playback device 300 may include an input unit 310, a communication unit 320, a display 330, a memory 340, and a control unit 350.

The input unit 310 generates input data in response to various user inputs and may include various input means. For example, the input unit 310 includes a keyboard, key pad, dome switch, touch panel, touch key, touch pad, and mouse. It may include (mouse), menu button, etc., but is not limited thereto.

The communication unit 320 is a component that performs communication with other devices. For example, the communication unit 320 may receive tracking information from a tracking device or server through the FreeD protocol. For example, the communication unit 320 supports 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energy (BLE), near field communication (NFC), Wireless communication such as WiFi communication may be performed, or wired communication such as cable communication may be performed, but is not limited thereto.

The display 330 displays various image data on a screen and may be composed of a non-emissive panel or an emissive panel. For example, the display 330 may display a first image or a second image. For example, the display 330 may be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a micro electromechanical system (MEMS). It may include, but is not limited to, a mechanical systems display, an electronic paper display, etc. Additionally, the display 330 may be combined with the input unit 310 and implemented as a touch screen or the like.

The memory 340 stores various information necessary for the operation of the playback device 300. For example, the stored information in the memory 340 may include a first image, tracking information, information about real-time rendering CG, a second image, a composite image, a control program, and information related to a processing method to be described later, but is limited thereto. It doesn't work.

For example, the memory 340 is classified into hard disk type, magnetic media type, CD-ROM (compact disc read only memory), and optical media type depending on its type. ), magneto-optical media type, multimedia card micro type, flash memory type, ROM type (read only memory type), or random RAM type. access memory type), etc., but is not limited thereto. Additionally, the memory 340 may be a cache, buffer, main memory, or auxiliary memory, or a separately provided storage system depending on its purpose/location, but is not limited thereto.

The control unit 350 can perform various control operations of the playback device 300. That is, the control unit 350 can control the execution of a control program stored in the memory 340 and a processing method to be described later. Additionally, the control unit 350 can control the operations of the remaining components of the playback device 300, that is, the input unit 310, communication unit 320, display 330, and memory 340. For example, the control unit 350 may include at least one processor, which is hardware, and a process, which is software executed on the processor, but is not limited thereto. For example, processors include microprocessors, central processing units (CPUs), processor cores, multiprocessors, application-specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). ), etc., but is not limited thereto.

Hereinafter, when a source (also referred to as a “flat source”) of a two-dimensional image (also referred to as a “flat image”) is displayed on the LED wall 100, the first image captured by the camera 100 is displayed in a more three-dimensional manner. We will explain how to image process the source of the two-dimensional image (i.e., make the flat source three-dimensional) so that it can be viewed.

First, we will explain the necessity of technology for flat source three-dimensionalization.

<Necessity of technology for three-dimensionalization of flat sources>

For filming virtual production, there is a limitation that it is difficult to quickly secure a high-quality photo-realistic rendering source displayed on the LED wall 100. This is because there is a difference in the method of producing 3D assets, which are the basic unit of virtual space, between the existing 'post-compositing method' and the 'real-time rendering' of virtual production. In other words, 3D assets in the existing 'free rendering' method create the shape of an object by increasing the number of vertices of the mesh and then use it for rendering. On the other hand, for 'real-time rendering' of virtual production based on a game engine, the complex vertex information that determines the shape of the object must be converted to a texture format as much as possible to reduce the number of vertices in the mesh, or go through an optimization process. Accordingly, virtual production requires additional production technology, manpower, and time in this process. Therefore, recently, VFX companies are using commercial markets that sell real-time assets (such as Quixel) or building their own asset libraries. Nevertheless, there are significant difficulties in freely using live-action filming sources containing real-world locations or characteristic backgrounds or quickly securing assets of hyper-realistic quality.

(1) Difficulties in optimizing 3D assets for real-time rendering

The key to creating 3D assets for real-time rendering of virtual production is that they must be lightweight. In other words, unlike the pre-rendering method, which allows rendering times of minutes to hours per frame on high-end servers, the game engine renders scenes at 30 to 60 frames per second every moment. This is the way to do it. Therefore, the number of vertices that make up the shape of the 3D asset that makes up the background should be minimized, and various information that determines visual details should be expressed through PBR Textures such as normal map and ambient occlusion map, which have relatively low load. do. In this way, the production of 3D assets for real-time rendering requires a separate process for performance optimization and human resources with expertise. One thing to add is that, unlike the production pipeline of game companies that are familiar with such optimization work, VFX companies that mainly deal with CG for movies and dramas lack the manpower or expertise to perform asset optimization work.

(2) Difficult to produce CG quality at the level of real-life action (Need to improve the game engine’s unique CG quality)

Virtual production basically has the great advantage of being able to instantly change visual elements such as spatial location, background composition, and light angle at the filming site through a game engine such as Unreal or Unity3d. The quality of graphics is also improving rapidly due to hardware acceleration technologies such as Realtime Raytracing and DLSS.

However, the CG quality of real-time rendering is inevitably somewhat lower compared to existing pre-rendering methods or live-action shooting. In particular, light reflection, surface details, and the naturalness of shadows still remain as tasks. Therefore, current game engine-based virtual production technology is mainly used in science fiction movies and fantasy genres where a little CG is allowed.

Of course, if you scan the entire real space itself through photogrammetry or laser scanning and use it optimally, a significant level of photorealistic expression is possible through a game engine. This also requires separate spatial scanning and asset optimization processes, and requires considerable time and technique. Therefore, in order to further expand the virtual production market and maximize efficiency, various technologies are needed to quickly and effectively produce photorealistic background assets.

Therefore, in the present invention, we would like to propose a technology for quickly and easily creating assets for space composition by utilizing sources of general two-dimensional images. In other words, by providing a method of creating a hierarchy by distance and giving a sense of thickness to flat sources (photos, videos), rather than configuring space through existing 3D assets, CG designers can choose options depending on conditions and situations. We want to expand the scope.

Next, we will explain the three-dimensional constraint problem of flat sources.

<Three-dimensional constraint issue for flat sources>

The virtual space of CG is filled with mesh data composed of three-dimensional information. Mesh is composed of three-dimensional points called vertices, and the surface is covered with an image called texture. At this time. The way the surface of the mesh reacts to light, i.e. the material, is determined by the shader code, which allows free movement and exploration within the virtual space composed of the mesh.

On the other hand, 'flat sources', such as general photos or videos, are a state in which a specific space is compressed into a square frame (Frustum) based on the angle of view of the camera lens, and free transformation is not possible. In other words, both distant and near objects are compressed and recorded on a plane called frustum, regardless of distance. In the case of a flat source photographed in this way, assuming that you are standing in the center of the photo and looking at it, it is perceived as a space to some extent due to perspective, where nearby objects appear large and distant objects appear small. However, when looking at a large photo frame in front of us, if we move left or right beyond the center of the photo, our eyes immediately recognize that it is not three-dimensional but flat. In order to understand this issue of three-dimensional disruption, it is first necessary to understand how people perceive space in three dimensions.

‘Spatial sense’ is what allows people to feel space as three-dimensional rather than flat. It goes through the process of combining the information on the plane coming from the two eyes to determine the sense of distance, or inferring space through partial information such as the Parallax Effect or vanishing point. Of course, physical movement through equilibrium recognition through the cochlea is also added, making it possible to understand the relationship between oneself and space in real time.

The important point here is that when people form a sense of space, they do not absolutely rely on a single method, but rather combine various information in real time and make a judgment through inference. A good example is motion sickness, which occurs when information from the sense of vision and sense of balance are different.

Accordingly, the present invention seeks to propose a method of making the source of a two-dimensional image look as three-dimensional as possible. To this end, we would like to present a technology that can quickly and easily create assets for space composition by modifying existing flat sources using as many elements as possible from the 'methods of how people create a sense of space' listed below.

Next, we will explain how people create a sense of space.

<How people form a sense of space>

1) Three-dimensional space perception due to binocular parallax

First, people can perceive a sense of space by perceiving three-dimensional space through the difference in images caused by the distance between the two eyes (i.e., binocular parallax). Using this principle, 3D movies that allow the left and right eyes to perceive two images shot with two cameras have become popular. However, the present invention does not cover 3D stereoscopic images. (3D stereoscopic images are not used much these days due to motion sickness, production efficiency, etc.)

2) Recognition of spatial sense due to Parallax Effect when moving left or right

Second, people can perceive a sense of space based on the Parallax Effect. At this time, the parallax effect refers to the phenomenon that when moving left or right while looking at a specific space, objects in the near field pass quickly and objects in the distance pass slowly. In movies, the technique of expressing the depth of space by slowly moving the camera sideways is often used. In the real world, this parallax effect is very natural, but if you move left and right with a large photo in front of you, all objects will pass at the same speed regardless of the near or far distance. In that case, people perceive it as a wall or plane rather than space. For this reason, the technique of shooting general photos or videos by displaying them on a large LED wall must be used very carefully. In particular, it doesn't matter if you film with the camera fixed, but as soon as you move the camera, it feels quite unnatural, and it looks like a movie made in the 50s or 60s with a painting set in the background.

Figure 6 shows an example of a vanishing point.

3) Perception of depth through vanishing point and perspective

Third, as shown in FIG. 6, a person can perceive a sense of space by perceiving a sense of depth based on a vanishing point. At this time, the vanishing point refers to the phenomenon where two parallel straight lines become closer as they become farther apart and eventually meet. This vanishing point can be explained as a phenomenon in which train tracks converge in one place as they become farther apart. In fact, people can naturally recognize vanishing points and perspectives through the shapes of objects placed in space or the patterns of the floor, and judge distance or nearness based on that. This virtual parallax structure (perspective) changes in real time depending on the viewer's location. However, in the case of flat photos, the vanishing point is already fixed based on the shooting point, so even if the person's viewing position changes, the vanishing point does not change, and the human brain immediately recognizes that it is not three-dimensional.

4) Identify three-dimensional shape through the direction of light and the light and dark of the object

Fourth, people can perceive a sense of space by determining whether it is three-dimensional through the direction of light and the light and dark of an object. In other words, people can estimate the three-dimensional effect of a specific object based on the principle that the side facing the light source is bright and the side opposite is dark. In the case of flat photos, although contrast is expressed, it has the limitation of not being able to change the direction of light because it is the light that was cast on the subject at the time of photography.

5) Perception of distance through changes in eye focus

Fifth, people can perceive a sense of space by perceiving distance through changes in eye focus. Even when filming a movie, a sense of space is expressed by adjusting the camera's focus from a close-up person to a distant person. In order to grasp an object by changing focus, whether in real or virtual space, the object must have three dimensions (i.e., a sense of volume). However, in the case of photos, unlike Mesh, which is three-dimensional information, all depth information is compressed in a flat frame called Frustum, so it cannot have a sense of volume and the focus cannot be changed.

Figure 7 shows an example of a fog phenomenon.

6) Estimation of depth due to fog phenomenon due to density of atmospheric dust

Lastly, people can perceive a sense of space by estimating depth based on the fog phenomenon that appears due to the density of atmospheric dust. In other words, in a landscape photo, objects in the background have high saturation and contrast and appear clear, while mountains in the distance have low saturation and contrast and appear blurry.

Next, we will explain the limitations of planar sources.

<Limitations of planar sources>

As we have seen, the way humans perceive space is complex, whereas flat sources (photos or videos) contain only a limited level of three-dimensional elements (perspective, vanishing point, and atmospheric fog). Therefore, in constructing a CG environment that is a three-dimensional space, 3D Mesh objects, which are complete three-dimensional information, are mainly used rather than using flat photos or videos as is. In the case of flat sources (2D photos, videos), 'panoramic images' are used to a limited extent to express distant scenes such as distant mountains or the sky. However, in recent large-scale LED studios (i.e., studios that provide a shooting space using LED walls), shooting methods using flat sources are also widely used. However, there are significant difficulties due to the three-dimensional limitations discussed earlier, and below are examples of limitations that occur when shooting using a flat source in an LED studio.

Figure 8 shows an example when a camera takes pictures from the side of an LED wall.

(1) The perspective structure in the photo collapses on both sides of the LED wall.

When taking photos by placing them on a large LED wall, relatively natural shooting results can be obtained in the center of the studio. This is because all flat images captured through lenses and sensors can only provide the same sense of perspective as reality only in the very center of the image. However, as shown in Figure 8, when shooting is performed with the camera moved to be adjacent to the left and right sides of the LED wall, the perspective structure in the photo of the corresponding planar image within the first image (V1) captured by the camera As it becomes different from reality, a significant sense of heterogeneity appears.

Figure 9 shows an example of shooting with a camera when the LED wall has a bent surface shape.

(2) The flat source is distorted depending on the surface shape of the LED wall.

When a general flat source is placed on an LED wall for filming, the image is mapped as is according to the shape of the LED wall surface. However, as shown in FIG. 9, when a flat image is placed on a bent portion of the LED wall, the image appears to be bent even in the first image V1 captured by the camera. Although this is physically natural, this limitation becomes a very serious limitation in fully utilizing an expensive LED studio.

Figure 10 shows an example of the absence of the parallax effect that appears when shooting with a camera while simply displaying the source of a two-dimensional image on an LED wall.

(3) Due to the absence of parallax effect, it is perceived as a simple wall rather than a space.

The Parallax Effect is a phenomenon in which, when moving left or right in a specific space, objects in the near field appear to pass quickly and objects in the distance appear to pass slowly. People can roughly perceive the depth of the target space through this parallax effect. However, as shown in FIG. 10, when a flat image is simply placed on an LED wall as a background and photographed at various locations, the parallax effect of the background naturally does not appear. Accordingly, people watching this immediately perceive the background displayed on the LED wall as a wall with a sense of space. For example, criticism that the CG compositing of the mountaineering scene in the drama 'Jiri Mountain' was awkward became an issue. In this case, too, live footage was used as the background for compositing, and the moment the camera moved, the unnatural parallax effect broke the sense of empathy. It happened.

(4) The conditions at the time of source shooting and the configuration of the LED wall must be the same.

Therefore, one way to effectively use a flat source is to make the camera configuration (placement, angle of view, distance, etc.) when shooting the flat source completely the same as the composition of the actual LED wall. In this case, quite natural shooting results can be obtained in special scenes, such as shooting inside a car. However, this method can also only shoot within a very limited and narrow area, such as the front of a vehicle or the interior. Additionally, there is a problem in that LED studios must be used only for very limited purposes, such as automobile scenes.

Next, we will explain the possibility of using planar sources.

<Possibility of using planar sources>

As we saw earlier, there are significant limitations to using planar sources in an LED studio. In other words, natural results are guaranteed only when the camera is fixed or any movement is minimized. If you break this and film while moving the camera freely, you will end up with an unsophisticated composite look like a 1950s movie.

However, the demand to use flat sources that can quickly and easily secure high-quality (realistic expression, high resolution) data as the background of an LED wall or as a spatial element of virtual production content is continuously growing. Accordingly, the present invention Based on the analysis of the advantages and disadvantages of flat sources, we try to maximize the advantages by giving various elements that make the flat source look three-dimensional.

Next, a processing method according to an embodiment of the present invention that utilizes the real-time spatial rendering characteristics of virtual production and at the same time increases the degree of freedom in shooting by applying as many three-dimensional effect forming elements as possible to the flat source (hereinafter referred to as “the present method”) (referred to as ").

This method is an image processing method performed under the control of the control unit 350 of the playback device 300, and is a method of image processing a source of a flat image (i.e., a flat source) so that the source of the flat image becomes three-dimensional. That is, this method performs three-dimensional image processing on the flat image so that the first image captured by the camera 100 can be viewed in a more three-dimensional manner. Accordingly, the image processed according to the method of viewing the source of the flat image is reproduced by the playback device 300 and displayed on the LED wall 100, and the camera 100 for photographing the display portion of the LED wall 100 The first image can be viewed in a more three-dimensional manner.

To this end, we will explain the three-dimensionalization process of the planar source used in this method. This three-dimensionalization process may be performed under the control of the control unit 350 of the playback device 300.

<Three-dimensionalization process of a flat source>

Figures 11 and 12 show examples of conceptual diagrams for anamorphic distortion.

(1) First process: Placing flat sources in the virtual space of virtual production

First, rather than displaying the flat source as is on the LED wall 100, it is placed within the processing space of the Virtual Production content. At this time, among the Virtual Production methods, ICVFX (In camera VFX) uses the LED wall 100 facing the camera 200 so that the CG can be viewed normally from the perspective of the camera 200, regardless of the shape of the LED wall 100. ) is a technique to reversely distort the graphics to be output on the LED wall 100 by processing the corresponding CG according to the position or shape of the image. This method is sometimes referred to as “Anamorphic distortion (or anamorphic mapping).”

For example, referring to FIG. 11, the CG portion displayed on the display bent toward the lower part of the LED wall 100 may be image-processed to be displayed elongated along the lower direction, and may be displayed in the left direction from the LED wall 100. The CG portion displayed on the bent display may be image-processed to be displayed stretched along the left direction. On the other hand, the CG portion displayed on the display of the LED wall 100 that faces the camera 200 almost directly may be image-processed to be displayed almost like the original CG portion.

When applying this method, in the first image captured by the camera 200, the shape of the LED wall 100 disappears, and the CG of the virtual world is displayed without change, so that the CG is the physical image of the LED wall 100. It appears as if there is a hole the size of the hole. In other words, it is the same concept as looking into another world beyond the open window of the LED wall 100. Therefore, the flat plate for the flat image proposed in the present invention and the image-processed object (i.e., curved plate or layered plate, etc.) for three-dimensionalization of the flat plate are basically placed in the virtual space (i.e., CG space). It is assumed that the image will be visualized normally regardless of the LED wall 100 through the ICVFX technique.

In particular, among the limitations of the flat source discussed above, in the case of a problem in which the flat source appears distorted depending on the physical shape of the LED wall 100, as shown in FIG. 12, a plate of the flat source in virtual space is used. (hereinafter also referred to as a "flat plate") is placed and the second image of the virtual camera (VC) that captured the flat plate is processed through anamorphic mapping centered on the viewpoint of the actual shooting camera (100) to create an LED wall. This can be solved naturally by marking (100). However, even if this method is used, distortion due to the shape of the LED wall 100 can be resolved, but the source itself is still flat, so it is not possible to obtain a three-dimensional car body. At the current stage, in the case of the first image captured by the camera 100, it will appear as if a large photo frame is placed beyond a large space open the size of the LED wall 100.

Figure 13 shows a conceptual diagram of creating a curved plate using a flat plate (reference plane).

(2) Second process: Creating a curved plate considering the three-dimensionality of real space

This is the step of adding various three-dimensional elements to the flat plate previously placed in virtual space (CG). Through this, when viewed from the perspective of the actual camera 200, it is possible to obtain a visual scene that is as close to the real thing as possible, and at the same time, it is possible to overcome many of the movement restrictions of the camera 200.

The first step is to create a mesh of a hemispherical curved plate that is concave (i.e., the side facing the flat plate is concave) with respect to the reference position using a flat plate in virtual space. At this time, the important point is that both the flat image mapped to the flat plate and the curved image mapped to the curved plate must look exactly the same when viewed from the shooting origin (i.e., reference position) of the virtual camera (VC). In other words, the reference position is the position where the virtual camera (VC) is spaced at regular intervals from the center point of the flat plate. At this time, when the virtual camera (VC) placed at the reference position photographs a flat plate spaced at a certain interval, the flat image corresponding to the flat plate is exactly matched to the second image captured by the virtual camera (VC) (i.e., the second image captured by the virtual camera (VC) is accurately matched to the flat plate. 2 The image is filled with a flat image (hereinafter, this arrangement is also referred to as the “standard arrangement”).

Additionally, when the virtual camera (VC) views the flat Plate on which the reference arrangement was performed at the reference position, the corresponding flat Plate may be referred to as a “reference plane.” Reference arrangement for this reference plane can be easily performed using information about the camera setting state (i.e., shooting information) at the time the plane image, which is the source of the corresponding plane plate, was captured. At this time, the shooting information may include the sensor size and focal length of the camera at the time of the shooting.

In order to create a curved plate, it is desirable that each vertex forming the reference plane and each vertex forming the curved plate correspond to the reference position as the origin. In other words, when drawing an extension line for each vertex from the reference position toward the reference plane, the image displayed at the vertex of the reference plane where the extension line meets the reference plane, and the image displayed at the vertex of the curved plate where the extension line meets the curved plate. must be completely identical. At this time, in each extension line, the distance between the reference position and each vertex of the curved plate has the same distance (k), so that the curved plate can be formed in a hemispherical shape.

That is, referring to FIG. 13, the first extension line meets the first point (A) of the reference plane and the second point (A'') of the curved plate, and the second extension line meets the third point (B) of the reference plane. It meets the fourth point (B'') of the curved plate. At this time, the image of the first point (A) is the same as the image of the second point (A''), and the image of the third point (B) is the same as the image of the fourth point (B''). In addition, the length (k) of the first extension line corresponding to the length between the reference position and the second point (A'') is the length (k) of the second extension line corresponding to the length between the reference position and the fourth point (B'') ) These conditions apply equally to the remaining reference planes and each vertex of the curved plate. Hereinafter, the distance (k) between the reference position and the curved plate may be referred to as the “curved surface separation distance.” This curved surface separation distance can be selected and input by the user through the input unit 310.

As an example, you can create a mesh of a curved plate by following the steps below (performed in virtual space).

1) Create a mesh for the reference plane from the plane source in virtual space. At this time, the number of vertices of the mesh is based on the set number of grids.

2) Based on the information at the time of shooting the plane source (camera sensor size, lens angle of view, etc.), the reference plane is placed at a certain distance from the reference position to perform reference arrangement.

3) Copy the mesh of the reference plane and create a layer mesh with the same vertices (vertex) and uv (image mapping information) at a specific location. The location is the layer distance value entered in advance. This value becomes k, the distance (curved surface separation distance) between the 'reference position' and the center point of the 'layer mesh'.

4) The position of each vertex of the layer mesh starts from the reference position and is readjusted to a point where the length becomes k while passing through the same vertex of the reference plane mesh.

5) Through this, the mesh of the curved plate is naturally reconstructed with each vertex in the center moving away and the edge vertices getting closer. In other words, as the angle of view increases and the distance (k) increases, the mesh shape of the created curved plate becomes closer to a hemisphere. Additionally, as the distance (k) increases, the created curved plate has a larger area.

6) Finally, when the virtual camera (VC) looks at the flat plate and the curved plate from the reference position, the image of the flat plate and the image of the curved plate appear completely identical.

The reason for creating a separate curved plate that looks the same as the flat plate of the reference plane when viewed from the reference position is to provide a more realistic perspective. In other words, in the case of a photograph, which is the result of compressing a three-dimensional space with a sense of depth into a single plane, the view is the same as the real thing only when you look at the exact center of the image. However, if you think of yourself as standing in the actual space in the photo, the entire environment surrounds the photographer in a circle.

Figure 14 shows an example of photographing a flat plate with a virtual camera, and Figure 15 shows an example of photographing a curved plate with a virtual camera.

Referring to Figures 14 and 15, the virtual camera (VC) is looking at each plate with a bias to the left rather than the exact center. Therefore, in the case of a curved plate with a structure surrounding a space, the second image is captured without an empty portion, as shown in FIG. 15, but in the case of a flat plate, an empty portion is included as shown in FIG. 14. You can see the second video being shot. In addition, it can be seen that the second image for the curved plate, which is centered around the virtual camera (VC) and surrounds the virtual camera (VC) like in the real world, is expressed more naturally than the second image for the single-sided plate.

This difference in naturalness becomes more noticeable when the virtual camera (VC) moves rather than a still image. In other words, when the virtual camera (VC) moves, the distant view of the flat plate looks like a picture frame hanging on the wall in front of it rather than feeling like it is far away. In conclusion, when using a curved plate, the range in which the virtual camera (VC) can move is significantly increased compared to a flat plate. In particular, it is possible to obtain a result without distortion (i.e., a second image) even if not only the position but also the direction (pan, tilt) of the virtual camera (VC) is changed.

Of course, the CG industry is already using a method of visualizing distant views such as the sky or very distant mountains by mapping 'HDRI panoramic photos' to the inside of a sphere. However, panoramic photos require separate shooting equipment or complex image processing and are mainly used only to express a complete distant view. On the other hand, the purpose of the present invention is to naturally arrange general photos with an angle of view (FOV) in space, rather than panoramic photos. However, in the case of general photos, unlike 360-degree panoramic photos that simply cover a sphere, it is very difficult to determine the exact shape or size of the curved plate being mapped. Therefore, in the present invention, for three-dimensional visualization of a flat source, the target flat image and the shooting information (sensor size, angle of view, distance), which is information about the camera setting state at the time the flat image was captured, are placed on the reference plane. After performing the reference arrangement, an accurate curved plate for the reference plane can be created as described above.

When the second image captured by a virtual camera (VC) on the curved plate created in this virtual space is played back on the LED wall 100 (of course, the anamorphic distortion described above is performed in this case as well), the camera that captured it (200 ) In the first image, the portion of the image played on the LED wall 100 can be recognized as a three-dimensional space with a certain degree of depth.

(3) Third process: Layer division through formation of Parallax Effect

Previously, a method of mapping a flat image to a curved plate was used to create a sense of space and overcome the location constraints of the virtual camera (VC). The advantages of this curved plate's hemispherical mapping method are further maximized for distant images. However, if near-ground, middle-ground, and far-view content exist simultaneously in the source of a flat image, if the virtual camera (VC) moves due to the lack of depth, a second image of an awkward scene will be produced. In other words, when the virtual camera (VC) moves left or right, the near view must appear to move quickly and the far view slowly due to the parallax effect, but this effect is not possible with a single flat image.

Therefore, the present invention proposes a method of generating a separate curved plate mesh for multiple images separated by distance from a flat source (hereinafter, each such image is referred to as a “separated image”).

FIG. 16 shows an example of generating each layer (Lyaer) for a plurality of separate images divided by distance with respect to the plane source of FIG. 7 in virtual space, and FIG. 17 shows each layer (Lyaer) according to FIG. 16 in virtual space. It shows how to create each curved plate for each layer of a separate image.

At this time, referring to FIG. 16, the first division image (Layer 1) may be an image including a portion corresponding to a closer view than the second division image (Layer 2) in the planar source, and the second division image may be an image in the plane source. It may be an image that includes a part corresponding to the near-field than the third division image (Layer 3). Accordingly, the first segmented image may be an image including a portion corresponding to the near field in the planar source, the second segmented image may be an image including a portion corresponding to the middle view in the planar source, and the third segmented image may be It may be an image containing a remote portion of a planar source. At this time, the first to third division images may be stored and utilized as separate layers, and each layer (Layer 1, Layer 2, and Layer 3) has the same plane size as the original plane source (i.e. has a two-dimensional size).

In addition, as shown in FIG. 17, a curved plate for the first segment image (hereinafter referred to as “first curved plate”) and a curved plate for the second segment image (hereinafter referred to as “second curved plate”) ) and a curved plate for the third segment image (hereinafter referred to as “third curved plate”) are created separately.

That is, the first curved plate is a curved plate for the first segment image created at a curved separation distance of the first distance (k1) with the reference plane of the first segment image placed as a reference. In addition, the second curved plate is a curved plate for the second segment image created at a curved separation distance of the second distance (k2) with the reference plane of the second segment image placed as a reference. Likewise, the third curved plate is a curved plate for the third segment image created at a curved separation distance of the third distance (k3) with the reference plane of the third segment image placed as a reference. Hereinafter, these curved plates for each distance (e.g., first to third curved plates) may be collectively referred to as “Layered Plate.”

Meanwhile, there may be various ways to generate each segment image divided by distance from a flat image. For example, the user can directly separate each segment image from the flat image using an image editing tool such as Photoshop, separate each segment image from the flat image using image processing techniques such as segmentation, or automatically use machine learning to separate each segment image from the flat image. A method of classifying each segment image by estimating depth from a flat image may be used, but the present invention is not limited to this.

As an example, the procedure for creating a Layered Plate is as follows. However, the process of creating the mesh of each curved plate for each classification image is the same as the second process described above.

1) Determine how many layers to be divided into depending on the distance from the target flat image (i.e., the number of images divided by distance). Hereinafter, a case where the images are divided into three (i.e., first to third division images) will be described.

2) In the target planar image, the near-ground, middle-ground, and far-field areas are separated, and the first to third segmented images each including the divided areas are stored in the memory 340.

3) Set information about the camera settings at the time of shooting (e.g., sensor size, angle of view, etc.). In general, since each segment image extracted from the same plane image is used, the angle of view value for each segment image is set to the same value. However, when inserting an image extracted from another plane image into a specific classification image, the size of the placed mesh can be adjusted by changing the angle of view value of the corresponding layer.

4) Set the distance at which each curved plate will be located according to each classification image (i.e., curved surface separation distance). That is, the smallest (i.e. short) curved surface separation distance is set for the first layer (Layer 1), the intermediate curved surface separation distance is set for the second layer (Layer 2), and the third layer (Layer 2) is set. For 3), set the curved separation distance to the largest value (i.e., farthest). At this time, it may be desirable to set a rough estimate for each layer and include a value as close to the actual value as possible. The more accurate the value of each curved surface separation distance is, the more closely the mesh of each curved plate can be created to the actual physical size of the object. For example, if you set a curved separation distance of 100,000cm for a layer of a separate image that actually contains a mountain 1km away, a mesh of a curved plate is created in real size at a distance of 1km from the reference location in virtual space.

FIG. 18 shows examples of second images captured while the virtual camera (VC) moves from left to right while facing each curved plate created according to FIG. 17 in virtual space. That is, Figure 18(a) shows the second image when the virtual camera (VC) is located on the left, and Figure 18(b) shows the second image when the virtual camera (VC) is moved to the right compared to the case of Figure 18(a). 18(c) shows the second image for the case where the virtual camera VC moves to the right compared to the case in FIG. 18(b).

5) Afterwards, when the virtual camera (VC) is moved left and right, as shown in Figure 18, the layers of each curved plate appear to move at different speeds due to the parallax according to the curved separation distance of each curved plate, Parallax effect may occur in the first and second images.

The important point here is that even if you place the layer images roughly in line with the eye, you can achieve a similar effect to a limited extent, but errors with actual physical phenomena cannot be prevented. In particular, when the virtual camera (VC) moves, the perspective may be broken or the perspective may feel unnatural. Therefore, as explained earlier in the curved plate mesh creation process, all vertices of the curved plate mesh of each layer (Layer 1, Layer 2, Layer 3) must start from the 'reference position' and end up at the same vertex of the reference plane mesh. A reference positioner located on a line passing through should be performed. Accordingly, no matter how many layers (Layer 1, Layer 2, ... are divided into several layers, and no matter what curved separation distance each layer (Layer 1, Layer 2, ...) is from the reference position, the virtual camera (VC) This means that the size of each curved plate appears the same when viewed from the 'reference position.' Through this, it is not a real 3D object, but is created separately according to each layer (Layer 1, Layer 2, ...) in virtual space. When the second image captured by the virtual camera (VC) of the curved plates is played back on the LED wall 100 (of course, the anamorphic distortion described above is performed in this case as well), the first image of the camera 200 that captured the curved plates is played back on the LED wall 100. Even if the portion of the image played on the LED wall 100 is used as a background for shooting people, the sense of space can be maximized without making it awkward.

(4) Fourth process: Expression of layer thickness considering the three-dimensionality of the target object

Previously, through the third process, we explained how to divide each separate image from a single flat image into layers (Layer 1, Layer 2, ...) by distance and arrange them in positions and sizes similar to the real thing. Now, in the next fourth process, The process is to give a sense of thickness to the curved Plate Mesh of each layer (Layer 1, Layer 2, ...). In other words, through the previous third process, each layer (Layer 1, Layer 2, ...) has its own angle of view and distance reflected. It is drawn on a curved Plate Mesh. This method is a good way to increase the overall sense of space, and is especially effective for distant scenes. However, when the virtual camera (VC) moves, each layer (Layer 1, Layer 2, … of my image) If the object is a structure that protrudes in the front direction (i.e., in the direction of the reference position) (e.g., a rock or mountain, etc., protrudes in the front direction), the second image captured by the virtual camera (VC) shows the actual structure. It feels like it's going inward (hereinafter referred to as the "protruding structure problem"). The reason is that the object is actually a structure that protrudes forward, but when mapped to the curved Plate Mesh, it is visualized as being pushed inward rather than in reality. Because it becomes.

FIG. 19 shows an example of the process of applying a depth map to each layer (Layer 1, Layer 2, and Layer 3) created according to FIG. 16. Additionally, FIG. 20 shows an example of applying a depth map to the third layer (Layer 3) among the layers (Layer 1, Layer 2, and Layer 3) created according to FIG. 16.

Therefore, the problem of the protrusion structure can be solved by applying a separate depth map to each layer (Layer 1, Layer 2, ..., as shown in Figure 20). Layers (Layers) Depth maps for Layer 1, Layer 2, ... are generated for each layer (for example, if there are 3 layers, there are 3 depth maps), and the application of each of these depth maps may be optional. Hereinafter, the first layer (Layer The depth map for 1) is referred to as the “first depth map”, the depth map for the second layer (Layer 2) is referred to as the “second depth map”, and the depth map for the third layer (Layer 3) is referred to as the “third depth map”.

That is, among the layers (Layer 1, Layer 2,...), there may be a layer to which a depth map is applied, and among the layers (Layer 1, Layer 2,...), there may be a layer to which a depth map is not applied. In this case, the depth The map can be created by painting with a brush in an image editing program such as Photoshop, or depth information can be extracted through depth estimation among machine learning methods, but the present invention is not limited to this.

The depth map has a plane size equal to the size of the corresponding layer, and each vertex (pixel) of the depth map corresponds to each vertex of the corresponding layer. At this time, each vertex of the depth map includes depth information about the degree of protrusion of each vertex of the corresponding layer. In other words, depending on the brightness of the image in the depth map, each vertex of the mesh of the corresponding layer may protrude forward.

For example, in the first depth map, the closer the vertex is to a light color (white) or a dark color (black), the corresponding first curved surface on which the vertex of the first curved surface Plate of the first layer (Layer 1) is placed as a reference. It may have information that causes it to protrude from the plate in a more forward direction (i.e., in the direction of the reference position). In other words, the vertex moves to a curved surface separation distance that is shorter than the original curved distance of the first curved plate. This can be equally applied to the remaining second and third depth maps, respectively. At this time, the important point is that, in any case, each vertex of each curved plate mesh must start from the 'reference position' and move to protrude according to the reference arrangement located on the line passing through the same vertex of the reference plane mesh. .

In addition, the maximum degree of protrusion can be set differently for each layer (Layer 1, Layer 2, ...). That is, the depth strength value can be adjusted for each layer (Layer 1, Layer 2, ...) to determine the thickness. You can adjust the strength of the effect.

Figures 21 and 22 are the results of shooting with a virtual camera after applying a depth map to the first layer (Layer 1) among the layers (Layer 1, Layer 2, and Layer 3) created according to Figure 16. Examples of the second image are shown. That is, Figure 21(a) shows a virtual camera (VC) photographing each curved plate from the front after applying a depth map to the first layer (Layer 1), and Figure 21(b) represents the second image for FIG. 21(a). In addition, Figure 22(a) shows the virtual camera (VC) being photographed from the right side of the reference position for each curved plate after applying the depth map to the first layer (Layer 1), and Figure 22 (b) shows the second image for FIG. 22(a). However, in Figures 21(b) and 22(b), the people are arbitrarily added to easily indicate each shooting direction.

Referring to Figures 21 and 22, when a Depth Map is applied to at least one of the layers (Layer 1, Layer 2, and Layer 3), thickness is immediately added to the curved Plate Mesh forming the corresponding layer.

In this way, if a flat image is layered into layers according to distance and a sense of thickness is given to each layer, it is possible to obtain a high level of three-dimensional effect and shooting freedom that existing flat sources cannot express. In particular, when used to give a sense of thickness to objects in the first layer (Layer 1), which is in the background, it not only provides high visual detail, but also helps improve rendering performance issues. In fact, compared to a typical 3D Asset composed of tens of thousands to millions of vertices, the curved Plate Mesh of each layer (Layer 1, Layer 2, …) uses only about 100 to 1,600 vertices, and the material The shader code that decides is also low in complexity, which is advantageous in securing performance.

(5) Real-time correction function for natural shooting

Through the previous steps, it was explained that a sense of space can be obtained by layering the planar source into each layer, giving it a sense of thickness, and then arranging it in space. Accordingly, the present invention is a stage in which the 'Layered Plate' made in this way is actually used, that is, in the stage of filming an image using a virtual production method in an LED studio equipped with an LED wall 100, according to the on-site situation or the director's opinion. It allows for easy real-time correction of the asset.

For example, real-time correction can be performed using the 'integrated control program' for virtual production shooting. Such a program may be installed in the playback device 300 or a control device (not shown) that is a separate electronic device connected to the playback device 300. The program may include a player that displays a virtual space on the LED wall 100 based on the physical movement of the camera 200 and the shape of the LED wall 100, and an 'integrated control program' that controls them. Through these programs, color differences between real space and actual space that may occur when playing CG content created by graphic designers on the LED wall (100), positional error adjustment between physical space and virtual space, and content timeline adjustment can be adjusted on site. You can do it.

The 'Layered Plate plug-in' for this can be added to 'N Player', which visualizes the second image played on the LED wall 100, and based on it, designers create several 'Layered Plate objects' and place them in space. and create content. In addition, a separate 'Layered Plate object' management interface has been added to the integrated control program, allowing the location, color, sensor size, angle of view, mesh grid, depth for each layer, etc. of each object to be changed and confirmed in real time. Through this, 'Layered Plate objects' created based on general photos or videos can be adjusted to match not only other CG spatial elements, but also physical elements (set, actor, lighting, etc.) within the filming location. If there is no separate correction interface, and urgent corrections are needed in the field, the content file must be opened directly, modified, distributed to each rendering server, and then played back on the LED wall (100) to check. This requires considerable time and procedures. However, considering the urgent shooting schedule, a method and interface that can control and correct in real time are essential.

So far, we have explained the methods for improving problems such as physical distortion or loss of three-dimensionality that occur when using flat sources by dividing them into several steps. The key is that by adding various types of three-dimensional elements to data from already flattened flat sources such as photos or videos, you can quickly and easily create a three-dimensional space using only existing flat sources, even if it is not a complete 3D asset.

Figure 23 shows an operation flowchart of a processing method according to an embodiment of the present invention.

This method includes the three-dimensional process of the above-described planar source. To this end, the present invention includes a first step of performing three-dimensional image processing on a flat image, and a second step of processing the image resulting from the three-dimensional image processing so that it is included in the second image and displayed on the LED wall 100. It can be included. At this time, the first step may include S110 to S140, as shown in FIG. 23. However, S130 may be selectively performed after S110 and S120 are performed, or S120 may be omitted and S110 and S130 may be performed. Afterwards, when S130 is performed, S140 may be optionally performed afterwards.

S110 is the step of placing the target flat source within the virtual space of virtual production. However, since this S110 is the same as described above according to the first process, detailed description thereof will be omitted.

S120 is the step of creating a curved plate in virtual space using a reference plane for the flat source arranged according to S110. However, since this S120 is the same as described above according to the second process, detailed description thereof will be omitted.

S130 is a step in which a separate curved plate is created in virtual space according to the separation distance between each curved surface for each layer of each separate image divided by distance from the flat source arranged according to S110. However, since this S130 is the same as described above according to the third process, detailed description thereof will be omitted.

S140 is the step of giving a sense of thickness to at least one of the curved plates using the corresponding depth map. However, since this S140 is the same as described above according to the fourth process, detailed description thereof will be omitted.

Meanwhile, the second step may include reproducing the second image by photographing (capturing) the curved plate while linking the virtual camera (VC) according to the movement of the camera 200 and the state of the lens. Of course, when multiple curved plates are arranged according to multiple separate image layers (Layer 1, Layer 2, etc.), the second step is when the virtual camera (VC) is linked according to the movement of the camera 200 and the lens state. It may include the step of photographing (capturing) the corresponding curved plates and playing the second image.

Hereinafter, the specific process of producing content according to virtual production shooting using this method will be described.

<Layered Plate object setting and use procedures>

In the present invention, a preparation process is different from the existing method of producing a full 3D asset or displaying an image on a general flat plate. At this time, the important point is that in order to revive the limited three-dimensionality of a specific image (photo or video), the more accurate the information (sensor size, lens angle of view) at the time of source shooting, the more precise the result will be. Based on such standard information, the location, size, and thickness of each layer are calculated. The specific procedures are as follows.

(1) Prepare the sauce

1. Prepare planar image source

Prepare a flat sauce for use in the Layered Plate. At this stage, you must first decide whether to create a 3D Mesh Asset in a general way, or to give a three-dimensional effect to a flat source such as a regular photo or video, considering production cost, period, or graphic quality. Both methods have pros and cons, so the designer can make a choice depending on the situation, and only the latter is covered in the present invention.

In the case of flat sources, you can use an existing image library for sale, or, if necessary, shoot the scene yourself. Since it is a source that will be divided by distance and placed in space, it is best to use a source where the spaces in the image are clearly divided by distance, such as near, middle, and far, if possible. Additionally, a higher degree of freedom can be achieved when using a horizontal image rather than an image looking down or looking up.

2. Create image files for each layer

The Layered Plate plug-in automatically creates a mesh for each layer with the appropriate location, size, and shape according to the image for each layer. To this end, first, prepare a layer image file processed according to at least one of the second to fourth processes based on the distance from a specific photo. In general, it is desirable to extract images from the same photo, but if necessary, images extracted from different photos can be used for a specific layer. In such cases, you must manually find a natural-looking value by adjusting the angle of view, distance, or thickness of the layer.

(2) Content production

This is the step of using the actual ‘Layered Plate object’ creation function after preparing the flat image source. In order to use this function, the 'Layered Plate plug-in' developed according to the present invention must be installed in Epic's Unreal Engine program. Afterwards, if you add a new 'Layered Plate object' on the Unreal Engine editing tool, an object that can be configured for various settings is created in the space.

1. Create Layered Plate object

When a Layered Plate object is created, the user must enter some basic information.

a) First, set how many layers there are.

B) Next, set how many vertices the mesh that determines the shape of each layer consists of. The setting method is to select the number of horizontal and vertical columns, such as 10x10 or 20x20. According to this value, the precision of the surface is determined when assigning depth to each layer later.

c) Next, enter the camera sensor size and angle of view information at the time the source image was captured. Only when this information is accurate can the actual physical size and shape be accurately calculated when creating a layer mesh later. However, even if you do not know the exact information, you can manually define an appropriate value by entering the estimate and comparing it with the size of other surrounding environments.

D) Next, the prepared images are assigned to each layer.

E) If there is a Depth Map image for each layer, apply it.

B) Next, for each layer, set the distance value where the mesh will be located. It is best to enter this value as accurately as possible based on physical facts. For example, if a specific mountain in the distance is located 3km from the shooting point, you must actually enter that value to create a layer mesh of the same size.

G) Lastly, enter the intensity at which the Depth Map is applied for each layer. Depth Map is a Gray Scale image with color values between 0 and 1, and sets the intensity of the color value at the actual distance. The farther the distance, the relatively larger the value.

2. Create layers

The values entered so far are used as base information for creating and visualizing each layer. What is especially important in the Layered Plate creation function is that all meshes and their vertices must be placed at accurately calculated positions based on the sensor size and angle of view (Fov) at the time of source shooting. In the first place, in the case of a flat image, various pieces of information in real space are compressed or some of them are missing or lost. For example, perspective is information that changes depending on the viewing position, but in a flat image, it exists only in a compressed form. Additionally, the actual sense of distance or thickness disappears altogether. Therefore, when some areas of the photo are cut out and roughly placed (by eye) or their size is adjusted, a perspective or perspective that deviates from the actual laws of physics can be created, creating a very incomplete image.

Accordingly, in the present invention, planar sources with limitations are divided into layers according to distance, and each provides a sense of thickness while preventing the angle of view and perspective of the original photo from being broken. For this, as explained earlier, all vertices of each layer mesh must be placed on a line passing through the corresponding vertices of the 'reference position' and 'reference plane mesh'.

The procedure for dividing layers and adding three-dimensional effect based on this layer mesh creation method is as follows. The contents below may be procedures performed within the program based on the basic information entered previously.

1) First, a 'reference plane' containing the original photo is created at, say, a 1m location. The number of vertices (grid configuration) previously set is reflected, and it is automatically generated considering the sensor size and lens angle of view.

2) The following procedures are then repeated n times depending on the number of layers.

3) First, create a layer mesh that copies the mesh information (vertex, uv, triangle) of the 'reference plane' as is.

4) Next, the copied planar mesh is moved to the set layer distance value position, and each vertex of the layer mesh is placed in the exact position in a plane form based on the sensor size and lens angle of view. At this time, when viewed from the reference position, the reference plane or the corresponding layer mesh must appear completely identical and overlap.

5) Next, make the shape of the layer mesh curved. All vertices of the layer mesh are rearranged based on the layer distance (i.e., curved surface separation) value, and the basic shape of the layer is transformed into a curved shape. Since each vertex is placed at the same distance from the reference position, when the angle of view (fov) increases, it approaches a hemispherical shape.

6) If there is a Depth Map assigned to each layer, a sense of thickness is given to each layer based on the corresponding image. This is a method of moving the vertex in the direction of the 'reference position' based on the pixel brightness of the corresponding point in the Depth Map. At this time, the key point is that the reference arrangement must be performed so that a specific vertex (A") of the layer mesh moves only on a line that passes through the reference plane (A) and the 'reference position'.

3. Place Layered Plate in virtual space

This is the step of placing the ‘Layered Plate objects’ created by the above procedure in the virtual space. In the case of a 'Layered Plate object', unlike a regular flat image, it is given some three-dimensional elements, but is not a complete 3D object. In other words, only one side of an object is fully visible, a similar concept to a movie set. Therefore, place the back or complete side of each object in an appropriate location where it is not visible. At this time, if necessary, values such as angle of view or distance of each layer can be adjusted to match the surrounding environment.

(3) LED studio shooting stage

The previous content creation stage explained what was covered when creating the first virtual space. At that stage, you can freely change or modify the content at any time. However, after the content is set up in Virtual Production mode in the LED studio, there are restrictions on modification or change. of course. Due to the nature of 'real-time rendering', important matters such as moving object positions or changing textures can be changed even at the shooting site. but. Due to tight filming schedules and concerns about errors, you should refrain from directly modifying the content itself if possible. Therefore, an interface that can correct the ‘Layered Plate object’ in real time can be added to the ‘integrated control program’. Through this, you can change colors such as gamma, brightness, and saturation, as well as the angle of view and distance for each layer, right at the desired time in the integrated control program without directly editing the content itself.

Each of the methods described above can be executed by loading the program into the memory 340 and executing the program under the control of the control unit 350. These programs may be stored in memory 340 on various types of non-transitory computer readable medium. Non-transitory computer-readable media includes various types of tangible storage media.

For example, non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), and CD-ROM (readable media). only memory), CD-R, CD-R/W, semiconductor memory (e.g., mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash ROM, random access memory (RAM)), It is not limited to this.

Additionally, the program may be supplied on various types of transitory computer readable medium. For example, temporary computer-readable media may include, but are not limited to, electrical signals, optical signals, and electromagnetic waves. That is, the temporary computer-readable medium can supply a program to the control unit 350 through a wired communication path such as an electric wire or optical fiber or a wireless communication path.

Hereinafter, the three-dimensional effect that appears according to the present invention configured as described above will be described.

<Comparison of three-dimensional effect before and after applying Layered Plate>

When placing and photographing an actual flat image (i.e., flat Plate) on the LED wall 100 (hereinafter referred to as the “first method”), and placing the Layered Plate in a virtual space according to the present invention and visualizing it using the Virtual Production method Compare the differences when doing this (hereinafter referred to as “the second method”). That is, the first method corresponds to the prior art, and the second method is a method proposed in the present invention.

In particular, we look at what differences arise between the two methods in terms of the three-dimensional effect formation elements listed below.

- Three-dimensional spatial perception due to binocular parallax

- Recognition of spatial sense due to Parallax Effect when moving left or right

- Perception of depth through vanishing point and perspective

- Understanding the shape of a three-dimensional object through the direction of light and the brightness and darkness of the object

- Recognition of distance through changes in eye focus

- Estimation of depth through fog phenomenon caused by density of atmospheric dust

Figure 24 shows an example of a shooting image according to a first method using a flat plate and a second method using a layered plate.

To explain the difference between the two shooting methods, as shown in FIG. 24, assume that the same camera and actor are set up and filmed in an LED shooting studio equipped with the same LED wall 100. In the first method, a large photo is placed behind the actor, and the studio camera is filmed while moving to the left and right of the stage. In the case of the second method, assets produced in the Layered Plate method are reproduced in the Virtual Production method, and the internal space of the LED wall is visualized in real time in response to the physical position of the actual camera and lens status (angle of view, focus).

(1) Related to three-dimensional spatial perception due to binocular disparity

In general, the way to use the sense of space using binocular parallax is to produce a stereoscopic image using two cameras. Through this, three-dimensional films such as Avatar (2009) are produced. However, this method is not used much at present, and if stereoscopic shooting is done in Virtual Production based on an LED wall, a significantly more three-dimensional scene can be produced due to the sense of depth of the second method compared to the first method. . Of course, even if this method is applied to real-time VR content rather than a movie, it is much more advantageous for the experiencer to perceive a sense of space due to the binocular parallax provided by the Layered Plate method.

Figure 25 shows an example of a shooting result according to the first method, and Figure 26 shows an example of a shooting result according to the second method.

2) Related to perception of space due to Parallax Effect when moving left or right

Referring to Figures 25 and 26, unlike the first method that uses an existing flat image, in the second method in which layers are separated, a parallax effect occurs when the camera moves. When creating a Layered Plate object, if you enter information such as camera sensor size, lens angle of view, and distance values for each layer with values similar to the actual ones, a layer mesh close to the actual position and size is created, creating a more physically accurate visual effect. can be obtained.

3) Related to perception of depth through vanishing point and perspective

Compared to the flat plate method, which does not change the vanishing point or perspective even when the shooting position changes, the layered plate object has each layer placed in a different position depending on the distance, and at the same time, when forming the thickness of each layer, the 'reference position' and 'reference plane' are used. The position of each vertex is calculated based on the line connecting one point. Accordingly, in the second method, a natural vanishing point and perspective can be formed depending on the shooting location.

Figure 27 shows examples of imaging according to the first and second methods.

4) Related to understanding the shape of a three-dimensional object through the direction of light and the light and dark of the object

Each layer in the second method can be given a sense of thickness through a depth map. This sense of volume can convey a more three-dimensional feeling by being influenced by the lights and shadows inside the CG. Even when the shadows of other objects within the CG are cast, unlike the first method, the second method can be visualized as being naturally cast on the surface.

5) Related to perception of distance through changes in eye focus

The human eye can adjust the thickness of the lens to focus on a specific part of an object and estimate the distance through this. The structure of the camera is similar to this, so when producing a video, the focus of the lens can be moved to convey the three-dimensional effect of objects in space to the audience. 'Out-focusing', in which areas that are in focus are made clear and other areas are blurred, is a very important element not only for creating a sense of space but also for emotional production. If a flat plate is used as the background according to the first method, it is naturally impossible to change the focus. On the other hand, since each layer of the Layered Plate object according to the second method is arranged in virtual space, it is naturally possible to focus on a desired point with a virtual camera and obtain an out-focusing effect.

6) Estimation of depth due to fog phenomenon caused by density of atmospheric dust

Due to the density of the atmosphere, distant objects have lower saturation and contrast, while nearby objects appear clearer and more colorful. For example, when looking at a landscape photo, people can grasp the approximate depth within the photo through this fog effect. Even in CG, atmospheric effects can be applied to objects in the distance through the Exponential Fog function. In the case of the second method, each layer is placed at a position corresponding to the actual physical distance, so when the Fog function of CG is turned on, the same atmospheric effect is applied. If there is already a sufficient fog effect on the mountains in the distance in the photo used as a source, reduce the fog value of the CG or use the color correction function (saturation, contrast, gamma, etc.) of the Layered Plate plug-in to adjust the color of the layer. Expression can be adjusted to an appropriate level.

In the detailed description of the present invention, specific embodiments have been described, but of course, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, but should be determined by the claims described below and their equivalents.

10: System 100: LED wall
200: Camera 300: Playback device
310: input unit 320: communication unit
330: Display 340: Memory
350: control unit 400: control device
VC: Virtual Camera

Claims (23)

When shooting based on virtual production, a second image containing content in a virtual space captured by a virtual camera linked according to the movement and lens status of the camera shooting the first image, which is a live-action image, is displayed using a light-emitting device (light-emitting device). An emitting diode (LED) device that reproduces the second image to be displayed on a wall,
A memory that stores information about a two-dimensional flat image; and
A control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image,
The control unit,
A flat plate for the flat image is placed in a virtual space, and the center point of the flat plate is placed at a position spaced apart from the reference position in the virtual space. Then, the flat plate is placed according to the curved separation distance. A concave hemispherical curved plate with respect to the flat plate is created at a location spaced apart from the flat plate using the flat plate,
In the flat image, separate curved plates for each layer of each separate image are created according to different curved surface separation distances,
A device that controls the virtual camera to capture the curved plates while interlocking with it according to the movement of the camera and the state of the lens.
When shooting based on virtual production, a second image containing content in a virtual space captured by a virtual camera linked according to the movement and lens status of the camera shooting the first image, which is a live-action image, is displayed using a light-emitting device (light-emitting device). An emitting diode (LED) device that reproduces the second image to be displayed on a wall,
A memory that stores information about a two-dimensional flat image; and
A control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image,
The control unit,
A flat plate for the flat image is placed in a virtual space, and the center point of the flat plate is placed at a position spaced apart from the reference position in the virtual space. Then, the flat plate is placed according to the curved separation distance. A concave hemispherical curved plate with respect to the flat plate is created at a location spaced apart from the flat plate using the flat plate,
Controls the virtual camera to capture the curved plate while interlocking according to the movement of the camera and the state of the lens,
Controlling the placement of the planar plate using information about the camera setting state at the time the planar image was captured,
For the first layer of the first segment image and the second layer of the second segment image that is farther than the first segment image, control is set so that the curved surface separation distance of the first layer is set shorter than the curved surface separation distance of the second layer. A device that does.
When shooting based on virtual production, a second image containing content in a virtual space captured by a virtual camera linked according to the movement and lens status of the camera shooting the first image, which is a live-action image, is displayed using a light-emitting device (light-emitting device). An emitting diode (LED) device that reproduces the second image to be displayed on a wall,
A memory that stores information about a two-dimensional flat image; and
A control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image,
The control unit,
A flat plate for the flat image is placed in a virtual space, and the center point of the flat plate is placed at a position spaced apart from the reference position in the virtual space. Then, the flat plate is placed according to the curved separation distance. A concave hemispherical curved plate with respect to the flat plate is created at a location spaced apart from the flat plate using the flat plate,
Controls the virtual camera to capture the curved plate while interlocking according to the movement of the camera and the state of the lens,
Controls the placement of the flat plate using information about the camera setting state at the time the flat image was captured,
For the first layer of the first segment image that is near, the second layer of the second segment image that is midground, and the third layer of the third segment image that is far, the curved separation distance of the first layer is the second layer. is shorter than the curved surface separation distance of and controls the curved surface separation distance of the second layer to be set shorter than the curved surface separation distance of the third layer.
According to any one of claims 1 to 3,
The control unit is a device that controls the curved plate to be created at the same matching position when the virtual camera looks at the flat plate and the curved plate from the reference position.
According to any one of claims 1 to 3,
The control unit controls each vertex forming the flat plate and each vertex forming the curved plate to create the curved plate at a position corresponding to the reference position as the origin.
According to any one of claims 1 to 3,
When the control unit draws an extension line for each vertex of the flat plate at the reference position, the control unit displays an image displayed at the vertex of the flat plate where the extension line meets the flat plate, and the curved plate where the extension line meets the curved plate. A device that controls the curved plate to be created at a location such that the images displayed at the vertices are the same.
According to any one of claims 1 to 3,
The control unit is a device that controls the placement of the flat plate using information about the camera setting state at the time the flat image is captured.
delete According to any one of claims 1 to 3,
The curved plate is a device that has an area that increases as the distance between the curved surfaces increases.
According to any one of claims 1 to 3,
The control unit uses a depth map for at least one of the curved plates to control so that at least one vertex of the curved plate protrudes from the curved plate in the direction of the reference position to provide a sense of thickness to the curved plate. Device.
When shooting based on virtual production, a second image containing content in a virtual space captured by a virtual camera linked according to the movement and lens status of the camera shooting the first image, which is a live-action image, is displayed using a light-emitting device (light-emitting device). emitting diode (LED) A method performed in a device that reproduces the second image to be displayed on a wall,
Performing stereoscopic image processing on a two-dimensional flat image; and
Processing the image according to the three-dimensional image processing to be included in the second image and displayed on the LED wall,
The steps performed above are:
Placing a flat plate for the flat image in a virtual space, wherein the center point of the flat plate is spaced apart from a reference position in the virtual space; and
It includes the step of creating a concave hemispherical curved plate with respect to the flat plate at a position spaced apart from the flat plate according to the curved surface separation distance using the flat plate,
The performing step includes generating separate curved plates for each layer of each separate image in the flat image according to different curved surface separation distances,
The processing step includes capturing the curved plates while the virtual camera is linked according to the movement of the camera and the state of the lens.
When shooting based on virtual production, a second image containing content in a virtual space captured by a virtual camera linked according to the movement and lens status of the camera shooting the first image, which is a live-action image, is displayed using a light-emitting device (light-emitting device). emitting diode (LED) A method performed in a device that reproduces the second image to be displayed on a wall,
Performing stereoscopic image processing on a two-dimensional flat image; and
Processing the image according to the three-dimensional image processing to be included in the second image and displayed on the LED wall,
The steps performed above are:
Placing a flat plate for the flat image in a virtual space, wherein the center point of the flat plate is spaced apart from a reference position in the virtual space; and
It includes the step of creating a concave hemispherical curved plate with respect to the flat plate at a position spaced apart from the flat plate according to the curved surface separation distance using the flat plate,
The processing step includes capturing the curved plates while the virtual camera is linked according to the movement of the camera and the state of the lens,
The performing step includes arranging the planar plate using information about the camera setting state at the time the planar image is captured,
The step of performing the above is that, for the first layer of the first segment image and the second layer of the second segment image that is farther than the first segment image, the curved surface separation distance of the first layer is the curved surface separation distance of the second layer. A method that includes shorter setup steps.
When shooting based on virtual production, a second image containing content in a virtual space captured by a virtual camera linked according to the movement and lens status of the camera shooting the first image, which is a live-action image, is displayed using a light-emitting device (light-emitting device). emitting diode (LED) A method performed in a device that reproduces the second image to be displayed on a wall,
Performing stereoscopic image processing on a two-dimensional flat image; and
Processing the image according to the three-dimensional image processing to be included in the second image and displayed on the LED wall,
The steps performed above are:
Placing a flat plate for the flat image in a virtual space, wherein the center point of the flat plate is spaced apart from a reference position in the virtual space; and
Comprising: creating a concave hemispherical curved plate with respect to the flat plate at a position spaced apart from the flat plate according to the curved surface separation distance using the flat plate,
The processing step includes capturing the curved plates while the virtual camera is linked according to the movement of the camera and the state of the lens,
The performing step includes arranging the planar plate using information about the camera setting state at the time the planar image is captured,
The performing step is a curved separation distance of the first layer with respect to the first layer of the first segment image that is the near-ground, the second layer of the second segment image that is the mid-ground, and the third layer of the third segment image that is the distance. is shorter than the curved surface separation distance of the second layer, and setting the curved surface separation distance of the second layer shorter than the curved surface separation distance of the third layer.
According to any one of claims 11 to 13,
The performing step includes generating the curved plate at a position where each of the flat plate and the curved plate are identically matched when the virtual camera views the flat plate and the curved plate from the reference position.
According to any one of claims 11 to 13,
The performing step includes creating the curved plate at a position where each vertex forming the flat plate and each vertex forming the curved plate correspond to the reference position as the origin.
According to any one of claims 11 to 13,
The step of performing the above is when drawing an extension line for each vertex of the flat plate at the reference position, an image displayed at the vertex of the flat plate where the extension line meets the flat plate, and the image displayed at the vertex of the flat plate where the extension line meets the curved plate. A method including the step of creating the curved plate at a location such that the images displayed at the vertices of the curved plate are the same.
According to any one of claims 11 to 13,
The performing step includes arranging the planar plate using information about the camera setting state at the time the planar image is captured.
According to clause 11,
The performing step uses a depth map for at least one of the curved plates to cause at least one vertex of the curved plate to protrude from the curved plate in the direction of the reference position, thereby giving a sense of thickness to the curved plate. How to include steps.
delete As a system used for virtual production-based filming,
A camera that captures a first image, which is a live action image; and
A playback device that reproduces the second image so that the second image containing the content of the virtual space captured by the virtual camera linked according to the movement of the camera and the state of the lens is displayed on a light-emitting diode (LED) wall. Contains ;,
The playback device is,
A memory that stores information about a two-dimensional flat image; and
A control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image,
The control unit,
A flat plate for the flat image is placed in a virtual space, and the center point of the flat plate is placed at a position spaced apart from a reference position in the virtual space, and then the flat plate is placed according to the curved separation distance. Creating a concave hemispherical curved plate with respect to the flat plate at a position spaced apart from the flat plate using the flat plate,
In the flat image, separate curved plates for each layer of each separate image are created according to different curved surface separation distances,
A system that controls the virtual camera to capture the curved plates while interlocking according to the movement of the camera and the state of the lens.
As a system used for virtual production-based filming,
A camera that captures a first image, which is a live action image; and
A playback device that reproduces the second image so that the second image containing the content of the virtual space captured by the virtual camera linked according to the movement of the camera and the state of the lens is displayed on a light-emitting diode (LED) wall. Contains ;,
The playback device is,
A memory that stores information about a two-dimensional flat image; and
A control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image,
The control unit,
A flat plate for the flat image is placed in a virtual space, and the center point of the flat plate is placed at a position spaced apart from the reference position in the virtual space. Then, the flat plate is placed according to the curved separation distance. A concave hemispherical curved plate with respect to the flat plate is created at a location spaced apart from the flat plate using the flat plate,
Controls the virtual camera to capture the curved plate while interlocking according to the movement of the camera and the state of the lens,
Controlling the placement of the planar plate using information about the camera setting state at the time the planar image was captured,
For the first layer of the first segment image and the second layer of the second segment image that is farther than the first segment image, control is set so that the curved surface separation distance of the first layer is set shorter than the curved surface separation distance of the second layer. A system that does.
As a system used for virtual production-based filming,
A camera that captures a first image, which is a live action image; and
A playback device that reproduces the second image so that the second image containing the content of the virtual space captured by the virtual camera linked according to the movement of the camera and the state of the lens is displayed on a light-emitting diode (LED) wall. Contains ;,
The playback device is,
A memory that stores information about a two-dimensional flat image; and
A control unit that performs stereoscopic image processing on the flat image and controls the image resulting from the stereoscopic image processing to be included in the second image,
The control unit,
A flat plate for the flat image is placed in a virtual space, and the center point of the flat plate is placed at a position spaced apart from a reference position in the virtual space, and then the flat plate is placed according to the curved separation distance. Creating a concave hemispherical curved plate with respect to the flat plate at a position spaced apart from the flat plate using the flat plate,
Controls the virtual camera to capture the curved plate while interlocking according to the movement of the camera and the state of the lens,
The planar plate is controlled to be placed using information about the camera setting state at the time the planar image was captured,
For the first layer of the first segment image that is near, the second layer of the second segment image that is midground, and the third layer of the third segment image that is far, the curved separation distance of the first layer is the second layer. A system that controls the curved surface separation distance of the second layer to be set shorter than the curved surface separation distance of the third layer.
delete