KR20200116224A - 360 degree image storing method and rendering method using viewpoint transformation - Google Patents

Abstract

Disclosed are a 360-degree image storage method and rendering method using a viewpoint change for removing holes generated in an occlusion area in the rendering process of an existing volumetric VR technique. The 360-degree image storage method according to an embodiment is characterized in performing a backward viewpoint change on images each photographed from a plurality of cameras included in a 360-degree camera system, and filling holes of each of the images on which the backward viewpoint change is performed to store the same. The 360-degree image rendering method according to an embodiment is characterized in loading the images each photographed from the plurality of cameras included in the 360-degree camera system and on which the backward viewpoint change is performed, and performing a forward viewpoint change on each of the images on which the backward viewpoint change is performed.

Description

360-degree image storage method and rendering method using viewpoint transformation {360 DEGREE IMAGE STORING METHOD AND RENDERING METHOD USING VIEWPOINT TRANSFORMATION}

The following description relates to a 360-degree image storage method and a rendering method, and more particularly, a technology for storing and rendering a 360-degree image using a viewpoint transformation in a 360-degree camera system.

The 360-degree camera system captures and provides 360-degree horizontal and 180-degree vertical angle of view images that allow you to experience virtual reality. To this end, the 360-degree camera system is composed of a plurality of cameras arranged in a circular or spherical shape, and the images captured by each of the plurality of cameras are aligned by using an optical flow or by detecting feature points. A monocular or binocular omnidirectional image is formed by stitching through the method.

An omnidirectional image provided by such a 360-degree camera system can be viewed mainly using a head-mounted display (HMD), which is a stereo display device. The HMD determines the user's gaze direction by measuring the movement of the user's head in 3 directions and rotation in 3 axes, and displays a partial image corresponding to the user's gaze direction among all-round images to the user.

However, since the omnidirectional image provides only the image from the viewpoint of the camera where the shooting was taken, there is a problem in that it can only provide the experience for the rotation of three axes among the movement of the user's head, and the experience for the movement in the three directions cannot be provided. .

Accordingly, a Volumetric VR technology has been proposed that provides an experience for movement of 6 degrees of freedom (movement in three directions and rotation in three axes). Volumetric VR technology saves a color image captured by a 360-degree camera system and a corresponding depth map, instead of pre-creating and storing omnidirectional images including left and right images, and storing the corresponding depth map (DIBR). It is a technology that renders and provides left and right images in real time using the based rendering) technique.

However, the method of rendering an image of a new viewpoint using the DIBR technique has a problem that a hole occurs in the occlusion area that was covered by the foreground according to the change of view, and to solve this problem, A method of filling the hole using the information of the area has been proposed. However, if only the corresponding method is used and the hole is filled, it can appear as noticeable artifacts in the image, so a new method is proposed to solve the disocclusion problem in the occlusion area. Need to be.

Accordingly, the following embodiments are intended to propose a technique capable of removing a hole generated in an occlusion area during a rendering process.

Embodiments propose a technique capable of removing a hole generated in an occlusion area in a rendering process of a conventional Volumetric VR technique.

In more detail, in one embodiment, after performing a retreat view conversion in which images captured by a plurality of cameras included in the 360 degree camera system are converted into a view retracted from the original view, each of the converted images A 360-degree image storage method that fills and stores a hole, and images captured by a plurality of cameras included in the 360-degree camera system are retracted by loading the converted images, and then retreating using the depth map of the images. A technique for minimizing and removing holes occurring in the occlusion region is proposed through a 360-degree image rendering method that performs a forward viewpoint transformation on each of the viewpoint-transformed images.

According to an embodiment, a 360-degree image storage method executed by a computer included in a 360-degree camera system includes images taken by a plurality of cameras included in the 360-degree camera system and a depth map. Obtaining a; Performing a retraction point conversion for each of the images using the depth map; Filling a hole of each of the converted images at the retreat point; And storing the images filled with the hole and the depth map.

According to an aspect, the filling of the holes of each of the images converted at the retreat point may include images that have been converted at the retreat point by using an image captured by a camera and an adjacent camera for each of the converted images at the retreat point. It may include filling each hole.

According to another aspect, the filling of the holes of each of the images converted at the retreat point may include using information on a background region adjacent to the hole of each of the converted images at the retreat point, It may include filling a hole of each of the converted images at the retreat point.

According to another aspect, the performing of the retraction point conversion for each of the images includes a focal length of each of the plurality of cameras, a length of a baseline between the plurality of cameras, and the Setting a viewpoint conversion equation based on a parallax between a plurality of cameras; Determining a movement parameter in the viewpoint conversion equation, wherein the movement parameter is a value related to a moving distance when a position where each of the images is photographed is moved to a virtual specific position; And converting the viewpoint of each of the images to match the viewpoint of the image captured at a position retracted from the position where each of the images was photographed, using the viewpoint conversion equation.

According to another aspect, in the determining of the movement parameter in the viewpoint conversion equation, the optical center of each of the plurality of cameras may be moved to a center position of a circle or sphere in which the plurality of cameras are arranged on the 360 degree camera system. It may be a step of determining the movement parameter based on the distance to be moved at the time.

According to another aspect, the step of converting the viewpoints of each of the images to coincide with the viewpoints of the images photographed at the positions retracted from the positions at which each of the images was photographed, comprises: It may be a step of converting a viewpoint of an image captured from a center position of a circle or sphere in which the plurality of cameras are arranged on the system.

According to another aspect, the method of storing the 360-degree image may further include performing a retraction point conversion for the depth map.

According to another aspect, the method of storing a 360-degree image may further include filling a hole of the depth map based on images filled with the hole.

According to another aspect, a plurality of depth maps are obtained corresponding to images respectively photographed by the plurality of cameras, and the step of filling the holes of the depth map includes filling holes of each of the plurality of depth maps. The 360-degree image storage method may further include generating a 360-degree omnidirectional depth map by stitching the depth maps filled with the holes.

According to another aspect, the method for storing a 360-degree image may further include generating a 360-degree omnidirectional image by stitching the images filled with the hole.

According to an embodiment, in a computer program recorded in a computer-readable recording medium to execute a 360-degree image storage method by being combined with a computer, the 360-degree image storage method comprises: Obtaining images and depth maps respectively photographed by the cameras; Performing a retraction point conversion for each of the images using the depth map; Filling a hole of each of the converted images at the retreat point; And storing the images filled with the hole and the depth map.

According to an embodiment, a 360-degree image storage system implemented by a computer included in a camera system includes at least one processor implemented to execute a command readable by a computer, and the at least one processor includes the 360-degree image storage system. An acquisition unit that acquires images captured by a plurality of cameras included in the camera system and a depth map, respectively; A viewpoint conversion performing unit for performing a retreat viewpoint conversion for each of the images using the depth map and filling a hole of each of the retracted viewpoint converted images; And a storage unit for storing images filled with the hole and the depth map.

According to an embodiment, a 360-degree image rendering method executed by a computer included in a 360-degree camera system includes images each captured by a plurality of cameras included in the 360-degree camera system are converted images at a retreat point. Loading a depth map; And performing a forward viewpoint conversion on each of the images converted from the retreat viewpoint using the depth map.

According to an aspect, the performing of the forward viewpoint conversion for each of the images converted at the retreat point is a step of removing a hole generated in the process of converting each of the images captured by the plurality of cameras at the retreat time point. I can.

According to another aspect, the step of performing the forward viewpoint conversion for each of the images converted at the retreat viewpoint includes a hole generated in an occlusion area during a rendering process, and an occlusion generated by the forward viewpoint conversion. It may be a step of hiding it into an area.

According to another aspect, the performing of the forward viewpoint conversion for each of the images converted at the retreat viewpoint includes information on a background region adjacent to a hole of each of the images converted at the advance viewpoint in each of the images converted at the forward viewpoint The method may further include filling a hole of each of the images converted at the forward point of view.

According to an embodiment, in a computer program recorded in a computer-readable recording medium to execute a 360-degree image rendering method by being combined with a computer, the 360-degree image rendering method includes: Loading images and depth maps, each of which images captured by the cameras are converted at retreat points; And performing a forward viewpoint conversion on each of the images converted from the retreat viewpoint using the depth map.

According to an embodiment, a 360-degree image rendering system implemented by a computer included in a camera system includes at least one processor implemented to execute a command readable by a computer, and the at least one processor includes the 360-degree image rendering system. A loading unit for loading images and depth maps, each of which images captured by a plurality of cameras included in the camera system are converted at retreat points; And a viewpoint conversion performing unit for performing a forward viewpoint conversion on each of the images converted from the retreat viewpoint using the depth map.

One embodiment may propose a technique capable of removing a hole generated in an occlusion area in a rendering process of a conventional Volumetric VR technique.

In more detail, in one embodiment, after performing a retreat point conversion for images captured by a plurality of cameras included in the 360-degree camera system, each hole of the converted images at the retraction point is filled and stored. After loading the pre-stored retreat view-transformed images by the 360-degree image storage method and the 360-degree image storage method, a 360-degree conversion is performed on each of the images converted at the retreat point by using the depth map of the images. Through the image rendering method, it is possible to propose a technique for minimizing and removing holes generated in the occlusion region.

1 is a block diagram showing a 360 degree camera system according to an embodiment.
2 to 3 are block diagrams illustrating examples of components that may be included in a processor of a 360 degree camera system according to an exemplary embodiment.
4 to 9 are diagrams for explaining a method for storing a 360 degree image and a method for rendering a 360 degree image according to an exemplary embodiment.
10 is a flowchart illustrating a method of storing a 360 degree image according to an exemplary embodiment.
11 is a flowchart illustrating a method of rendering a 360 degree image according to an exemplary embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing a 360 degree camera system according to an embodiment.

Referring to FIG. 1, a 360-degree camera system 100 according to an embodiment may include a plurality of cameras 110, a memory 120, a processor 130, and an input/output interface 140. Hereinafter, the 360-degree camera system 100 is described as including a plurality of cameras 110 (eg, six or more cameras) arranged along a circle, but is now limited or not limited, and implementing Volumetric VR For this purpose, it may be a variety of camera systems including a plurality of cameras 110 arranged to simultaneously photograph the same subject.

The memory 120 is a computer-readable recording medium, and may include a non-volatile mass storage device such as a random access memory (RAM), read only memory (ROM), and a disk drive). In addition, the memory 120 includes an operating system, an image captured by each of the plurality of cameras 110, and at least one program code (for example, a 360-degree image storage method and/or a 360-degree image rendering method to be described later). Can be saved. These software components may be loaded from a computer-readable recording medium separate from the memory 120. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 120 through a communication module (not shown in the drawing) other than a computer-readable recording medium.

The processor 130 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 130 by the memory 120 or the communication module. For example, the processor 130 may execute a command received according to a program code stored in a recording device such as the memory 120 (for example, a code for performing a 360-degree image storage method and/or a 360-degree image rendering method described later). It can be configured to run.

The input/output interface 140 may be a means for interfacing with an input/output device (not shown in the drawing). For example, the input device may include a device such as a touch screen and a button provided in the 360-degree camera system 100, and the output device may include a device such as a display displaying captured images. For another example, the input/output interface 140 may be a means for an interface with a device in which input and output functions are integrated into one, such as a touch screen.

In other embodiments, the 360 degree camera system 100 may include more components than the components of FIG. 1. However, there is no need to clearly show most of the prior art components. For example, the 360-degree camera system 100 may further include other components such as a transceiver, a Global Positioning System (GPS) module, various sensors, and a database.

Hereinafter, specific embodiments of a 360-degree image storage method and a 360-degree image rendering method executed by the processor 130 included in the 360-degree camera system 100 will be described.

The 360-degree image storage method described below is that the 360-degree image rendering method in Volumetric VR technology is performed in advance in order to remove the hole generated in the occlusion area, so the 360-degree image storage method and the 360-degree image rendering method are one. It will be described as being performed by the processor 130 of. In addition, since one processor 130 is composed of components that perform each step of the 360-degree image storage method and components that perform each step of the 360-degree image rendering method, it is assumed that different components are included. Although described, each component is only classified for convenience in order to describe the subject performing each step. That is, it is assumed that both the 360-degree image storage method and the 360-degree image rendering method are performed by the processor 130.

2 to 3 are block diagrams illustrating examples of components that may be included in a processor of a 360 degree camera system according to an exemplary embodiment. In more detail, FIG. 2 is a block diagram illustrating components when a processor performs a 360-degree image storage method, and FIG. 3 is a block diagram illustrating components when a processor performs a 360-degree image rendering method.

Referring to FIG. 2, a 360-degree image storage system according to an embodiment may be configured to include the processor 130 of the 360-degree camera system 100 described above with reference to FIG. 1. Specifically, the 360-degree image storage system may include an acquisition unit 210, a viewpoint conversion performing unit 220, and a storage unit 230, which are components of the processor 130, as shown in the drawing. Depending on the embodiment, components of the processor 130 may be selectively included or excluded from the processor 130. In addition, according to embodiments, components of the processor 130 may be separated or merged to express the function of the processor 130. For example, at least some of the components of the processor 130 may be implemented in a processor included in a separate server that is connected to the 360-degree camera system 100 through communication.

Components of the processor 130 are code of an operating system included in the memory 120 and at least one program (e.g., 360-degree image storage method) to perform the steps (S1010 to S1030) included in the 360-degree image storage method of FIG. It may be implemented to execute an instruction of (a program composed of code that executes an image storage method).

Here, the components of the processor 130 may be expressions of different functions according to instructions provided by the program code stored in the memory 120. For example, the acquisition unit 210 may be used as a functional expression for an operation of acquiring images respectively captured by a plurality of cameras 110 included in the 360 degree camera system 100.

Accordingly, each component of the processor 130 may perform a 360-degree image storage method by operating as follows. For example, images captured by the plurality of cameras 110 included in the 360 degree camera system 100 by the acquisition unit 210 (hereinafter, images respectively captured by the plurality of cameras 110 are (Meaning images captured simultaneously by each of the cameras 110) and a depth map, and the viewpoint conversion performing unit 220 performs a retreat viewpoint conversion for each of the images using the depth map. And, by filling the holes of each of the converted images at the retreat point, and by stitching and storing the images filled with the holes by the storage unit 230, the image to be used in the 360-degree image rendering method described later (after the retraction point is converted, the hole The filled images or images stored by stitching and storing the filled images after the retraction point is converted may be generated and stored in advance. A detailed description of the operation of each component will be described with reference to FIG. 10, and a detailed description of the operating principle of the viewpoint conversion performing unit 220 will be described with reference to FIGS. 4 to 9.

Referring to FIG. 3, the 360-degree image rendering system according to an embodiment may be configured to include the processor 130 of the 360-degree camera system 100 described above with reference to FIG. 1. Specifically, the 360-degree image rendering system may include a load unit 310 and a viewpoint conversion performing unit 320, which are components of the processor 130, as shown in the drawing. Depending on the embodiment, components of the processor 130 may be selectively included or excluded from the processor 130. In addition, according to embodiments, components of the processor 130 may be separated or merged to express the function of the processor 130. For example, at least some of the components of the processor 130 may be implemented in a processor included in a separate server that is connected to the 360-degree camera system 100 through communication.

Components of the processor 130 are code of an operating system included in the memory 120 and at least one program (e.g., 360-degree image rendering method) to perform the steps (S1110 to S1120) included in the 360-degree image rendering method of FIG. It may be implemented to execute an instruction of a program composed of code that executes the image rendering method).

Here, the components of the processor 130 may be expressions of different functions according to instructions provided by the program code stored in the memory 120. For example, images captured by a plurality of cameras 110 included in the 360-degree camera system 100 are respectively converted to retreat points (a storage unit in the 360-degree image storage system described above with reference to FIG. 2) The loading unit 310 may be used as a functional expression for an operation of loading images stored in the hole 230) or images filled with the hole are stitched after the retraction time is converted.

Accordingly, each component of the processor 130 may perform a 360-degree image rendering method by operating as follows. For example, images captured by the plurality of cameras 110 included in the 360-degree camera system 100 by the load unit 310 are respectively converted to retreat points (stored in the 360-degree image storage method described above). , The retreat point is converted and images filled with each hole) and a depth map are loaded (or images and the depth map is stored by stitching the images filled with the hole after the retreat point is transformed), and the viewpoint conversion performing unit 320 By performing a forward viewpoint conversion for each of the images converted at the retreat point using a depth map (or by performing a forward viewpoint conversion using a depth map for the image stored by stitching the images filled with holes after the retraction point conversion) , Image rendering may be performed. A detailed description of the operation of each component will be described with reference to FIG. 11, and a detailed description of the operating principle of the viewpoint conversion performing unit 320 will be described with reference to FIGS. 4 to 9.

4 to 9 are diagrams for explaining a method for storing a 360 degree image and a method for rendering a 360 degree image according to an exemplary embodiment.

Referring to FIG. 4, in the 360-degree image storage system according to an embodiment, the optical center of each of a plurality of cameras 110 constituting the 360-degree camera system 100 is matched to a specific virtual position 410, The image captured by each of the cameras 110 of may be converted as if it was captured at the same location 410. The virtual specific location 410 may be an arbitrary point retracted by a predetermined distance or more to the rear of the plurality of cameras 110 in consideration of the arrangement of the plurality of cameras 110, and the 360 degree camera system 100 It may mean a center position of a circle in which the plurality of cameras 110 are arranged.

At this time, when the optical centers of each of the plurality of cameras 110 are matched, the viewpoints of images captured by the plurality of cameras 110 are also matched.

In other words, the 360-degree image storage system does not actually move each of the plurality of cameras 110, but the viewpoint of the images captured at the same time by the plurality of cameras 110 of the image captured at the virtual specific location 410. By converting it as if it were a viewpoint (converting a retreat viewpoint), a 360-degree image storage method is performed.

Similar to the retraction viewpoint transformation, the 360-degree image rendering system according to an embodiment may perform a forward viewpoint transformation that is an inverse transformation of the retraction viewpoint transformation. In this regard, referring to FIG. 5, the 360-degree image rendering system includes images that have been converted from the retraction point stored by the 360-degree image combining system (or images converted from the retreat point by the 360-degree image combining system are stitched and stored). The image-hereinafter referred to as a stitched image -) may be converted as if it was photographed at an arbitrary point 510 advanced by a predetermined distance or more so as to be closer to the foreground from a virtual specific location 410.

In this way, the 360-degree image rendering system converts the viewpoints of the converted images (or stitching images) to the retracted viewpoint stored by the 360-degree image combining system as if they were the viewpoints of images photographed at an advanced point 510 (advancing viewpoint). Conversion) to perform a 360-degree image rendering method.

As a principle explaining the transformation of the forward and retreat viewpoints, referring to FIG. 6, the point w at a distance d _f from the camera at a specific position 610 is an optical axis in the background at a distance d _b ( 620) at a distance of X. This position is when the camera is placed in the rearward at a particular location 610 r a position 630 retracts by the changes to the X ^'position, the position 640, the camera is moved to the side by s at a particular location 610 When placed in the optical axis 650, it is changed to a position separated by X ^" . Assuming that these X, X ^' , X ^" appear at a position that is x, x ^' , x ^" away from the center of the image in the image, these positions are It can be calculated using a pinhole camera model as shown in FIGS. 7 to 9.

For example, when the camera moves from a specific position 610 to a position 630 retracted by r, X ^′ may be calculated as shown in Equation 1 below from FIG. 6.

<Equation 1>

In addition, by using a pinhole camera model X ^'is 7 to 8, which is calculated as in equation 1, the image can be expressed by the following expression (2).

<Equation 2>

의 관계를 가지므로(여기서, f는 초점 거리를 의미하고, B는 360도 카메라 시스템(100)에 포함되는 복수의 카메라들(110) 사이의 베이스라인의 길이, D는 시차를 의미함), 식 2는 아래 식 3과 같이 표현될 수 있다.Meanwhile, in the depth map, the depth d and the parallax D are

(Here, f denotes the focal length, B denotes the length of the baseline between the plurality of cameras 110 included in the 360-degree camera system 100, and D denotes the parallax), Equation 2 can be expressed as Equation 3 below.

<Equation 3>

For another example, when the camera moves from the specific position 610 to the position 640 that is s apart from the specific position 610, X ^" may be calculated as shown in Equation 4 below from FIG. 6.

<Equation 4>

In addition, X ^" calculated as in Equation 4 can be expressed as Equation 5 below in the image by using the pinhole camera model of FIGS. 7 to 9, and can be expressed as Equation 6 below according to the relationship between the depth d and the parallax D. I can.

<Equation 5>

<Equation 6>

는 도 7 내지 9의 핀홀 카메라 모델을 이용하여 영상에서 아래 식 7과 같이 계산될 수 있다.Therefore, when the camera retreats r from a specific position 610 backward and moves to a position s away from the specific position 610

Can be calculated as in Equation 7 below in the image using the pinhole camera model of FIGS. 7 to 9.

<Equation 7>

In addition, Equation 7 can be expressed as Equation 8 below according to the relationship between the depth d and the parallax D.

<Equation 8>

From Equation 8, the time point at which the camera converts an image captured at a specific location 610 into a captured image when the camera moves from the specific location 610 in the x, y, and z directions by m _x , m _y and m _z respectively A transformation equation can be obtained. The original image I (x, y) captured by the camera at a specific position 610 and the converted image I ^' (which is an image captured when the camera moves from a specific position 410 by m _x , m _y , m _z ) For x ^' , y ^' ), if x, y, x ^' , y ^' are the pixel positions in each image, forward warping can be expressed as Equation 9 below, and backward warping ) Can be expressed as Equation 10.

<Equation 9>

<Equation 10>

및

는 광축(420, 450)이 지나는 영상 중심의 좌표를 의미한다.In equations 9 and 10

And

Denotes the coordinates of the image center through which the optical axes 420 and 450 pass.

The 360-degree image storage system and the 360-degree image rendering system according to an embodiment may perform conversion of a retreat viewpoint or a forward viewpoint of an image by using Equations 9 and 10 derived as described above. A detailed description thereof will be described with reference to FIGS. 10 to 11.

10 is a flowchart illustrating a method of storing a 360 degree image according to an exemplary embodiment. Hereinafter, the 360-degree image storage method is described as being performed by components of the processor 130 included in the 360-degree camera system 100, and the components of the processor 130 are recorded in a computer-readable recording medium. It can be implemented in the form of a computer program.

Referring to FIG. 10, in step S1010, the acquisition unit 210 included in the 360-degree image storage system according to the embodiment is, respectively, a plurality of cameras 110 included in the 360-degree camera system 100 Acquire the captured images and a depth map. Here, the images respectively captured by the plurality of cameras 110 may be images captured simultaneously by the plurality of cameras 110 operating simultaneously. At this time, since the plurality of cameras 110 are arranged along a circle or sphere and constitute the 360 degree camera system 100, there is a parallax between the images (images taken by each of the plurality of cameras 110). Can occur.

Hereinafter, the depth map for the images may be acquired as an image defining a relationship between a parallax and a depth over all pixels of an image through an algorithm for obtaining a conventional depth map while the images are captured.

In step S1020, the viewpoint conversion performing unit 220 included in the 360-degree image storage system performs a retraction viewpoint conversion for each of the images using the depth map. Specifically, the viewpoint conversion performing unit 220 is based on the depth map, the focal length of each of the plurality of cameras 110, the length of the baseline between the plurality of cameras 110, and between the plurality of cameras 110 By setting a viewpoint conversion equation (Equation 9 or 10 described above) based on the parallax of, and determining a movement parameter in the viewpoint conversion equation, each viewpoint of the images may be converted to match the viewpoint using the viewpoint conversion equation. Hereinafter, converting the viewpoints of the images to match may mean matching the images.

Here, the movement parameter is a value related to the moving distance when the position where each of the images is captured is moved to a virtual specific position, and corresponds to m _x , m _y , and m _z in Equations 9 and 10. Therefore, the fact that the viewpoint conversion performing unit 220 determines the movement parameter means that the optical center of each of the plurality of cameras 110 is the center of the circle on which the plurality of cameras 110 are disposed on the 360 degree camera system 100. It means that the moving distance when moving to the position is determined as a moving parameter. For example, the viewpoint conversion performing unit 220 may determine only the value of m _{z as} a movement parameter in Equation 9 or 10, and the 360 degree camera system 100 is a stereo 360 degree camera system in which two cameras are arranged as a set. In the case of, m _x or m _y may be determined as half of the baseline size.

In this way, in order to determine the moving distance when the optical center of each of the plurality of cameras 110 is moved to a virtual specific position of the camera system 100 as a movement parameter, the optical center of each of the plurality of cameras 110 The step to confirm is essential. Although this step is not shown in the drawing, it may be adaptively performed at any time before step (S1020), and a rotation axis in which parallax does not occur when the rotation axis of each of the plurality of cameras 110 is moved back and forth. This can be done by experimentally exploring the location of.

At this time, when the optical centers of each of the plurality of cameras 110 are matched, the viewpoints of images captured by the plurality of cameras 110 are also matched. Accordingly, in step S1020, the viewpoint conversion performing unit 220 converts and matches each viewpoint of the images to the viewpoint of the image captured at the center position of the 360-degree camera system 100, thereby matching the images. I can.

That is, the viewpoint conversion performing unit 220 performs a virtual transformation in which the position of each of the plurality of cameras 110 is moved to a point where the position of each of the plurality of cameras 110 is retracted to the rear (the center point of the circle where the plurality of cameras 110 are placed). I can. This retreat point conversion is to minimize the occurrence of occlusion.

After the retreat viewpoint conversion is performed on the images, the viewpoint conversion performing unit 220 fills a hole in each of the images converted at the retreat viewpoint in step S1030. For example, the viewpoint conversion performing unit 220 may fill a hole of each of the converted images at the retreat by using an image photographed by a camera that is adjacent to a camera photographing each of the converted images at the retreat viewpoint. For a more specific example, if there is a hole in the first image (the image captured by the first camera is converted at the retreat and retreat point in step S1020), the view conversion performing unit 220 2 The hole of the first image may be filled with a pixel value of an area corresponding to the hole of the first image among areas of the image captured by the camera (or the image captured by the second camera is converted at a retreat point).

For another example, the view conversion performing unit 220 may fill a hole of each of the converted images at the retreat view by using information on a background region adjacent to each hole of the converted images at the retreat view. For a more specific example, if there is a hole in the first image (the image captured by the first camera is converted at the retreat and retreat time point in step S1020), the view conversion performing unit 220 includes regions of the first image. The hole of the first image may be filled with pixel values of the background area adjacent to the middle hole.

Further, although not shown as a separate step, the viewpoint conversion performing unit 220 may perform the step S1030 and then perform the retreat viewpoint conversion on the depth map. Since the conversion of the retreat point of the depth map is also performed in the same manner as in step S1020, a detailed description thereof will be omitted.

In addition, the viewpoint conversion performing unit 220 may fill the hole of the depth map by referring to images filled with the hole (images filled with the hole after the retraction viewpoint is converted). Since this process is similar to the method of filling the hole in step S1030, a detailed description thereof will be omitted.

Then, in step S1040, the storage unit 230 stores the images filled with the hole and the depth map. As described above, the hole-filled images stored by the storage unit 230 (images filled with the hole after the retraction point is converted) and the depth map are used in step S1110 of the 360-degree image rendering method to be described later.

In addition, in step S1040, the storage unit 230 creates a 360-degree omnidirectional image by stitching the images filled with holes, and stores the stitched image and a depth map to be used in step S1110 of the 360-degree image rendering method described later. You can do it.

In this case, when a plurality of stored depth maps are acquired corresponding to images respectively captured by the plurality of cameras 110, the viewpoint conversion performing unit 220 may fill a hole of each of the plurality of depth maps, The storage unit 230 may stitch depth maps filled with holes to generate and store a 360-degree all-round depth map.

11 is a flowchart illustrating a method of rendering a 360 degree image according to an exemplary embodiment. Hereinafter, it is assumed that the 360-degree image rendering method described is performed after the 360-degree image storage method described with reference to FIG. 10 is previously performed. However, the present invention is not limited or limited thereto, and the images captured by the plurality of cameras 110 included in the 360-degree camera system 100 are performed after various image storage methods in which the converted images are stored at the retreat point. Can be. In addition, hereinafter, the 360-degree image rendering method is described as being performed by components of the processor 130 included in the 360-degree camera system 100, and the components of the processor 130 are stored in a computer-readable recording medium. It can be implemented in the form of a recorded computer program.

Referring to FIG. 11, in step S1110, the load unit 310 included in the 360-degree image rendering system according to an embodiment is, respectively, in a plurality of cameras 110 included in the 360-degree camera system 100. Each of the captured images loads the converted images and the depth map at the retreat point. Here, the images converted from the retreat point may be images in which the hole is filled after the retreat point is converted in step S1040 of the 360-degree image storage method described with reference to FIG. 10.

Likewise, the depth map for the images is a depth map obtained in the 360-degree image storage method described with reference to FIG. 10, and in particular, may be a depth map in a state where a hole is filled.

At this time, in step S1110, the load unit 310 may load the stitching image and the depth map. In this case, the loaded depth map may also be a 360-degree omni-directional depth map in which depth maps are stitched, and in step S1120 to be described later, the viewpoint conversion performing unit 320 performs a forward viewpoint transformation for a stitched image that is a 360-degree omni-directional image. Can be done.

Thereafter, in step S1120, the viewpoint conversion performing unit 320 included in the 360-degree image rendering system performs a forward viewpoint conversion for each of the images converted from the retreat viewpoint using the depth map. Specifically, the viewpoint conversion performing unit 320 is based on the depth map, the focal length of each of the plurality of cameras 110, the length of the baseline between the plurality of cameras 110, and between the plurality of cameras 110 By setting a viewpoint conversion equation (Equation 9 or 10 described above) based on the parallax of, and determining a movement parameter in the viewpoint conversion equation, each viewpoint of the images is converted to the viewpoint of the image photographed from the forward position using the viewpoint conversion equation. You can convert it to match. For example, the viewpoint conversion performing unit 320 has the optical center of each of the plurality of cameras 110 advanced from the center position of a circle or sphere on which the plurality of cameras 110 are placed on the 360 degree camera system 100. It is possible to perform a forward point conversion to be placed in a position.

The forward viewpoint transformation may be an inverse transformation of the retraction viewpoint transformation performed in step S1020 of the 360-degree image storage method described above with reference to FIG. 10.

Through this forward point conversion, a hole generated in the process of converting each of the images captured by the plurality of cameras 110 to the retreat point (performed in step S1020 of the 360-degree image storage method described above with reference to FIG. 10) Holes that may occur in the transition of the retracted point) can be eliminated.

In addition, a hole generated in the occlusion area during the rendering process may also be hidden as an occlusion area generated by a forward viewpoint transformation.

Holes that are not removed or hidden even through the step S1120 may be filled using information on a background region adjacent to the hole of each of the images converted from the forward viewpoint in each of the images converted from the forward viewpoint. For example, after step S1120, the viewpoint conversion performing unit 320 may fill a hole of each of the forward viewpoint converted images by using information on a background region adjacent to each hole of the forward viewpoint converted images. . For a more specific example, if there is a hole in the image converted from the forward view, the view conversion performing unit 320 may fill the hole of the image with pixel values of a background area adjacent to the hole among the areas of the image.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments are a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable gate array (PLU). It may be implemented using one or more general purpose computers or special purpose computers, such as a logic unit), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be embodyed in any type of machine, component, physical device, computer storage medium or device to be interpreted by the processing device or to provide instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be one that continuously stores a program executable by a computer, or temporarily stores a program for execution or download. In addition, the medium may be a variety of recording means or storage means in a form in which a single or several pieces of hardware are combined, but is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks, and And a ROM, RAM, flash memory, and the like, and may be configured to store program instructions. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or a storage medium managed by a server.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (18)

In the 360-degree image storage method executed by a computer provided in a 360-degree camera system,
Acquiring images and depth maps respectively captured by a plurality of cameras included in the 360 degree camera system;
Performing a retraction point conversion for each of the images using the depth map;
Filling a hole of each of the converted images at the retreat point; And
Storing the images filled with the hole and the depth map
360-degree image storage method comprising a. The method of claim 1,
Filling the holes of each of the converted images at the retreat point,
Filling a hole of each of the converted images at the retreat time using an image photographed by a camera and an adjacent camera photographing each of the converted images at the retreat time point
360-degree image storage method comprising a. The method of claim 1,
Filling the holes of each of the converted images at the retreat point,
Filling a hole of each of the converted images at the retreat time using information on a background region adjacent to each hole of the converted images at the retreat time point
360-degree image storage method comprising a. The method of claim 1,
The step of performing a retraction point conversion for each of the images,
Setting a viewpoint conversion equation based on a focal length of each of the plurality of cameras, a length of a baseline between the plurality of cameras, and a parallax between the plurality of cameras based on the depth map;
Determining a movement parameter in the viewpoint conversion equation, wherein the movement parameter is a value related to a moving distance when a position where each of the images is photographed is moved to a virtual specific position; And
Converting a viewpoint of each of the images to match a viewpoint of an image photographed at a position retracted from a position where each of the images was photographed using the viewpoint conversion equation
360-degree image storage method comprising a. The method of claim 4,
The step of determining a movement parameter in the viewpoint conversion equation,
A step of determining the movement parameter based on a moving distance when the optical center of each of the plurality of cameras is moved to the center position of a circle or sphere in which the plurality of cameras are arranged on the 360° camera system. How to save. The method of claim 4,
The step of converting the viewpoint of each of the images to match the viewpoint of the image photographed at a position retracted from the position where each of the images was photographed,
Converting a viewpoint of each of the images into a viewpoint of an image photographed at a center position of a circle or a sphere in which the plurality of cameras are arranged on the 360 degree camera system. The method of claim 1,
Performing a retraction point conversion for the depth map
360-degree image storage method further comprising a. The method of claim 1,
Filling a hole of the depth map based on images filled with the hole
360-degree image storage method further comprising a. The method of claim 8,
The depth map,
A plurality of images are acquired corresponding to images respectively captured by the plurality of cameras,
Filling the hole of the depth map,
Filling a hole of each of the plurality of depth maps,
The 360-degree image storage method,
Creating a 360 degree omnidirectional depth map by stitching the depth maps filled with the holes
360-degree image storage method further comprising a. The method of claim 1,
Creating a 360 degree omnidirectional image by stitching the images filled with the hole
360-degree image storage method further comprising a. In a computer program recorded in a computer-readable recording medium to execute a 360-degree image storage method combined with a computer,
The 360-degree image storage method,
Acquiring images and depth maps respectively captured by a plurality of cameras included in the 360 degree camera system;
Performing a retraction point conversion for each of the images using the depth map;
Filling a hole of each of the converted images at the retreat point; And
Storing the images filled with the hole and the depth map
A computer program recorded on a recording medium including a. In the 360-degree image storage system implemented by a computer provided in the camera system,
At least one processor implemented to cause a computer to execute readable instructions
Including,
The at least one processor,
An acquisition unit for acquiring images and depth maps respectively captured by a plurality of cameras included in the 360 degree camera system;
A viewpoint conversion performing unit for performing a retreat viewpoint conversion for each of the images using the depth map and filling a hole of each of the retracted viewpoint converted images; And
A storage unit that stores images filled with the hole and the depth map
360-degree image storage system comprising a. In the 360-degree image rendering method executed by a computer provided in a 360-degree camera system,
Loading images and depth maps, respectively, in which images captured by a plurality of cameras included in the 360-degree camera system are converted at retreat points; And
Performing a forward viewpoint conversion for each of the images converted at the retreat viewpoint using the depth map
360 degree image rendering method comprising a. The method of claim 13,
The step of performing the forward viewpoint conversion for each of the images converted at the retreat viewpoint,
A method for rendering a 360-degree image, which is a step of removing a hole generated in a process in which images respectively photographed by the plurality of cameras are converted to retreat points. The method of claim 13,
The step of performing the forward viewpoint conversion for each of the images converted at the retreat viewpoint,
A method for rendering a 360-degree image, which is a step of hiding a hole generated in an occlusion area during a rendering process as an occlusion area generated by the forward viewpoint transformation. The method of claim 13,
The step of performing the forward viewpoint conversion for each of the images converted at the retreat viewpoint,
Filling a hole of each of the forward-time-converted images by using information on a background area adjacent to the hole of each of the forward-time-transformed images
A 360-degree image rendering method further comprising a. In a computer program recorded in a computer-readable recording medium to execute a 360-degree image rendering method in combination with a computer,
The 360-degree image rendering method,
Loading images and depth maps, respectively, in which images captured by a plurality of cameras included in the 360-degree camera system are converted at retreat points; And
Performing a forward viewpoint conversion for each of the images converted at the retreat viewpoint using the depth map
A computer program recorded on a recording medium including a. In the 360-degree image rendering system implemented by a computer provided in the camera system,
At least one processor implemented to cause a computer to execute readable instructions
Including,
The at least one processor,
A loading unit that loads images and depth maps, each of which images captured by a plurality of cameras included in the 360-degree camera system are converted into retreat points; And
A viewpoint conversion performing unit that performs a forward viewpoint conversion for each of the images converted at the retreat viewpoint using the depth map
360 degree image rendering system comprising a.

Images