KR102242710B1 - Apparatus for providing half free-viewpoint video - Google Patents

Abstract

An apparatus for providing a semi-free viewpoint image includes: an image receiving unit for receiving a plurality of actual content images and a plurality of background images obtained by photographing an object, from a plurality of actual cameras arranged at preset camera intervals; an input unit that receives an input of a viewing viewpoint interval from a user terminal; a virtual content image synthesizing unit that determines whether to generate a virtual viewpoint based on the camera interval and the viewing viewpoint interval, and synthesizes a virtual content image corresponding to the virtual viewpoint based on actual camera parameters for the actual content images and the actual cameras according to the result of determination; a semi-free viewpoint image synthesizing unit that extracts the object from the actual content images and the virtual content images, and synthesizes a semi-free viewpoint image based on the extracted object and background images; and an image transmission unit for transmitting the generated semi-free viewpoint image to the user terminal.

Description

Device that provides semi-free view video {APPARATUS FOR PROVIDING HALF FREE-VIEWPOINT VIDEO}

The present invention relates to an apparatus and method for providing a semi-free viewpoint image.

Free-view video is a technology that produces a specific site or object as a multi-view video through a plurality of cameras, and refers to a technology that allows the user to select and view a desired viewpoint by widening the range of viewpoint selection for the user. The free viewpoint image has the advantage that a new viewpoint can be selected and viewed, and the user can feel a sense of realism and immersion through the free viewpoint image.

In relation to such a free view image, Korean Patent No. 10-1556741 discloses an apparatus and method for forming a hemispherical smart camera for providing a realistic free view image.

Conventionally, for free view images, user guides and view synthesis related software and data sets have been standardized through MPEG-FTV (Free Viewpoint TV). However, the data set provided by MPEG-FTV is a super multi-view video captured at intervals of a stereo camera, and the view synthesis software is optimized for the data set. It had the disadvantage of requiring a treatment process.

Receives a plurality of real content images and a plurality of background images photographing an object from a plurality of real cameras arranged at preset camera intervals, and a semi-free viewpoint image that combines the received plurality of real content images and a plurality of background images. It is intended to provide an apparatus and method to provide.

When the viewing point interval is input from the user terminal, it is determined whether to generate a virtual viewpoint based on the camera interval and the viewing viewpoint interval, and based on the actual camera parameters for the plurality of actual content images and the plurality of real cameras according to the determination result. Thus, an apparatus and method for providing a semi-free viewpoint image for synthesizing a virtual content image corresponding to a virtual viewpoint is provided.

An object is extracted from a plurality of real content images and virtual content images, and an apparatus and method for providing a semi-free view image for synthesizing a semi-free view image based on the extracted object and a plurality of background images are provided.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems as described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention is an image receiving unit for receiving a plurality of real content images and a plurality of background images photographing an object from a plurality of real cameras arranged at preset camera intervals , It is determined whether to generate a virtual viewpoint based on the input unit receiving the viewing viewpoint interval from the user terminal, the camera interval and the viewing viewpoint interval, and the plurality of actual content images and the plurality of real cameras according to the determination result. A virtual content image synthesizing unit for synthesizing a virtual content image corresponding to the virtual view based on an actual camera parameter for, extracting the object from the plurality of real content images and the virtual content image, and extracting the object and the plurality of And a semi-free view image synthesizing unit for synthesizing a semi-free view image based on the background image of and an image transmission unit for transmitting the generated semi-free view image to the user terminal.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the above-described problem solving means of the present invention, it is possible to provide an apparatus and method for providing a semi-free viewpoint image for synthesizing viewpoints by obtaining data on a free viewpoint image from a content image having a wide camera distance.

Apparatus for providing a semi-free view image capable of improving the user's viewing satisfaction for the free view image by receiving a desired view interval from the user and providing a semi-free view image synthesized as an image of the user’s desired view; and Can provide a way.

Apparatus and method for providing a semi-free view image that enables efficient management of the image providing device by not having to perform view synthesis by generating a camera parameter for a virtual view in advance when the view of the free view image is switched in the image providing device Can provide.

1 is a block diagram of an image providing system according to an embodiment of the present invention.
2 is a block diagram of an image providing apparatus according to an embodiment of the present invention.
3 is an exemplary diagram showing coordinate data derived from a background image according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a process of generating a background image according to an embodiment of the present invention.
5 is a flowchart of a method of providing a semi-free viewpoint image in an image providing apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, and one or more other features, not excluding other components, unless specifically stated to the contrary. It is to be understood that it does not preclude the presence or addition of any number, step, action, component, part, or combination thereof.

In the present specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized by using two or more hardware, or two or more units may be realized by one piece of hardware.

In this specification, some of the operations or functions described as being performed by the terminal or device may be performed instead in a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an image providing system according to an embodiment of the present invention. Referring to FIG. 1, an image providing system 1 may include a plurality of actual cameras 110, an image providing device 120, and a user terminal 130. The plurality of actual cameras 110, the image providing apparatus 120, and the user terminal 130 exemplarily show components that can be controlled by the image providing system 1.

Each component of the image providing system 1 of FIG. 1 is generally connected through a network. For example, as shown in FIG. 1, the image providing apparatus 120 may be connected to a plurality of actual cameras 110 and user terminals 130 at the same time or at intervals of time.

A network refers to a connection structure that enables information exchange between nodes such as terminals and servers, and is a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, and ultrasound. Communication, Visible Light Communication (VLC), LiFi, and the like are included, but are not limited thereto.

The plurality of actual cameras 110 may be arranged in parallel in one dimension at a preset camera interval (eg, 15 cm) in a specific space. In this case, the plurality of actual cameras 110 may be installed to converge by focusing the focus 115 on one point of the wall.

For the plurality of real cameras 110, an interval between the cameras and the number of the plurality of real cameras 110 may be arbitrarily set by an administrator. For example, the plurality of real cameras 110 may be sequentially designated as camera 1, camera 2, camera 3, etc. from the actual camera located on the left side, and the preset camera spacing is narrowed according to the size of the content. It may be set to be disposed or may be set to be disposed widely.

The plurality of real cameras 110 may transmit a plurality of real content images and a plurality of background images photographing an object to the image providing device 120. For example, when three actual cameras 110 are arranged at a preset camera interval, the second camera located in the middle captures a checkerboard image, and the first and third cameras are in a specific space (for example, An empty stadium) and actual content images may be photographed and transmitted to the image providing device 120.

The image providing device 120 may receive a plurality of actual content images and a plurality of background images photographing an object from a plurality of real cameras 110 arranged at preset camera intervals.

The image providing apparatus 120 may derive internal camera parameters based on the checkerboard image, and may derive external camera parameters from coordinate data and internal camera parameters derived from a background image. Here, the actual camera parameters may include internal camera parameters and external camera parameters.

The image providing apparatus 120 may derive an ideal camera parameter for converting an optical axis to be parallel by performing perspective transformation between neighboring images among a plurality of actual content images, from the actual camera parameters. For example, the image providing apparatus 120 may derive a relationship matrix between an ideal camera parameter and an actual camera parameter, and normalize a plurality of actual content images and a plurality of background images based on the derived relationship matrix.

The video providing device 120 may receive a viewing time interval from the user terminal 130.

The image providing apparatus 120 may determine whether to generate a virtual viewpoint based on the camera interval and the viewing viewpoint interval. Here, the image providing apparatus 120 may synthesize a virtual content image based on a plurality of normalized real content images. For example, when the camera interval is greater than the viewing viewpoint interval, the image providing apparatus 120 may determine that the virtual viewpoint is generated to correspond to the viewing viewpoint interval positioned between the camera intervals.

The image providing apparatus 120 may synthesize a virtual content image corresponding to a virtual viewpoint based on a plurality of real content images and real camera parameters for the plurality of real cameras 110 according to the determination result. For example, the image providing device 120 generates a depth image based on a plurality of real content images and real camera parameters, and if a virtual camera parameter for a virtual viewpoint does not exist, the virtual camera based on the real camera parameter A parameter may be derived, and a virtual content image corresponding to a virtual viewpoint may be synthesized based on the depth image and the virtual camera parameter.

The image providing device 120 extracts feature points from a plurality of background images, matches the extracted feature points to match the matched feature points, generates a panorama image based on the corresponding feature points, and generates a virtual image based on the generated panorama image. A background image corresponding to the viewpoint may be generated. For example, the image providing device 120 sets an ROI corresponding to the real cameras at both ends of the plurality of real cameras from the panoramic image, and derives an arrangement ratio between the real and virtual views of the plurality of real cameras, An ROI corresponding to the virtual viewpoint may be set from the panoramic image based on the derived arrangement ratio, and a background image corresponding to the virtual viewpoint may be generated based on the set ROI.

The image providing apparatus 120 may extract an object from a plurality of real content images and virtual content images, and synthesize a semi-free view image based on the extracted object and a plurality of background images. For example, the image providing apparatus 120 may generate a semi-free view image by synthesizing an object extracted from the virtual content image and a background image generated corresponding to the virtual view.

The image providing apparatus 120 may derive an image frame of a semi-free viewpoint image to be provided at each viewpoint based on the number of real viewpoints corresponding to the plurality of real cameras and the number of virtual viewpoints generated between the actual viewpoints. Here, in the semi-free view image, image frames to be provided at each view may be sequentially reproduced while each view is switched according to an arrangement position of each view.

The image providing device 120 may transmit a semi-free view image generated to the user terminal 130.

2 is a block diagram of an image providing apparatus according to an embodiment of the present invention. Referring to FIG. 2, the image providing device 120 includes an image receiving unit 210, a parameter deriving unit 220, a normalizing unit 230, an input unit 240, a virtual content image synthesizing unit 250, and a background image generating unit. 260, a semi-free view image synthesizer 270 and an image transmission unit 280 may be included.

The image receiving unit 210 may receive a plurality of real content images and a plurality of background images photographing an object from a plurality of real cameras 110 arranged at preset camera intervals. For example, when three real cameras 110 are arranged at preset camera intervals, the image receiving unit 210 includes a plurality of real content images taken from the first camera and the third camera among the three real cameras 110. And a plurality of background images, and a checkerboard image photographed from a camera 2 located in the middle of the three actual cameras 110 may be received. Here, the plurality of actual content images refer to actual images to be provided to the user terminal 130, and the plurality of background images (eg, empty stadiums) refer to image data for estimating external parameters of the camera, and checkers The board image may mean image data for deriving internal parameters of the camera. The reason why the checkerboard image is received from a plurality of real cameras 110, for example, from an actual camera arranged in the middle such as camera 2 is that the internal parameters obtained from camera 2 are transferred to the inside of camera 1 and camera 3 This is to ensure that each image has the same interior by using it.

The parameter derivation unit 220 may derive internal camera parameters based on the checkerboard image. For example, the parameter derivation unit 220 may derive internal parameters using camera calibration. At this time, for convenience of calculation, it is assumed that there is no distortion in the checkerboard image, and an internal parameter may be derived.

The parameter derivation unit 220 may derive external camera parameters from coordinate data derived from the background image and internal camera parameters. Here, the parameter derivation unit 220 may derive 3D coordinates from the background image in order to derive external parameters. The process of deriving the three-dimensional coordinates will be described in detail with reference to FIG. 3.

3 is an exemplary diagram showing coordinate data derived from a background image according to an embodiment of the present invention. Referring to FIG. 3, when the image receiving unit 210 receives a multi-view image with an empty basketball arena as a background from a plurality of real cameras 110, the parameter deriving unit 220 includes eight corners and a basketball of the basketball arena. Using the standard of the stadium, one point is the origin, and the actual three-dimensional coordinate values for each of the eight points (310 to 317) of the basketball arena are obtained with respect to the world coordinate system in the same direction as the camera coordinate system. A total of eight 3D-2D coordinate pairs per camera can be derived by acquiring the 2D image coordinates in which the corresponding points are projected from the captured background image of each empty basketball stadium.

For example, the parameter derivation unit 220 includes an internal parameter obtained from a camera located in the middle of the plurality of real cameras 110, for example, camera 2, eight coordinate pairs of each real camera, and a distortion coefficient of '0'. For', it is possible to derive external parameters of the camera including rotation matrix, camera position information, etc. using the Perspective n point (PNP) algorithm. Here, the PNP algorithm is an algorithm that estimates the relative pose between a camera and a 3D object calibrated from a set of visible 3D points known as object coordinates (X, Y, Z) and a 2D projection known as pixel coordinates (u, v). it means. For example, the parameter derivation unit 220 may derive the rotation angle, position information, etc. between the world coordinate system based on the designated origin and the coordinate system of each actual camera as a matrix or vector (external parameter) using a PNP algorithm. have.

Through this process, the parameter derivation unit 220 may derive internal parameters and external parameters of each of the plurality of actual cameras 110 by using the checkerboard image and the background image photographing a specific space.

Returning to FIG. 2 again, the normalization unit 230 may derive a relationship matrix between an ideal camera parameter and an actual camera parameter, and normalize a plurality of actual content images and a plurality of background images based on the derived relationship matrix. Here, since the plurality of real cameras 110 are used to generate the semi-free view image, the normalization may be performed to correct the height and distortion of the image captured by each real camera.

The parameter derivation unit 220 may derive an ideal camera parameter for converting the optical axis to be parallel by performing perspective transformation between neighboring images among the plurality of actual content images, from the actual camera parameters. Here, the actual camera parameters may include internal camera parameters and external camera parameters.

For example, the parameter derivation unit 220 designates a neighboring image, such as a camera image 1 and 2, or a camera 2 and 3, as a reference image among a plurality of actual content images. A part of my area can be set as a reference patch. In this case, when the next image is assumed to be the target image, the parameter derivation unit 220 may detect a region having the highest similarity to the reference patch in the image. For example, the parameter derivation unit 220 uses the four points of the reference patch to form a homography matrix between two images of the image of camera 1 and the image of camera 2, the image of camera 2 and the image of camera 3 Is estimated, and perspective transformation can be performed on the target image using the estimated homography matrix. At this time, the perspective conversion is performed sequentially with the camera 1 image-2, camera 2 video-3, and camera 3-4, so that the heights of the images are all the same. Since the optical axis of each camera is not parallel, it is possible to convert the optical axis of each camera to be parallel using the derived internal parameters and external parameters of each camera.

By setting a camera located in the middle of the plurality of actual cameras 110 as a reference image, internal parameters and external parameters of an ideal camera may be derived based on the internal parameters and external parameters of the camera located in the center. For example, when three actual cameras 110 are arranged at preset intervals, the second camera is set as the reference image, and the internal parameters and external parameters of the ideal camera are set based on the internal parameters and external parameters of the second camera. Can be derived.

Here, the internal parameters and external parameters of the ideal camera have the same all internal parameters, and the position information of each ideal camera decreases or increases the x-coordinate value to a constant value, and the y-coordinate value or the z-coordinate value may be the same. Here, the reason that the y-coordinate value or the z-coordinate value is the same is because a plurality of actual cameras 110 are installed in parallel in one dimension.

Through this process, the internal parameters and external parameters of the ideal camera can be derived, and the relationship between the derived internal and external parameters of the actual camera and the internal and external parameters of the ideal camera can be estimated as a homography matrix. In this case, a final image may be obtained through perspective transformation for images whose height is corrected using a homography matrix.

In addition, normalization may be performed through perspective transformation in the same manner for an empty stadium image.

The input unit 240 may receive a viewing time interval from the user terminal 130. Here, the viewing viewpoint interval may represent an interval between viewpoints when a virtual viewpoint is finally synthesized and a semi-free viewpoint image is provided to a user. The reason for receiving the viewing point interval is that when a plurality of actual cameras 110 are arranged at 15cm intervals and photographed, the image is cut off when the viewpoint is switched to camera 1, camera 2, and camera 3. Therefore, by receiving an input from the user terminal 130 as to how many virtual viewpoints to be generated between the intervals of the plurality of actual cameras 110, it is to ensure that the image is seamlessly switched.

The virtual content image synthesizing unit 250 may determine whether to generate a virtual viewpoint based on the camera interval and the viewing viewpoint interval. For example, when the camera interval is greater than the viewing viewpoint interval, the virtual content image synthesizing unit 250 may determine that the virtual viewpoint is generated to correspond to the viewing viewpoint interval positioned between the camera intervals. For example, when a plurality of real cameras 110 are arranged at 15cm intervals, and the viewing point interval is input from the user terminal 130 in the input unit 240 as 1cm, the virtual content image synthesizing unit 250 is Can generate a total of 14 viewpoint images between camera 1 and camera 2.

The virtual content image synthesizing unit 250 may synthesize a virtual content image corresponding to a virtual viewpoint based on a plurality of real content images and real camera parameters for the plurality of real cameras 110 according to the determination result.

For example, the virtual content image synthesizing unit 250 may generate a depth image based on a plurality of actual content images and actual camera parameters. In this case, if there is no virtual camera parameter for the virtual viewpoint, the virtual camera parameter may be derived based on the actual camera parameter. This is because a plurality of actual cameras 110 are linearly arranged in one dimension, and thus can be estimated using an arrangement ratio. For example, if a virtual viewpoint for 1.5 between viewpoint 1 and viewpoint 2 needs to be synthesized, but there is no virtual parameter for the virtual viewpoint, the internal parameters and external parameters of camera 1 and camera 2 It is possible to define a virtual camera parameter for the virtual viewpoint of 1.5 times as the median value of the internal parameter and the external parameter. In this way, the camera parameters for the virtual viewpoint can use the internal parameters of the real camera closest to both sides.

The virtual content image synthesizing unit 250 may synthesize a virtual content image corresponding to a virtual viewpoint based on the depth image and the virtual camera parameter. In this case, the virtual content image synthesizing unit 250 may synthesize a virtual content image based on a plurality of normalized real content images.

The virtual content image synthesizing unit 250 may generate a saliency map from the synthesized virtual content image, perform binarization, and set an ROI for the object to separate only the object. Here, the importance map is generated as a binary image by separating important objects from the background from the image, and is used in various fields such as image or video compression, object recognition, scene classification, visual tracking, and behavior detection.

The background image generator 260 may extract feature points from a plurality of background images, match the extracted feature points, and associate the matched feature points with each other.

The background image generator 260 may generate a panorama image based on a corresponding feature point. For example, the background image generator 260 may generate one panorama image by warping a plurality of background images through perspective transformation using any four points among corresponding points.

The background image generator 260 may generate a background image corresponding to a virtual viewpoint based on the generated panorama image. The process of generating the background image will be described in detail with reference to FIG. 4.

4 is an exemplary diagram for explaining a process of generating a background image according to an embodiment of the present invention. Referring to FIG. 4, the background image generator 260 sets an ROI corresponding to the real cameras at both ends of the plurality of real cameras from the panoramic image, and calculates an arrangement ratio between the real and virtual views of the plurality of real cameras. It is possible to derive and set a region of interest corresponding to the virtual viewpoint from the panoramic image based on the derived arrangement ratio, and generate a background image corresponding to the virtual viewpoint based on the set region of interest.

For example, the background image generator 260 sets the image corresponding to the first camera 400 as the first ROI 410, and the same image corresponding to the neighboring camera 2 401 is generated. 2 It can be set as the region of interest 411.

In this case, the background image generation unit 260 uses an area of interest corresponding to the virtual view from the panoramic image by using the arrangement ratio between the real view and the virtual view of the plurality of real cameras 110 with respect to the generated virtual view 402. 412) may be set, and a background image corresponding to the virtual viewpoint may be generated based on the set region of interest.

Returning to FIG. 2 again, the semi-free view image synthesizing unit 270 may generate a semi-free view image by synthesizing an object extracted from the virtual content image and a background image generated corresponding to the virtual view. For example, the semi-free view image synthesizing unit 270 extracts an object from a plurality of real content images and virtual content images, and synthesizes the semi-free view image through a blending process for the extracted object and a plurality of background images. I can. In this case, the semi-free viewpoint image may be synthesized by applying the position of the object to the background image as it is to the coordinates set as the ROI in the binarized importance map.

The semi-free view image synthesizing unit 270 may derive an image frame of a semi-free view image to be provided at each view based on the number of real views corresponding to the plurality of real cameras and the number of virtual views generated between the actual views. I can.

In the conventional free view image, whenever the user switches the view, an internal parameter and an external parameter of the camera of the corresponding virtual view must be generated, and a virtual view must be synthesized using this. Accordingly, the amount of transmission may increase.

Accordingly, the semi-free viewpoint image proposed by the present invention has the advantage of efficiently maximizing the effect in terms of the management of the image providing apparatus 120.

For example, the semi-free view video receives the desired viewing time interval and frame rate from the user terminal 130 before the content is played, so that the number and frames of virtual views to be generated between the plurality of real cameras 110 By calculating the speed, it is possible to calculate the number of frames to be provided to the user terminal 130 at each time point.

For example, when the predetermined arrangement interval for the plurality of actual cameras 110 is 15 cm, and the viewing point interval received from the user terminal 130 is 3 cm, the semi-free view image synthesizing unit 270 is the 1.2 th, The 1.4th, 1.6th, and 1.8th images may be synthesized, and a first frame, a second frame, etc. may be grouped for each view section to generate a single image and transmitted to the user terminal 130.

The image transmission unit 280 may transmit a semi-free view image generated to the user terminal 130. Here, in the semi-free view image, image frames to be provided at each view may be sequentially reproduced while each view is switched according to an arrangement position of each view.

5 is a flowchart of a method of providing a semi-free viewpoint image in an image providing apparatus according to an embodiment of the present invention. The method of providing a semi-free viewpoint image in the image providing apparatus 120 shown in FIG. 5 includes steps processed in a time series by the image providing system 1 according to the embodiment shown in FIGS. 1 to 4. . Accordingly, even if omitted below, the method of providing a semi-free view in the image providing apparatus 120 according to the exemplary embodiment illustrated in FIGS. 1 to 4 is also applied.

In step S510, the image providing apparatus 120 may receive a plurality of real content images and a plurality of background images photographing an object from a plurality of real cameras 110 arranged at preset camera intervals.

In step S520, the image providing device 120 may receive a viewing time interval from the user terminal 130.

In step S530, the image providing device 120 determines whether to generate a virtual viewpoint based on the camera interval and the viewing viewpoint interval, and based on the actual camera parameters for the plurality of actual content images and the plurality of actual cameras according to the determination result. Thus, a virtual content image corresponding to a virtual viewpoint can be synthesized.

In operation S540, the image providing apparatus 120 may extract an object from a plurality of real content images and virtual content images, and synthesize a semi-free view image based on the extracted object and a plurality of background images.

In step S550, the image providing device 120 may transmit the generated semi-free view image to the user terminal 130.

In the above description, steps S510 to S550 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be switched.

The method of providing a semi-free viewpoint image in the image providing apparatus described with reference to FIGS. 1 to 5 may be implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by a computer. I can. In addition, the method of providing a semi-free viewpoint image in the image providing apparatus described with reference to FIGS. 1 to 5 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and are not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

110: multiple real cameras
120: video providing device
210: image receiving unit
220: parameter derivation unit
230: normalization unit
240: input
250: virtual content image synthesis unit
260: background image generation unit
270: Semi-free viewpoint image synthesis unit
280: image transmission unit

Claims (5)

In the apparatus for providing a semi-free viewpoint image,
An image receiver configured to receive a plurality of real content images and a plurality of background images photographing an object from a plurality of real cameras arranged at preset camera intervals;
An input unit receiving a viewing time interval from a user terminal;
Determine whether to generate a virtual viewpoint based on the camera interval and the viewing viewpoint interval, and respond to the virtual viewpoint based on the plurality of real content images and real camera parameters for the plurality of real cameras according to the determination result. A virtual content image synthesizing unit for synthesizing the virtual content image;
A semi-free view image synthesizing unit for extracting the object from the plurality of real content images and the virtual content images, and synthesizing a semi-free view image based on the extracted object and the plurality of background images; And
An image transmission unit that transmits the generated semi-free view image to the user terminal
Including,
A feature point is extracted from the plurality of background images, the extracted feature points are matched to match the matching feature points, a panoramic image is generated based on the corresponding feature point, and the virtual view is corresponded to the virtual viewpoint based on the generated panoramic image. A background image generator that generates a background image
Including more,
The background image generation unit,
Set an ROI corresponding to the real cameras at both ends of the plurality of real cameras from the panoramic image,
Derive an arrangement ratio between the actual viewpoint and the virtual viewpoint of the plurality of real cameras,
Set an ROI corresponding to the virtual viewpoint from the panoramic image based on the derived arrangement ratio,
To generate a background image corresponding to the virtual viewpoint based on the set region of interest.
The method of claim 1,
When the camera interval is greater than the viewing viewpoint interval, the virtual content image synthesizing unit determines that a virtual viewpoint is generated to correspond to the viewing viewpoint interval positioned between the camera intervals,
Generate a depth image based on the plurality of actual content images and the actual camera parameters,
If the virtual camera parameter for the virtual viewpoint does not exist, the virtual camera parameter is derived based on the actual camera parameter,
The image providing apparatus for synthesizing a virtual content image corresponding to the virtual viewpoint based on the depth image and the virtual camera parameter.
The method of claim 1,
A parameter derivation unit that derives internal camera parameters based on the checkerboard image, and derives external camera parameters from coordinate data derived from the background image and the internal camera parameters
Including more,
The image providing apparatus, wherein the actual camera parameter includes the internal camera parameter and the external camera parameter.
The method of claim 1,
The semi-free view image synthesizing unit generates the semi-free view image by synthesizing an object extracted from the virtual content image and a background image generated corresponding to the virtual view. delete

Images