KR20140045636A - Apparatus for generating layered panorama image - Google Patents

Abstract

The present invention relates to an apparatus for generating a layered panorama image. The apparatus for generating a layered panorama image includes a multi-view image obtaining unit capturing one scene in various angles through a plurality of cameras and obtaining a multi-view image; an additional image obtaining unit capturing a region of interest including an object which is a target of interest of a user in the multi-view image and obtaining an additional image; a panorama image generating unit extracting at least one image information among feature point information, focal distance information and image array information from the multi-view image and generating a panorama image by synthesizing the multi-view image on the basis of the extracted image information; and a layered panorama image generating unit matching the panorama image and the additional image and generating a layered panorama image on the basis of position information between the panorama image and the additional image. [Reference numerals] (111) Multi-view image obtaining unit; (112) Additional image obtaining unit; (120) Image information extracting unit; (130) Panorama image generating unit; (140) Layered image generating unit; (150) Layered panorama image generating unit

Description

Layered panoramic image generator {APPARATUS FOR GENERATING LAYERED PANORAMA IMAGE}

The present invention relates to a panorama image technology, and more particularly, to a layered panorama image generating apparatus and method for providing an enhanced service for a region of interest including an object of interest of a user in a panorama image.

A panoramic image that generates a single image by capturing an image using a plurality of cameras or optical devices and then synthesizing it has an advantage of providing a much wider angle of view than a general image.

On the other hand, since panoramic images are aimed at capturing a wide viewing angle, operations such as camera zoom and tilt are difficult. Therefore, there is a difficulty in photographing such as a zooming operation on a specific area in the panoramic image.

As a result, most panoramic video services are remote video, and in order to solve the monotony of the panoramic video, some panoramic players provide a zoom function of the panoramic video. However, since this is simply a method of digitally enlarging a part of the panoramic image, there is a problem in that image quality deteriorates.

Republic of Korea Patent Publication No. 10-2012-0046802

The present invention provides an apparatus and method for generating a layered panoramic image for providing an enhanced image service of a region of interest of a user in a panoramic image.

The present invention provides a layered panoramic image playback method and apparatus for providing an enhanced image of a user's ROI in a panoramic image.

The present invention relates to a layered panoramic image generating device, a multi-view image acquisition unit for acquiring a multi-view image by photographing a scene at a plurality of angles through a plurality of cameras, an object of interest to the user in the multi-view image An additional image obtaining unit which acquires an additional image by capturing a region of interest including: extracting at least one image information of a feature point, a focal length, and image array information from the multi-view image, and based on the extracted image information A panoramic image generator for generating a panoramic image by synthesizing a multi-view image and a layered panoramic image generator for generating a layered panoramic image by matching the panoramic image and the additional image based on position information between the panoramic image and the additional image Include.

Not only a panoramic image having a wide angle but also a high magnification, high quality, and 3D image of a region of interest including an object of interest of the user in the panoramic image may be provided.

1 is a block diagram schematically illustrating an apparatus for generating a layered panoramic image according to an exemplary embodiment of the present invention.
FIG. 2 is a diagram illustrating a configuration of an image acquisition unit illustrated in FIG. 1.
3 illustrates an example of a hierarchical structure of a layered panoramic image according to the present invention.
4 is a block diagram schematically illustrating a terminal device for playing a layered panoramic image according to an exemplary embodiment of the present invention.
5 is a diagram illustrating a layered panoramic image generating method according to an exemplary embodiment of the present invention.
6 is a diagram illustrating an example of a process of playing a layered panoramic image according to an embodiment of the present invention.
7 illustrates an example of an operation of enlarging an image part corresponding to an ROI.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. Further, the present invention is not limited to the embodiments described below, but can be applied in various different forms within the scope of the technical idea of the present invention.

The components described in this specification may include components other than those described below as needed, and a detailed description of parts that are not directly related to the present invention will be omitted. In addition, the arrangement of each component described in this specification can be adjusted as necessary, and one component may be included in another component, and one component may be divided into two or more components.

Layered panorama image is to provide enhanced panorama services such as high magnification, high-definition, and 3D image services for the region of interest (ROI) within the panorama image that provides the user with a wide viewing angle. Refers to a hierarchical structure of additional video such as high magnification, high quality, and 3D video based on a panoramic image.

The user's region of interest refers to an area including an object of interest such as a person or a thing in the panoramic foreground when the panorama image is played.

1 is a block diagram schematically illustrating an apparatus for generating a layered panoramic image according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the layered panoramic image generating apparatus 100 includes an image obtaining unit 110, an image information extracting unit 120, a panoramic image generating unit 130, a layer image generating unit 140, and a layered panoramic image generating. The unit 150 is included.

The image acquirer 110 acquires a multiview image and an additional image for providing an enhanced service for the ROI of the user. The image acquirer 110 may include a multi-view image acquirer 111 and an additional image acquirer 112.

The multi-view image acquisition unit 111 may be configured of a plurality of camera modules, a rig for fixing the camera module at a predetermined geometric position, and a synchronization module for controlling shooting synchronization of each of the plurality of camera modules. Each of the plurality of camera modules performs shooting in accordance with the synchronization control signal of the synchronization module and acquires a multi-view image. The multi-view image is an image obtained by simultaneously photographing a scene from various angles using the plurality of camera modules described above.

The additional image acquirer 112 additionally photographs a specific area in the multiview image acquired by the multiview image acquirer 111 to obtain an additional image. The specific area is a part for providing an enhanced image to the user.

The additional image acquirer 112 may be configured as a camera module for capturing a specific area and a synchronization module for synchronizing shooting synchronization between the camera module of the multi-view image acquirer 111.

In this case, the camera module for additionally photographing a specific area may be configured with one or more cameras as necessary. In addition, a specific area may be captured by using a high magnification camera, a 3D camera, or the like according to the type of the image to be provided to the user.

The additional image acquisition unit 112 captures an image in consideration of a geometric position for spatial matching with the panoramic image, and captures an image according to a synchronization control signal of the multi-view image acquisition unit 111 to synchronize synchronization with a shooting time. To be guaranteed.

FIG. 2 is a diagram illustrating a configuration of an image acquisition unit illustrated in FIG. 1.

As shown in FIG. 2, the image acquisition unit 110 includes a multi-view image acquisition unit 111 including a plurality of cameras, at least one stereoscopic camera and / or high magnification for acquiring 3D images. / An additional image acquisition unit 112 having at least one high magnification camera for acquiring a high quality image.

Referring back to FIG. 1, the image information extractor 120 extracts image information from the multiview image and the additional image acquired by the image acquirer 110.

The multiview image is an individual image obtained from each of the plurality of camera modules of the multiview image acquisition unit 111. These individual multi-view images are synthesized into one panoramic image by the panoramic image generating unit 130 to be described later.

Accordingly, the image information extractor 120 extracts image information for synthesizing the multiview image into one panorama image from the multiview image. For example, feature points, focal lengths, image arrangement information, and the like may be extracted from the multiview image.

The additional image is to provide an enhanced image of a specific region in the panoramic image, and is matched with the panoramic image to form a layered panoramic image.

Accordingly, the image information extractor 120 extracts matching information for matching with the panoramic image or image information for generating an enhanced image such as a 3D image from the additional image. For example, a depth value, a texture value, a color value, a geometric position value, a feature point, and the like may be extracted from the additional image.

The panorama image generator 130 synthesizes individual multi-view images into one panorama image. In this case, the plurality of multi-view images are synthesized based on the image information of the multi-view image extracted by the image information extractor 120 to generate one panorama image.

The layer image generator 140 generates a layer image based on the image information of the additional image extracted by the image information extractor 120.

For example, when a 3D image is to be provided for a specific area in the panoramic image, the 3D image is obtained based on the image information of the additional image obtained by the additional image obtaining unit 112 and the additional information extracted by the image information extracting unit 120. An image can be generated. Alternatively, the layer information may be generated by configuring the image information of the additional image extracted by the image information extractor 120 in a structured form.

The layered panoramic image generator 150 generates a layered panoramic image by matching the panoramic image and the layer image. That is, the panoramic image and the layer image are spatially matched based on the positional information between the panoramic image and the layer image, and the hierarchical structure is configured.

The matching between the panorama image and the layer image may generate a layered panorama image by performing a pattern matching through a direct matching process or image processing of a user. However, in case of direct matching of the user, when the object included in the layer image is moved or the camera module of the additional image acquisition unit 112 acquires the additional image by performing an operation such as zooming or tilting, the user directly in every frame of the image It's hard to match.

Therefore, an automatic matching technique for automatically matching the panorama image and the layer image based on the image information extracted by the image information extractor 120 may be used. For example, an automatic matching technique such as pattern matching of an image or an object tracking technique may be used. Such an automatic matching technique of an image may automatically reflect and match position information between the panorama image and the layer image without user intervention even if an object existing in the image moves.

3 illustrates an example of a hierarchical structure of a layered panoramic image according to the present invention.

Referring to FIG. 3, layer 1 is a panorama image generated by synthesizing a multi-view image, and layer 2 and layer 3 are layer images for providing an enhanced service for a user's ROI. A layered panoramic image is hierarchically generated by spatially matching these images based on the location information of each of layer 1, layer 2, and layer 3.

However, the layering step of FIG. 3 is just an example, and the layered panoramic image may be configured by matching the layer images according to the complexity and diversity of the services to be provided to the user.

4 is a block diagram schematically illustrating a terminal device for playing a layered panoramic image according to an exemplary embodiment of the present invention.

Referring to FIG. 4, the terminal device 200 includes an input unit 210, an analyzer 220, a renderer 230, a user event processor 240, and a layer image processor 250.

The input unit 210 receives the layered panoramic image to reproduce the layered panoramic image according to the present invention, and initializes the terminal device.

The analyzer 220 analyzes the layered panoramic image and analyzes the hierarchical structure.

For example, it is possible to analyze information such as how many layers the layered panoramic image is composed of, what region in the panoramic image is connected to the layer image, and whether the layer image is a high magnification image or a 3D image.

The renderer 230 renders the panorama image based on the information analyzed by the analyzer 220. In this case, only the panoramic image is rendered until the user selects a region of interest.

The user event processor 240 processes an event generated when a user selects an ROI through a screen of the terminal device 200, and periodically monitors whether the ROI is selected by the user. If the user selects a specific ROI during the playback of the panorama image, an event for the specific ROI is generated and processed.

For example, when an event according to the user's region of interest occurs, the user event processor 240 detects location information of the region of interest selected by the user in the panoramic image and transmits the location information to the analyzer 220. The analyzer 220 checks whether there is a layer image that can be provided in the corresponding region based on the location information of the region of interest.

The layer image processor 250 is to provide a layer image by an event according to a user's selection of a region of interest and to play the layer image hierarchically on the panoramic image. That is, the layer image may be matched to a position corresponding to the ROI selected by the user in the panorama image to be rendered.

5 is a diagram illustrating a layered panoramic image generating method according to an exemplary embodiment of the present invention. This method may be performed by the layered panoramic image generating apparatus 100 shown in FIG. 1.

Referring to FIG. 5, a multiview image 10 is obtained by photographing a scene at various angles through a plurality of cameras (S100). This may be performed by the multiview image acquisition unit 111 of FIG. 1.

In operation S110, an additional image 20 is obtained by capturing a partial region in a scene photographed by the plurality of cameras through a high magnification camera or a stereoscopic camera. This may be performed by the additional image acquisition unit 112 of FIG. 1.

The additional image 20 is to provide an enhanced service for the region of interest of the user, as described above, and is obtained by separately photographing a partial region including an object of interest to the user, such as a person and an object, in the multi-view image. It is a video.

In steps S100 and S110, the plurality of cameras for capturing the multi-view image and the camera for capturing the additional image exchange images with each other to perform synchronization.

At least one image information of a feature point, a focal length, and image array information is extracted from the multiview image, and a panorama image is generated by synthesizing the multiview image based on the extracted image information (S120). This may be performed by the image information extractor 120 and the panorama image generator 130 of FIG. 1.

The layered panoramic image 30 is generated by matching the generated panoramic image with the additional image (S130). That is, the panoramic image and the layer image are spatially matched based on the positional information between the panoramic image and the layer image, and the hierarchical structure is configured. This may be performed by the layered panoramic image generator 150 of FIG. 1.

6 is a diagram illustrating an example of a process of playing a layered panoramic image according to an embodiment of the present invention. This may be performed by the terminal device 200 of FIG.

Referring to FIG. 6, first, the terminal device 200 renders a panorama image and outputs it to the screen.

If the user selects a region of interest in the panoramic image during playback of the panoramic image (S200), the terminal device 200 checks whether there is a layer image corresponding to the region of interest selected by the user, and the layer corresponding to the region of interest selected by the user. If there is an image, the image portion corresponding to the ROI is zoomed as shown in FIG. 6B (S210).

An example of an operation of enlarging an image part corresponding to the ROI is illustrated in FIG. 7. Referring to FIG. 7, when a user selects a region of interest 710 in the panorama image 700, the image portion corresponding to the selected region of interest 710 is gradually enlarged and replaced with a high quality layer image 720. Thereafter, the ROI 710 in the panoramic image 700 is matched with the high quality layer image 720 and output on the screen.

Referring back to FIG. 6, the terminal device 200 matches a portion of an image corresponding to the enlarged ROI to a layer image of high quality / high magnification as shown in FIG. (S220).

If the object in the ROI selected by the user moves, the layer image in the panoramic image is automatically matched and output on the screen as the object moves as shown in FIG. 6 (d). That is, the layer image may be rendered by matching the panorama image with the reproduction synchronization by matching the location information between the panorama image and the layer image.

In a conventional terminal device, when performing a zoom operation on a region of interest when playing a panoramic image, a digital zoom function, which is a simple image magnification, is performed, and thus a problem of image quality deterioration occurs when the region of interest is enlarged.

However, in the layered panoramic image according to the present invention, the high magnification / high definition image is pre-recorded on the areas of interest of the user in the panoramic image together with the basic panorama image, and the high magnification / high definition image and the basic panorama image are To match hierarchically. Therefore, even when the terminal device performs a zoom operation on the ROI when the panoramic image is played, the image quality deterioration problem can be solved because the high magnification / high definition image in the panoramic image is provided to the user.

In addition, according to the intention and purpose of the author, a binocular or holographic service as well as a high magnification / high definition image may be provided for a region of interest in the panoramic image.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the claims, and all technical ideas within the scope of the claims should be construed as being included in the scope of the present invention.

100: layered panoramic image generating device
110: image acquisition unit
111: multi-view image acquisition unit 112: additional image acquisition unit
120: image information extraction unit
130: panorama image generating unit
140: layer image generating unit
150: layered panorama image generator
200: terminal device
210: input unit
220: analysis unit
230: renderer
240: user event processing unit
250: layer image processing unit

Claims (1)

A multi-view image acquisition unit for acquiring a multi-view image by photographing one scene at various angles through a plurality of cameras;
An additional image acquisition unit configured to obtain an additional image by capturing a region of interest including an object of interest of the user in the multi-view image;
A panoramic image generator which extracts at least one image information of a feature point, a focal length, and image arrangement information from the multiview image, and generates a panoramic image by synthesizing the multiview image based on the extracted image information; And
And a layered panoramic image generator configured to generate a layered panoramic image by matching the panoramic image and the additional image based on location information between the panoramic image and the additional image.

Images