KR20220101565A - Apparatus and Method for 360°Image Stitching - Google Patents

Abstract

An apparatus and method for 360-degree image stitching are disclosed. The apparatus for 360-degree image stitching according to an embodiment of the present invention includes a memory having at least one program recorded, and a processor for executing the program, wherein the program creates a single 360-degree image by stitching features of multiple input images based on at least one parameter included in a 360-degree stitching function description template, the 360-degree stitching function description template includes a setting parameter, which is an array of function parameters, the setting parameter includes a stitching parameter, a camera parameter, a feature parameter, and a projection parameter, the feature parameter includes a feature extraction method from each of the plurality of input images, and the projection parameter includes a projection type that is a type of projection surface onto which the plurality of input images are projected. The apparatus defines the feature parameter as information for post-processing of a stitching result to correct a 360-degree image stitching error, and the apparatus is executed based on the defined feature parameter.

Description

Apparatus and Method for 360°Image Stitching}

The described embodiment relates to a 360-degree image stitching technique for generating a single 360-degree panoramic image by combining a plurality of images.

In general, an image stitching function requires internal and external characteristics of a capture device to estimate a geometrical relationship between capture devices, such as a camera, that has acquired each of a plurality of input images. However, since there may be inconsistencies that cannot be calculated using the inner and outer matrices in the plurality of input images, there must be a slight gap. Most image stitching functions apply feature extraction and matching technology to further improve stitching quality.

In addition, in most image stitching algorithms, after calculating homography with a feature matching point randomly selected from a feature matching point set, it is determined whether most of the feature points satisfy the homography transformation model. However, when the feature matching points are randomly selected as described above, the calculated homography may not be global because the selected feature points appear at a locally concentrated position. Therefore, in this case, the homography transformation model should be updated to be global by reselecting the matching feature points. Therefore, in order to correct the stitching error as described above, feature parameters are defined as information for post-processing such as boundary information adjustment by reselection or update of feature correspondences. There is a need.

Meanwhile, in the current 360-degree stitching function, the output range parameter is a parameter defined in OMAF (“ISO/IEC 23090-2:2019 Information technology — Coded representation of immersive media — Part 2: Omnidirectional media format”, Jan. 2019.) are materialized by However, while OMAF handles various projection formats such as Equirectangular and CubeMap, the current 360-degree stitching function does not provide parameters for these various projection formats.

The described embodiment defines a feature parameter as information for post-processing of a stitching result for correcting a 360-degree image stitching error, and proposes a 360-degree image stitching apparatus and method performed based on the defined feature parameter. do it with

In addition, the described embodiment defines projection parameters for projecting an input stream having various types of projection shapes, and aims to propose a 360-degree image stitching apparatus and method performed based on the defined projection parameters.

A 360-degree image stitching apparatus according to an embodiment includes a memory in which at least one program is recorded and a processor executing the program, wherein the program is multi-input based on at least one parameter included in the 360-degree stitching function description template. Stitching features of images to create a single 360-degree image, wherein the 360-degree stitching function description template includes a setting parameter that is an array of function parameters, wherein the setting parameter includes a stitching parameter, a camera parameter, a feature parameter, and a projection parameter. The feature parameter may include a method of extracting a feature from each of the plurality of input images, and the projection parameter may include a projection type that is a type of a projection surface onto which the plurality of input images are projected.

In this case, the program detects a key point from each of the multiple input images using one of at least one feature extraction method and extracts a descriptor for the key point, wherein the feature extraction method is SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), nonlinear diffusion filter, accelerated KAZE algorithm, Oriented FAST and Rotated BRIEF (Oriented FAST and Rotated BRIEF) based on FAST key point detection and combination of BRIEF descriptors, scale space key points and rotation invariant key points BRISK (Binary Robust Invariant Scalable Keypoints) algorithm using descriptors, feature descriptors using BRIEF (Binary Independent Elementary Feature) descriptors for matching points, and feature detection using Laplacian of Gaussian (LoG) of images. can

In this case, the feature extraction method is a 'URI' type, and the URN may indicate the feature extraction method.

In this case, the feature parameter may further include a feature point number, feature point positions, and feature correspondence.

In this case, the program includes the steps of decoding a plurality of input images, extracting a feature point of each of the plurality of decoded input images, extracting a camera parameter based on the feature point, based on the camera parameter, the decoded multiple generating a 360-degree image by projecting the input images onto a projection object, determining boundary line information of the 360-degree image based on camera parameters, post-processing the 360-degree image based on the boundary line information, and the The step of encoding the post-processed 360-degree image is performed, but in the step of extracting the feature point, the feature point is extracted based on the feature extraction method of the feature parameter, and in the post-processing step, the number of feature points of the feature parameter (feature point number) ), feature point positions, and feature correspondence may be used.

At this time, the projection type includes a table-type omnidirectional projection format list including a square (Equirectangular) and a cubemap (Cubemap), but when Id is '0' in the table, the omnidirectional projection is a square (Equirectangular) and , when Id is '1', the omni-directional projection is a cubemap, and when Id is '2', the omni-directional projection may be in another projection format.

In this case, the program includes the steps of decoding a plurality of input images, extracting a feature point of each of the plurality of decoded input images, extracting a camera parameter based on the feature point, based on the camera parameter, the decoded multiple generating a 360-degree image by projecting the input images onto a projection object, determining boundary line information of the 360-degree image based on camera parameters, post-processing the 360-degree image based on the boundary line information, and the The encoding of the post-processed 360-degree image may be performed, but in the step of generating the 360-degree image, it may be performed based on the projection format of the projection parameter.

A 360-degree image stitching method according to an embodiment includes the steps of decoding a plurality of input images, extracting feature points of each of the plurality of decoded input images, extracting camera parameters based on the feature points, Based on the decoded multi-input images by projecting the image to the projection, generating a 360-degree image, based on the camera parameter, determining the boundary line information of the 360-degree image, based on the boundary line information, the 360-degree image Post-processing and encoding the post-processed 360-degree image, wherein the steps are performed based on a 360-degree stitching function description template, and include a setting parameter that is an array of function parameters, wherein the setting parameter comprises: a stitching parameter, a camera parameter, a feature parameter, and a projection parameter, wherein the feature parameter includes a method of extracting a feature from each of a plurality of input images, and the projection parameter is a projection that is a type of a projection surface onto which the plurality of input images are projected. It can contain types.

At this time, in the step of extracting the key point, a key point is detected from each of the multiple input images using one of at least one feature extraction method of the feature parameter, and a descriptor for the key point is extracted, wherein the feature extraction method is SIFT ( Scale Invariant Feature Transform), Speeded Up Robust Features (SURF), nonlinear diffusion filter, accelerated KAZE algorithm, Oriented FAST and Rotated BRIEF (ORB), scale based on FAST key point detection and combination of BRIEF descriptors BRISK (Binary Robust Invariant Scalable Keypoints) algorithm using spatial keypoints and rotation invariant keypoint descriptors, using BRIEF (Binary Independent Elementary Feature) descriptors for matching points. Features using Laplacian of Gaussian (LoG) descriptors and images. at least one of detection.

In this case, the feature extraction method is a 'URI' type, and the URN may indicate the feature extraction method.

In this case, the feature parameter may further include a feature point number, feature point positions, and feature correspondence.

In this case, in the post-processing step, the feature point number, feature point positions, and feature correspondence of the feature parameter may be used.

At this time, the projection type includes a table-type omnidirectional projection format list including a square (Equirectangular) and a cubemap (Cubemap), but when Id is '0' in the table, the omnidirectional projection is a square (Equirectangular) and , when Id is '1', the omni-directional projection is a cubemap, when Id is '2', the omni-directional projection is another projection format, in the step of generating a 360-degree image, the projection of the projection parameters It may be performed based on the format.

The method for generating a 360-degree image stitching workflow according to an embodiment may include: obtaining a request for 360-degree image stitching and a 360-degree image parameter necessary for generating a 360-degree image stitching workflow; It can be applied to a 360-degree image stitching workflow A multimedia processing entity required to process tasks constituting a 360-degree image stitching workflow, obtaining a list of existing functions, based on functions selected from the function list, and generating a 360-degree image stitching workflow Determining the number of and according to the determined number of multimedia processing entities, generating a plurality of multimedia processing entities and assigning tasks constituting a 360-degree image stitching workflow to the plurality of multimedia processing entities, 360 Operations constituting the figure image stitching workflow are performed based on at least one parameter included in the 360 degree stitching function description template, wherein the 360 degree stitching function description template includes a setting parameter that is an array of function parameters, The parameters include a stitching parameter, a camera parameter, a feature parameter, and a projection parameter, wherein the feature parameter includes a method of extracting a feature from each of a plurality of input images, and the projection parameter includes: It can contain a projection type that is a kind.

According to the described embodiment, 360-degree image stitching can be performed based on the feature parameter defined as information for post-processing of the stitching result for correcting the 360-degree image stitching error, so that the quality of the 360-degree stitched image can be improved. can

In addition, according to the described embodiment, 360-degree image stitching can be performed in various projection forms, so that a 360-degree image stitching function can be improved.

1 is a schematic block diagram illustrating a 360-degree image stitching apparatus according to an embodiment.
2 is a diagram for explaining an image correction process according to an embodiment.
3 is a detailed description of a method of setting a workflow for performing a 360-degree image stitching operation according to an exemplary embodiment.
4 is a diagram showing the configuration of a computer system according to an embodiment.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Although "first" or "second" is used to describe various elements, these elements are not limited by the above terms. Such terms may only be used to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the spirit of the present invention.

The terminology used herein is for the purpose of describing the embodiment and is not intended to limit the present invention. In this specification, the singular also includes the plural, unless specifically stated otherwise in the phrase. As used herein, “comprises” or “comprising” implies that the stated component or step does not exclude the presence or addition of one or more other components or steps.

Unless otherwise defined, all terms used herein may be interpreted with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, a 360-degree image stitching apparatus and method according to an embodiment will be described in detail with reference to FIGS. 1 to 4 .

1 is a schematic block diagram illustrating a 360-degree image stitching apparatus according to an embodiment.

1, a multi-media processing entity (Multi-Media Processing Entity) 110 corresponding to a 360-degree image stitching apparatus according to an embodiment is a network-based media processing (NBMP) source (hereinafter referred to as NBMP). An application that processes a 360-degree image of a 360-degree stitched image that is a result of a 360-degree stitching task based on parameters transmitted together with a plurality of images transmitted from the 'media processing source') 100 (120) can be provided.

First, the media processing source 100 may include configuration information 102 , a media source 104 , and metadata 106 .

The setting information 102 and the metadata 106 represent information related to a plurality of input video streams.

The media source 104 contains the raw image from which the 360-degree image is made. In this case, the media source 104 may include a texture image and a depth image, or may include only a texture image.

The multimedia processing entity 110 may be set by a workflow manager 202 of FIG. 3 to be described later. In this case, if it is determined that the image stitching workflow cannot be processed by a single processing entity, two or more multimedia processing entities 110 may be set. If it is determined that the image stitching workflow can be processed by a single processing entity, only one multimedia processing entity 110 may be set. The multimedia processing entity 110 may be created in the cloud platform 208 of FIG. 2 by the workflow manager 204 of FIG. 2 .

If two or more multimedia processing entities 110 are set, tasks according to steps 130 to 142 of FIG. 1 may be assigned to a plurality of multimedia processing entities 110 . Accordingly, an image stitching operation requiring a lot of computing resources can be quickly processed by the plurality of multimedia processing entities 110 .

When an image stitching task is processed by a plurality of multimedia processing entities 110, for assignment of the tasks, for example, image decoding, feature point extraction, camera parameter extraction, image projection, boundary information extraction related to image stitching , setting information, input and output descriptors, and metadata for detailed operations such as blending, post-processing and image encoding are required. The information is used to interface between different tasks in a plurality of processing entities.

According to an embodiment, detailed information and parameters applied to each of the detailed tasks as described above may be defined by a 360-degree image stitching function description template to be described later. Accordingly, the multimedia processing entity 110 may generate a single 360-degree image by stitching features of multiple input images based on at least one parameter included in the 360-degree stitching function description template. A detailed description of the 360-degree stitching function description template according to this embodiment will be described later with reference to <Table 2> to <Table 7>.

Prior to this, detailed steps (steps 130 to 142) of the 360-degree image stitching method will be described in detail.

In step 130, the encoded image is decoded. Specifically, in step 130 , the encoded raw image of the media source 104 is decoded according to the operation of multiple threads or GPUs. In addition, the media data feed encoded from the cloud parameters as well as the media source 104 may be used for decoding in step 130 . The raw image represents an original image to which post-processing is not applied.

In step 132 , a feature point is extracted from the raw image decoded in step 130 .

The feature point refers to a point serving as a reference point for matching an image with an image when tracking or recognizing a specific object in a plurality of images. By analyzing the values of the pixels distributed based on the feature point, similar portions between different images are detected. Accordingly, by comparing feature points between adjacent raw images in operation 132 , a corresponding point between adjacent raw images may be determined. And according to the corresponding point, a stitching process may be performed between adjacent raw images.

At this time, in step 132, a key point is detected from each of the multiple input images by using one of the feature extraction methods included in the feature parameters defined according to the embodiment (refer to <Table 6> to be described later), A descriptor for a keypoint may be extracted. At this time, the feature extraction method is ORB based on SIFT (Scale Invariant Feature Transform), SURF (Speed Up Robust Features), nonlinear diffusion filter, accelerated KAZE algorithm, FAST key point detection, and combination of BRIEF descriptors (Oriented FAST and Rotated BRIEF), BRISK (Binary Robust Invariant Scalable Keypoints) algorithm using scale space keypoints and rotation invariant keypoint descriptors, feature descriptors using BRIEF (Binary Independent Elementary Feature) descriptors for matching points, and LoG of images It may be performed based on a feature extraction method including at least one of feature detection using (Laplacian of Gaussian).

In this case, the feature parameter input in step 132 may be continuously used as needed in subsequent steps.

In step 134, extrinsic and intrinsic camera parameters are calculated according to the set of feature points and corresponding points. The external camera parameter includes a photographing angle of the camera, a photographing position, and the like. The internal camera parameters include a focal length of a camera, a principal point, an asymmetry coefficient, and the like.

According to an embodiment, by analyzing the feature point and the corresponding point, the difference between the photographing angle and the photographing position between two cameras used to photograph two adjacent images may be calculated. Accordingly, an external parameter of the camera may be determined according to the feature point and the corresponding point.

According to an embodiment, by analyzing the feature point and the corresponding point, a difference between a focal length, a main point, and an asymmetry coefficient of two cameras may be calculated. Accordingly, an internal parameter of the camera may be determined according to the feature point and the corresponding point.

Also, in step 134 , an image distortion correction process based on camera parameters may optionally be performed. According to an embodiment, the image distortion correction process may be a process of correcting image distortion according to lens characteristics.

In step 136 , the raw images decoded in step 130 are projected on the image projection plane based on the camera parameter determined in step 134 . Specifically, based on the camera parameter and the image projection plane, the images are geometrically aligned. And the geometrically aligned images are projected on the image projection plane. As a result of the image projection, a 360-degree image is generated.

In this case, step 136 may be performed based on projection parameters defined according to an embodiment (refer to Table 7 to be described later). That is, the decoded raw images may be projected onto one of various types of projection surfaces including a sphere, a cube, a polyhedron, and an ERP (EquiRectangular Projection).

Accordingly, various types of input streams can be covered. That is, the output-coverage of the stitching parameters of <Table 3> to be described later is "Coverage of the output image may be specified by following parameters defined in OMAF 7.3.5, in the following order: [coverage_shape_type, center_azimuth, center_elevation, center_tilt, azimuth_range, elevation_range]". That is, OMAF (“ISO/IEC 23090-2:2019 Information technology — Coded representation of immersive media — Part 2: Omnidirectional media format”, Jan. 2019.) dealing with various projection formats such as Equirectangular and CubeMap. It is embodied by the parameters defined in , and the 360-degree stitching function can provide parameters for these various projection shapes by the administration parameters according to the embodiment, so that various types of input streams handled by OMAF can be covered. .

In operation 138 , a boundary line of adjacent images is extracted based on the stitching parameter initially extracted from the 360-degree image generated in operation 136 . In this case, the stitching parameter may include a boundary line position parameter for the position of the boundary line or a boundary line mask parameter indicating a region corresponding to the boundary line in the image. The stitching parameter for the boundary line may be determined based on the camera parameter determined in step 134 .

In operation 138 , when an object that did not exist at a previous time suddenly appears in the overlapping region of an adjacent image, the stitching parameter may be updated. Accordingly, according to the change of the overlapping area, the boundary line position parameter or the boundary line mask parameter may be changed.

In operation 140 , blending, color correction, and noise filtering processes may be performed on the overlapping region of the 360-degree image generated in operation 136 . Blending, color correction, and noise filtering processes may be performed based on the boundary information extracted in step 138 .

The blending process is a process of mixing two partial images corresponding to overlapping regions of a 360-degree image so that the 360-degree image can be visually recognized as one continuous image. The color correction process refers to a process of adjusting color parameters such as color, brightness, and saturation so that images constituting a 360-degree image can be recognized as a single image. The noise filtering process refers to a process of removing noise from an image. According to the process of step 140 , a 360-degree image generated from a plurality of images may be visually recognized as one image.

2 is a diagram for explaining an image correction process according to an embodiment.

Referring to FIG. 2 , in most image stitching algorithms, after calculating a homography with a feature matching point randomly selected from a feature matching point set, it is determined whether most of the feature points satisfy the homography transformation model. However, when the feature matching points are randomly selected as described above, the calculated homography may not be global because the selected feature points appear at a locally concentrated position. Therefore, in this case, the homography transformation model should be updated to be global by reselecting the matching feature points. Therefore, in order to correct the stitching error as described above, feature parameters are defined as information for post-processing such as boundary information adjustment by reselection or update of feature correspondences. There is a need.

Accordingly, according to the embodiment, feature parameters are defined as the function parameters of the setting parameters, and in step 140 , it may be performed based on the feature parameters defined according to the embodiment.

In this case, the feature parameter may further include a feature point number, feature point positions, and feature correspondence.

In step 142, the 360-degree image processed in step 140 is encoded. The encoding in step 142 may be performed according to cloud parameters. And the encoded 360-degree image may be output by the cloud platform.

Through steps 130 to 142, a video stream composed of an coded 360-degree image is generated from a video stream composed of an coded 2D image.

According to an embodiment, the order of steps 130 to 142 may be partially changed, and a predetermined step may be omitted. Also, in the middle of steps 130 to 142, an image processing technique conventional in the image processing field may be applied. Commands corresponding to steps 130 to 142 may be performed in the cloud platform 208 of FIG. 3 .

Hereinafter, in <Table 1>, descriptions, inputs and outputs for each operation of steps 130 to 142 are described.

Task Description Input Output Decoding(130) Involves video type conversion from encoded video stream feed from media source to raw video stream using several threads or GPUs. Encoded media data feed from media source, and media and cloud parameters Decoded media data like raw images Feature point extraction(132) Involves feature point extraction and matching corresponding points process between neighboring raw images. Raw images to be aligned(stitched), and feature extraction method Extracted feature points and correspondence sets Camera parameter extraction(134) Involves camera extrinsic and intrinsic parameter calculation using feature points and correspondence sets. Optionally, involves image distortion correction process using cameral parameters. Extracted feature points and correspondence sets Extrinsic/intrinsic camera parameters and Lens distortion corrected images Projection(136) Involves image projection on the sphere, cube, polyhedron, and ERP with geometric alignment based on the camera parameters. Lens distortion corrected images and projection and camera parameters Projected 360 images Seam information extraction(138) Involves seam extraction or update process required when suddenly an object appears on the overlapped region. Projected 360 images and initially extracted stitching parameters including seam position or seam mask Extracted or updated stitching parameters including seam position or seam mask Blending and post-processing(140) Involves blending, color correction, and noise filtering in the overlapped region of projected 360 images Projected 360 images, and stitching parameters Blended and post-processed 360 images Encoding(142) Involves video encoding of blended and post-processed 360 images using several threads or GPUs. Blended and post-processed 360 images and media and cloud parameters Encoded 360 video
*It could be the output of cloud platform

The media service provider may provide a 360-degree image stitching service to the user by using the workflow according to steps 130 to 142 described above.

The media service provider's working directory may include details about each job in <Table 1>. In addition, the workflow manager 204 of FIG. 3 may select each task in <Table 1> to prepare the workflow described in FIG. 1 .

Hereinafter, a workflow for performing the task of image stitching in the media system for distributing the image stitching process according to steps 130 to 142 described in FIG. 1 to a plurality of multimedia processing entities 110 will be described.

3 is a detailed description of a method of setting a workflow for performing a 360-degree image stitching operation according to an exemplary embodiment.

3 is performed through interworking of some components of a media system for image stitching according to a plurality of processing entities according to an embodiment, a media source (NBMP Source) 202 and a workflow manager (Workflow Manager) 204 ), a Function Registry 206 , a Cloud Platform 208 , and a Media Task Manager (NBMP Task) 210 .

In this case, the media source 202 , the workflow manager 204 , the function registry 206 , and the media task manager 210 may be operated by separate processors. Alternatively, all or part of the workflow manager 204 , the media source 202 , the function registry 206 , and the media operation manager 210 may be operated by one processor. In addition, the workflow manager 204 , the media source 202 , the function registry 206 , and the media operation manager 210 may each include a memory device for storing information for image stitching.

The workflow manager 204 may create a workflow for image stitching. In addition, the workflow manager 204 may create a plurality of multimedia processing entities for performing a plurality of tasks included in the workflow in the cloud platform 208 .

The cloud platform 208 includes a plurality of processors. In addition, tasks necessary for image stitching may be performed by the plurality of multimedia processing entities set in the cloud platform 208 .

Then, a method of setting a workflow of image stitching by the workflow manager 204 will be described in detail.

Referring to FIG. 3 , in step S302 , the media source 202 uses the CreateWorkflow API (Application Programming Interface) to generate a workflow for image stitching together with a request for image stitching. The CreateWorkflow API is a dedicated API for creating a workflow. In addition, the media source 202 generates a workflow description document describing the generated workflow, and transmits the workflow description document to the workflow manager 204 .

The workflow manager 204 creates a workflow according to the sent workflow description document.

In step S304 , the workflow manager 204 sends a query or query set to the function registry 206 to find a function to be placed in the workflow for image stitching. The query or query set describes the functionality of the workflow required by the workflow description document of step 302 . In addition, the function registry 206 stores a list of functions supported by the media system 200 .

In step S306, for each query, the function registry 206 provides the workflow manager 204 with a list of functions that can be used for image stitching, and their description and configuration information. The function registry 206 compares the list of functions supported by the media system 200 with the description of the functions of the workflow described in the query or query set.

In addition, the function registry 206 may provide a list of functions applicable to the workflow among functions supported by the media system 200 to the workflow manager 204 .

In step 308 , the workflow manager 204 selects functions necessary for the workflow from the list of functions provided from the function registry 206 in step 306 . Then, the workflow manager 204 may connect to the cloud platform 208 and create one or more media processing entities in the cloud platform 208 according to the requirements of the selected functions.

In step S310, the cloud platform 208 confirms the creation of one or more multimedia processing entities. The cloud platform 208 may then confirm the generation of network access information for one or more multimedia processing entities. The cloud platform 208 communicates to the workflow manager 204 that one or more multimedia processing entities have been created.

In step S312, the workflow manager 204 creates setting information for each task. And the workflow manager 204 transmits the setting information for each job to the job manager 210 . In order to transmit the setting information to the job manager 210 , a job API may be used. The operation API is a dedicated API for generating configuration information.

In step S314, the task manager 210 checks whether successful setting for each task has been made. If successful setting of the current task is made, the task manager 210 generates access information so that the workflow manager 204 can access the next task. And the task manager 210 generates access information to the workflow manager 204 . As steps 312 and 314 are performed for each task, it can be confirmed whether successful setting for the entire workflow has been made.

In step 316, the workflow manager 204 confirms the creation of the workflow, and notifies the media source 202 that media processing for image stitching can be started. In addition, as the media source 104 provided from the media source 202 is processed according to a workflow, a 360-degree image may be generated.

As described above, in order to set a function used in the workflow, create an appropriate number of multimedia processing entities for performing the task of the workflow, and determine the setting information of each task of the workflow, Many parameters are needed for the tasks of the workflow.

Hereinafter, <Table 2> shows an example of a parameter list of a 360-degree image stitching function description template.

Descriptor Parameter Name Type Description General Description string stitches multiple input frames into a 360 equirectangular projected output. nbmp-brand String "urn:mpeg:mpegi:nbmp:2020:360stitcher" input-ports Array of objects Multiple input ports shall be defined for input media (e.g. images or videos). Specific media types supported by the implementation shall be defined by Input Descriptor (e.g. mime-type and the correspondent media type parameters). output-ports Array of objects One output port shall be defined for monoscopic output (image or video). Any other optional outputs for other output types if supported. Specific media types supported by the implementation shall be defined by Output Descriptor (e.g. mime-type and the correspondent media type parameters). The output media type shall be the same as the input media type. Processing Keywords Array of string [
"360-degree panoramic stitcher",
"360-degree stitcher"
] Configuration Parameters Array of parameters Function parameters:
· stitching parameters
· camera parameters
· feature parameters
· projection parameters

Referring to <Table 2>, the 360-degree stitching function description template according to the embodiment may include a general descriptor, a processing descriptor, and a configuration descriptor. The configuration descriptor includes various function parameters as sub-parameters for configuration information of tasks required for an image stitching workflow.

For example, the stitching parameters may be expressed as data in the form of an array including sub-parameters related to a boundary point position and a boundary point mask.

In addition, the camera parameters are an array type including sub-parameters related to the focal length, principal point, skew coefficient, and translation and rotation of the camera. can be expressed as data of

In addition, the feature parameters proposed according to the embodiment are subordinate to a feature extraction method, a feature point number, feature point positions, and feature correspondence. It can be expressed as data in the form of an array including parameters.

In addition, the projection parameter proposed according to the embodiment may be expressed as data in the form of an array including sub-parameters related to a projection type and the like.

Hereinafter, examples of parameters required for each operation of image stitching are described in <Table 3> to <Table 7>.

<Table 3> shows examples of stitching parameters.

Name definition Unit Type Valid Range stitching-type Output panoramic type, one of "monoscopic" and "stereoscopic". The default value is "monoscopic". N/A String N/A camera-parameters Camera and capturing-related parameters N/A Array of camera objects N/A v-seam-positions Interpolated　vertical areas where overlapped field of views occurs between neighbouring frames. The number of positions is the same as the number of overlaps. Each position can be represented by a pair of position number in degree (starting and ending degrees) and the total size of the array equals to the seam number multiplied by 2. degree Array of numbers N/A h-seam-positions Interpolated　horizontal areas where overlapped field of views occurs between neighbouring frames. The number of positions is the same as the number of overlaps. Each position can be represented by a pair of position number in degree (starting and ending degrees) and the total size of the array equals to the seam number multiplied by 2. degree Array of numbers N/A seam-mask Interpolated area locations may be represented by mask image, which has only 1 or 0 value.This parameter is optional. URL String N/A stitching-method Specific stitching algorithm may be specified for fast or fine stitching approaches. The value can be one of "fast", "normal", and "fine". The default value is "fast". N/A String N/A seam-extent-freedom The seam margin may be expanded in degree. The size of the array must be the half of the seam position. The default value for each seam is 0 (zero). pixel Array of numbers N/A convergence-distance Convergence distance selection criteria may be specified. It determines the handling of near and far detail reproduction around seam positions. The value is one of "auto", "near", "far". The default value is "auto". N/A String N/A camera-weighting The weighting value of the input in stitching process. The higher the weighting value is, the more important the input is. The value shall be normalized to the scale 0 to 100. The output quality changes when the value is changed. The value for each input shall be ordered in the same as the input-ports' order and the size of the array is equal to the number of input media streams. N/A Array of numbers 0..100 output-coverage Coverage of the output image may be specified by following parameters defined in OMAF 7.3.5, in the following order: [coverage_shape_type, center_azimuth, center_elevation, center_tilt, azimuth_range, elevation_range] degree Array of numbers -360..360

<Table 4> shows an example of camera object parameters as camera parameters.

Name definition Unit Type Valid Range camera-intrinsic Camera lens intrinsic parameters representing the optical center, focal length of the camera, as well as the skew coefficient N/A Object N/A camera-extrinsic Camera transformation parameters representing a rigid transformation from 3-D world coordinate system to the 3-D camera's coordinate system. The structure is a 4x4 homogeneous transformation matrix in an array, which is in row major order. N/A Array of Numbers N/A camera-shutter-type Camera shutter type, one of "rolling" and "global". The default value is "rolling". N/A String N/A camera-sync-skew Camera shutter synchronization skew value with respect to the first camera when multiple cameras are used.The default value is 0. millisecond Number N/A capturing-environment The capturing scene type, one of "indoor", "outdoor", or "unknown/unspecified". The default is "unknown". N/A String N/A shooting-type Capturing shooting type, one of "long shoot", "medium shoot", "close up", or "unspecified". The default is "unspecified". N/A String N/A

<Table 5> shows an example of camera intrinsic object parameters as camera parameters.

Name definition Unit Type Valid Range camera-type Camera type, one of "pinhole", "fisheye", "omnidirectional".
The default value is "pinhole". N/A String N/A focal length Focal length (x, y) pixel Array of numbers N/A principal-point Principal point (optical center) (x, y) pixel Array of numbers N/A distortion Coefficients of various radial and other distortions. The coefficients array follows the structure used by OpenCV like one-dimensional vector of (k ₁ , k ₂ , p ₁ , p ₂ [,k ₃ [,k ₄ , k ₅ , k ₆ [, e1, e2 [, e3 . .. ]]]]), with lengths like 4, 5, 8, 10 and more. The "e" coefficients are parameters for Entrance Pupil (EP) distortion correction, which is defined in MPEG-I Part 7 and OMAF N/A Array of numbers N/A

<Table 6> shows examples of feature parameters.

Name definition Unit Type Valid Range feature-extraction-method URN indicating the feature extraction method and take one of the following values according to the feature extraction methods in OpenCV
1. urn:opencv:cv:feature2d:sift
- extracting key points and computing descriptors using the Scale Invariant Feature Transform (SIFT) algorithm
2. urn:opencv:cv:feature2d:surf
- extracting Speeded Up Robust Features (SURF) from an image
3. urn:opencv:cv:feature2d:kaze
- keypoint detector and descriptor extractor using nonlinear diffusion filter in the scale space
4. urn:opencv:cv:feature2d:akaze
- keypoint detector and descriptor extractor using accelerated KAZE algorithm
5. urn:opencv:cv:feature2d:orb
- keypoint detector and descriptor extractor using ORB (Oriented FAST and Rotated BRIEF) based on the fusion of Oriented FAST keypoint detector and BRIEF descriptor
6. urn:opencv:cv:feature2d:brisk
- keypoint detector and descriptor extractor using BRISK (Binary Robust Invariant Scalable Keypoints) algorithm that utilizes scale space key point detection and rotation invariant key point descriptor
7. urn:opencv:cv:feature2d:brief
- feature descriptor using Binary Robust Independent Elementary Feature (BRIEF) descriptor for matching points described
8. urn:opencv:cv:feature2d:log
- feature detector using Laplacian of Gaussian (LoG) filter of an image, described
9. 'other URN'
- other URN indicating other feature extraction method

NOTE Additional methods may become available. We need an extensive mechanism to support them. N/A URI N/A

<Table 7> shows examples of projection parameters.

Name definition Unit Type Valid Range projection-type Projection type information for the following tasks which perform any video processing with the output of this stitching function, for instance OMAF Packager.
Takes one of the following values according to the projection format list:
- 'Equirectangular'
- 'Cubemap'

Default is 'Equirectangular'
Omnidirectional projection formats
Id: Omnidirectional projection
0 : Equirectangular
1: Cubemap
2: Other projection format N/A String N/A

<Table 8> shows an embodiment of the JSOM schema in which various function parameters are defined in Javascipt language as sub-parameters for configuration information of tasks required for the image stitching workflow included in the configuration descriptor. This JSOM schema may be delivered to the aforementioned multimedia processing entity and used for creating 360-degree image stitching.

{
"parameters": [
{
"name": "stitching-type",
"id": 1,
"datatype": "string",
"values": [
{
"id": 11,
"restrictions": [
"monoscopic"
]
},
{
"id": 12,
"restrictions": [
"stereoscopic"
]
}
]
},
{
"name": "camera-parameters",
"id": 2,
"datatype": "array",
"schema": {
"type": "object",
"properties": {
"camera-intrinsic": {
"type": "object",
"$ref": "#/definitions/camera-intrinsic"
},
"camera-extrinsic": {
"type": "array",
"items": {
"type": "number",
"minItems": 16,
"maxItems": 16
}
},
"camera-shutter-type": {
"type": "string",
"desciption": "The camera shutter type. One of 'rolling', and 'global'. Default is 'rolling'"
},
"camera-sync-skew": {
"type": "number"
},
"capturing-environment": {
"type": "string"
},
"shooting-type": {
"type": "string"
}
}
}
},
{
"name": "v-seam-positions",
"id": 3,
"datatype": "array",
"schema": {
"type": "number"
}
},
{
"name": "h-seam-positions",
"id": 4,
"datatype": "array",
"schema": {
"type": "number"
}
},
{
"name": "seam-mask",
"id": 5,
"datatype": "string"
},
{
"name": "stitching-method",
"id": 6,
"datatype": "string"
},
{
"name": "seam-extent-freedom",
"id": 7,
"datatype": "array",
"schema": {
"type": "number"
}
},
{
"name": "convergence-distance",
"id": 8,
"datatype": "string"
},
{
"name": "camera-weighting",
"id": 9,
"datatype": "array",
"schema": {
"type": "integer"
}
},
{
"name": "output-coverage",
"id": 10,
"datatype": "array",
"schema": {
"type": "number"
}
},
{
"name": "feature-parameter",
"id": 11,
"datatype": "object",
"schema": {
"$ref": "#/definitions/feature-parameter"
}
},
{
"name": "projection-parameters",
"id": 12,
"datatype": "array",
"schema": {
"$ref": "#/definitions/projection-parameter"
}
}
],
"definitions": {
"camera-intrinsic": {
"type": "object",
"properties": {
"camera-type": {
"type": "string",
"description": "Camera type. One of 'pinhole', 'fisheye', 'omnidirectional'. Default is 'pinhole'"
},
"focal-length": {
"type": "array",
"items": {
"type": "number",
"minItems": 2,
"maxItems": 2
}
},
"principal-point": {
"type": "array",
"items": {
"type": "number",
"minItems": 2,
"maxItems":2
}
},
"distortion": {
"type": "array",
"items": {
"type": "number",
"minItems": 4
}
}
}
},
"feature-parameter": {
"type": "object",
"properties": {
"feature-extraction-method": {
"type": "string",
"format": "uri",
"patternProperties": {
"^urn:": {"type": "string"}
},
"additionalProperties": false
}
}
},
"projection-parameter": {
"type": "string",
"properties": {
"projection-type": {
"type": "string",
"description": "Projection type. One of Equirectangular, Cubemap, or other projection type. Default is Equirectangular."
}
}
}
}
}

4 is a diagram showing the configuration of a computer system according to an embodiment. The 360-degree image stitching apparatus according to the embodiment may be implemented in the computer system 1000 such as a computer-readable recording medium.

Computer system 1000 may include one or more processors 1010 , memory 1030 , user interface input device 1040 , user interface output device 1050 , and storage 1060 that communicate with each other via bus 1020 . can In addition, the computer system 1000 may further include a network interface 1070 coupled to the network 1080 . The processor 1010 may be a central processing unit or a semiconductor device that executes programs or processing instructions stored in the memory 1030 or storage 1060 . The memory 1030 and the storage 1060 may be a storage medium including at least one of a volatile medium, a non-volatile medium, a removable medium, a non-removable medium, a communication medium, and an information delivery medium. For example, the memory 1030 may include a ROM 1031 or a RAM 1032 .

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains may practice the present invention in other specific forms without changing its technical spirit or essential features. You will understand that there is Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (19)

a memory in which at least one program is recorded; and
a processor for executing a program;
program,
A 360-degree image is generated by stitching features of a plurality of input images based on at least one parameter included in the 360-degree stitching function description template,
The 360 degree stitching function description template is,
a configuration parameter that is an array of functional parameters;
The setting parameters are
including stitching parameters, camera parameters, feature parameters and projection parameters,
The characteristic parameter is
A method for extracting features from each of a plurality of input images,
The projection parameters are
A 360-degree image stitching apparatus, including a projection type, which is a type of a projection surface on which a plurality of input images are projected. The method of claim 1, wherein the program
Detecting a key point from each of multiple input images using one of at least one feature extraction method, and extracting a descriptor for the key point,
The feature extraction method is
Oriented FAST and Rotated BRIEF (ORB) based on Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), nonlinear diffusion filter, accelerated KAZE algorithm, FAST key point detection and combination of BRIEF descriptors , BRISK (Binary Robust Invariant Scalable Keypoints) algorithm using scale space keypoints and rotation invariant keypoint descriptors, feature descriptors using BRIEF (Binary Independent Elementary Feature) descriptors for matching points, and Laplacian of Gaussian (LoG) of images. A 360-degree image stitching device comprising at least one of feature detection. The method of claim 2, wherein the feature extraction method comprises:
A 360-degree image stitching device, in which the URN represents a feature extraction method as a 'URI' type. The method of claim 2, wherein the feature parameter is
360-degree image stitching apparatus, further comprising a feature point number, feature point positions, and feature correspondence. The method of claim 3, wherein the program
decoding a plurality of input images;
extracting a feature point of each of the plurality of decoded input images;
extracting a camera parameter based on the feature point;
generating a 360-degree image by projecting the decoded multi-input images onto a projector based on the camera parameters;
determining boundary line information of a 360-degree image based on the camera parameter;
Post-processing a 360-degree image based on the boundary line information; and
Performing the step of encoding the post-processed 360-degree image,
In the step of extracting the feature points,
Extracting feature points based on the feature extraction method of feature parameters,
In the post-processing stage,
A 360-degree image stitching apparatus using a feature point number, feature point positions, and feature correspondence of a feature parameter. The method of claim 1, wherein the projection type is:
Includes a list of omnidirectional projection formats in the form of a table, including square (Equirectangular) and cubemap (Cubemap),
If Id is '0' in the table, the omnidirectional projection is square (Equirectangular),
If Id is '1', the omnidirectional projection is a Cubemap,
When Id is '2', the omnidirectional projection is another projection format, 360 degree image stitching device. 7. The method of claim 6, wherein the program comprises:
decoding a plurality of input images;
extracting a feature point of each of the plurality of decoded input images;
extracting a camera parameter based on the feature point;
generating a 360-degree image by projecting the decoded multi-input images onto a projector based on the camera parameters;
determining boundary line information of a 360-degree image based on the camera parameter;
Post-processing a 360-degree image based on the boundary line information; and
Performing the step of encoding the post-processed 360-degree image,
In the step of creating a 360-degree image,
A 360-degree image stitching apparatus that performs based on the projection format of the projection parameter. decoding a plurality of input images;
extracting a feature point of each of the plurality of decoded input images;
extracting a camera parameter based on the feature point;
generating a 360-degree image by projecting the decoded multi-input images onto a projector based on the camera parameters;
determining boundary line information of a 360-degree image based on the camera parameter;
Post-processing a 360-degree image based on the boundary line information; and
Including the step of encoding the post-processed 360 degree image,
The above steps are performed based on the 360 degree stitching function description template,
a configuration parameter that is an array of functional parameters;
The setting parameters are
including stitching parameters, camera parameters, feature parameters and projection parameters,
The characteristic parameter is
A method for extracting features from each of a plurality of input images,
The projection parameters are
A 360-degree image stitching method, including a projection type, which is a type of a projection plane on which a plurality of input images are projected. The method of claim 8, wherein in the step of extracting the feature points,
Detecting a key point from each of multiple input images using one of at least one feature extraction method of a feature parameter, and extracting a descriptor for the key point,
The feature extraction method is
Oriented FAST and Rotated BRIEF (ORB) based on Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), nonlinear diffusion filter, accelerated KAZE algorithm, FAST key point detection and combination of BRIEF descriptors , BRISK (Binary Robust Invariant Scalable Keypoints) algorithm using scale space keypoints and rotation invariant keypoint descriptors, feature descriptors using BRIEF (Binary Independent Elementary Feature) descriptors for matching points, and Laplacian of Gaussian (LoG) of images. A 360-degree image stitching method comprising at least one of feature detection. The method of claim 9, wherein the feature extraction method comprises:
A 360-degree image stitching method, in which the URN represents a feature extraction method as a 'URI' type. The method of claim 9, wherein the feature parameter is
A 360-degree image stitching method, further comprising a feature point number, feature point positions, and feature correspondence. 12. The method of claim 11, wherein in the post-processing step,
A 360-degree image stitching method using a feature point number, feature point positions, and feature correspondence of a feature parameter. 10. The method of claim 9, wherein the projection type is:
Includes a list of omnidirectional projection formats in the form of a table, including square (Equirectangular) and cubemap (Cubemap),
If Id is '0' in the table, the omnidirectional projection is square (Equirectangular),
If Id is '1', the omnidirectional projection is a Cubemap,
If Id is '2', forward projection is another projection format,
In the step of creating a 360-degree image,
A 360-degree image stitching method performed based on a projection format of a projection parameter. obtaining a 360-degree image parameter required for a request for 360-degree image stitching and creation of a 360-degree image stitching workflow;
obtaining a list of functions applicable to a 360-degree image stitching workflow;
generating a 360-degree image stitching workflow based on the functions selected from the function list;
determining the number of multimedia processing entities required to process tasks constituting the 360-degree image stitching workflow, and generating a plurality of multimedia processing entities according to the determined number of multimedia processing entities; and
assigning tasks constituting a 360-degree image stitching workflow to a plurality of multimedia processing entities;
Operations constituting the 360-degree image stitching workflow are performed based on at least one parameter included in the 360-degree stitching function description template,
The 360 degree stitching function description template is,
a configuration parameter that is an array of functional parameters;
The setting parameters are
including stitching parameters, camera parameters, feature parameters and projection parameters,
The characteristic parameter is
A method for extracting features from each of a plurality of input images,
The projection parameters are
A method of generating a 360-degree image stitching workflow, including a projection type, which is a type of a projection plane on which a plurality of input images are projected. The method of claim 14, wherein the tasks constituting the 360-degree image stitching workflow are:
decoding a plurality of input images;
extracting a feature point of each of the plurality of decoded input images;
extracting a camera parameter based on the feature point;
generating a 360-degree image by projecting the decoded multi-input images onto a projector based on the camera parameters;
determining boundary line information of a 360-degree image based on the camera parameter;
Post-processing a 360-degree image based on the boundary line information; and
Including the step of encoding the post-processed 360 degree image,
The above steps are performed based on the 360 degree stitching function description template,
A method of creating a 360 degree image stitching workflow, comprising a setup parameter that is an array of functional parameters. The method of claim 15, wherein in the step of extracting the feature point,
Detecting a key point from each of multiple input images using one of at least one feature extraction method of a feature parameter, and extracting a descriptor for the key point,
The feature extraction method is
Oriented FAST and Rotated BRIEF (ORB) based on Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), nonlinear diffusion filter, accelerated KAZE algorithm, FAST key point detection and combination of BRIEF descriptors , BRISK (Binary Robust Invariant Scalable Keypoints) algorithm using scale space keypoints and rotation invariant keypoint descriptors, feature descriptors using BRIEF (Binary Independent Elementary Feature) descriptors for matching points, and Laplacian of Gaussian (LoG) of images. A method of generating a 360-degree image stitching workflow, comprising at least one of feature detection. The method of claim 16, wherein the feature extraction method comprises:
'URI' type, where the URN represents the feature extraction method, a 360-degree image stitching workflow creation method. The method of claim 16, wherein the feature parameter is:
It further includes a feature point number, feature point positions, and feature correspondence,
In the post-processing stage,
A 360-degree image stitching workflow generating method using feature point numbers, feature point positions, and feature correspondence of feature parameters. The method of claim 15, wherein the projection type is:
Includes a list of omnidirectional projection formats in the form of a table, including square (Equirectangular) and cubemap (Cubemap),
If Id is '0' in the table, the omnidirectional projection is square (Equirectangular),
If Id is '1', the omnidirectional projection is a Cubemap,
If Id is '2', forward projection is another projection format,
In the step of creating a 360-degree image,
A method of creating a 360-degree image stitching workflow, performed based on a projection format of a projection parameter.

Images