KR20200111639A - Method and apparatus for providing 360 stitching workflow and parameter - Google Patents

Abstract

Provided is a method for providing an image stitching workflow. The method comprises the steps of: making a request for image stitching and obtaining a 360 degree VR image parameter required to generate an image stitching workflow; obtaining a list of functions applicable to the image stitching workflow; generating the image stitching workflow based on the functions selected from the function list; determining the number of media processing subjects required to process tasks constituting the image stitching workflow, and generating a plurality of media processing subjects according to the determined number of media processing subjects; and allocating the tasks constituting the image stitching workflow to the plurality of media processing subjects. The method can provide a 3 degree of freedom experience to a user.

Description

Video stitching workflow and parameter provision method and device {METHOD AND APPARATUS FOR PROVIDING 360 STITCHING WORKFLOW AND PARAMETER}

The present disclosure relates to a high-definition 360-degree VR image processing apparatus and method. More specifically, the present disclosure is an apparatus for distributing each task of the workflow to a plurality of media processing subjects so that a video stitching workflow for a high-definition 360 degree VR broadcasting service can be performed by a plurality of media processing subjects of a cloud platform. And a method. In addition, the present disclosure provides parameters related to a video stitching workflow for a high-definition 360-degree VR broadcasting service.

The advent of digital broadcasting has changed the traditional broadcasting method, which used to unilaterally watch signals transmitted by broadcasting stations, into a form in which users can selectively watch only the desired content at the desired time. In addition, the development of broadband transmission technology has made it possible to provide a realistic broadcasting service that can provide high-definition realistic media (e.g., UHDTV, 3DTV, etc.) to viewers over 4K while overcoming bandwidth limitations.

360-degree virtual reality (VR) media is a media that allows viewers to select and view a desired view by providing an omnidirectional image. Recently, efforts to provide 360-degree VR media through broadcasting network are accelerating. . In the case of 360-degree VR media, which is currently generally applied, full 4K or FHD-level VR media can be provided at the same time. The advantage of providing viewers with a desired view or region of interest (RoI) without delay. However, there is a problem that the image quality of the reproduced view is deteriorated according to the actual viewer's movement. In addition, as another form, a 360-degree VR service that streams for a corresponding area based on the viewer's motion or the information of the view selected by the viewer can provide a relatively high-definition view, but in response to the user's movement, There is a problem that the delay time (Motion to Photon, MTP) for playing the video becomes longer.

As a 360-degree VR media-related technology, a technology that provides signaling for recognizing a panoramic video broadcasting service and indicating related video characteristics, first transmits a thumbnail image, and transmits data of a specific area requested based on the thumbnail in the receiver. There is a technology or a technology that classifies a panoramic video into tiles of a certain area and transmits only data of a tile for an area selected by the user. These existing technologies propose signaling, tiling transmission, and synchronization methods to provide panoramic broadcasting services, but provide 360-degree VR broadcasting services based on user movement or user selection in a broadcasting network environment for high-definition 360-degree VR images. There is a problem that it is difficult to provide.

The present disclosure relates to a high-definition 360-degree VR image processing apparatus and method. In the present disclosure, various parameters are provided for creating the workflow of image stitching. In addition, since many computing resources are required for the image stitching process, a method and apparatus for allocating a task required for image stitching to a plurality of media processing subjects are proposed.

In the present disclosure, obtaining a 360-degree VR image parameter required for a request for image stitching and generation of an image stitching workflow, obtaining a list of functions applicable to the image stitching workflow, the function list Generating the image stitching workflow based on the functions selected from, determining the number of media processing subjects required to process the tasks constituting the image stitching workflow, and determining the number of media processing subjects according to the determined number of media processing subjects. And generating a plurality of media processing subjects, and allocating tasks constituting the image stitching workflow to the plurality of media processing subjects. A method for generating an image stitching workflow is provided.

According to an embodiment, the method for generating the image stitching workflow includes: determining setting information for a current job constituting the image stitching workflow, and transmitting setting information for the current job to a job manager, and the When the successful setting of the current task is confirmed by the task manager, acquiring access information for the next task performed immediately after the current task from the task manager.

According to an embodiment, the method of generating the video stitching workflow includes setting information on a task related to the changed content-dependent parameter when there is a change in a content-dependent parameter that is changed according to a content among the 360 degree VR video parameters. It may include the step of updating.

According to an embodiment, when a request for image stitching from a media source is stopped, acquiring the 360 degree VR image parameter or allocating tasks constituting the image stitching workflow to the plurality of media processing subjects The execution of the step may be interrupted.

According to an embodiment, the operations constituting the image stitching workflow include: decoding an encoded raw image, extracting a feature point of the decoded raw image, and extracting a camera parameter based on the feature point. , Based on the camera parameter, generating a 360-degree VR image by projecting the decoded unprocessed image onto a projecting object, determining boundary information of the 360-degree VR image based on the camera parameter, the It may include post-processing the 360-degree VR image based on boundary line information, and encoding the post-processed 360-degree VR image.

According to an embodiment, the 360-degree VR image parameter is a general descriptor parameter, an input descriptor parameter, an output descriptor parameter, a processing descriptor parameter, a requirement descriptor parameter, a client support descriptor parameter, a failover descriptor parameter, a monitoring descriptor. It may include at least one of a parameter and a report descriptor parameter.

According to an embodiment, the 360 degree VR image parameter includes a setting parameter, and the setting parameter may include at least one of a media parameter, a feature parameter, a camera parameter, a projection parameter, a stitching parameter, and a cloud parameter. .

According to an embodiment, the media parameter includes at least one of a codec parameter, a chroma parameter, an fps parameter, a gop parameter, and a resolution parameter, and the feature parameter is a feature extraction method parameter, a feature point number parameter, a feature point location parameter, and an optional feature. Including at least one of the corresponding point parameter, the camera parameter includes at least one of a focal length parameter, a main point parameter, an asymmetry coefficient parameter, a movement parameter, and a rotation parameter, the projection parameter includes a projection type parameter, and the stitching The parameter may include at least one of a boundary point position parameter and a boundary point mask parameter, and the cloud parameter may include at least one of a number of threads parameter and a number of GPUs parameter.

According to an embodiment, the cloud parameter indicates a computing resource required to perform the image stitching workflow, determines the number of media processing subjects required to process tasks constituting the image stitching workflow, and the determined In the step of generating a plurality of media processing subjects according to the number of media processing subjects, the number of media processing subjects may be determined according to the cloud parameter.

In the present disclosure, in an image stitching workflow generating apparatus for generating an image stitching workflow, including one or more processors and one or more memory devices, the one or more processors generate an image stitching workflow stored in the one or more memory devices. Execute one or more commands of a program for, and the one or more memory devices store a program for generating the image stitching workflow, and the one or more commands are required to request for image stitching and to generate the image stitching workflow. Obtaining a VR image parameter, obtaining a list of functions applicable to the image stitching workflow, generating the image stitching workflow based on functions selected from the function list, the Determining the number of media processing subjects necessary to process the tasks constituting the image stitching workflow, generating a plurality of media processing subjects according to the determined number of media processing subjects, configuring the image stitching workflow An apparatus for generating an image stitching workflow is provided, comprising the step of allocating jobs to the plurality of media processing subjects.

According to an embodiment, the one or more commands determine setting information on a current job constituting the image stitching workflow, and transmit setting information on the current job to a job manager, and to the job manager Thus, when successful setting of the current task is confirmed, acquiring access information for a next task performed immediately after the current task from the task manager.

According to an embodiment, the one or more commands update setting information on a task related to the changed content-dependent parameter when there is a change in a content-dependent parameter that is changed according to a content among the 360-degree VR video parameters. It may include steps.

According to an embodiment, when a request for image stitching from a media source is stopped, execution of the one or more commands may be stopped.

In the present disclosure, in a computer-readable recording medium including a bitstream including a 360-degree VR image parameter required for image stitching of a 360-degree VR image, the 360-degree VR image parameter is a general descriptor parameter, an input descriptor parameter, A computer-readable recording medium comprising at least one of an output descriptor parameter, a processing descriptor parameter, a requirement descriptor parameter, a client support descriptor parameter, a failover descriptor parameter, a monitoring descriptor parameter, and a reporting descriptor parameter. Is provided.

By converting multiple images synchronized by the camera into one stitched 360-degree VR image, a 3DoF (degree of freedom) experience can be provided to the user. However, when there is a large amount of data in 3DoF 360 content, it is difficult to process 3DoF 360 content in a standalone server. Therefore, the above problem can be solved by network-based image stitching according to a plurality of processing subjects.

1 is a diagram illustrating a processing method of a 360 degree VR image using a camera image and a camera depth image according to an embodiment of the present disclosure.
2 is a diagram illustrating a media system for image stitching according to a plurality of processing subjects according to an embodiment of the present disclosure.
3 is a detailed description of a workflow for performing an image stitching operation according to an embodiment of the present disclosure.
4 shows a method of determining a Preserved Region according to a multi-path type.
5 is a flowchart illustrating a method of generating an image stitching workflow according to an embodiment of the present disclosure.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals in the drawings refer to the same or similar functions over several aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. For a detailed description of exemplary embodiments described below, reference is made to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from each other but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of exemplary embodiments, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed by the claims.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The term and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component of the present invention is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components exist in the middle. It should be understood that it may be possible. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

Components shown in the embodiments of the present invention are independently shown to represent different characteristic functions, and does not mean that each component is formed of separate hardware or a single software component. That is, each constituent part is listed and included as a constituent part for convenience of explanation, and at least two constituent parts of each constituent part are combined to form one constituent part, or one constituent part may be divided into a plurality of constituent parts to perform functions Integrated embodiments and separate embodiments of the components are also included in the scope of the present invention unless departing from the essence of the present invention.

The terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present invention, terms such as "comprises" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance. That is, in the present invention, the description of “including” a specific configuration does not exclude configurations other than the corresponding configuration, and means that additional configurations may be included in the scope of the implementation of the present invention or the technical idea of the present invention.

Some of the components of the present invention are not essential components that perform essential functions in the present invention, but may be optional components only for improving performance. The present invention can be implemented by including only the components essential to implement the essence of the present invention excluding components used for performance improvement, and a structure including only essential components excluding optional components used for performance improvement Also included in the scope of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing the embodiments of the present specification, if it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present specification, the detailed description thereof will be omitted, and the same reference numerals are used for the same elements in the drawings. Used and redundant descriptions for the same components are omitted.

In the present disclosure, various embodiments of a stitching method for generating a 360 degree VR image will be described. In the present disclosure, various embodiments of a method for implementing the stitching method in a cloud environment will be described.

1 is a diagram illustrating a method of processing a 360 degree VR image according to an embodiment of the present disclosure.

The media processing source 100 includes setting information 102, a media source 104, and metadata 106. The setting information 102 and metadata 106 represent information related to a plurality of input video streams. The media source 104 includes a raw image that becomes a material for a 360 degree VR image. According to an embodiment, the media source 104 may include a texture image and a depth image. Alternatively, according to an embodiment, the media source 104 may include only a texture image. The setting information 102, the media source 104, and the metadata 106 of the media processing source 100 are transmitted to the media processing subject 110.

The media processing subject 110 stitches the image of the media source 104 according to the setting information 102 and the metadata 106. 1 shows detailed steps necessary for image stitching. A 360-degree VR image is generated according to the image stitching according to FIG. 1. In addition, the 360-degree VR image may be transmitted to the application 120 that processes the 360-degree VR image.

The media processing subject 110 is set by the workflow manager 202 of FIG. 2 to be described below. If it is determined that the image stitching workflow cannot be processed by a single processing subject, two or more media processing subjects 110 may be set. If it is determined that the image stitching workflow can be processed by a single processing subject, only one media processing subject 110 may be set. The media processing subject 110 may be created in the cloud platform 210 of FIG. 2 by the workflow manager 202 of FIG. 2.

If two or more media processing subjects 110 are set, the jobs according to steps 130 to 142 of FIG. 1 may be allocated to a plurality of media processing subjects 110. Accordingly, an image stitching operation requiring a large amount of computing resources can be quickly processed by the plurality of media processing subjects 110.

When an image stitching task is processed by a plurality of media processing subjects 110, for the assignment of the tasks, for example, image decoding, feature point extraction, camera parameter extraction, image projection, boundary information extraction related to image stitching , Blending, post-processing, and video encoding, setting information for detailed operations, input and output descriptors, and metadata are required. The information is used by a plurality of processing entities to process interfaces between various tasks. To this end, detailed information such as media, functions, cameras, projections, stitching and cloud parameters to be used for each operation of the image stitching process is applied to each detailed step of image stitching. Hereinafter, detailed steps of image stitching (steps 130 to 142) will be described in detail.

In step 130, the coded image is decoded. Specifically, in step 130, the encoded raw image of the media source 104 is decoded according to an operation of several threads or GPUs. In addition, not only the media source 104 but also a media data feed encoded from a cloud parameter may be used for decoding in step 130. The raw image represents an original image to which post-processing is not applied.

In step 132, feature points are extracted from the raw image decoded in step 130. The feature point refers to a point that becomes a reference point for matching an image and an image when tracking or recognizing a specific object in a plurality of images. By analyzing the values of the pixels distributed based on the feature points, similar portions between different images are detected. Accordingly, by comparing feature points between adjacent raw images in step 132, a corresponding point between adjacent raw images may be determined. In addition, a stitching process may be performed between adjacent raw images according to the corresponding point.

In step 134, according to the set of feature points and corresponding points, external and internal camera parameters are calculated. The external camera parameters include a camera's photographing angle and a photographing position. The internal camera parameters include a focal length of the camera, a main point, and an asymmetry coefficient.

According to an embodiment, by analyzing the feature point and the corresponding point, a difference between a photographing angle and a photographing position between two cameras used for photographing two adjacent images may be calculated. Accordingly, the external parameters of the camera may be determined according to the feature points and corresponding points.

According to an embodiment, by analyzing the feature point and the corresponding point, a difference between a focal length, a main point, and an asymmetry coefficient of the two cameras may be calculated. Accordingly, the internal parameters of the camera may be determined according to the feature points and corresponding points.

In addition, in step 134, an image distortion correction process may be optionally performed based on camera parameters. According to an embodiment, the image distortion correction process may be a process of correcting image distortion according to lens characteristics.

In step 136, the raw images decoded in step 130 are projected on a spherical surface, a cube, a polyhedron, and an ERP (EquiRectangular Projection) based on the camera parameter determined in step 134. Specifically, images are geometrically aligned based on the camera parameters and the image projection plane. And the images arranged geometrically are projected onto the image projection surface. As a result of the image projection, a 360 degree VR image is generated.

In step 138, stitching parameters for the boundary lines of the adjacent images are extracted from the 360-degree VR image generated in step 136. The stitching parameter may include a boundary line position parameter with respect to the position of the boundary line or a boundary line mask parameter indicating an area corresponding to the boundary line in the image. The stitching parameter for the boundary line may be determined based on the camera parameter determined in step 134.

In step 138, when an object that did not exist at a previous viewpoint suddenly appears in the overlapping region of the adjacent image, the stitching parameter may be updated. Accordingly, according to the change of the overlapping area, the boundary line position parameter or the boundary line mask parameter may be changed.

In step 140, blending, color correction, and noise filtering processes may be performed on the overlapped region of the 360-degree VR image generated in step 136. The blending, color correction, and noise filtering process may be performed based on the boundary information extracted in step 138.

The blending process is a process of mixing two partial images corresponding to an overlapping area of a 360 degree VR image so that a 360 degree VR image can be visually recognized as a single continuous image. The color correction process refers to a process of adjusting color parameters such as hue, brightness and saturation so that images constituting a 360 degree VR image can be recognized as one image. The noise filtering process refers to a process of removing noise from an image. According to the process of step 140, a 360 degree VR image generated from a plurality of images may be visually recognized as one image.

In step 142, the 360 degree VR image processed in step 140 is encoded. The encoding in step 142 may be performed according to a cloud parameter. In addition, the encoded 360-degree VR image may be output by a cloud platform.

Through the steps 130 to 142, a video stream composed of an encoded 360 degree VR image is generated from a video stream composed of an encoded 2D image. Depending on embodiments, the order of steps 130 to 142 may be partially changed, and certain steps may be omitted. In addition, an image processing technique conventional in the image processing field may be applied between steps 130 and 142. Commands corresponding to steps 130 to 142 may be executed in the cloud platform 210 of FIG. 2. Hereinafter, in Table 1, descriptions, inputs, and outputs for each operation of steps 130 to 142 are described.

work Explanation input Print Decryption Includes video type conversion, using multiple threads or GPUs, to convert a feed of an encoded video stream from a media source into a raw video stream. Media source and feed of encoded media data from media and cloud parameters Decoded media data such as raw video Feature point extraction The process of extracting feature points and matching corresponding points between adjacent raw images is included. Raw image to be aligned (stitched) and feature extraction method Extracted feature points and corresponding set Camera parameter extraction Includes calculation of external and internal camera parameters using feature points and corresponding sets. Optionally, an image distortion correction process using camera parameters is included. Extracted feature points and corresponding set External and internal camera parameters and lens distortion correction video Projection Based on camera parameters, image projection according to geometric alignment is included in spherical, cube, polyhedron and ERP (EquiRectangular Projection). External and internal camera parameters and lens distortion correction video 360 degree VR video Extract border information Includes the process of extracting boundary information and updating necessary when objects suddenly appear in overlapping areas. The first extracted stitching parameters for 360 degree VR image and boundary line position and boundary line mask Extracted or updated stitching parameters including border position or border mask Blending and post processing Includes blending, color correction, and noise filtering of overlapping areas of 360-degree VR images. 360 degree VR video and stitching parameters Blended and post-processed 360 degree VR video encoding Includes video encoding of blended and post-processed 360 degree VR images using multiple threads or GPUs. Blended and post-processed 360 degree VR images and cloud parameters Coded 360 projection image (may be output from cloud platform)

The media service provider may provide an image stitching service to a user by using the workflow according to steps 130 to 142 above. The media service provider's working directory may contain details for each job in Table 1. In addition, the workflow manager 202 of FIG. 2 may prepare the workflow described in FIG. 1 by selecting each task in Table 1. Hereinafter, the media system 200 for distributing the image stitching process according to steps 130 to 142 described in FIG. 1 to a plurality of media processing subjects 110 will be described.

2 illustrates a media system 200 for image stitching according to a plurality of processing subjects according to an embodiment of the present disclosure.

The media system 200 includes a workflow manager 202, a media source 204, a function registry 206, a media task manager 208, a cloud platform 210, and the like. The media system 200 of FIG. 2 shows only some components necessary to describe the present embodiment, and components included in the media system 200 of the present disclosure are not limited to the above-described examples. For example, two or more components may be implemented within a single component, or an operation executed by one component may be divided and implemented to be executed by two or more components. In addition, some components may be omitted or additional components may be added.

The workflow manager 202, the media source 204, the function registry 206, and the media task manager 208 may each be operated by separate processors. Alternatively, all or some of the workflow manager 202, media source 204, function registry 206, and media task manager 208 may be operated by a single processor. In addition, the workflow manager 202, the media source 204, the function registry 206, and the media task manager 208 may each include a memory device for storing information for image stitching.

The workflow manager 202 may create a workflow for image stitching. In addition, the workflow manager 202 may create a plurality of media processing subjects for performing a plurality of tasks included in the workflow on the cloud platform 210.

The cloud platform 210 includes a plurality of processors. In addition, tasks necessary for image stitching may be performed by a plurality of media processing subjects set in the cloud platform 210. Hereinafter, a method of setting a workflow for image stitching by the workflow manager 202 will be described in detail with reference to FIG. 3.

3 is a detailed description of a workflow for performing an image stitching operation according to an embodiment of the present disclosure.

In step S302, the media source 204 generates a workflow for image stitching together with a request for image stitching using a CreateWorkflow API (Application Programming Interface). The CreateWorkflow API is a dedicated API for creating a workflow. Then, the media source 204 generates a workflow description document describing the generated workflow, and transmits the workflow description document to the workflow manager 202. The workflow manager 202 creates a workflow according to the transmitted workflow description document.

Workflow description documents include general descriptor parameters, input descriptor parameters, output descriptor parameters, processing descriptor parameters, requirements descriptor parameters, client support descriptor parameters, failover descriptor parameters, monitoring descriptor parameters, and reporting descriptor parameters. can do.

General descriptor parameters include the workflow ID, name, descriptor, brand, priority, etc. The input descriptor parameter includes media parameters such as codec type, frame rate, resolution, number of input views, metadata stream, and metadata collection format of an image to be input. The output descriptor parameter includes media parameters such as codec type, frame rate, posting format, metadata stream, and metadata collection format of the stitched image to be output by the workflow. The processing descriptor parameter includes a parameter for a processing type of each job of image stitching and connection information between each job. The requirements descriptor parameter contains parameters for requirements for the entire workflow, such as delay requirements. The client support descriptor parameter contains parameters for client support for the workflow. The alternate operation descriptor parameter contains parameters for the system alternate operation of the workflow. The monitoring descriptor parameter contains parameters for the monitoring type of the workflow. The report descriptor parameter includes parameters indicating the type, period, and start time of reporting the results of the monitoring of the workflow.

Hereinafter, an embodiment of each descriptor parameter of the workflow description document and sub-parameters included in the descriptor parameter is described in Table 2.

number parameter Further explanation One General Descriptor Details of the required workflow are provided. The following list of parameters is described: ID: m-360VR stitching
● Name: m-360VR stitching
● Descriptor: 360VR stitching workflow across multiple processing subjects
● Brand: urn:mpeg:nbmp:2019:functions: m-360VRStitcher
● Priority:2 2 InputDescriptor Information is provided describing the input that the media source intends to use in the workflow. Media parameters:
○ Total number of media descriptors N. Each media descriptor contains the following media parameters.
■ Media Stream Id: MediaStreamId of the media stream
■ Codec Type: HEVC
■ Media Type: Video
■ Codec clock rate: 60fps
■ Protocol: MMT
■ Origin: URL of media stream
● Metadata parameters:
○ Metadata Stream Id: MetadataStreamId of the metadata description for the view stream of interest
○ Metadata Dictionary: MORE
○ Protocol: <ingest-protocol>
○ Meta data collection format: JSON
○ Origin: URL of the metadata stream 3 OutputDescriptor Information is provided describing the output the media source intends to use for the workflow. Media parameters:
○ Stream information
■ Media stream Id: MediaStreamId of output media stream
■ Stream tag: stream-tag
■ Bandwidth: Bandwidth
■ Codec type: AVC, HEVC, VVC
■ Media Type: Video
■ Clock rate: 60fps
■ Protocol: <protocol>
■ Destination: <destination-location>
■ Publication format: OMAF
● Metadata parameters:
○ Metadata Stream Id: n/a
○ Metadata Dictionary: MPEG-I Metadata
○ Protocol: n/a
○ Meta data collection format: JSON
○ Origination: URL of the Metadata stream 4 ProcessingDescriptor Details on the type of processing you want to set up in the media source are provided.● Keyword Search for Tasks: 360VR stitching, multiple processing entities
● TaskConnectionMap: Connection information between different tasks in a static workflow
<OutputTask.OutputPort,InputTask.InputPort>

· Decoding[0..n].outputFrame, FeaturepointExtraction[0..n].inputFrame
· FeaturepointExtraction[0..n].outputFeatureArray, CameraParameterExtraction.inputFeatureArray
· CameraParameterExtraction.outputCameraParameters, Projection.inputCameraParameters
· Decoding[0..n].outputFrame, Projection[0..n].inputFrame
· CameraParameterExtraction.outputCameraParameters, SeamInformationExtraction.inputCameraParameters
· Projection[0..n].outputFrame, SeamInformationExtraction.inputFrame
· SeamInformationExtraction.outputSeamMasks, BlendingandPostProcessing.inputSeamMasks
· Projection[0..n].outputFrame, BlendingandPostProcessing.inputFrame
● BlendingandPostProcessing.outputFrame, Encoding.inputFrame 5 RequirementDescriptor Requirements information for the entire workflow is specified. The above information includes the following: QoS requirements: Detailed QoS requirements for end-to-end workflow
○ Delay requirements: real time
● Processing requirements:
○ Hardware requirements:
■ GPU: 4
■ CPU core: 8
○ Deployment requirements:
■ Location: <network-edge>
● Security requirements: not required 6 ClientAssistance Client support information is provided for the workflow. Device-capabilities: n/a
● User-preferences: n/a 7 FailoverDescriptor Information on failover cases of the above workflow is provided. FailoverMode:
○ ExecuteBackupDeployment
● FailoverDelay: 0
● State persistence descriptor:
● BackupDeployment: 8 MonitoringDescriptor Information about the type of monitoring of the above workflow is provided. The above information includes the following:● Event: CRITICAL event
● Variable: No need
● System: No need 9 ReportingDescriptor ● Report-type: <Consumption, QoS>● Reporting-Interval: 120
● Reporting-start-time: <start-time>
● URL: <reporting-server-URL>

In step 304, the workflow manager 202 transmits a query or set of queries to the function registry 206 in order to find a function to be placed in the workflow for image stitching. The query or set of queries describes the function of the workflow required by the workflow description document of step 302. In addition, the function registry 206 stores a list of functions supported by the media system 200.

In step 306, for each query, the function registry 206 provides the workflow manager 202 with a list of functions that can be used for image stitching, and their description and configuration information. The function registry 206 compares a list of functions supported by the media system 200 with a description of the functions of the workflow described in the query or query set. In addition, the function registry 206 may provide a list of functions applicable to the workflow among functions supported by the media system 200 to the workflow manager 202.

In step 308, the workflow manager 202 selects functions required for the workflow from the list of functions provided from the function registry 206 in step 306. In addition, the workflow manager 202 may access the cloud platform 210 and create one or more media processing subjects in the cloud platform 210 according to the requirements of the selected functions.

In step 310, the cloud platform 210 confirms the creation of one or more media processing subjects. In addition, the cloud platform 210 may check the generation of network access information for one or more media processing subjects. The cloud platform 210 communicates to the workflow manager 202 that one or more media processing entities have been created.

In step 312, The workflow manager 202 generates setting information for each task. Then, the workflow manager 202 transmits the setting information for each job to the job manager 208. In order to transmit the setting information to the task manager 208, a task API may be used. Operation API is a dedicated API for creating configuration information.

In step 314, the task manager 208 checks whether successful settings for each task have been made. If successful setting of the current task is made, the task manager 208 generates access information so that the workflow manager 202 can access the next task. And the task manager 208 generates access information to the workflow manager 202. Steps 312 and 314 are performed for each task, so that it can be confirmed whether or not the entire workflow has been successfully set.

In step 316, the workflow manager 202 confirms the creation of the workflow, and informs the media source 204 that media processing for image stitching can be started. In addition, by processing the media source 104 provided from the media source 204 according to the workflow, a 360 degree VR image may be generated.

According to an embodiment, the workflow manager 202 may continuously monitor a content-dependent parameter related to output of each job. The content-dependent parameter means a parameter that changes according to the content. When it is necessary to change some content-dependent parameters of each job for the next segment of the video, the workflow manager 202 may update the setting information using the job API. Further, the workflow manager 202 may transmit the updated setting information to the corresponding job.

According to an embodiment, when there is no request for image stitching (or when the request is stopped), the media source 204 may stop image stitching by using the DeleteWorkflow API.

In FIGS. 2 and 3, it has been described that in steps 302 to 316, the process of creating a stitching workflow and a process of the stitching workflow is performed by interacting with a plurality of subjects. However, according to an embodiment, unlike FIGS. 2 and 3, the functions of the workflow manager 202, the media source 204, the function registry 206, and the media task manager 208 are all in the cloud platform 210. Can be processed. Therefore, according to an embodiment, all tasks necessary for creating a workflow for stitching may be performed in the cloud platform 210.

As described above, in order to set the function used in the workflow, create an appropriate number of media processing subjects for performing the work of the workflow, and determine the setting information of each work of the workflow, Many parameters are required for the tasks of the workflow.

Hereinafter, Table 3 shows an embodiment of a parameter list of an image stitching function reference template. In Table 3, the types and descriptions of parameters for the workflow are described in more detail.

explainer Parameter name type Explanation General ID String Provided from the feature store Name String "m-360VRStitcher" Description String "360 VR stitcher with multiple processing entities" Brand String "urn:mpeg:nbmp:2019:functions:m-360VRStitcher" InputPorts Map Input metadata may be optionally included along with at least one input media. OutputPorts Map Output metadata may be optionally included along with at least one output media. Input Media Parameters Object Image or video to be stitched (encoded). Input media capabilities of m-360VRstitcher provided
● Stream overview: CMAF, OMAF, ISOBMFF
● Media Stream Id: n/a
● Codec type: AVC, HEVC, VVC
● Media Type: Video, Image
● Codec clock rate: 24fps, 25fps, 30fps, 50fps, 60fps
● Protocol: n/a
● Origination: n/a Metadata parameters Object Input metadata capability of m-360VRstitcher is provided ● Stream overview: MORE, MPEG-V [Company W1]
● Metadata Stream Id: n/a
● Metadata dictionary: MORE, MPEG-V
● Protocol: n/a
● Meta data collection format: JSON, ISOBMFF, XML
● Origination: URL of the Metadata stream Output Media Parameters Object Output media capabilities of the m-360VRstitcher are provided ● Stream overview: OMAF
● Media Stream Id: n/a
● Codec type: AVC, HEVC, VVC
● Media Type: Video, Image
● Codec clock rate: 24fps, 25fps, 30fps, 50fps, 60fps
● Protocol: MPEG-DASH, MPEG-MMT, MPEG2-TS
● Origination: n/a Metadata parameters Object Output metadata capability of m-360VRstitcher is provided ● Metadata overview: MPEG-I Metadata, MORE
● Metadata Stream Id: n/a
● Metadata dictionary MPEG-I Metadata, MORE
● Protocol: n/a
● Meta data collection format: JSON, ISOBMFF, XML
● Origination: URL of the Metadata stream Processing Keywords Array Number of keywords· "m-360vrstitching"
· "M-360 stitching" TaskConnectionMap Connection information between different tasks in static workflow <OutputTask.OutputPort,InputTask.InputPort>
● Decoding[0..n].outputFrame, FeaturepointExtraction[0..n].inputFrame
● FeaturepointExtraction[0..n].outputFeatureArray, CameraParameterExtraction.inputFeatureArray
● CameraParameterExtraction.outputCameraParameters, Projection.inputCameraParameters
● Decoding[0..n].outputFrame, Projection[0..n].inputFrame
● CameraParameterExtraction.outputCameraParameters, SeamInformationExtraction.inputCameraParameters
● Projection[0..n].outputFrame, SeamInformationExtraction.inputFrame
● SeamInformationExtraction.outputSeamMasks, BlendingandPostProcessing.inputSeamMasks
● Projection[0..n].outputFrame, BlendingandPostProcessing.inputFrame
● BlendingandPostProcessing.outputFrame, Encoding.inputFrame Requirements QoS requirements Object ● All input video must be synchronized based on MORE Processing Requirements Object ● Processing requirements: ○ Hardware requirements:
■ GPU: 4
■ CPU core: 8
○ Deployment requirements:
■ Location: <network-edge> Security Requirements Object n/a Configuration Parameters Array Function parameters Media parameters (codec, chroma, fps, gop, resolution)
● Feature parameters (feature extraction method, number of feature points, feature point location, optional feature correspondence points)
● Camera parameters (internal and external parameters: camera focal length, main point, asymmetry coefficient, camera movement and rotation parameters)
● Projection parameters (projection type)
● Stitching Parameters (Border Point Position and Boundary Point Mask)
● Cloud parameters (number of threads, number of GPUs) ClientAssistance clientAssistanceFlag Boolean coefficient Device Capabilities String User Preferences String Monitoring Variable Array Assertion Assertions Object

The TaskConnectionMap parameter of the processing descriptor parameter of Table 3 includes connection information for each task of the workflow of image stitching described in steps 130 to 142. According to the description of TaskConnectionMap, the connection information is provided in the form of “<OutputTask.OutputPort,InputTask.InputPort>”. OutputTask represents an output task, and InputTask represents an input task. In addition, OutputPort represents the type of output information according to the output operation, and InputPort represents the type of input information for the input operation.

For example, ”Decoding[0..n].outputFrame, FeaturepointExtraction[0..n].inputFrame” indicates that the output operation is decoding, and an output frame is generated according to the end of the decoding operation. Further, the syntax indicates that the input operation is feature point extraction, and an input frame is input for the feature point extraction operation. In addition, according to the above syntax, it can be seen that output information of the decoding operation is input to the feature point extraction operation.

When implementing the image stitching workflow according to the connection information according to the TaskConnectionMap in Table 3, it can be seen that detailed steps of image stitching are implemented in the same manner as steps 130 to 142 of FIG. 1.

Table 3 defines configuration parameters. The setting parameters include sub-parameters for setting information on tasks necessary for the image stitching workflow. For example, the media parameter may be expressed as data in the form of an array including sub-parameters related to codec, chroma, frame per second (fps), group of pictures (gop), and resolution. The feature parameter may be expressed as data in the form of an array including sub-parameters related to the feature extraction method, the number of feature points, the feature point location, and the optional feature correspondence point. The camera parameter can be expressed as data in the form of an array including sub-parameters related to the focal length of the camera, the principal point, the skew coefficient, and the translation and rotation of the camera. have. The projection parameter may be expressed as data in the form of an array including sub-parameters related to the projection type. The stitching parameter may be expressed as data in the form of an array including sub-parameters related to a boundary point position and a boundary point mask. The cloud parameter may be expressed as data in the form of an array including sub-parameters related to the number of threads and the number of GPUs.

Hereinafter, embodiments of parameters required for each operation of image stitching are described in Tables 4 to 9.

Table 4 shows examples of media parameters.

Parameter name type Parameter description Codec String Video codec type. Ex) Video codec type such as h.264/avc, h.265/hevc. Ex) png, jpg, etc. Chroma String Chroma subsampling types yuv420, yuv422, yuv444 etc. Fps number Frame Frames per second, 30, 60, etc. Gop number GOP size, 15, 30, etc. Resolution String Resolution 3840x2160, 7680 x 4320, etc.

Table 5 shows an example of a feature parameter.

Parameter name type Parameter description Feature extraction method String Feature point detection method ex) SIFT, SURF, KAZE, AKAZE, ORB, BRISK, BRIEF, LoG, etc. Feature point number number Number of feature points Feature point positions Array The location of the feature point specified according to x and y coordinates Feature correspondence String Correspondence point of each feature point

Table 6 shows examples of camera parameters.

Parameter name type Parameter description Camera_shutter_type String "rolling" or "global" Camera_sync_skew number 0 if in synch, milliseconds for out of synch, -1 if not known Capturing_settings Object Scene type (indoor or outdoor), exposure of ambient lighting, etc. Camera_extrinsics Object Camera conversion parameters for aligning images in 3D space (camera movement and rotation parameters) Camera_intrinsics Object Aligning images in 3D space Camera internal parameters (focal length of camera, main point, asymmetry coefficient)

Table 7 shows examples of projection parameters.

Parameter name type Parameter description Projection_type String Projection type ex) equirectangular, square, circular cylinder type, etc.

Table 8 shows examples of stitching parameters.

Parameter name type Parameter description Seam_positions Array Interpolation area affecting the final stitching quality. The area structure can be represented as a series of pixel points (start point, intersection point, end point). Seam_mask Object Optionally, the interpolated region position can be represented as a mask image with only 1 or 0 values for a more sophisticated stitching process. You can also place a mask image by URL or URI. Preserved_regions Array When multiple media are processed as one of the following types, an important area is provided:-Input media stream id
-RectangleType{left, right, top, bottom}
-RegionMask{Mask_image}
-DepthMask{Mask_Grey_image}
-MultipathType(Array of 2D Points)

Stitching_method String A specific stitching algorithm that is specified as a partial or full stitching approach. Seam_extent_of_freedom number The degrees of freedom the boundary area can move, eg Horizontal direction

4 shows a method of determining a Preserved Region according to a multi-path type. Referring to FIG. 4, an important area may be determined according to five points corresponding to (x1, y1) to (x5, y5). Depending on the embodiment, the number of points for determining the important area may be set differently.

Table 9 shows examples of cloud parameters.

parameter type Parameter description Thread number number Number of threads used GPU number number Number of GPUs used

According to the parameters of the embodiments described in Tables 4 to 9, each operation of image stitching may be set.

5 is a flowchart illustrating a method of generating an image stitching workflow according to an embodiment of the present disclosure.

In step 502, a request for image stitching and a 360 degree VR image parameter necessary for generating an image stitching workflow are obtained.

In step 504, a list of functions that can be applied to the image stitching workflow is obtained.

In step 506, based on the functions selected from the function list, an image stitching workflow is generated.

In step 508, the number of media processing subjects required for processing the tasks constituting the image stitching workflow is determined, and a plurality of media processing subjects are generated according to the determined number of media processing subjects.

In step 510, tasks constituting the image stitching workflow are assigned to a plurality of media processing subjects.

The video stitching workflow generation method includes determining setting information on a current job constituting the video stitching workflow, transmitting setting information on the current job to a job manager, and setting the current job by the job manager. When it is confirmed, the step of obtaining access information for a next task performed immediately after the current task from the task manager may be further included.

The video stitching workflow generation method further includes updating setting information on a task related to the changed content-dependent parameter when there is a change in a content-dependent parameter that is changed according to a content among 360 degree VR video parameters. I can.

The method of generating the image stitching workflow may be set such that when a request for image stitching is stopped from a media source, execution of each of the steps of FIG.

The tasks constituting the image stitching workflow include: decoding the encoded raw image, extracting feature points of the decoded raw image, extracting camera parameters based on the feature points, and decoded based on camera parameters. Generating a 360-degree VR image by projecting the raw image onto the projectile, determining boundary line information of the 360-degree VR image based on camera parameters, and post-processing the 360-degree VR image based on the boundary line information. The step may include encoding the post-processed 360-degree VR image.

The 360-degree VR video parameters include: general descriptor parameters, input descriptor parameters, output descriptor parameters, processing descriptor parameters, requirements descriptor parameters, client support descriptor parameters, failover descriptor parameters, monitoring descriptor parameters, and reporting descriptor parameters. It may include at least one of.

The 360 degree VR image parameter includes a setting parameter, and the setting parameter may include at least one of a media parameter, a feature parameter, a camera parameter, a projection parameter, a stitching parameter, and a cloud parameter. The cloud parameter represents a computing resource required to perform the image stitching workflow, and the number of media processing subjects may be determined according to the cloud parameter.

In the above-described embodiments, the methods are described on the basis of a flow chart as a series of steps or units, but the present invention is not limited to the order of steps, and certain steps may occur in a different order or concurrently with those described above. have. In addition, those of ordinary skill in the art understand that the steps shown in the flowchart are not exclusive, other steps are included, or one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You can understand.

The above-described embodiments include examples of various aspects. Although not all possible combinations for representing the various aspects can be described, those of ordinary skill in the art will recognize that other combinations are possible. Accordingly, the present invention will be said to cover all other replacements, modifications and changes falling within the scope of the following claims.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic-optical media such as floptical disks. media), and a hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

In the above, the present invention has been described by specific matters such as specific elements and limited embodiments and drawings, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Anyone with ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all modifications that are equivalently or equivalent to the claims as well as the claims to be described later fall within the scope of the spirit of the present invention. I would say.

Claims (20)

Obtaining a 360-degree VR image parameter required to request for image stitching and to create an image stitching workflow;
Obtaining a list of functions applicable to the image stitching workflow;
Generating the image stitching workflow based on functions selected from the function list;
Determining the number of media processing subjects necessary to process tasks constituting the image stitching workflow, and generating a plurality of media processing subjects according to the determined number of media processing subjects; And
And allocating tasks constituting the image stitching workflow to the plurality of media processing subjects.
The method of claim 1,
The image stitching workflow generation method,
Determining setting information on a current job constituting the image stitching workflow, and transmitting setting information on the current job to a job manager; And
When the setting of the current task is confirmed by the task manager, the step of obtaining access information for a next task executed immediately after the current task from the task manager, further comprising: creating an image stitching workflow. Way.
The method of claim 2,
The image stitching workflow generation method,
If there is a change in a content-dependent parameter that is changed according to a content among the 360-degree VR video parameters, updating setting information on a job related to the changed content-dependent parameter. How to create a flow.
The method of claim 1,
When the request for the image stitching is stopped from the media source, performing the step of acquiring the 360-degree VR image parameter or the step of allocating tasks constituting the image stitching workflow to the plurality of media processing subjects are performed. A method of creating an image stitching workflow, characterized in that it is stopped.
The method of claim 1,
Tasks constituting the image stitching workflow,
Decoding the encoded raw image;
Extracting feature points of the decoded raw image;
Extracting a camera parameter based on the feature point;
Generating a 360 degree VR image by projecting the decoded raw image onto a projectile based on the camera parameter;
Determining boundary information of the 360 degree VR image based on the camera parameter;
Post-processing the 360 degree VR image based on the boundary line information; And
And encoding the post-processed 360-degree VR image.
The method of claim 1,
The 360-degree VR video parameters include: general descriptor parameters, input descriptor parameters, output descriptor parameters, processing descriptor parameters, requirements descriptor parameters, client support descriptor parameters, failover descriptor parameters, monitoring descriptor parameters, and reporting descriptor parameters. An image stitching workflow generation method comprising at least one of.
The method of claim 1,
The 360 degree VR video parameter includes a setting parameter,
The setting parameter includes at least one of a media parameter, a feature parameter, a camera parameter, a projection parameter, a stitching parameter, and a cloud parameter.
The method of claim 7,
The cloud parameter represents a computing resource required to perform the image stitching workflow,
Determining the number of media processing subjects required to process the tasks constituting the image stitching workflow, and generating a plurality of media processing subjects according to the determined number of media processing subjects,
The number of media processing subjects is determined according to the cloud parameter.
In the image stitching workflow generating apparatus for generating an image stitching workflow, comprising at least one processor and at least one memory device,
The one or more processors execute one or more instructions of a program for generating an image stitching workflow stored in the one or more memory devices,
The at least one memory device stores a program for generating the image stitching workflow,
The one or more commands,
Obtaining a 360 degree VR image parameter required for a request for image stitching and generation of an image stitching workflow;
Obtaining a list of functions applicable to the image stitching workflow;
Generating the image stitching workflow based on functions selected from the function list;
Determining the number of media processing subjects necessary to process the tasks constituting the image stitching workflow, and generating a plurality of media processing subjects according to the determined number of media processing subjects; And
And allocating tasks constituting the image stitching workflow to the plurality of media processing subjects.
The method of claim 9,
The one or more commands,
Determining setting information on a current job constituting the image stitching workflow, and transmitting setting information on the current job to a job manager; And
When the successful setting of the current task is confirmed by the task manager, obtaining access information for a next task executed immediately after the current task from the task manager is generated. Device.
The method of claim 10,
The one or more commands,
If there is a change in a content-dependent parameter that is changed according to a content among the 360-degree VR video parameters, updating setting information on a task related to the changed content-dependent parameter. Generating device.
The method of claim 9,
An apparatus for generating an image stitching workflow, wherein when a request for image stitching from a media source is stopped, execution of the one or more commands is stopped.
The method of claim 9,
Tasks constituting the image stitching workflow,
Decoding the encoded raw image;
Extracting feature points of the decoded raw image;
Extracting a camera parameter based on the feature point;
Generating a 360 degree VR image by projecting the decoded raw image onto a projectile based on the camera parameter;
Determining boundary information of the 360 degree VR image based on the camera parameter;
Post-processing the 360 degree VR image based on the boundary line information; And
And encoding the post-processed 360-degree VR image.
The method of claim 9,
The 360-degree VR video parameters include: general descriptor parameters, input descriptor parameters, output descriptor parameters, processing descriptor parameters, requirements descriptor parameters, client support descriptor parameters, failover descriptor parameters, monitoring descriptor parameters, and reporting descriptor parameters. An image stitching workflow generating apparatus comprising at least one of.
The method of claim 9,
The 360 degree VR video parameter includes a setting parameter,
The setting parameter includes at least one of a media parameter, a feature parameter, a camera parameter, a projection parameter, a stitching parameter, and a cloud parameter.
The method of claim 15,
The cloud parameter represents a computing resource required to perform the image stitching workflow,
The one or more commands,
An apparatus for generating an image stitching workflow, characterized in that the number of media processing subjects is determined according to the cloud parameter.
A computer-readable recording medium comprising a bitstream including a 360 degree VR image parameter required for image stitching of a 360 degree VR image, comprising:
The 360-degree VR video parameter is one of a general descriptor parameter, an input descriptor parameter, an output descriptor parameter, a processing descriptor parameter, a requirement descriptor parameter, a client support descriptor parameter, a failover descriptor parameter, a monitoring descriptor parameter, and a reporting descriptor parameter. Computer-readable recording medium comprising at least one.
The method of claim 17,
The 360 degree VR video parameter includes a setting parameter,
The setting parameter comprises at least one of a media parameter, a feature parameter, a camera parameter, a projection parameter, a stitching parameter, and a cloud parameter.
The method of claim 18,
The cloud parameter represents a computing resource required to perform the image stitching workflow,
According to the cloud parameter, the number of media processing subjects for processing the image stitching workflow is determined.
The method of claim 18,
The media parameter includes at least one of a codec parameter, a chroma parameter, an fps parameter, a gop parameter, and a resolution parameter,
The feature parameter includes at least one of a feature extraction method parameter, a feature point number parameter, a feature point location parameter, and an optional feature correspondence point parameter,
The camera parameter includes at least one of a focal length parameter, a main point parameter, an asymmetric coefficient parameter, a movement parameter, and a rotation parameter,
The projection parameter includes a projection type parameter,
The stitching parameter includes at least one of a boundary point position parameter and a boundary point mask parameter,
The cloud parameter comprises at least one of a number of threads parameter and a number of GPUs parameter.

Images