KR20180073778A - Method for processing virtual image and device thereof - Google Patents

Abstract

Provided are a device and an operation method for image processing. The virtual reality image processing method according to an embodiment of the present invention includes a process of calling a stitching pipeline having a designated shape; a process of calling an original image with respect to each of at least two stitching modules included in the stitching pipeline; a step of converting the original image called with respect to each of the stitching modules for matching with the designated shape based on each of the stitching modules; and a step of creating an exploded view of one virtual reality image by combining each of the converted images at a designated position with respect to each of the corresponding stitching modules.

Description

TECHNICAL FIELD [0001] The present invention relates to a virtual reality image processing method,

The present invention relates to a virtual reality image processing method and apparatus, and more particularly, to a method and apparatus for generating, modifying, or processing a virtual reality image.

A 360 ° panoramic image is used to implement panoramic virtual reality. Panoramic virtual reality requires two steps. The first is to obtain a 360 degree panoramic virtual reality image (vr image), and the second is to display through a panoramic vr viewer.

The panoramic vr viewer receives the panoramic image as an input, and when the user designates a specific direction of the sight, the view from the panoramic image is generated in real time from the viewpoint in the sight direction.

The vr image is usually generated by fixing the position of the camera, attaching successively obtained images while appropriately rotating the body of the camera, or attaching images taken through a plurality of cameras to each other. At this time, A variety of image processing methods are being developed for generating vr images linked together.

Patent Document 10-1077584 discloses an image processing apparatus for matching images acquired from a plurality of cameras, comprising: at least two or more cameras arranged so that a photographing region partially overlaps between adjacent cameras; A video signal receiving unit for receiving at least two or more input video signals obtained from the plurality of cameras; A lookup table for storing image mapping data of a relation of each image pixel constituting a plurality of input image signals obtained from the plurality of cameras to image pixels of a composite image signal; And an image matching unit for receiving a plurality of input image signals from the image signal receiving unit and constructing image pixels of a composite image signal for each of the image pixels constituting each input image signal with reference to the lookup table; And a video signal output unit for generating and outputting an output video based on the composite video signal formed by the video matching unit.

Such an image processing apparatus generates an output image by mapping pixels using a lookup table to a plurality of images. When the number of images changes or a change occurs in the vr image generation environment, such as image correction, there is a limitation that the process of generating the vr image must be performed from the beginning.

10-1077584 (registered patent publication)

According to various embodiments of the present invention, a virtual reality image processing apparatus and an operation method thereof for generating a vr image using a plurality of images can be provided.

According to an embodiment of the present invention, it is possible to provide a virtual reality image processing apparatus and an operation method thereof capable of effectively coping with a change of a vr image generation environment in generating a vr image using a plurality of images.

According to an embodiment of the present invention, in generating a vr image using an image, a virtual reality image processing apparatus and an operation method thereof for modifying a part of a generated vr image can be provided.

According to an embodiment of the present invention, there is provided a method of calling a stitching pipeline of a specified type; Calling an image source for each of the at least two stitching modules included in the stitching pipeline; Converting the called image source for each of the at least two stitching modules to match a specified type based on each of the at least two stitching modules; And merging the converted two or more images at a designated position with respect to each of the two or more corresponding stitching modules to generate a developed view of one virtual reality image.

The method of claim 1, wherein transforming the called image source for each of the at least two stitching modules to match a specified type based on each of the at least two stitching modules comprises: It is possible to distort the image by applying it to some distorted maps that are positioned and shaped in a plate based on a green gradient.

The step of transforming the called image for each of the at least two stitching modules to match the designated type based on each of the at least two stitching modules further comprises: distorting the called image source; And performing a preprocessing of a designated channel for merging the distorted image.

The step of merging the converted two or more images at a designated position with respect to each of the two or more corresponding stitching modules to generate a developed view of one virtual reality image may further include the step of merging the transformed images, And processing the multiplication operation.

In addition, the designated channel may be determined corresponding to an overlapped area of the at least two distorted images arranged in the developed view of the virtual reality image.

Also, the pre-processing may apply an alpha value to an overlapping area of the at least two distorted images arranged in the developed view of the virtual reality image.

The virtual reality image processing method may further include adding at least one stitching module to the stitching pipeline.

Further, the step of adding at least one stitching module to the stitching pipeline may be further performed in a process of calling the stitching pipeline of the specified type, or the process of adding the at least one stitching module to each of the two or more stitching pipes May be performed after merging at a specified position for each module to generate a developed view of one virtual reality image.

Selecting a specific location in the developed view of the virtual reality image; And calling at least one of a distorted image and an image source corresponding to the specific location.

Meanwhile, according to an embodiment of the present invention, an image calling module for calling an image source; A transformation processing module that transforms the image source and merges two or more distorted images; And calling the image source for each of the at least two stitching modules included in the stitching pipeline through the image calling module, and calling the image source through the conversion processing module for each of the two or more stitching modules The image source is converted to match the designated type based on each of the two or more stitching modules, and each of the two or more converted images is merged at a designated position with respect to each of the corresponding two or more stitching modules, And a control module for processing the generated virtual reality image to generate a developed view of the virtual reality image.

According to various embodiments, the transformation processing module may distort the image by applying the image source to some distorted maps that are positioned and shaped in a plate based on red and green gradients.

The transformation processing module may also perform a preprocessing of a designated channel for distorting the called image source and merging the distorted image.

In addition, the conversion processing module may process the multiplication of the designated channel for merging the converted image.

In addition, the control module may determine the designated channel corresponding to an overlapped area of the two or more distorted images arranged in the developed view of the virtual reality image.

In addition, the conversion processing module may perform the pre-processing by applying an alpha value to an overlapped region of the two or more distorted images arranged in the developed view of the virtual reality image.

In addition, the control module may add at least one stitching module to the stitching pipeline.

In addition, the control module may select a specific position in the developed view of the virtual reality image, and may call at least one of the distorted image and image source corresponding to the specific position.

Each of the two or more stitching modules further includes an image node calling an image source photographed with respect to a specified direction for each of the two or more stitching modules; A sample node for determining at least one of a position and a distortion shape placing the image source called in the image node; A distortion node for distorting the image source based on the sample node; A channel setting node for performing preprocessing of a designated channel for merging the distorted image; And a correction node processing a product operation of the designated channel for merging the distorted image; Lt; RTI ID = 0.0 > a < / RTI >

According to various embodiments of the present invention, by creating a stitching pipeline that includes two or more stitching modules that call a pre-created stitching pipeline or process an image for one direction, It is possible to efficiently generate the vr image based on the vr image.

According to various embodiments of the present invention, the quality of the generated vr image is effectively controlled by adding and / or removing stitching modules for the synthesis of images photographed in various directions through the stitching pipeline, and vr You can handle the modification of the image.

According to various embodiments of the present invention, by providing various formats for generating vr images based on the stitching pipeline, it is possible to provide a variety of vr image generation and further improve the freedom of creation of vr images .

According to various embodiments of the present invention, processing of vr image generation through a stitching file link and a stitching module included in a stitching pipeline improves flexibility in the generation of vr images and solves the problems that are generated, The degree of completeness of the image can be improved.

1 shows a schematic configuration of a system including an image processing apparatus and an image processing apparatus according to an embodiment of the present invention.
2 is a diagram showing a structure of a vr image generated in an image processing apparatus according to an embodiment of the present invention.
3 is a diagram illustrating a structure of a stitching module for processing an image in an image processing apparatus according to an embodiment of the present invention and a stitching pipeline including the stitching module.
FIG. 4 is a diagram showing images processed in the generation of a vr image in the image processing apparatus according to an exemplary embodiment of the present invention.
5 is a diagram showing a vr image developed view generated based on a distorted image in an image processing apparatus according to an embodiment of the present invention.
6 is a diagram showing a stitching module generated in the stitching pipeline in the image processing apparatus according to the embodiment of the present invention.
7 is a view for processing an image for a selected region of a vr image development plan in an image processing apparatus according to an embodiment of the present invention.
8 shows a flow of an operation of generating a vr image in an image processing apparatus according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings.

The embodiments of the present invention are capable of various changes and various embodiments, and specific embodiments are illustrated in the drawings and the detailed description is described with reference to the drawings. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all changes and / or equivalents and alternatives falling within the spirit and scope of the invention.

The same reference numerals can be used for the same or similar components in connection with the description and the description of the drawings.

It should be noted that the terms such as " comprising " or " may include " may be used among the various embodiments of the present invention to indicate the presence of a corresponding function, operation or component, Not limited.

Also, in various embodiments of the invention, the terms "comprise" or "having" are intended to specify the presence of stated features, integers, steps, operations, components, parts or combinations thereof, But do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In various embodiments of the present invention, the term "or" or the like includes any and all combinations of words listed together. For example, 'A or B' may comprise A, include B, or both A and B.

In various embodiments of the present invention, expressions such as 'first', 'second', 'first' or 'second' may modify various elements of the present invention, but the order and / . In addition, the representations may be used to distinguish one component from another.

In the various embodiments of the present invention, when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, It is to be understood that the components may also be present. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terms "module," "module," and the like, as used in various embodiments of the invention, refer to a unit that processes at least one function or operation and may be implemented in hardware or software, or in a combination of hardware and software .

In various embodiments of the present invention, the description may not exactly match the information presented, such as a cited property, a variable, or a value, depending on the expression "substantially", " It should not be limited to embodiments of the invention in accordance with various embodiments of the present invention, such as variations, including limitations of measurement accuracy, and other factors commonly known.

The terminology used in the following description is used to describe various embodiments of the present invention and is not intended to limit the present invention. Also, in the context of the present invention, the singular expressions may include plural expressions unless the context clearly dictates otherwise.

Also, all terms used herein, including technical or scientific terms, should be interpreted to have the same or similar meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Unless expressly defined in the examples, they should not be construed or interpreted in an ideal or overly formal sense.

According to various embodiments of the present invention, the image processing apparatus may be provided with a control unit (or a processing unit), for example, a processor or the like, for performing image processing operations according to various embodiments of the present invention, For example, a memory, a server, or the like that stores data related to object processing operations.

The image processing apparatus may be configured in the form of one or more independent modules. Further, the image processing apparatus may be configured to be connected to or combined with at least one other module associated with, or included as a part of, the module.

According to one embodiment, the image processing apparatus may be provided as part of a program (or software), a device (or hardware) and / or a system As shown in FIG.

According to one embodiment, the system may be a 3D image processing system (or 3D animation system), in particular a maya, a 3d max, a poser, a pose-studio, Nuke, houdini, and the like. Further, the system is not limited to the program, but may be provided to a computer apparatus in which at least one of the above-mentioned programs is installed.

According to various embodiments of the present invention, the image processing apparatus may be configured to perform a pre-processing or a post-processing to perform various tasks or functions for processing various images constituting a virtual reality image for a head- And performing intermediate operations, such as creating, retrieving, collecting, invoking, loading (or loading), modifying, storing, and linking data used in a task and / or generating, outputting, , Storage, connection, and so on.

Here, an image can be described as one still image (or image frame), and can also include the meaning of a moving image composed of two or more connected (or arranged) images.

For example, an image can be described as a cut (or a frame) for a still image, a shot (a shot or a scene) for an moving image, and can be defined as an image by adding a temporal flow to the image.

A virtual reality image processing apparatus (hereinafter referred to as an image processing apparatus) is a device that implements a virtual reality image (or a panoramic virtual reality image (hereinafter, referred to as a vr image)) using a plurality of previously captured images Lt; / RTI >

Furthermore, although the vr image has been described as an image used through a head-mounted display device, the present invention is not limited thereto and can be provided as a vr image used in various display devices.

1 shows a schematic configuration of a system 10 including an image processing apparatus 100 and an image processing apparatus 100 according to an embodiment of the present invention. 1, an image processing apparatus 100 includes a control module 110, an image calling module 120, a positioning module 130, a sink processing module 140, a conversion processing module 150, Module < RTI ID = 0.0 > 160 < / RTI >

Here, each of the configurations included in the image processing apparatus 100 can be described as a unit that is divided into tasks and functions. In this case, the module may be configured with a software module or a hardware module as described above.

According to one embodiment, the system 10 including the image processing apparatus 100 comprises at least one of an input 201, an output 203, and a database 105 connected to the image processing apparatus 100 Element.

The control module 110 may provide a stitching pipeline for generating a vr image based on a plurality of two-dimensional images photographed to have mutual offsets for objects in the real space using at least one camera.

According to one embodiment, the vr image may be generated as an image on a two-dimensional plane that can be represented as a spherical image 301 as shown in Fig. At this time, the vr image can be generated by arranging and distorting a plurality of images photographed in various directions according to a designated form.

According to one embodiment, a plurality of images photographed in various directions can be photographed using one camera, or two or more cameras.

For example, a plurality of images photographed in various directions may be images taken at the same time by a camera combined with a single device toward two or more cameras in a designated direction, such as the camera 311 in Fig.

According to one embodiment, the vr image comprises an image for two or more directions and is preferably selected from the group consisting of front, back, left, right, zenith, (nidar) images based on images of six directions.

According to various embodiments of the present invention, the vr image is described as being generated using images for six directions as described above, but the present invention is not limited thereto. For example, It is obvious that vr image can be generated using The stitching pipeline 40 provided through the control module 110 may include a node and a link connecting the node as shown in FIG. According to one embodiment, the stitching pipeline 40 may include a node (hereinafter, a stitching module) for calling an image (or asset) for at least one direction to generate a vr image.

3, the stitching pipeline 40 provided through the control module 110 according to an exemplary embodiment of the present invention includes stitching modules 410, 420, 430, 440, 450, 460).

At this time, each stitching module may include at least one child node that invokes and transforms an image to create a natural vr image by merging (or splicing) multiple images for various directions.

For example, referring to the forward stitching module 410, the stitching module 410 may include an image node 511 that calls for footage images (footage images) A sample node 513 that determines the location and distortion type that places the called image source in the node 511, a distortion node 515 that distorts the image source, a pre-processing of the designated channel for merging the distorted image A channel setting node 517, and a correction node 519 that processes the operation of the specified channel for merging the distorted image.

The two or more stitching modules included in the stitching pipeline may be connected through at least one merge node.

According to one embodiment, the image node 511 may output the called image via the image calling module 120. For example, in the case of the stitching module 410, the called image source (e.g., image 411 of FIG. 4) may be output from images in various directions for the designated project.

According to one embodiment, each stitching module may be configured to automatically invoke an image source for a specified direction for a selected project and output it to the image node 511.

Here, when a project is selected, the stitching module can be set to call an image source including a specified phrase among various image sources included in the selected project in the file name.

For example, in selecting an image source, the stitching module 410 may be set to invoke an image source (e.g., image 411 of FIG. 4) that includes cam1 in the filename.

Similarly, each of the stitching modules 420, 430, 440, 450, and 450 may include an image source such as cam2, cam3, cam4, cam5, and cam6 in filenames such as images 421, 431, 441, 451, 461 < / RTI >

The image node 511 may display a thumb nail image for at least a portion of the called image source.

The sample node 513 is used to transform (e.g., distort) the image source called to the stitching module 513 in the developed view for generating the vr image (or the developed view of the generated vr image, hereafter the vr image developed view) Distortion information.

According to one embodiment, the distortion information may be information (e.g., an instruction word) that is configured to process at least one of a change in shape, resize, and compositing, combining, and combining with at least one other image of the image source.

Here, the sample node 513 may include information on the position where the image original is placed and the degree of the distortion, or the position where the called image source is placed and information about the degree of distortion.

According to one embodiment, information about the location and the degree of distortion at which the called image source is placed may be determined via a map. For example, the map is provided based on a newkeep program and can be provided in stmap format. According to one embodiment, a map in stmap format can be provided as an image in the form of x-axis and y-axis coordinates by applying vector2 to each position on the screen.

Here, the map of the stmap format can be set with the formula of 1 / resolution according to the position of the end of each screen based on the lower left point (0,0). Generally, the x and y axes of the screen can be set to correspond to red (red, R), and green (green, G) as colors. In this case, the value of green (0, 1, 0) increases toward (0,1) and the value of red (1,0,0) increases toward (1,0) Lt; / RTI > plate.

Here, the map in the stmap format can be generated using at least some of the functions and resources of the camera that photographed the image source.

The sample node 513 applies the position (or area) information designated to the stitching module 410 or the like in the whole of the map (e.g., the map of the stmap format) It is possible to determine a map (hereinafter referred to as a partial distortion map) in a state in which the remaining area outside the position (or area) specified by the stitching module in the distortion map is removed. According to one embodiment, the size of the total distortion map may be provided equal to or similar to the size of the vr image exploded view.

The sample node 513 may include a position, distortion and / or shape on the global distortion map for placing the image source called through the stitching module 410, such as the map provided (e.g., the map 413 shown in FIG. 4) Or a thumbnail image of a map in which a standard is displayed (hereinafter, some distortion map).

The distortion node 515 includes instructions to process the image source 511 to cause the image source 511 to distort the image source according to the information contained in the sample node 513, And / or may be distorted in accordance with the standard.

According to one embodiment of the distortion, some distortion maps are processed to distort (or translate) the position information of the image source to a specified position using pixel information (e.g., pixel coordinate information) of some distortion map for the image source Information. Thus, if an image source (image source 411 of FIG. 4) is applied to some distortion map 413 via distortion node 515, a distorted image (or distorted footage image, hereinafter referred to as distorted image) can do.

Distortion node 515 receives the distortion information value via sample node 513 and then distorts and / or transforms the image source as described above to generate a distorted image (e.g., to process image 415 of FIG. 4) Command.

In accordance with one embodiment, the distortion node 515 is configured to determine whether the distortion of the image (e.g., the distortion) of the image based on the information set in the sample node 513 and / You can handle the distortion of the original.

The channel setting node 517 can process the channels used for merging the distorted image through the stitching module 410 and the distorted image 415 through the stitching module.

According to one embodiment, the distorted image 415 generated using the image source 411 and some distortion map 413 may be modified such that the distorted image 415 through the stitching module 410, (E.g., an alpha value) associated with the transparency of the channel (e.g., the overlapping region (or overlapping region) in the merging process of the distorted image) determined for the action to be combined with the distorted image.

The channel setting node 517 then applies the alpha value for the overlapping region of the distorted image 415 and / or some distortion map 413 to the distorted image in generating the vr image developed view, It is possible to perform control so as not to process the calculation of the remaining area except for the image 415.

According to one embodiment, the channel setting node 517 may be provided with at least some functions of a shuffle copy of the neque program.

Correction node 519 may include an instruction to process the multiplication operation used for merging the distorted image through the stitching module 410 and the distorted image 415 through another stitching module.

According to one embodiment, the correction node 519 performs a multiplication operation through the stitching module 410 to process the alpha value in the course of merging the distorted image 415 with the distorted image through another stitching module .

At this time, the correction node 519 performs a multiplication operation to process the information of the alpha value included in the distorted image 415 and / or some distortion map 413, thereby controlling the processing to be unnecessary can do.

According to one embodiment, the correction node 519 may be provided with at least some of the functionality of the line-of-sight premultiplied

The control module 110 may generate a distorted image by distorting the image source for each of the stitching modules included in the stitching pipeline 40, as described above.

5, the control module 110 may also process each generated distorted image 415, 425, 435, 445, 455, 465 in a composite manner to produce a single vr image development view, A lat-long image 600 may be generated.

1 again, the image calling module 120 determines whether or not each of the stitching modules 410, 420, 430, 440, 450, 460 of the stitching pipeline 40 corresponds to the direction specified for each of the stitching modules It can call the image source and output it to the image node 511.

Here, a method of calling an image source corresponding to a specified direction with respect to each of the stitching modules may be such that, when a position of the designated project is input, the image source including the code specified for the stitching module and / Or various data calling methods such as drag and drop using the input unit 201 or calling using a path of an image source can be used.

According to one embodiment, the image calling module 120 may invoke, for each stitching module, an image source comprising a phrase specified in a file name among various image sources included in the selected project.

The image calling module 120 can output the called image source through the image node of the corresponding stitching module.

The positioning module 130 may determine information about a position on the vr image based on the setting information of each of the stitching modules. The positioning module 130 can process the image source called at the image node to be located at a specified location based on the developed view of the vr image.

According to one embodiment, the positioning module 130 may use a map (e.g., stmap) in determining the location to place the called image source. For example, referring to the map 413 of FIG. 4, the stitching module 410 may make a call based on the set location information (e.g., location information set in the sample node 513) and / The position of the original image 411 can be determined.

Likewise, based on the location information and / or some distortion maps 423, 433, 443, 453, 463 set in the stitching modules 420, 430, 440, 450 and 460 included in the stitching pipeline 40 The position of the image source can be determined.

The control module 110 may perform synchronization using the called image source via the stitching pipeline 40. [ According to one embodiment, through the sink processing module 140, when each called image source is presented as an image, the playback of the called image source can be synchronized to each of the stitching modules.

The sink processing module 140 may synchronize a sink of the image source and / or the distorted images that are connected to generate the vr image exploded view. According to one embodiment, the sync processing module 140 may use the audio included in each of the image sources and / or distorted images, for example, the audio source and / Or the playback of the distorted images.

The control module 110 may distort and merge the image source through the conversion processing module 150 to generate the vr image development view. At this time, the conversion processing module 150 may combine distorted images based on each stitching module to generate a vr image development view.

The conversion processing module 150 can acquire a reference to an overlapping area (or overlapping area of view angle) in each distorted image.

For example, when merging two or more distorted images, at least some of the areas where the image source is applied (e.g., the image source is distorted) in each of the distorted images are different from the area where the image source is applied At least a portion of < / RTI > The conversion processing module 150 can confirm the reference to the overlapping area generated in the merging operation.

 According to one embodiment, the reference of the overlapping region can be obtained using a grid plate.

Further, the conversion processing module 150 can determine an overlapping area of the view angle based on the camera view angle, and can determine the view angle to be, for example, 10 to 20 degrees, more preferably 15 degrees.

The transformation processing module 150 may modify the line for the overlapping regions of the distorted images to match the completed vr image exploded view based on the acquired reference.

According to one embodiment, the transformation processing module 150 may detect lines and objects, such as horizontal lines, horizon lines, in an image source and / or distorted image. At this time, the conversion processing module 150 may modify the line for the overlapping area based on at least one of the detected horizon lines, horizontal lines, outlines forming the object, and / or the boundary lines.

In addition, the conversion processing module 150 can correct a difference (e.g., a vignetting effect) caused by the refraction of the camera lens and / or the exposure difference of light generated upon photographing the image source.

The control module 110 determines whether or not abnormalities (e.g., errors, errors) are generated during the operation of generating the vr image using the stitching module included in the stitching pipeline 40, through the anomaly detection module 160 Can be confirmed.

The anomaly detection module 160 is configured to detect whether the images processed through at least one of the control module 110, the image calling module 120, the positioning module 130, the sink processing module 140 and the conversion processing module 150 Identity, similarity, and / or consistency of the data.

According to one embodiment, the anomaly detection module 160 may compare at least some areas of the overlapping areas of each image (e.g., a distorted image or an image source) that make up the vr image exploded view. The abnormality detection module 160 can check overlapping areas among images called through each of the stitching modules to check whether synchronization is synchronized and / or whether distorted images are connected.

Also, the abnormality detection module 160 can check the occurrence of a problem in the generation of the vr image development view or the vr image using the image source called through the plurality of stitching modules included in the stitching pipeline 40 .

For example, in the process of reproducing at least one of the image sources, such as error screen output such as non-callable, non-reproducible and / or black screen, or in the process of merging images distorted and / or distorted Problems such as vignetting and chromatic aberration can be detected.

The abnormality detection module 160 can process the image stiction and / or merging operation with respect to the stitching module in which the abnormality has occurred. According to one embodiment, when the abnormality detecting module 160 determines that an abnormality has occurred in the left image source 421 in the stitching pipeline 40, It is possible to process the output of the vr image without performing the image processing of the vr image.

At this time, the anomaly detection module 160 detects at least some of the regions overlapping some of the distortion maps 423 among some of the distortion maps 413, 433, 443, 453, and 463 in the vr image developed view, ) And the image distortion for the merging operation with the distorted image generated.

Referring back to FIG. 1, the control module 110 may add at least one stitching module to the stitching pipeline 40. According to one embodiment, the control module 110 may add a stitching module 470 to the stitching pipeline 40, as shown in FIG.

For example, the control module 110 may add at least one stitching module in the creation or paging operation of the stitching pipeline. In addition, the control module 110 may specify a location in the global distortion map used to generate the pre-generated vr image development view and / or the vr image development map, and may add a stitching module to the specified location.

The control module 110 may generate at least one child node among the image node, the sample node, the distortion node, the channel setting node, and the correction node with respect to the generated stitching module.

In addition, the control module 110 determines at least one of the stitching modules for the overlapped area of some distortion maps and / or distorted images with respect to the stitching module 470 generated among the stitching modules included in the stitching pipeline 40 The image processing content of the sample node 513 and / or the distortion node 515 can be modified.

According to various embodiments, the control module 110 may modify at least a portion of the generated vr image or the vr image generated through the vr image development view. The control module 110 can select a partial area 601 of the vr image development view and perform image processing on the selected area as shown in Fig.

The control module 110 may generate an image of at least one of the distorted image 475, some distortion map 473 and the image source 471 for the selected region, for the selected region.

At this time, in generating the distorted image 475, some distortion map 473, and / or the image source 471 for the selected area in the vr image development view, the image source may be used to generate the distorted image and / It is possible to perform inverse processing in relation to the function of at least part of the operation of generating the developed view.

Also, the control module 110 can generate a stitching module corresponding to the selected area. For example, the control module 110 may determine whether the image region, the sample node, the distortion node, the channel setting node, and / or the image node based on the selected region (e.g., A correction node, and a correction node.

In addition, the control module 110 may modify at least some of the image source called through the stitching pipeline or the distorted image generated based on the image source. According to one embodiment, the control module 110 may select at least one of the image nodes of the stitching pipeline 40 output through the output 203 to call the image source called through the stitching module .

According to various embodiments, the control module 110 may select at least one of the regions of the distorted image displayed in the vr image developed view output through the output 203 to call the distorted image corresponding to the selected location have.

The control module 110 may provide for modifying the distorted image and / or image source (e.g., to modify it on a per-pixel basis) and thus may be used to modify the image, such as matching of the sync, matching of the lines, The correction can be effectively processed.

The control module 110 may also select at least one of the sample nodes of the stitching pipeline 40 to call the distortion node and / or the channel setting node of the stitching module.

The control module 110 may store the modified information in the image and / or the node if the information in the called image and / or distortion node and / or the channel setting node has been modified.

According to various embodiments, it is apparent that the control module 110 is not limited to adding the stitching module in the stitching pipeline 40, but can exclude (remove or remove) the stitching module.

In addition, the control module 110 may change the location at which the image is displayed in some distortion maps. According to one embodiment, the control module 110 may store, via the positioning module 130, information about the changed position in the distortion node and / or the channel setting node.

According to various embodiments, the control module 110 may perform additional distortion on the distorted image. According to one embodiment, the control module 110 may perform an operation of matching colors to the overlapping regions in an operation of merging two or more distorted images.

Also, in accordance with various embodiments, the control module 110 may be configured to distort the image so that it is distorted so as to be joined to an area that is not completely joined due to a difference in the sync of the distorted image in the operation of merging two or more distorted images vr image quality can be improved.

According to various embodiments, the control module 110 may include at least one of the image calling module 120, the positioning module 130, the sink processing module 140, the conversion processing module 150, Can invoke the components of < / RTI >

In addition, the control module 110 can call data input from the outside (or an external device) and / or data stored in the database 150, and process the called data through various components.

According to various embodiments, the system 10 includes at least one component of an input 201, an output 203, and a storage 205, May be coupled to the device 100.

According to one embodiment, the input unit 201 may be provided as an apparatus of at least one of a keyboard, a mouse, a note pad, a track pad, and a microphone connected to the computer, and the output unit 203 may include a monitor, Speaker, or the like.

Here, the system 10 including the image processing apparatus 100 may further include a communication unit or may be connected to a communication unit. The image processing apparatus 100 can connect communication between at least one external device and an external program module through a communication unit. For example, the communication unit may be connected to the network through wireless communication or wired communication to communicate with the devices.

In the case of wireless communication, the communication unit is configured to perform at least one of wireless communication among wireless fidelity (Wifi) communication, worldwide interoperability formicrowave access (WMax) communication, Bluetooth (BT) communication, and Zigbee communication And a communication module.

In the case of wired communication, the communication unit may include at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), or a plain old telephone service Communication module. At this time, the communication unit can be included in one integrated chip (IC) or an IC package.

The storage unit 205 may store images photographed through at least one camera, for example, an image source. Further, the storage unit 205 may store images and related information processed through the image processing apparatus 100. [

According to one embodiment, the storage unit 205 stores information about at least one stitching module included in the stitching pipeline 40, for example, image nodes, sample nodes, distortion nodes, Lt; RTI ID = 0.0 > a < / RTI >

Also, the storage unit 205 may include information on at least one of a distorted image, a vr image, a vr image developed view, an entire distortion map, and a partial distortion map generated based on the image source.

According to one embodiment of the present invention, a vr image can be generated based on the components of the image processing apparatus 100 described above and their functions. According to one embodiment, the image processing apparatus 100 may generate a vr image exploded view and a vr image based on the stitching pipeline as described above.

8 shows a flow of an operation of generating a vr image in the image processing apparatus 100 according to an embodiment of the present invention.

The image processing apparatus 100 can call the specified stitching pipeline (S801). According to one embodiment, the image processing apparatus 100 may call the stitching pipeline stored in the storage unit 205 and / or create a new stitching pipeline.

At this time, when calling the stitching pipeline (for example, the stitching pipeline 40 in Fig. 3) stored in the storage unit 205, the stitching pipeline 40 may include at least two stitching modules.

The image processing apparatus 100 may call an image source for generating a vr image based on the called stitching pipeline (S803). According to one embodiment, the image processing apparatus 100 may call the photographed image source for a specified direction for each of the two or more stitching modules included in the stitching pipeline.

The image processing apparatus 100 may convert the image source set for each stitching module according to one embodiment (S805) Information and / or some distortion maps.

The image processing apparatus 100 may generate a developed view of one virtual reality image by merging each of the converted two or more images at a designated position (S807). The image processing apparatus 100 may process additional image distortions for overlapping regions in merging two or more transformed images.

According to one embodiment, in merging the transformed image, the image processing apparatus 100 may apply a specified alpha value to the overlapping region.

The image processing apparatus 100 can perform a post-processing operation such as an image flattening (stabilization) operation, a horizontal line included in the vr image, a point tracking using a horizon, a fixed object, have.

According to various embodiments, the image processing apparatus 100 may add or control a stitching module to the stitching pipeline. According to one embodiment, the image processing apparatus 100 may be configured to provide a captured image to at least a portion of the generated vr image as an image source in order to improve quality and / or apply additional effects. A stitching module for processing can be added.

According to various embodiments, in addition to the stitching pipeline, the image processing apparatus 100 is not limited to creating a stitching module for adding a photographed image source to at least one direction, An image or vr image development can be used to modify the image for the selected area.

According to various embodiments of the present invention, the image processing apparatus 100 may generate a stitching pipeline that includes two or more stitching modules that call the pre-created stitching pipeline or process an image for one direction, The generation of the vr image based on the photographed image in various directions can be effectively performed.

In addition, the imaging processing apparatus 100 can effectively control the quality of the generated vr image by processing the stitching module for synthesizing the photographed images in various directions via the stitching pipeline so as to be additionally and / vr can handle the modification of the image.

According to one embodiment, the modification of the image may be performed on a selected image using computer graphics, such as digital processing, e.g., special effect (s), visual effects-special effects ). ≪ / RTI >

As described above, the image processing apparatus 100 can provide various types of vr image generation by providing various formats for generating vr images based on the stitching pipeline, and further, it is possible to greatly increase the freedom of creation of vr images Can be improved.

In addition, when the distortion and / or merging process of the vr image and the abnormal state occurring in the processed image are checked, if the vr image is generated by distorting the rest of the image, the space of the generated vr image is distorted It can lose reality.

When detecting the abnormal state, the image processing apparatus 100 does not perform image processing (e.g., image distortion, merging, etc.) on the region corresponding to the image, The image can be processed so as not to lose the reality.

As described above, the image processing apparatus 100 improves the flexibility of creating a vr image and solving the problems that are generated by processing vr image generation through a stitching file included in the stitching file profile and the stitching pipeline , The perfection of the generated vr image can be improved.

Various embodiments of the invention described above may be implemented in software. When implemented in software, the storage may be provided as a computer-readable storage medium (or a computer-readable storage medium) that stores one or more programs (or programming modules, applications). For example, the storage unit is a computer-readable storage medium in the form of a programming module, and the software may be embodied as instructions stored in the storage unit.

The one or more programs may include instructions that cause the image processing apparatus 100 to perform the methods according to the embodiments of the invention and / or the claims of the present invention. When executed by one or more control units, the one or more control units may perform a function corresponding to the command. The computer readable storage medium may be, for example, a memory. At this time, the database 150 may be included in the memory.

At least a portion of the programming module may be implemented (e.g., executed) by, for example, a control unit. At least some of the programming modules may include, for example, modules, programs, routines, sets of instructions or processes, etc. to perform one or more functions.

For example, according to an embodiment of the present invention, the image processing apparatus 100 may include a javascript object notation (json) format used in an open standard format for delivering data objects, a NoSQL (Mongodb), which is one of the MongoDBs.

Also, according to an embodiment of the present invention, the image processing apparatus 100 can be implemented using a lightweight directory access protocol (LDAP) protocol to manage various user apparatuses to be connected.

The computer-readable recording medium includes a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc) A magneto-optical medium such as a floppy disk and a program command such as a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, (EEPROM), a magnetic disc storage device, or other form of optical storage (e. G., A computer readable storage medium) Devices, magnetic cassettes, and the like. Or a combination of some or all of these. In addition, a plurality of constituent memories may be included.

In addition, the software may be stored in the input device as a communication network such as the Internet, an Intranet, a LAN (Local Area Network), a WLAN (Wide Area Network), or a SAN (Storage Area Network) May be stored in an attachable storage device that can be accessed through a communication network. Such a storage device can be connected to the image processing apparatus 100 through an external port. Further, a separate storage device on the communication network may be connected to the image processing apparatus 100. [

The hardware devices described above may be configured to operate as one or more software modules to perform operations on the various embodiments of the present invention, and vice versa.

Modules or programming modules according to various embodiments of the present invention may include at least one or more of the elements described above, some of which may be omitted, or may further include other additional elements. Operations performed by modules, programming modules, or other components in accordance with various embodiments of the invention may be performed in a sequential, parallel, iterative, or heuristic manner. Also, some operations may be performed in a different order, omitted, or other operations may be added.

The embodiments of the present invention disclosed in the present specification and drawings are merely illustrative examples of the present invention and are not intended to limit the scope of the present invention in order to facilitate understanding of the present invention. Accordingly, the scope of the present invention should be construed as being included in the scope of the present invention, all changes or modifications derived from the technical idea of the present invention.

100: image processing apparatus 110: control module
120: image calling module 130: positioning module
140: sink processing module 150: conversion processing module
160: abnormality detection module 201: input part
203: output unit 205: storage unit

Claims (18)

Calling the stitching pipeline of the specified type;
Calling an image source for each of the at least two stitching modules included in the stitching pipeline;
Converting the called image source for each of the at least two stitching modules to match a specified type based on each of the at least two stitching modules; And
And merging the converted two or more images at a designated position with respect to each of the two or more corresponding stitching modules to generate a developed view of the one virtual reality image. The method according to claim 1,
Wherein the step of transforming the called image source for each of the at least two stitching modules to match the specified shape based on each of the at least two stitching modules,
Wherein the image source is distorted by applying the image source to a partially distorted map positioned and shaped in a plate based on red and green gradients. The method according to claim 1,
Wherein converting the invoked image for each of the at least two stitching modules to match a specified type based on each of the at least two stitching modules,
Distorting the called image source; And
And performing pre-processing of a designated channel for merging the distorted image. The method of claim 3,
The step of merging the converted two or more images at a designated position with respect to each of the two or more corresponding stitching modules to generate a developed view of one virtual reality image,
And processing the product operation of the designated channel for merging the transformed image. The method according to claim 1,
Wherein the designated channel is determined corresponding to an overlapping one of the two or more distorted images disposed in an exploded view of the virtual reality image. The method of claim 3,
Wherein the preprocessing applies an alpha value to an overlapping region of the at least two distorted images disposed in a developed view of the virtual reality image. The method according to claim 1,
And adding at least one stitching module to the stitching pipeline. The method according to claim 1,
Wherein the step of adding at least one stitching module to the stitching pipeline comprises:
Or in the process of calling the stitching pipeline of the specified type or merging the converted two or more images at the designated positions for each of the corresponding two or more stitching modules to generate a developed view of one virtual reality image The virtual reality image processing method comprising: The method according to claim 1,
Selecting a specific location in the developed view of the virtual reality image; And
And calling at least one of a distorted image and an image source corresponding to the specific location. An image calling module that calls the image source;
A transformation processing module that transforms the image source and merges two or more distorted images; And
Calling the stitching pipeline of a specified type and invoking the image source for each of the at least two stitching modules included in the stitching pipeline through the image calling module, Wherein the image processing unit converts the called image source to match a specified type based on each of the two or more stitching modules and merges the converted two or more images at a designated position with respect to each of the two or more corresponding stitching modules, And a control module that processes the generated virtual reality image to generate a developed view of the real image. 11. The method of claim 10,
Wherein the transformation processing module applies the image source to a portion of the distortion map that is positioned and shaped in a plate based on a red and green gradient to distort the image. 11. The method of claim 10,
Wherein the transformation processing module performs a preprocessing of a designated channel for distorting the called image source and merging the distorted image. 13. The method of claim 12,
Wherein the conversion processing module processes multiplication of the specified channel for merging the transformed image. 11. The method of claim 10,
Wherein the control module determines the designated channel corresponding to an overlapping area of the at least two distorted images disposed in the developed view of the virtual reality image. 13. The method of claim 12,
Wherein the conversion processing module performs the pre-processing by applying an alpha value to an overlapping area of the two or more distorted images arranged in the developed view of the virtual reality image. 11. The method of claim 10,
Wherein the control module adds at least one stitching module to the stitching pipeline. 11. The method of claim 10,
Wherein the control module selects a specific location in the developed view of the virtual reality image and calls at least one of the distorted image and the image source corresponding to the specific location. 11. The method of claim 10,
Wherein each of the two or more stitching modules comprises:
An image node calling an image source photographed for a specified direction for each of the at least two stitching modules;
A sample node for determining at least one of a position and a distortion shape placing the image source called in the image node;
A distortion node for distorting the image source based on the sample node;
A channel setting node for performing preprocessing of a designated channel for merging the distorted image; And
A correction node processing a product operation of the specified channel for merging the distorted image; And at least one child node of the virtual reality image processing apparatus.

Images