WO2018006669A1

WO2018006669A1 - Parallax fusion method and apparatus

Info

Publication number: WO2018006669A1
Application number: PCT/CN2017/086950
Authority: WO
Inventors: 袁梓瑾
Original assignee: 腾讯科技（深圳）有限公司
Priority date: 2016-07-04
Filing date: 2017-06-02
Publication date: 2018-01-11
Also published as: CN106162143B; CN106162143A

Abstract

The present application relates to a parallax fusion method and apparatus. The method comprises: acquiring a first direction flow field movement relationship and a second direction flow field movement relationship of an overlapping area between a first image and a second image; correcting the first direction flow field movement relationship so as to obtain a corrected first direction flow field movement relationship, and correcting the second direction flow field movement relationship so as to obtain a corrected second direction flow field movement relationship; respectively performing forward and backward transformation conversions on an overlapping area of the first image and an overlapping area of the second image by using the corrected first direction flow field movement relationship and second direction flow field movement relationship; and fusing the overlapping area of the first image and the overlapping area of the second image, which have been subjected to the forward and backward transformation conversions, to obtain a final image of the overlapping area between the first image and the second image.

Description

Parallax fusion method and device

The present application claims the priority of the Chinese Patent Application, the entire disclosure of which is hereby incorporated by reference.

Technical field

The present application relates to the field of image processing, and in particular to a parallax fusion method and apparatus.

background

The 360-degree panoramic video has gradually become one of the main contents of the virtual reality field because it can provide users with a more realistic and immersive viewing experience that is different from the traditional limited-view video. Since the single-lens system that currently collects panoramic video is rare, it is generally composed of video captured by multiple cameras or multiple lens systems.

Technical content

The technical solution of the example of the present application is implemented as follows:

The present application proposes a parallax fusion method and apparatus, which can reduce or eliminate ghosting, ghosting virtual edges, continuous line misalignment fractures and the like due to parallax.

A parallax fusion method comprising:

Obtaining a first direction flow field motion relationship and a second direction flow field motion relationship of the first image and the second image overlap region;

Performing forward and backward deformation transformation on the overlapping area of the first image and the overlapping area of the second image by using the first direction flow field motion relationship and the second direction flow field motion relationship, respectively;

An overlap region and a second image of the first image transformed by forward and backward deformation The overlapping regions are fused to obtain a final image of the first image and the second image overlap region.

The application also provides a parallax fusion device, the device comprising:

One or more memories;

One or more processors; among them,

The one or more memories storing one or more instruction modules configured to be executed by the one or more processors; wherein

The one or more instruction modules include:

a motion relationship acquiring module, configured to acquire a first direction flow field motion relationship and a second direction flow field motion relationship of the first image and the second image overlap region;

a deformation module, configured to perform forward and backward deformation transformation on the overlapping area of the first image and the overlapping area of the second image by using the first direction flow field motion relationship and the second direction flow field motion relationship, respectively;

And a fusion module, configured to fuse the overlap region of the first image transformed by the forward and backward deformations and the overlap region of the second image to obtain a final image of the first image and the second image overlap region.

The present application also proposes a non-transitory computer readable storage medium storing computer readable instructions that cause at least one processor to:

Obtaining a first direction flow field motion relationship and a second direction flow field motion relationship of the first image and the second image overlap region; respectively, using the first direction flow field motion relationship and the second direction flow field motion relationship respectively to overlap the first image The overlapping area of the area and the second image is transformed by forward and backward deformation; the overlapping area of the first image transformed by the forward and backward deformation is merged with the overlapping area of the second image to obtain the first image and the second image The final image of the image overlap area.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the examples of the present application, the drawings used in the description of the examples will be briefly described below. Obviously, the drawings in the following description For some examples of the present application, other drawings may be obtained from those of ordinary skill in the art without departing from the drawings. among them,

1 is a schematic diagram showing the internal structure of an electronic device in an example;

2 is a flow chart of a parallax fusion method in an example;

3 is a structural block diagram of a parallax fusion device in an example;

4 is a block diagram showing the internal structure of a correction module in an example;

Figure 5 is a block diagram showing the internal structure of a deformation module in an example;

Figure 6 is a block diagram showing the internal structure of a fusion module in an example.

Implementation

In the process of implementing the examples of the present invention, the inventors have found that for a panoramic video stitched by a plurality of cameras or a plurality of lens systems, two non-co-optical cameras are obtained from the optical optical perspective principle of the lens. The content imaged by the two-dimensional imaging sensor always has a certain parallax in their common field of view. At different depths, the degree of parallax is different, which ultimately causes the stitched video content to appear visually in the region where parallax exists. Unacceptable defects, such as ghosting, ghosting, broken lines of continuous lines, etc. Even in the manner of "splicing significant feature points in the overlapping regions of the left and right images, and performing left and right matching, and then solving a deformation function, so that the feature points of the left and right image matching coincide, and the image deformation cost is minimal", although it can be made The main objects in the overlap area are roughly coincident, but the parallax can not be completely solved, and there are still problems such as ghosting virtual edges and continuous lines being broken.

In order to make the objects, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the accompanying drawings and examples. It is understood that the specific examples described herein are merely illustrative of the application and are not intended to be limiting.

It will be understood that the terms "first", "second" and the like, as used herein, may be used to describe various elements, but these elements are not limited by these terms. These terms are only used for Distinguish the first component from another component. For example, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client, without departing from the scope of the present application. Both the first client and the second client are clients, but they are not the same client.

FIG. 1 is a schematic diagram showing the internal structure of an electronic device in an example. As shown in FIG. 1, the electronic device includes a processor connected via a system bus, a non-volatile storage medium, an internal memory, a network interface, a display screen, and an input device. The non-volatile storage medium of the electronic device stores an operating system, and further includes a parallax fusion device, and the parallax fusion device is used to implement a parallax fusion method. The processor is used to provide computing and control capabilities to support the operation of the entire terminal. An internal memory in an electronic device provides an environment for operation of a parallax fusion device in a non-volatile storage medium, the internal memory being storable with computer readable instructions that, when executed by the processor, can cause The processor performs a parallax fusion method. The network interface is used to communicate with other devices, and the like. The display screen of the electronic device may be a liquid crystal display or an electronic ink display screen, and the input device may be a touch layer covered on the display screen, or may be a button, a trackball or a touchpad provided on the outer casing of the electronic device, or may be An external keyboard, trackpad, or mouse. The electronic device can be a cell phone, a personal computer, a tablet, a personal digital assistant, a wearable device, or a server. A person skilled in the art can understand that the structure shown in FIG. 1 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may be It includes more or fewer components than those shown in the figures, or some components are combined, or have different component arrangements.

2 is a flow chart of a parallax fusion method in an example. As shown in FIG. 2, a parallax fusion method is implemented on an electronic device, including:

Step 202: Acquire a first direction flow field motion relationship and a second direction flow field motion relationship of the first image and the second image overlap region.

The captured panoramic video image is generally captured by multiple cameras or multiple lens systems. The video stitching is to stitch the first image and the second image. There is a partial overlap when the first image and the second image are spliced, that is, an overlapping area of the first image and the second image. A portion of the overlapping area of the first image and the second image belonging to the first image is referred to as an overlapping area of the first image, and a portion of the overlapping area of the first image and the second image belonging to the second image is referred to as an overlap of the second image Area. Each pixel point coordinate of the overlapping area of the first image and the second image, each pixel point coordinate of the overlapping area of the first image, and each pixel point coordinate of the overlapping area of the second image are the same.

The first direction flow field motion relationship of the first image and the second image overlap region is a flow field motion relationship from the first image to the second image direction.

The second direction flow field motion relationship of the first image and the second image overlap region is a flow field motion relationship from the second image to the first image direction.

For example, if the overlapping regions of the first image and the second image overlap in the horizontal direction, the flow field motion relationship from left to right is Flow _l2r , and the flow motion relationship from right to left is Flow _r2l . L2r is the abbreviation of left2right, the mark is from left to right, r2l is the abbreviation of right2left, and the mark is from right to left.

The pixel-by-pixel dense matching relationship of the overlapping area of the first image and the overlapping area of the second image may be calculated according to the classical flow field algorithm, that is, the first direction flow field motion relationship and the second direction flow field motion relationship.

Step 204: correcting the motion relationship of the first direction flow field to obtain a corrected motion relationship of the first direction flow field, and correcting the motion relationship of the flow field in the second direction to obtain a corrected motion relationship of the second direction flow field.

In this example, the motion relationship of the first direction flow field is corrected to obtain a modified first direction flow field motion relationship, including: performing a transition to the non-overlapping region of the first image in the first direction flow field motion relationship The convergence correction obtains the corrected first-direction flow field motion relationship.

Specifically, the non-overlapping area of the first image means that the first image does not overlap with the second image. Area. The transitional joint correction is to smooth the transition between the overlap region and the non-overlapping region image after deformation.

Correcting the motion relationship of the flow field in the second direction to obtain the corrected motion relationship of the motion field in the second direction, comprising: performing a transitional connection correction on the motion relationship of the flow field in the second direction with the non-overlapping region of the second image, The corrected second-direction motion flow field motion relationship.

Specifically, the non-overlapping area of the second image refers to an area in the second image that does not overlap with the first image.

Step 206: Perform a forward and backward deformation transformation on the overlapping area of the first image and the overlapping area of the second image by using the corrected first direction flow field motion relationship and the second direction flow field motion relationship, respectively.

In this example, the corrected first direction flow field motion relationship and the second direction flow field motion relationship are respectively used to perform forward and backward deformation transformation on the overlap region of the first image and the overlap region of the second image to obtain the first image. a forward-transformed transformed image of the overlapping region, a backward-transformed transformed image of the overlapping region of the first image, a forward-transformed transformed image of the overlapping region of the second image, and a backward-transformed transformed image of the overlapping region of the second image A deformation transform image.

If the modified first-direction flow field motion relationship is used for deformation transformation, the first image is a reference image and the second image is a target image.

If the modified second-direction flow field motion relationship is used for deformation transformation, the second image is a reference image, and the first image is a target image.

The forward deformation changes to transform the image from the reference image to the target image according to the input flow field data, and the backward deformation changes into a deformation transformation from the target image to the reference image.

It can be understood that the purpose of correcting the first direction flow field motion relationship and the second direction flow field motion relationship in the above step 204 is to obtain a better connection effect between the non-overlapping area and the overlap area, in practice step 204 It doesn't have to be done, it can't be done. The first direction flow field motion relationship and the second direction flow field motion relationship obtained in step 202 are corrected, so that in step 206, the first direction flow field motion relationship and the second direction flow field obtained in step 202 can be directly adopted. The motion relationship may be performed by performing forward and backward deformation transformation on the overlapping area of the first image and the overlapping area of the second image.

Step 208: merging the overlap region of the first image after the forward and backward deformation transformation and the overlap region of the second image to obtain a final image of the first image and the second image overlap region.

In this example, the four transformed transformed images are fused to obtain a final image of the first image and the second image overlap region.

In the above parallax fusion method, by acquiring the first direction flow field motion relationship and the second direction flow field motion relationship of the first image and the second image overlap region, the first direction flow field motion relationship and the second direction flow field motion relationship are used. The overlapping area of the first image and the second image overlapping area perform forward and backward deformation transformation, and the overlapping area of the first image transformed by the forward and backward deformation and the overlapping area of the second image are merged to obtain a first image. And the final image of the overlapping area of the second image is transformed and re-fused by using the occlusion area of the flow field data including the transition of the different depth planes to obtain the final image. The images of the overlapping regions obtained by the deformation transformation have different 瑕疵, and the 瑕疵 regions obtained by the forward and backward transformations are just complementary, so the final image obtained after the fusion reduces or avoids the continuation due to the parallax. Breaking of lines, ghosting of virtual edges, ghosting, etc.

In an example, if the first image and the second image are overlapped in the horizontal direction, the motion relationship of the first direction flow field is modified to be connected with the non-overlapping region of the first image, and the corrected first direction flow field is obtained. The motion relationship includes: multiplying the horizontal direction component of the first direction flow field motion relationship by the first coefficient factor including the horizontal pixel width coordinate of the overlapping area and the horizontal pixel width relationship of the overlapping area to obtain the corrected first direction flow field The horizontal direction component of the motion relationship and the vertical direction component of the motion relationship of the first direction flow field are used as the vertical direction component of the corrected first direction flow field motion relationship.

In this example, the first coefficient factor including the pixel point horizontal coordinate of the overlap region and the horizontal pixel width relationship of the overlap region may be a ratio between a pixel point horizontal coordinate of the overlap region and a horizontal pixel width of the overlap region, or an overlap region. The ratio between the horizontal coordinate of the pixel and the horizontal pixel width of the overlap minus one.

The first coefficient factor is the ratio between the horizontal coordinate of the pixel point of the overlap region and the horizontal pixel width of the overlap region minus one.

When calculated by the formula (1).

In formula (1),

a horizontal component representing the corrected first-order flow field motion relationship,

a vertical component representing the motion relationship of the corrected first direction flow field,

The horizontal direction component representing the motion relationship of the flow direction in the first direction before correction,

A vertical direction component representing a motion relationship of the first direction flow field before correction, N is a horizontal pixel width of the first image and the second image overlap region, x is a horizontal coordinate of the image pixel point, and y is a vertical coordinate of the image pixel point, The superscript h represents the horizontal direction component, the superscript v represents the vertical direction component, and l2r is the abbreviation of left2right, marking the direction from the first image to the second image (ie, left to right direction).

In other examples, the first coefficient factor is the ratio between the horizontal coordinate of the pixel of the overlap region and the horizontal pixel width of the overlap region.

When using equation (2).

In an example, if the first image and the second image are overlapped in the horizontal direction, the motion relationship of the flow field in the second direction is modified to be in transition with the non-overlapping region of the second image, and the corrected second direction motion flow is obtained. The field motion relationship includes: multiplying a horizontal direction component of the second direction flow field motion relationship by a second coefficient factor including a horizontal pixel width coordinate of the overlapping area and a horizontal pixel width relationship of the overlapping area to obtain a corrected second direction flow The horizontal direction component of the field motion relationship and the vertical direction of the motion relationship of the second direction flow field The quantity is used as the vertical direction component of the corrected flow direction relationship in the second direction.

In this example, the second coefficient factor including the pixel point horizontal coordinate of the overlap region and the horizontal pixel width relationship of the overlap region may be a difference between a predetermined constant and a ratio between a pixel point horizontal coordinate of the overlap region and a horizontal pixel width of the overlap region. The value relationship is either a difference relationship between a predetermined constant and a ratio between a pixel point horizontal coordinate of the overlap region and a horizontal pixel width of the overlap region minus one.

The second coefficient factor is a difference relationship between a predetermined constant and a ratio between a pixel point horizontal coordinate of the overlap region and a horizontal pixel width of the overlap region minus one.

When using equation (3).

In formula (3),

a horizontal component representing a corrected motion relationship of the flow field in the second direction,

a vertical component representing a corrected motion relationship of the flow field in the second direction,

The vertical direction component representing the motion relationship of the flow direction in the second direction before correction, N is the horizontal pixel width of the overlap region of the first image and the second image, x is the horizontal coordinate of the pixel of the image, and y is the vertical coordinate of the pixel of the image. The superscript h represents the horizontal direction component, the superscript v represents the vertical direction component, and r2l is the abbreviation of right2left, marking the direction from the second image to the first image (ie, right to left direction). In the above formula (3), the predetermined constant is 1, and of course, the predetermined constant is not limited to 1.

In other examples, the second coefficient factor is a difference relationship between a predetermined constant and a ratio between a pixel point horizontal coordinate of the overlap region and a horizontal pixel width of the overlap region, that is,

Calculated using equation (4).

In other examples, the first coefficient factor is

The second coefficient factor is

Or, the first coefficient factor can be

The second coefficient factor is

Not limited to this.

In one example, if the first image and the second image overlap in a vertical direction, the transition relationship between the first direction flow field motion relationship and the non-overlapping region of the first image is corrected, and the corrected first direction flow is obtained. The field motion relationship includes: multiplying a vertical direction component of the first direction flow field motion relationship by a third coefficient factor including a vertical coordinate of a pixel point of the overlap region and a vertical pixel height relationship of the overlap region to obtain a corrected first direction The vertical direction component of the flow field motion relationship and the horizontal direction component of the first direction flow field motion relationship are used as the horizontal direction component of the corrected first direction flow field motion relationship. In the above formula (4), the predetermined constant is 1, and of course the predetermined constant is not limited to 1.

In this example, the third coefficient factor may be a ratio relationship between the vertical coordinates of the pixel points of the overlap region and the vertical pixel height of the overlap region, or between the vertical coordinates of the pixel points of the overlap region and the vertical pixel height of the overlap region minus one. Ratio relationship.

The third coefficient factor may be a ratio relationship between the vertical coordinate of the pixel of the overlap region and the vertical pixel height of the overlap region minus one.

Calculated using equation (5).

In formula (5),

A vertical direction component representing a motion relationship of the first direction flow field before correction, M is a vertical pixel height of the first image and the second image overlap region, x is a horizontal coordinate of the image pixel point, and y is a vertical coordinate of the image pixel point, The superscript h represents the horizontal direction component, the superscript v represents the vertical direction component, and u2d is an abbreviation of up2down, marking the direction from the first image to the second image (ie, the top to bottom direction).

The third coefficient factor may be a ratio relationship between the vertical coordinates of the pixel points of the overlap region and the vertical pixel height of the overlap region.

Calculated using equation (6).

In an example, if the first image and the second image overlap in a vertical direction, the motion relationship of the second direction flow field is modified to be connected with the non-overlapping region of the second image, and the corrected second direction motion is obtained. The flow field motion relationship includes: multiplying a vertical direction component of the second direction flow field motion relationship by a fourth coefficient factor of a vertical relationship between a pixel point vertical coordinate of the overlap region and a vertical pixel height of the overlap region, and a second corrected coefficient The vertical direction component of the directional flow field motion relationship and the horizontal direction component of the second direction flow field motion relationship are used as the horizontal direction component of the corrected second direction flow field motion relationship.

In this example, the fourth coefficient factor may be a difference relationship between a predetermined constant and a ratio of a vertical coordinate of a pixel of the overlap region to a vertical pixel height of the overlap region, or may be a predetermined constant and a vertical coordinate of the pixel of the overlap region and overlap The difference between the vertical pixel height of the region minus the ratio between the values.

The fourth coefficient factor is a difference relationship between a predetermined constant and a ratio between a vertical coordinate of a pixel of the overlap region and a vertical pixel height of the overlap region minus one.

Calculated using equation (7).

In formula (7),

A vertical direction component representing a motion relationship of the first direction flow field before correction, M is a vertical pixel height of the first image and the second image overlap region, x is a horizontal coordinate of the image pixel point, and y is a vertical coordinate of the image pixel point, The superscript h represents the horizontal direction component, the superscript v represents the vertical direction component, and d2u is the abbreviation of down2up, marking the direction from the second image to the first image (ie, the bottom-up direction). In the above formula (7), the predetermined constant is 1, and of course the predetermined constant is not limited to 1.

The fourth coefficient factor is a difference relationship between a predetermined constant and a ratio between a vertical coordinate of a pixel point of the overlap region and a vertical pixel height of the overlap region.

When using equation (8).

In other examples, the third coefficient factor can be

The fourth coefficient factor can be

Alternatively, the third coefficient factor can be

The fourth coefficient factor can be

Not limited to this. In the above formula (8), the predetermined constant is 1, and of course, the predetermined constant is not limited to 1.

In one example, the modified first direction flow field motion relationship and the second direction flow field motion relationship are respectively used to perform forward and backward deformation transformations on the overlap region of the first image and the overlap region of the second image, including:

(1) performing a forward deformation transformation on the overlapping region of the first image by using the corrected first-direction flow field motion relationship to obtain a forward-transformed transformed image of the overlapping region of the first image;

(2) performing a backward deformation transformation on the overlapping region of the first image by using the corrected flow direction relationship in the second direction to obtain a backward deformation transformation image of the overlapping region of the first image;

(3) performing a forward deformation transformation on the overlapping region of the second image by using the corrected second-direction flow field motion relationship to obtain a forward-transformed transformed image of the overlapping region of the second image;

(4) Performing a backward deformation transformation on the overlapping region of the second image by using the corrected first-direction flow field motion relationship to obtain a backward-transformed transformed image of the overlapping region of the second image.

It should be noted that if the modified first-direction flow field motion relationship is used for deformation transformation, the first image is a reference image and the second image is a target image; if the modified second-direction flow field motion relationship is used for deformation transformation The second image is a reference image, and the first image is a target image.

The forward deformation is transformed into a deformation transformation from the reference image to the target image, and the backward deformation is changed. Change to the deformation transformation from the target image to the reference image.

Taking the first image and the second image overlapping in the horizontal direction as an example, the overlapping area of the first image is labeled I _L , and the overlapping area of the second image is labeled I _R , according to the modified first direction flow field motion relationship rFlow ( x, y) _l2r and the modified second-direction flow field motion relationship rFlow(x, y) _r2l transforms I _L and I _R to obtain the following data:

R'=FL2R=Forwardwarp(rFlow _l2r , I _L )

R”=BL2R=Backwardwarp(rFlow _r2l , I _L )

L'=FR2L=Forwardwarp(rFlow _r2l , I _R )

L"=BR2L=Backwardwarp(rFlow _l2r , I _R )

Where R' is a forward-transformed transformed image of the overlapping region I _L of the first image, R" is a backward-transformed transformed image of the overlapping region I _L of the first image, and L' is an overlapping region I _R of the second image The forward deformation transforms the image, L" is the backward deformation transformed image of the overlapping region I _R of the second image, Forwardwarp represents the forward deformation transformation, and Backwardwarp represents the backward deformation transformation.

Forwardwarp (flow, I) refers to the forward deformation transformation of the image I from the reference image to the target image according to the input flow field data flow. Because the flow field data flow includes occlusion areas due to different depth plane transitions, R' and L' obtained by Forwardwarp transformation will have a first type of 瑕疵 area such as a void area and an overlap area, and the first type of 瑕疵 area will be Marked as region ^hole .

Backwardwarp (flow, I) refers to the backward deformation transformation of the image I from the target image to the reference image according to the input flow field data flow. Since the flow field data flow includes occlusion areas due to transitions of different depth planes, the R" and L" obtained by the Backwardwarp transformation no longer have holes, but a ghosting phenomenon occurs.

The four overlapping area images obtained by the deformation transformation have R', L', R" and L", each with a void area, an overlapping area, an overlapping area, etc., and the 瑕疵 area obtained by the Backwardwarp transformation and the Forwardwarp transformation happens to have Complementarity, so they are sourced to obtain the final image of the overlapping region.

In one example, the overlapping regions of the first image and the overlapping regions of the second image after the forward and backward deformation are merged to obtain a final image of the first image and the second image overlapping region, including:

Combining the forward deformation transformed image and the backward deformation transformed image of the overlapping region of the first image to obtain a first fused image;

The forward-transformed transformed image and the backward-transformed transformed image of the overlapping region of the second image are fused to obtain a second fused image.

The calculation formula is as shown in formula (9).

among them,

For the first fused image,

For the second fused image. Fusion(x,y) is a fusion function.

Further, when the pixel point horizontal coordinate value of the overlapping area is smaller than the intermediate pixel point horizontal coordinate value of the overlapping area, the final image of the first image and the second image overlapping area is the first fused image;

When the pixel point horizontal coordinate value of the overlapping area is equal to or larger than the intermediate pixel point horizontal coordinate value of the overlapping area, the final image of the first image and the second image overlapping area is the second fused image.

Specifically, the calculation formula is as shown in the formula (10).

In formula (10), I _overlap represents the final image. Mid indicates the intermediate separation position of the overlap region (the horizontal coordinate value of the intermediate pixel of the overlap region), which can be obtained by taking the center line and obtaining other methods.

In one example, the forward transformed transformed image and the backward transformed transformed image of the overlapping region of the first image are merged to obtain a first fused image, including:

If the pixel of the overlapping area does not belong to the first type of 瑕疵 region, the first fused image is a forward morphing transformed image of the overlapping region of the first image;

If the pixel points of the overlapping area belong to the first type of 瑕疵 region, the first fused image is a backward morphing transformed image of the overlapping region of the first image.

In this example, an implementation of the fusion(x,y) function can be as follows (11):

In the formula (11), R'(x, y) is R', and R"(x, y) is R".

In one example, the forward-transformed transformed image and the backward-transformed transformed image of the overlapping region of the second image are merged to obtain a second fused image, including:

If the pixel of the overlapping area does not belong to the first type of 瑕疵 region, the second fused image is a forward morphing transformed image of the overlapping region of the second image;

If the pixel points of the overlapping area belong to the first type of 瑕疵 region, the second fused image is a backward morphing transformed image of the overlapping region of the second image.

In this example, an implementation of the fusion(x,y) function can be as follows (12):

In the formula (12), L'(x, y) is L', and L"(x, y) is L".

The first type of 瑕疵 region and the non-first 瑕疵 region are distinguished from the pixel points, and the pixels belonging to the first 瑕疵 region are replaced by the backward deformation transformation image, and the backward deformation transformation image has no first 瑕疵 region, which can be eliminated. The first type of 瑕疵将将像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素像素, ghosting virtual edges, continuous lines, broken joints and other stitching.

It should be noted that, in the above example, the process of overlapping the first image and the second image in the horizontal direction is used, and the first image and the second image may be overlapped and merged in the vertical direction, and the fusion process is the same.

FIG. 3 is a structural block diagram of a parallax fusion device in an example. As shown in FIG. 3, a disparity fusion device is a device configured to implement a parallax fusion method, and the device includes:

One or more memories;

One or more processors; among them,

The one or more instruction modules include: a motion relationship acquisition module 310, a correction module 320, a deformation module 330, and a fusion module 340. among them:

The motion relationship acquiring module 310 is configured to acquire a first direction flow field motion relationship and a second direction flow field motion relationship of the first image and the second image overlap region.

The correction module 320 is configured to correct the first direction flow field motion relationship to obtain a corrected first direction flow field motion relationship, and modify the second direction flow field motion relationship to obtain a corrected second direction flow. Field sports relationship.

The deformation module 330 is configured to perform forward and backward deformation transformation on the overlapping area of the first image and the overlapping area of the second image by using the corrected first direction flow field motion relationship and the second direction flow field motion relationship, respectively.

The fusion module 340 is configured to fuse the overlap region of the first image transformed by the forward and backward deformations and the overlap region of the second image to obtain a final image of the first image and the second image overlap region.

Similar to the method, the function of the above-mentioned correction module 320 is to obtain a better connection effect between the non-overlapping area and the overlapping area. In fact, the correction module 320 is not an essential instruction module, so that the deformation module 330 directly acquires the motion relationship module. The first direction flow field motion relationship and the second direction flow field motion relationship acquired by 310 may be forward and backward deformation transformation of the overlap region of the first image and the overlap region of the second image.

Figure 4 is a block diagram showing the internal structure of the correction module in one example. As shown in Figure 4, the correction Module 320 includes a first modification unit 3202 and a second modification unit 3204. among them:

The first correcting unit 3202 is configured to perform a transition joint correction with the non-overlapping region of the first image in the first direction flow field motion relationship, to obtain a corrected first direction flow field motion relationship;

The second correcting unit 3204 is configured to perform a transitional connection correction with the non-overlapping region of the second image in the second direction flow field motion relationship, and obtain a corrected second direction motion flow field motion relationship.

In one example, if the first image and the second image are overlapped in the horizontal direction, the first correcting unit 3202 is further configured to multiply the horizontal direction component of the first direction flow field motion relationship by the pixel point horizontal coordinate including the overlapping area. a first coefficient factor in relation to a horizontal pixel width of the overlap region obtains a corrected horizontal direction component of the first direction flow field motion relationship, and a vertical direction component of the first direction flow field motion relationship as the corrected first The vertical component of the directional flow field motion relationship;

The second correcting unit 3204 is further configured to multiply the horizontal direction component of the second direction flow field motion relationship by a second coefficient factor including a pixel point horizontal coordinate of the overlapping area and a horizontal pixel width relationship of the overlapping area. The horizontal direction component of the two-direction flow field motion relationship and the vertical direction component of the second direction flow field motion relationship are used as the vertical direction component of the corrected second-direction flow field motion relationship.

In one example, if the first image and the second image overlap in a vertical direction, the first correcting unit 3202 is further configured to multiply the vertical direction component of the first direction flow field motion relationship by the pixel point vertical coordinate including the overlap region. a third coefficient factor relating to a vertical pixel height relationship of the overlap region obtains a corrected vertical direction component of the first direction flow field motion relationship, and a horizontal direction component of the first direction flow field motion relationship as the corrected first The horizontal component of the directional flow field motion relationship;

The second correcting unit 3204 is further configured to multiply the vertical direction component of the second direction flow field motion relationship by the pixel vertical coordinate of the overlapping area and the vertical pixel of the overlapping area. The fourth coefficient factor of the degree relationship obtains the corrected vertical direction component of the second direction flow field motion relationship, and the horizontal direction component of the second direction flow field motion relationship is used as the corrected second direction flow field motion relationship. Horizontal component.

Figure 5 is a block diagram showing the internal structure of the deformation module in one example. As shown in FIG. 5, the deformation module 330 includes a first deformation unit 3302, a second deformation unit 3304, a third deformation unit 3306, and a fourth deformation unit 3308. among them:

The first deformation unit 3302 is configured to perform forward deformation transformation on the overlapping region of the first image by using the corrected first direction flow field motion relationship to obtain a forward deformation transformation image of the overlapping region of the first image;

The second deformation unit 3304 is configured to perform backward deformation transformation on the overlapping region of the first image by using the corrected second-direction flow field motion relationship to obtain a backward deformation transformation image of the overlapping region of the first image;

The third deforming unit 3306 is configured to perform forward deformation transformation on the overlapping region of the second image by using the corrected second-direction flow field motion relationship to obtain a forward-transformed transformed image of the overlapping region of the second image;

The fourth deforming unit 3308 is configured to perform backward deformation transformation on the overlapping region of the second image by using the corrected first direction flow field motion relationship to obtain a backward deformation transformed image of the overlapping region of the second image.

Figure 6 is a block diagram showing the internal structure of a fusion module in an example. As shown in FIG. 6, the fusion module 340 includes a first fusion unit 3402 and a second fusion unit 3404. among them:

The first merging unit 3402 is configured to fuse the forward-transformed transformed image and the backward-transformed transformed image of the overlapping region of the first image to obtain a first fused image;

The second merging unit 3404 is configured to fuse the forward-transformed transformed image and the backward-transformed transformed image of the overlapping region of the second image to obtain a second fused image;

When the pixel point horizontal coordinate value of the overlapping area is smaller than the intermediate pixel point horizontal coordinate value of the overlapping area, the final image of the first image and the second image overlapping area is the First fused image;

In one example, if the pixel points of the overlap region do not belong within the first type of 瑕疵 region, the first fused image is a forward morphing transformed image of the overlap region of the first image, and if the overlap The pixel of the region belongs to the first type of 瑕疵 region, and the first fused image is a backward deformation transformed image of the overlapping region of the first image;

If the pixel of the overlapping area does not belong to the first type of area, the second fused image is a forward-transformed transformed image of the overlapping area of the second image, and if the pixel of the overlapping area belongs to In the first type of 瑕疵 region, the second fused image is a backward morphing transformed image of the overlapping region of the second image.

One of ordinary skill in the art can understand that all or part of the process of implementing the above example method can be completed by a computer program to instruct related hardware, and the program can be stored in a non-volatile computer readable storage medium. The program, when executed, may include a flow of an instance of each of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or the like.

An example of the present application provides a non-transitory computer readable storage medium storing computer readable instructions, which may cause at least one processor to perform the parallax fusion method proposed in the above example, for example, acquiring a first image and a second image. The first direction flow field motion relationship and the second direction flow field motion relationship in the overlap region; the first direction flow field motion relationship and the second direction flow field motion relationship respectively are used to overlap the first image overlap region and the second image overlap region respectively Forward and backward deformation transformation is performed; the overlapping region of the first image transformed by the forward and backward deformation and the overlapping region of the second image are merged to obtain a final image of the first image and the second image overlapping region.

The above examples only express several embodiments of the present application, and the description thereof is more specific. And the detailed description is not to be construed as limiting the scope of the claims. It should be noted that a number of variations and modifications may be made by those skilled in the art without departing from the spirit and scope of the present application. Therefore, the scope of the invention should be determined by the appended claims. .

Claims

A parallax fusion method comprising:

Obtaining a first direction flow field motion relationship and a second direction flow field motion relationship of the first image and the second image overlap region;

Performing forward and backward deformation transformation on the overlapping area of the first image and the overlapping area of the second image by using the first direction flow field motion relationship and the second direction flow field motion relationship, respectively;

The overlapping region of the first image transformed by the forward and backward deformation and the overlapping region of the second image are fused to obtain a final image of the first image and the second image overlapping region.
The method according to claim 1, wherein said adopting a first direction flow field motion relationship and a second direction flow field motion relationship respectively perform forward and backward directions of the overlap region of the first image and the overlap region of the second image Before the transformation, the method further includes:

Correcting the first direction flow field motion relationship to obtain a corrected first direction flow field motion relationship, and correcting the second direction flow field motion relationship to obtain a corrected second direction flow field motion relationship;

Correspondingly, the first direction flow field motion relationship and the second direction flow field motion relationship respectively perform forward and backward deformation transformation on the overlap region of the first image and the overlap region of the second image, respectively:

The modified first direction flow field motion relationship and the second direction flow field motion relationship are used to perform forward and backward deformation transformation on the overlap region of the first image and the overlap region of the second image, respectively.
The method according to claim 2, wherein the correcting the first direction flow field motion relationship to obtain the corrected first direction flow field motion relationship comprises:

Performing a transitional joint correction with the non-overlapping region of the first image in the first direction flow field motion relationship, and obtaining a corrected first direction flow field motion relationship;

Correcting the second-direction flow field motion relationship to obtain a corrected second direction Sports flow field motion relationships, including:

Performing a transitional joint correction with the non-overlapping region of the second image in the second-direction flow field motion relationship, and obtaining a modified second-direction motion flow field motion relationship.
The method according to claim 3, wherein if the first image and the second image are overlapped in a horizontal direction, the moving relationship between the first direction flow field and the non-overlapping region of the first image is corrected, The corrected first-order flow field motion relationship is obtained, including:

Multiplying the horizontal direction component of the first direction flow field motion relationship by a first coefficient factor including a pixel point horizontal coordinate of the overlap region and a horizontal pixel width relationship of the overlap region to obtain a corrected first direction flow field motion relationship level a direction component, and a vertical direction component of the motion relationship of the first direction flow field as a vertical direction component of the corrected first direction flow field motion relationship;

Performing a transitional joint correction relationship with the non-overlapping region of the second image in the second direction flow field motion relationship, and obtaining a modified second direction motion flow field motion relationship, including:

Multiplying the horizontal direction component of the second direction flow field motion relationship by a second coefficient factor including a pixel point horizontal coordinate of the overlap region and a horizontal pixel width relationship of the overlap region to obtain a corrected horizontal flow field motion relationship level The direction component, and the vertical direction component of the motion relationship of the flow direction in the second direction are used as the vertical direction component of the motion relationship of the corrected second direction flow field.
The method according to claim 3, wherein if the first image and the second image overlap in a vertical direction, the moving relationship between the first direction flow field and the non-overlapping region of the first image is corrected. The corrected first-order flow field motion relationship is obtained, including:

Multiplying the vertical direction component of the first direction flow field motion relationship by a third coefficient factor including a vertical coordinate of a pixel point of the overlap region and a vertical pixel height relationship of the overlap region to obtain a vertical relationship of the corrected first direction flow field motion relationship Direction component, and the first The horizontal direction component of the directional flow field motion relationship is used as the horizontal direction component of the corrected first direction flow field motion relationship;

Performing a transitional joint correction relationship with the non-overlapping region of the second image in the second direction flow field motion relationship, and obtaining a modified second direction motion flow field motion relationship, including:

Multiplying the vertical direction component of the second-direction flow field motion relationship by a fourth coefficient factor including a vertical relationship between a pixel point vertical coordinate of the overlap region and a vertical pixel height of the overlap region, and obtaining a vertical relationship of the corrected second-direction flow field motion relationship The directional component and the horizontal component of the flow relationship in the second direction are used as the horizontal component of the corrected second-direction flow field motion relationship.
The method according to claim 2, wherein said adopting the corrected first direction flow field motion relationship and the second direction flow field motion relationship respectively forward the overlap region of the first image and the overlap region of the second image And backward deformation transformations, including:

Performing a forward deformation transformation on the overlapping region of the first image by using the corrected first-direction flow field motion relationship to obtain a forward-transformed transformed image of the overlapping region of the first image;

The modified second-direction flow field motion relationship is used to perform backward deformation transformation on the overlapping area of the first image to obtain a backward deformation transformed image of the overlapping area of the first image;

Performing a forward deformation transformation on the overlapping region of the second image by using the corrected second-direction flow field motion relationship to obtain a forward-transformed transformed image of the overlapping region of the second image;

The modified first-order flow field motion relationship is used to perform backward deformation transformation on the overlapping region of the second image to obtain a backward-transformed transformed image of the overlapping region of the second image.
The method according to claim 6, wherein the overlapping region of the first image subjected to the forward and backward deformation transformation and the overlapping region of the second image are fused to obtain a final image of the overlap region of the first image and the second image ,include:

Combining the forward deformation transformed image and the backward deformation transformed image of the overlapping region of the first image to obtain a first fused image;

Forward deformation transformed image and backward deformation transformed image of overlapping region of the second image Performing fusion to obtain a second fused image;

When the pixel point horizontal coordinate value of the overlapping area is smaller than the intermediate pixel point horizontal coordinate value of the overlapping area, the final image of the first image and the second image overlapping area is the first fused image;

When the pixel point horizontal coordinate value of the overlapping area is equal to or larger than the intermediate pixel point horizontal coordinate value of the overlapping area, the final image of the first image and the second image overlapping area is the second fused image.
The method according to claim 7, wherein the fusing the forward-transformed transformed image and the backward-transformed transformed image of the overlapping region of the first image to obtain the first fused image comprises:

If the pixel of the overlapping area does not belong to the first type of 瑕疵 region, the first fused image is a forward morphing transformed image of the overlapping region of the first image;

If the pixel points of the overlap region belong to the first type of 瑕疵 region, the first fused image is a backward deformation transformed image of the overlap region of the first image;

And combining the forward-transformed transformed image and the backward-transformed transformed image of the overlapping region of the second image to obtain a second fused image, including:

If the pixel of the overlapping area does not belong to the first type of 瑕疵 region, the second fused image is a forward morphing transformed image of the overlapping region of the second image;

If the pixel points of the overlapping area belong to the first type of 瑕疵 region, the second fused image is a backward morphing transformed image of the overlapping region of the second image.
A parallax fusion device comprising:

One or more memories;

One or more processors; among them,

The one or more memories storing one or more instruction modules configured to be executed by the one or more processors; wherein

The one or more instruction modules include:

a motion relationship acquiring module, configured to acquire a first direction flow field motion relationship and a second direction flow field motion relationship of the first image and the second image overlap region;

a deformation module, configured to perform forward and backward deformation transformation on the overlapping area of the first image and the overlapping area of the second image by using the first direction flow field motion relationship and the second direction flow field motion relationship, respectively;

And a fusion module, configured to fuse the overlap region of the first image transformed by the forward and backward deformations and the overlap region of the second image to obtain a final image of the first image and the second image overlap region.
The apparatus of claim 9 wherein said apparatus further comprises:

a correction module, configured to modify the motion relationship of the first direction flow field after the deformation module performs forward and backward deformation transformation on the overlap region of the first image and the overlap region of the second image a first direction flow field motion relationship, and a correction of the second direction flow field motion relationship to obtain a corrected second direction flow field motion relationship;

Correspondingly, the deformation module is configured to perform forward and backward directions on the overlapping area of the first image and the overlapping area of the second image by using the corrected first direction flow field motion relationship and the second direction flow field motion relationship, respectively. Deformation transformation.
The apparatus of claim 10 wherein said correction module comprises:

a first correcting unit, configured to perform a transitional joint correction with the non-overlapping region of the first image in the first direction flow field motion relationship, to obtain a corrected first direction flow field motion relationship;

The second correcting unit is configured to perform a transitional connection correction with the non-overlapping region of the second image in the second direction flow field motion relationship, and obtain a modified second direction motion flow field motion relationship.
The apparatus according to claim 11, wherein said first correcting unit is further configured to multiply a horizontal direction component of said first direction flow field motion relationship to include if said first image and said second image overlap in a horizontal direction Pixel horizontal coordinate and overlap area of overlapping area The first coefficient factor of the horizontal pixel width relationship obtains the corrected horizontal direction component of the first direction flow field motion relationship, and the vertical direction component of the first direction flow field motion relationship is used as the corrected first direction flow field The vertical component of the motion relationship;

The second correcting unit is further configured to multiply the horizontal direction component of the second direction flow field motion relationship by a second coefficient factor including a pixel point horizontal coordinate of the overlapping area and a horizontal pixel width relationship of the overlapping area. The horizontal direction component of the second direction flow field motion relationship and the vertical direction component of the second direction flow field motion relationship are used as the vertical direction component of the corrected second direction flow field motion relationship.
The apparatus according to claim 11, wherein said first correcting unit is further configured to multiply a vertical direction component of said first direction flow field motion relationship to include if said first image and said second image overlap in a vertical direction a third coefficient factor of a vertical coordinate relationship between a pixel point vertical coordinate of the overlap region and a vertical pixel height of the overlap region obtains a corrected vertical direction component of the first direction flow field motion relationship, and a horizontal direction of the motion relationship of the first direction flow field The component is used as the horizontal direction component of the corrected first direction flow field motion relationship;

The second correcting unit is further configured to multiply the vertical direction component of the second direction flow field motion relationship by a fourth coefficient factor including a vertical coordinate of a pixel point of the overlapping area and a vertical pixel height relationship of the overlapping area. The vertical direction component of the second direction flow field motion relationship and the horizontal direction component of the second direction flow field motion relationship are used as the horizontal direction component of the corrected second direction flow field motion relationship.
The apparatus of claim 10 wherein said deformation module comprises:

a first deformation unit, configured to perform forward deformation transformation on the overlapping area of the first image by using the corrected first direction flow field motion relationship, to obtain a forward deformation transformation image of the overlapping area of the first image;

a second deformation unit, configured to perform backward deformation transformation on the overlapping region of the first image by using the corrected second-direction flow field motion relationship, to obtain a backward deformation transformation image of the overlapping region of the first image;

a third deformation unit, configured to perform forward deformation transformation on the overlapping area of the second image by using the corrected second direction flow field motion relationship, to obtain a forward deformation transformation image of the overlapping area of the second image;

The fourth deformation unit is configured to perform backward deformation transformation on the overlapping region of the second image by using the corrected first direction flow field motion relationship to obtain a backward deformation transformation image of the overlapping region of the second image.
The apparatus of claim 14, wherein the fusion module comprises:

a first merging unit, configured to fuse the forward-transformed transformed image and the backward-transformed transformed image of the overlapping region of the first image to obtain a first fused image;

a second merging unit, configured to fuse the forward-transformed transformed image and the backward-transformed transformed image of the overlapping region of the second image to obtain a second fused image;

When the pixel point horizontal coordinate value of the overlapping area is smaller than the intermediate pixel point horizontal coordinate value of the overlapping area, the final image of the first image and the second image overlapping area is the first fused image;

When the pixel point horizontal coordinate value of the overlapping area is equal to or larger than the intermediate pixel point horizontal coordinate value of the overlapping area, the final image of the first image and the second image overlapping area is the second fused image.
The apparatus according to claim 15, wherein the first fused image is a forward-transformed transformed image of an overlapping region of the first image if a pixel point of the overlapping region does not belong within the first type of 瑕疵 region And if the pixel points of the overlap region belong to the first type of 瑕疵 region, the first fused image is a backward deformation transformed image of the overlap region of the first image;

If the pixel of the overlapping area does not belong to the first type of area, the second fused image is a forward-transformed transformed image of the overlapping area of the second image, and if the pixel of the overlapping area belongs to In the first type of 瑕疵 region, the second fused image is a backward morphing transformed image of the overlapping region of the second image.
A non-transitory computer readable storage medium, wherein computer readable instructions are stored, the at least one processor being operative to:

Obtaining a first direction flow field motion relationship and a second direction flow field motion relationship of the first image and the second image overlap region;

Performing forward and backward deformation transformation on the overlapping area of the first image and the overlapping area of the second image by using the first direction flow field motion relationship and the second direction flow field motion relationship, respectively;

The overlapping region of the first image transformed by the forward and backward deformation and the overlapping region of the second image are fused to obtain a final image of the first image and the second image overlapping region.