JP6350357B2 - Image joining apparatus, image joining method, and image joining program - Google Patents

Description

  The present invention relates to an image joining apparatus, an image joining method, and an image joining program.

  In recent years, a method of moving a camera by hand so as to cover a long subject to shoot a moving image and joining a plurality of frames constituting the moving image to create a panoramic image of a still image has been put into practical use. It is generalized in the form of additional functions of digital cameras. In Patent Document 1, when a panorama image is created by panning or tilting the camera to create a panoramic image, the second and subsequent fields are overwritten by shifting the position corresponding to the motion vector of the image from the previous field. It is described to do.

Japanese Patent Laid-Open No. 06-284321

However, in the panorama image according to the background art, the height of the panorama image is fixed to the height of each frame to be joined. For this reason, the panoramic image does not reflect the slowness of the movement of the camera and is a featureless panoramic image.
The present invention has been made to solve such a problem, and is an image joining apparatus, an image joining method, and an image joining capable of creating a characteristic panoramic image reflecting the steepness and trajectory of the movement of a camera. The purpose is to provide a program.

  Therefore, the present invention provides a vector detection unit that detects a global motion vector between a first frame and a second frame that form a continuous image, and an entire second frame based on the magnitude of the vector. The position corresponding to the vector is shifted with respect to the size changing unit for changing the size of the second texture constituting at least a part and the first texture constituting at least a part of the entire first frame. Thus, an image joining apparatus including a texture overwriting unit that overwrites the second texture whose size has been changed is provided.

  The present invention also includes a step of detecting a global motion vector between a first frame and a second frame constituting a continuous image, and at least one of the entire second frame based on the magnitude of the vector. The step of changing the size of the second texture constituting the part and the first texture constituting at least a part of the entire first frame are shifted by the position corresponding to the vector, And overwriting the second texture with the modified image.

  The present invention also includes a step of detecting a global motion vector between a first frame and a second frame constituting a continuous image, and at least one of the entire second frame based on the magnitude of the vector. The step of changing the size of the second texture constituting the part and the first texture constituting at least a part of the entire first frame are shifted by the position corresponding to the vector, An image joining program for causing a computer to execute the step of overwriting the second texture having been changed is provided.

  According to the present invention, it is possible to provide an image joining apparatus, an image joining method, and an image joining program for creating a characteristic panoramic image that reflects the slowness and trajectory of camera movement.

1 is a block diagram showing a schematic configuration of an image joining apparatus 1 according to an embodiment. It is a figure for demonstrating operation | movement of the size change part 30 which concerns on embodiment. It is another figure for demonstrating operation | movement of the size change part 30 which concerns on embodiment. It is a figure for demonstrating operation | movement of the texture overwriting part 40 which concerns on embodiment. It is a figure which shows the completed panoramic image which concerns on embodiment. It is a flowchart which shows the process sequence of the image joining method which concerns on embodiment.

Hereinafter, an image bonding apparatus and an image bonding method according to the present embodiment will be described with reference to the drawings.
First, the image bonding apparatus according to the present embodiment will be described.
FIG. 1 is a block diagram showing a schematic configuration of an image joining apparatus 1 according to the present embodiment.
The image joining apparatus 1 includes an image input unit 10, a vector detection unit 20, a size changing unit 30, a texture overwriting unit 40, an image output unit 50, and the like.

The image input unit 10 inputs a moving image captured by the user freely moving the camera two-dimensionally by hand or rotating the camera about an axis orthogonal to the central axis of the camera lens.
The vector detection unit 20 detects a global motion vector between the first frame and the second frame constituting the moving image input by the image input unit 10. A known matching evaluation method is used to detect the global motion vector. Further, the first frame and the second frame may be continuous frames or two frames having a predetermined number of frames therebetween. In other words, the first frame and the second frame may be based on the order in which the images were taken, with the first taken image as the first frame and the later taken image as the second frame. One frame and the second frame may be set at intervals of a plurality of frames without being continuous.

  The size changing unit 30 changes the size of the texture constituting the second frame based on the size of the global motion vector detected by the vector detecting unit 20. The texture whose size is changed by the size changing unit 30 may constitute the entire second frame, or may constitute a part of the second frame. Here, as an example, the texture refers to a part of the image of the frame, and as an example, refers to a circular image, a triangle, or other polygonal image obtained by cutting out a part of the image in the frame. In the present embodiment, it is assumed that the texture constitutes the entire second frame, and the size of the texture, that is, the size of the entire frame is a height H and a width W.

At this time, the size changing unit 30 changes the size of the texture in the vertical direction, that is, the height H, based on the size of the horizontal component u of the detected global motion vector V = (u, v).
FIG. 2 is a diagram for explaining the operation of the size changing unit 30 according to the present embodiment.
In the present embodiment, the size changing unit 30 compares the horizontal component u with the horizontal size of the frame, that is, the width W of the frame. Then, according to the change rule, when u is, for example, W / 4 or more and less than W / 2, the texture height H is changed to 0.8H. The change rules are as follows.

When 0 ≦ u <W / 4 H → H (No change)
When W / 4 ≦ u <W / 2 H → 0.8H
When W / 2 ≦ u H → 0.7H

According to this change rule, when the horizontal component u is large, that is, when the movement in the horizontal direction is large, the result is that the height of the texture becomes small.
The size changing unit 30 deletes the vertical ends of the texture, that is, the upper and lower ends of the texture, when changing the height of the texture.

In addition, the size changing unit 30 changes the size of the texture in the horizontal direction, that is, the width W, based on the size of the vertical component v of the detected global motion vector V = (u, v).
FIG. 3 is another diagram for explaining the operation of the size changing unit 30 according to the present embodiment.
In the present embodiment, the size changing unit 30 compares the vertical component v with the vertical size of the frame, that is, the height H of the frame. Then, according to the change rule, the texture width W is changed to 0.7 W when v is H / 2 or more, for example. The change rules are as follows.

When 0 ≦ v <H / 4 W → W (No change)
When H / 4 ≦ v <H / 2 W → 0.8W
When H / 2 ≦ v W → 0.7W

According to this change rule, when the vertical component v is large, that is, when the movement in the vertical direction is large, the result is that the width of the texture becomes small.
The size changing unit 30 deletes the horizontal ends of the texture, that is, the left and right ends of the texture, when changing the width of the texture.

The texture overwriting unit 40 shifts the position by the amount of the global motion vector detected by the vector detection unit 20 with respect to the first frame, and the texture related to the second frame whose size is changed by the size changing unit 30 Is overwritten.
FIG. 4 is a diagram for explaining the operation of the texture overwriting unit 40 according to the present embodiment.
Here, it is assumed that the global motion vector V = (u, v) = (W / 4, H / 2) between the first frame 100 and the second frame (not shown).

The texture overwriting unit 40 moves the first frame 100 in the horizontal and vertical directions by the amount of the global motion vector, that is, (W / 4, H / 2), and has a height of 0.8H. The texture 210 related to the second frame having a width of 0.7 W is overwritten.
At this time, the texture 210 related to the second frame is not reduced in size and changed in size, but is changed in size by deleting the upper, lower, left, and right edges, so that the texture 210 related to the first frame 100 is changed. The texture and the texture related to the second frame overlap and match. That is, the texture according to the first frame 100 and the texture according to the second frame can be joined using a motion vector without causing a shift in the correspondence between the pixels.

Similarly, the texture overwriting unit 40 shifts the position by the amount of the global motion vector between the second frame and the third frame detected by the vector detection unit 20 with respect to the texture related to the second frame. Thus, the size changing unit 30 overwrites the texture related to the third frame whose size has been changed.
Further, the texture overwrite unit 40 operates in the same manner for the fourth and subsequent frames to complete the panoramic image. Of course, if the texture overwrite unit 40 has two or more frames, it can create a panoramic image.

The image output unit 50 outputs the panoramic image created by the texture overwriting unit 40 to the outside.
FIG. 5 is a diagram showing a completed panoramic image according to the present embodiment. FIG. 5 shows that the completed panoramic image is a characteristic image reflecting the slowness and trajectory of the movement of the camera, and the horizontal and vertical dimensions are not necessarily accurate.
It can be seen that the panoramic image created using 13 frames has a texture height or width that is small where the movement of the camera is large, and looks like a brushstroke drawn by the camera.

  Each component realized by the image bonding apparatus 1 can be realized by executing a program under the control of an arithmetic device (not shown) provided in the image bonding apparatus 1 which is a computer, for example. More specifically, the image joining apparatus 1 is realized by loading a program stored in a storage unit (not shown) into a main storage device (not shown) and executing the program under the control of the arithmetic unit. In addition, each component is not limited to being realized by software by a program, and may be realized by any combination of hardware, firmware, and software.

  The above-described program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) are included.

  Further, the program may be supplied to the computer by various types of temporary computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

Next, an image joining method according to the present embodiment will be described.
FIG. 6 is a flowchart showing a processing procedure of the image joining method according to the present embodiment.
When the image joining apparatus 1 starts processing, the image input unit 10 allows the user to freely move the camera two-dimensionally by hand or rotate the camera around an axis orthogonal to the central axis of the camera lens. Each frame of the captured moving image is input (step S10).

Next, the vector detection unit 20 uses the global motion vector V = b between the m-th frame and the n-th frame (m and n are continuous or discontinuous) in each frame constituting the input moving image. (U, v) is detected (step S20).
Next, the size changing unit 30 changes the size (H, W) of the texture constituting at least a part of the nth frame based on the size of each component u, v of the global motion vector V ( Step S30). As described above, the texture size is changed by deleting the upper and lower edges of the texture or the left and right edges of the texture based on the horizontal or vertical component of the global motion vector V and a predetermined change rule. .

Next, the texture overwrite unit 40 shifts the position by the global motion vector V with respect to the m-th frame and overwrites the texture related to the n-th frame whose size has been changed (step S40).
Finally, the texture overwrite unit 40 determines whether the nth frame is the final frame (step S50). When the n-th frame is the final frame (Yes in step S50), the image overwritten by the image output unit 50, that is, the panorama image is output, and the process ends.
When the n-th frame is not the final frame (No in step S50), the process returns to step S20, and the vector detection unit 20 selects the n-th frame among the frames constituting the input moving image. The global motion vector V between the oth frame (n and o are continuous or discontinuous) is detected, and the processing is continued.

  In the image joining apparatus or the image joining method according to the present embodiment, the size of each component u, v of the global motion vector V and the size of the texture after the change are proportional to each other or more than necessary depending on the threshold value. Settings such as restricting the change in the size of the file may be performed. These settings may be adjusted with reference to the completed panoramic image.

  As described above, the image joining apparatus 1 according to the present embodiment includes the vector detection unit 20 that detects the global motion vector V between the first frame 100 and the second frame that form a continuous image; Based on the size of the vector V, the size changing unit 30 for changing the size of the second texture constituting at least part of the entire second frame, and at least part of the entire first frame 100 are configured. A texture overwriting unit 40 is provided that overwrites the second texture 210 whose size has been shifted by shifting the position corresponding to the vector V with respect to the first texture 100.

With such a configuration, it is possible to create a characteristic panoramic image that reflects the slowness and trajectory of the camera movement.
Moreover, in the image joining apparatus 1 according to the present embodiment, it is preferable that the size changing unit 30 does not change the size of the second texture when the size of the vector V is smaller than a predetermined threshold.
Further, in the image joining apparatus 1 according to the present embodiment, the size changing unit 30 changes the vertical size of the second texture based on the horizontal component u of the vector V, so that the vertical component v of the vector is changed. Based on this, it is preferable to change the horizontal size of the second texture.
In the image joining apparatus 1 according to the present embodiment, it is preferable that the size changing unit 30 changes the size of the second texture by deleting the end of the second texture.

In addition, the image joining method according to the present embodiment includes a step S20 of detecting a global motion vector V between the first frame 100 and the second frame constituting a continuous image, and the size of the vector V. Based on step S30 for changing the size of the second texture constituting at least a part of the entire second frame and the first texture 100 constituting at least a part of the entire first frame 100. And step S40 for overwriting the second texture 210 whose size has been changed by shifting the position corresponding to the vector V.
With such a configuration, it is possible to create a characteristic panoramic image that reflects the slowness and trajectory of the camera movement.

In addition, the image joining program according to the present embodiment includes step S20 for detecting the global motion vector V between the first frame 100 and the second frame constituting a continuous image, and the size of the vector V. Based on step S30 for changing the size of the second texture constituting at least a part of the entire second frame and the first texture 100 constituting at least a part of the entire first frame 100. The computer executes step S40 in which the position corresponding to the vector V is shifted and the second texture 210 whose size has been changed is overwritten.
With such a configuration, it is possible to create a characteristic panoramic image that reflects the slowness and trajectory of the camera movement.

1 Image Joining Device 20 Vector Detection Unit 30 Size Changing Unit 40 Texture Overwriting Unit

Claims (6)

A vector detection unit for detecting a global motion vector between a first frame and a second frame constituting a continuous image;
A size changing unit that changes the size of a second texture constituting at least a part of the entire second frame based on the size of the vector;
A texture overwriting unit for overwriting the second texture whose size has been shifted by shifting the position corresponding to the vector with respect to the first texture constituting at least a part of the entire first frame. It equipped with a door,
The size changing unit is an image joining device that does not change the size of the second texture when the size of the vector is smaller than a predetermined threshold . A vector detection unit for detecting a global motion vector between a first frame and a second frame constituting a continuous image;
A size changing unit that changes the size of a second texture constituting at least a part of the entire second frame based on the size of the vector;
A texture overwriting unit for overwriting the second texture whose size has been shifted by shifting the position corresponding to the vector with respect to the first texture constituting at least a part of the entire first frame. It equipped with a door,
The size changing unit
Changing the vertical size of the second texture based on the horizontal component of the vector;
An image joining apparatus that changes a horizontal size of the second texture based on a vertical component of the vector . The resizing unit, by removing the end portion of the second texture image joining apparatus according to claim 1 or claim 2 changes the size of the second texture. Detecting a global motion vector between a first frame and a second frame constituting a continuous image;
Changing the size of a second texture comprising at least a portion of the entire second frame based on the size of the vector;
Overwriting the second texture whose size has been changed by shifting the position corresponding to the vector with respect to the first texture constituting at least a part of the entire first frame. And
In the step of changing the size of the second texture,
An image joining method in which the size of the second texture is not changed when the size of the vector is smaller than a predetermined threshold . Detecting a global motion vector between a first frame and a second frame constituting a continuous image;
Changing the size of a second texture comprising at least a portion of the entire second frame based on the size of the vector;
Overwriting the second texture whose size has been changed by shifting the position corresponding to the vector with respect to the first texture constituting at least a part of the entire first frame. And
In the step of changing the size of the second texture,
Changing the vertical size of the second texture based on the horizontal component of the vector;
An image joining method for changing a horizontal size of the second texture based on a vertical component of the vector . Detecting a global motion vector between a first frame and a second frame constituting a continuous image;
Changing the size of a second texture comprising at least a portion of the entire second frame based on the size of the vector;
A step of overwriting the second texture whose size has been changed by shifting the position corresponding to the vector with respect to the first texture constituting at least a part of the entire first frame. an image joining program to be executed by,
In the step of changing the size of the second texture,
Changing the vertical size of the second texture based on the horizontal component of the vector;
Based on the vertical component of the vector, the horizontal size of the second texture is changed.
Image joining program.

Images