JP2006119730A - Image stitching - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent a connection part of a continuous image formed by stitching a plurality of images from deteriorating in picture quality. <P>SOLUTION: When 1st image data Dp1 and 2nd image data Dp2 are overlapped one each other, the distribution state of high frequency components of an overlap part is detected to set the width length WL of an overlap area with a 1st length for a part where a high frequency component quantity H is small or the width length WL of an overlap area having a 2nd length shorter than the 1st length for a part where the high frequency component quantity H is large. Then the 1st image data Dp1 and 2nd image data Dp2 are composited in an overlap area having the width length WL. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for generating image data of continuous images by connecting first image data and second image data captured so that a part of an image overlaps.

  In recent years, with the widespread use of digital cameras (digital still cameras) and personal computers, it has become possible to variously process image data captured by digital cameras using a personal computer. As one of various processes, there is a panoramic image generation process for generating a panoramic image by connecting a plurality of image data captured by a digital camera.

  In this panorama image generation process, a subject image is divided into a plurality of images so that a part of the image overlaps with a digital camera, and in the obtained plurality of images, a continuous image is obtained by joining together adjacent images. Create a panorama image. Such panoramic image generation processing is disclosed in, for example, Patent Document 1.

JP 2000-90232 A

  As a method for joining two images, a composite area having a predetermined width (hereinafter also referred to as an “overlapping area”) is provided at the boundary between the two images, and the composition ratio of both images is gradually increased in this overlapping area. A configuration to change is known.

  FIG. 16 is an explanatory diagram illustrating a composition method for setting a superposition region and changing a composition ratio of both images. As shown in the figure, in the first image, the composition ratio gradually decreases from 100% to 0% toward the second image in the width direction of the overlapping region, while the second image has the overlapping region. In the width direction, the composition ratio gradually decreases from 100% to 0% toward the first image, and the sum of the composition ratios of both images is 100%. According to this configuration, both images can be mixed smoothly within the width of the overlapping region. As a result, the boundary line portion between the two images has a smooth color change. Note that the “width of the overlapping region” described above is the length in the direction in which the two images to be joined are arranged in the overlapping region (composite region). The same applies to “the width of the composite region (or the width of the composite region)” in this specification.

  However, in the latter technique, there is a problem that the image quality is rather poor depending on the pattern when the pattern is not sufficiently aligned between the first image and the second image. . For example, when the latter technique is used for landscape photographs of the sky and forest, the image quality of the sky part is small even if the above alignment is insufficient, but the picture pattern is complicated like a forest. If the above-mentioned alignment is insufficient, such a portion is smoothed in a state where a complicated pattern is displaced, resulting in a blurred image.

  The problem to be solved by the present invention is to prevent the deterioration of the image quality of the joint portion of continuous images obtained by connecting a plurality of images.

  As means for solving at least a part of the problems described above, the following configuration is adopted.

The image stitching device of the present invention includes:
An image stitching device that creates image data of continuous images by stitching first image data and second image data captured so that part of an image overlaps,
Analyzing means for analyzing the complexity of the picture depending on the location of the overlapping part of at least one of the two image data;
A width control unit that determines a width of a synthesis area for combining the two image data when the two image data are joined at the overlapping portion, based on an analysis result by the analysis unit;
A combination unit configured to set a combined region having a width defined by the width control unit for the overlapping portion of the two image data, and combine the two image data using the combined region. It is a summary.

  According to the image stitching device having the above-described configuration, the complexity of the picture corresponding to the location of the overlapping portion between the first image data and the second image data is analyzed, and based on the analysis result, the first The width of the combined area of the image data and the second image data is determined for each location. Both image data are synthesized in the synthesis area having the predetermined width.

  For this reason, the width of the synthesis region is determined based on the complexity of the pattern for each place of the overlapping portion. In general, alignment between two image data is not easy, and alignment is often insufficient. However, according to the above configuration, even if the alignment is insufficient, it is possible to reduce the synthesis area of the image data in a place where the complexity of the pattern is large. There is no excessive smoothing. As a result, a complicated pattern portion such as a forest can be synthesized with less blur, and deterioration of the image quality at the joint portion of both image data can be prevented.

  In the image stitching device having the above-described configuration, the width length control unit receives a specific command from an input device operated by an operator, and determines the width length of the synthesis region regardless of the analysis result. It is good also as a structure provided with the width length fixing means fixed to the shortest length which can be determined by a control means.

  According to this configuration, when the operator confirms the composite result of the image data and is concerned about multiple images (double image), the operator makes the width of the composite region independent of the analysis result. Fixed to the shortest length. For this reason, it is possible to easily avoid the multiple shots generated in the synthesis area due to insufficient alignment.

  The width length fixing means may include a switch that is displayed on a display device and is turned on and off by a pointing device as the input device.

  According to this configuration, since the operator only has to operate a pointing device such as a mouse, the operability is excellent.

  The analysis unit may include a high frequency component distribution state detection unit that analyzes a spatial frequency of the overlapping portion and detects a distribution state of a high frequency component as a parameter indicating the complexity of the pattern.

  According to this configuration, it is possible to easily analyze the complexity of the pattern according to the location of the overlapping portion from the distribution state of the high frequency components.

The high-frequency component distribution status detecting means is
Line extraction means for sequentially extracting each one line along the width direction of the composite region from the overlapping portion;
High-frequency component amount detection means for detecting the amount of the high-frequency component for each extracted line;
High-frequency component amount determination means for determining whether or not the amount of the high-frequency component of each line detected by the high-frequency component amount detection means is equal to or less than a predetermined threshold;
The width control means is
The line for which the amount of the high-frequency component is determined to be less than or equal to the threshold is set to the first length by the high-frequency component amount determination unit, and the line that has been determined that the amount of the high-frequency component is not less than or equal to the threshold. May be configured to include setting means for setting a second length shorter than the first length as the width length.

  According to this configuration, the high-frequency component amount is detected from the data for each one line along the width direction of the composite region, and whether or not the high-frequency component amount is equal to or less than a predetermined threshold value is determined. The length is determined for each line as the first length or the second length shorter than the first length. As a result, with a simple configuration, the width of the synthesis region can be determined according to the distribution state of the high frequency components over the entire image.

  The synthesizing means may be configured to synthesize both the image data so that the synthesis ratio gradually decreases as it goes to the image on the other side in the width direction of the synthesis area.

  According to this configuration, the first image data and the second image data can be smoothly mixed within the width of the synthesis area.

The image stitching method of the present invention includes:
An image stitching method for creating image data of continuous images by stitching first image data and second image data captured so that a part of an image overlaps,
(A) a process of analyzing the complexity of the pattern according to the location of the overlapping part of at least one of the two image data;
(B) a step of determining, for each location, a width of a synthesis region for synthesizing the two image data when the two image data are joined at the overlapping portion, based on an analysis result by the analysis unit;
(C) setting a composite area having a width defined in the process (b) for the overlapping portion of the two image data, and combining the two images with the composite area. Is the gist.

The computer program of the present invention is:
A computer program for generating image data of continuous images by connecting first image data and second image data captured so that a part of an image overlaps,
(A) a function of analyzing the complexity of the pattern according to the location of the overlapping part of at least one of the two image data;
(B) a function of determining, for each location, a width of a synthesis area for synthesizing both image data when the two image data are joined at the overlapping portion, based on an analysis result by the analysis unit;
(C) a function of setting a composite region having a width defined by the function (b) for the overlapping portion of the two image data, and combining the two image data in the composite region; It is a computer program for realizing.

  According to the image stitching method and computer program of the present invention, as in the image stitching apparatus of the present invention, it is possible to prevent the deterioration of the image quality at the joint portion of continuous images obtained by stitching two pieces of image data. Play.

  The recording medium of the present invention is characterized by a computer-readable recording medium that records the computer program of the present invention. This recording medium has the same operation and effect as each computer program of the present invention.

  The present invention includes other aspects as follows. The 1st aspect is an aspect as a program supply apparatus which supplies the computer program of this invention via a communication path. In this first aspect, the above-described method and apparatus are realized by placing a computer program on a server or the like on a computer network, downloading a necessary program to a computer via a communication path, and executing the program. Can do.

The best mode for carrying out the present invention will be described based on examples. This embodiment will be described in the following order.
1. Device configuration:
2. Computer processing:
2-A. Overall processing:
2-B. Panorama image generation processing:
3. Action / Effect:
4). Other embodiments:

1. Device configuration:
FIG. 1 is an explanatory diagram showing a schematic configuration of a computer system to which an embodiment of the present invention is applied. The computer system of this embodiment mainly includes a personal computer 10 constituting the image stitching device of the present invention, and includes a display 20, a keyboard 22, and a mouse 24 as peripheral devices. Furthermore, a digital camera 26, a CD drive 28, and a printer 29 are connected to the personal computer 10. The mouse 24 can be replaced with other pointing devices such as a trackball, a trackpad, and a tablet.

  The personal computer 10 includes a memory 13, a display image memory 14, a hard disk drive 15, an input control unit 16, a display control unit 17, an output control unit 18, and the like that are mutually connected by a bus 12 around a CPU 11 as a central processing unit. Is provided. The memory 13 stores various data and becomes a work area of the CPU 11. The display image memory 14 is a memory that temporarily stores image data of an image to be displayed on the display 20.

  The hard disk drive 15 stores a computer program Pr as software of the panoramic image synthesizer. The hard disk drive 15 stores a plurality of image data Dpi (i = 1 to n, n is a positive integer) used for panoramic image synthesis. The image data Dpi is image data of a captured image taken by the digital camera 26 and is stored in a predetermined area (for example, a holder) of the hard disk drive 15. The captured image is a color image.

  FIG. 2 is an explanatory diagram illustrating an example of a plurality of captured images captured by the digital camera 26. As shown in the drawing, the plurality of captured images are images obtained by dividing a landscape as a subject into a total of two in the horizontal direction so that a part of the captured images overlaps. That is, the first image P1 and the second image P2 that are a plurality of photographed images have a photographing range so that a part of the photographed image overlaps (hereinafter, this overlapping part is referred to as an “overlapping part”). Each was taken while moving to the right. The first and second images P1, P2 are stored in the hard disk drive 15 as first and second image data Dp1, Dp2 (see FIG. 1). The subject is a photograph of a forest (mountain forest), and the sky is reflected in the background of the forest.

  Returning to FIG. 1, the input control unit 16 is a control unit that captures input operations from the keyboard 22 and mouse 24, captures image data from the digital camera 26, and captures data from the CD drive 28. The display control unit 17 is a control unit that controls signal output to the display 20. The output control unit 18 is a control unit that controls printing on the printer 29.

  The computer program Pr is originally stored in a CD-ROM (not shown) as a recording medium. The computer program Pr can be read from the CD-ROM and installed in the hard disk drive 15 by setting the CD-ROM in the CD drive 28 and starting a predetermined installation program. By executing the computer program Pr by the CPU 11, various constituent requirements of the image stitching apparatus of the present invention are realized.

  In FIG. 1, various constituent requirements are indicated by functional blocks implemented in the CPU 11. That is, the CPU 11 includes the analysis unit 31, the width length control unit 32, and the synthesis unit 33 as functions.

  The computer program Pr is provided as a configuration stored in another portable recording medium (portable recording medium) such as a flexible disk, a magneto-optical disk, and an IC card in place of the CD-ROM. It can be. Further, the computer program Pr can be provided via a network from a specific server connected to an external network. The network may be the Internet or a computer program obtained by downloading from a specific homepage. Alternatively, it may be a computer program supplied in the form of an email attachment.

2. Computer processing:
2-A. Overall processing:
By executing the computer program Pr by the CPU 11 of the personal computer 10, the analysis unit 31, the width length control unit 32, and the synthesis unit 33 described above are realized. The control process according to this computer program Pr will be described in detail below.

  FIG. 3 is a flowchart showing the overall flow of this control process. This routine is started when an instruction to start the computer program Pr is given. As shown in the figure, when the process is started, the CPU 11 first performs a process of displaying the application window WD on the CRT display 12 (step S100).

  FIG. 4 is an explanatory diagram showing the application window WD. As shown in the figure, in the process menu column MN on the left side of the application window WD, four types of buttons BT1, BT2, BT3, and BT4 of “input”, “panorama”, “print”, and “output” are directed downward. The operator clicks these buttons BT1 to BT4 with the mouse 24 in order, and the image captured by the digital camera 26 is combined into one panoramic image on the screen of the CRT display 12. You can proceed with the output. That is, as shown in FIG. 3, after executing step 100, the CPU 11 takes in an operation command in which the buttons BT <b> 1 to BT <b> 4 are clicked by the mouse 24, and performs input processing, panoramic image generation processing, printing corresponding to the operation command. Processing and output processing are executed in order (steps S200, S300, S400, S500).

  The input process executed in step S200 captures a plurality of image data used for panoramic image synthesis. Here, the above-described image data Dpi (image data Dp1) captured by the digital camera 26 and stored in the hard disk drive 15 is used. , Dp2). This image data Dpi import operation is performed by receiving an operation command in which the “file” button BT11 provided in the menu bar BR1 of the application window WD shown in FIG. This is performed for each image data. In addition, it is good also as a structure which takes in several image data collectively by designating a folder etc. Further, it is possible to adopt a configuration in which the digital camera 26 directly captures the data without going through the hard disk drive 15 once.

  The plurality of image data Dp1 and Dp2 captured as described above are stored in a predetermined area of the memory 13, and are also displayed in the work field FDW of the application window WD as shown in FIG. .

  The panoramic image generation process executed in step S300 is to generate a single panoramic image by connecting a plurality of image data Dp1 and Dp2 input in step S200, and details will be described later.

  The printing process executed in step S400 outputs the panorama image data created in step S300 to the printer driver as a print command. This printing process has a well-known configuration and will not be described in detail here. However, panorama image data representing a panorama image obtained by connecting the image data Dp1 and Dp2 is printed from the printer 29.

  The output process executed in step S500 outputs (saves) the panoramic image data created in step S300 as a file to an output destination such as the HDD 15 or the like. As a result, the panoramic image data is stored in a desired storage medium.

2-B. Panorama image generation processing:
The panorama image generation process executed in step S300 will be described in detail below. FIG. 5 is a flowchart showing the panoramic image generation process. When the process shifts to this routine, as shown in the figure, the CPU 11 first performs a process of counting the number n of image data Dpi captured in the input process of step S200 (step S310). For example, in the example shown in FIG. Although not shown in the drawing, when the number n is 1, the panoramic image generation process is terminated, assuming that image composition is unnecessary.

  Next, the CPU 11 performs a process of displaying the connection process dialog box DB on the CRT display 12 (step S320).

  FIG. 6 is an explanatory diagram showing a dialog box DB for stitching processing. As shown in the figure, this dialog box DB is provided with a processing instruction field FD1 and a work field FD2. In the upper part of the work field FD2, a list of image data Dpi captured by the input process in step S200 is displayed. Here, as shown in the figure, the first image data Dp1 and the second image data Dp2 are displayed. Note that a plurality of rectangular regions R are displayed in a row in the lower part of the work column FD2. The processing instruction field FD1 is provided with a “Smooth overlapping portion” check box CB1 and a “Create” button BT10.

  Returning to FIG. 5, after execution of step S320, the CPU 11 performs processing for designating in what arrangement the plurality of image data Dpi captured in the input processing of step S200 are to be connected (step S330). Here, the arrangement is an arrangement of a plurality of image data Dpi at the time of creating a panoramic image. In this embodiment, it is possible to specify how to arrange the plurality of image data Dpi within a horizontal row. It has become.

  Specifically, the above designation is made by an operation using the mouse 24 of the operator in the dialog box DB for the joining process shown in FIG. The operator drags each image data Dpi (Dp1, Dp2) displayed in the upper part of the work field FD2 and drops it on the rectangular area R displayed in the lower part of the work field FD2, thereby causing the first image to be displayed. It is possible to specify how the data Dp1 and the second image data Dp2 are arranged. In the illustrated example, the first image data Dp1 is the left part of the subject, and the second image data Dp2 is the right part of the subject. Therefore, the image data Dp1 and Dp2 may be arranged as they are. From the bottom as shown by the broken line in the figure.

  For example, if the photographer of the subject has taken pictures in the reverse order to the example shown in FIG. 2, the first image data Dp1 is the right part of the subject, and the second image data Dp2 is the left side of the subject. Therefore, the operator drags and drops the second image data Dp2 into the first rectangular area R from the left and the first image data Dp1 into the second rectangular area R from the left. It will be.

  Returning to FIG. 5, when the designation of the arrangement is completed in step S330, the CPU 11 then determines whether the [Create] button BT10 provided in the connection process dialog box DB is clicked by the operation of the mouse 24 by the operator. It is determined whether or not (step S340). Here, if it is determined that it has not been clicked, the process of step S340 is repeated to wait for the operator to click. If it is determined in step S340 that the button has been clicked, the CPU 11 advances the process to step S350.

  In step S350, the CPU 11 performs a process of reading the first image data Dpi from the left in the array of the image data Dpi displayed in the lower part of the work field FD2 of the dialog box DB for the joining process. In the example of FIG. 6, the first image data Dp1 is read. Next, the CPU 11 performs a process of substituting the value 2 for the variable i (step S360).

  Thereafter, the CPU 11 performs a process of reading the i-th image data Dpi determined by the variable i from the left in the array of image data Dpi displayed in the lower part of the work field FD2 (step S370). In the example of FIG. 6, the second image data Dp2 is read. Next, a joining process for joining the image data Dpi read in step S370 and the adjacent image data Dpi read before is performed (step S380). Details of the joining process will be described later.

  After executing step S380, the CPU 11 increments the variable i by 1 (step S390), and determines whether the variable i is greater than the number n of the image data Dpi (step S392).

  If it is determined in step S392 that the variable i is equal to or less than the number n, the process returns to step S370, assuming that all the image data Dp1 to Dpn input in step S200 have not been joined yet. , Steps S370 to S392 are repeatedly executed. On the other hand, if it is determined in step S392 that the variable i exceeds the number n, it is assumed that the joining of all the image data Dp1 to Dpn is completed, and the joining result of the image data Dp1 to Dpn is set as a panoramic image. This is displayed on the display 20 (step S394). This display may be displayed, for example, in a result display dialog box (not shown). Alternatively, a panoramic image display column (not shown) may be prepared in advance in the stitching processing dialog box DB, and the stitching result may be displayed in the panoramic image display column. After the execution of step S394, the process returns to “Return” to end the panoramic image generation process.

  Although not shown in the flowchart of FIG. 5, the operator looks at the display of the stitching result on the display 20 in step S394 and determines that the stitching result is not preferable. When the check box CB1 is turned on / off, the processing is started again from step S340.

  FIG. 7 is a flowchart showing details of the joining process executed in step S380. As shown in the figure, when the process shifts to the joining process, the CPU 11 first performs a process for obtaining a positional relationship between the two image data Dpi to be joined (step S601). This process is performed using an image position relation calculation program incorporated in advance in the panorama image generation software.

The image position relation calculation program basically includes the following procedure.
(1) Each of the two input images is converted to gray scale.
(2) A plurality of points (for example, 100 to 200) that are corners on the gray scale image are acquired.
(3) Regarding the points of one image, the points that may correspond in the other image are examined. Specifically, the local cross-correlation is calculated for all combinations of each point of one image and each point of the other image, and the calculated value is greater than a predetermined threshold. The combination of points becomes a combination of possibilities of matching.

(4) Processing is performed so that the combination of the points becomes one-to-one. If a certain point is a correct combination, there is a high possibility that a nearby point in the vicinity is also a correct combination. Using this feature, the correct combination of points is obtained based on how much the surrounding points indicate the matching degree of a certain combination of points.

  In step S601, the above calculation is actually performed after designating the connection direction of the two image data Dpi as the horizontal direction from left to right. The reason why the connection direction is the horizontal direction from left to right is that the two image data Dpi to be joined in the joining process are the image data read in order from the left of the array in step S370. As a result, since a one-to-one combination of points is obtained in (4), the positional relationship between the two image data Dpi is obtained using this information.

  After execution of step S601, the CPU 11 performs a process of extracting an overlapping portion for one of the two image data Dpi to be joined. In this embodiment, overlapping portions are extracted from the image data Dpi located on the left side of the array. In the example of FIG. 6, the image data Dpi located on the left side of the array is the first image data Dp1. The overlapping portion is obtained from the positional relationship between the two image data Dpi obtained in step S601.

  FIG. 8 is an explanatory diagram showing the first image data Dp1. In the figure, the hatched portion is an overlapping portion OL1 with the second image data Dp2. In step S601, image data included in the overlapping portion OL1 is extracted.

  FIG. 9 is an explanatory diagram showing the second image data Dp2. In the figure, the hatched portion is an overlapping portion OL2 with the first image data Dp1. In step S601, the image data included in the overlapping portion OL2 can be extracted. In step S601, in short, it is only necessary to extract the overlapping portion of either one of the image data to be joined.

  Returning to FIG. 7, after executing step S602, the CPU 11 assigns a value 1 to the variable j (step S603). Next, a process of extracting data for one line extending in the horizontal direction from the overlapping portion extracted in step S602 is performed (step S602).

  FIG. 10 is an explanatory diagram showing how the width length WL of the composite area (hereinafter also referred to as “overlapping area”) is determined by the joining process. As shown in the figure, the overlapping portion OL1 of the first image data Dp1 is composed of Nx pixels in the horizontal direction and Ny pixels in the vertical direction. 1 pixel) of data LD is read in the vertical direction from top to bottom. Specifically, the data LDj of the j-th line from the top determined by the variable j is extracted.

  Returning to FIG. 7, after executing step S604, the CPU 11 analyzes the spatial frequency of the data LDj of the jth line extracted in step S604, and performs a process of detecting the high frequency component amount H (step S605). Since the “spatial frequency” is a numerical value representing the fineness of the periodic structure of the image, the spatial frequency of the line data LDj is analyzed to detect the high frequency component amount H, thereby reducing the complexity of the image pattern. I can know. In step S605, specifically, how much high-frequency components are included in the line data LDj by separating high-frequency components and low-frequency components by discrete cosine transform (a kind of Fourier transform). This ratio is determined as the high frequency component amount H.

  Thereafter, the CPU 11 determines whether or not the flag F is a value 1 (step S606). The value of the flag F is determined depending on whether or not a check box CB1 for smoothing the overlapping portion provided in the dialog box DB for joining processing is checked. When the check is on, F = “1”, and when the check is not on, F = “0”. That is, in step S606, it is determined whether or not the “Smooth overlapping portion” check box CB1 is checked.

  If it is determined in step S606 that the “Smooth overlapping portion” check box CB1 is checked, the process proceeds to step S607, and the high-frequency component amount H detected in step S605 is a predetermined value. It is determined whether or not it is equal to or lower than H0. Here, when it is determined that the high-frequency component amount H is equal to or less than the predetermined value H0, the width WL of the overlapping region (in other words, the combined region to be combined) to be overlapped when the two image data are connected. Is obtained according to the following equation (1) (step S608).

WL = Nx · W1 (1)
Here, Nx is the horizontal length of both image data Dpi to be superimposed, and corresponds to the number of pixels in the horizontal direction of the overlapping portion OL1 of the first image data Dp1 in the above-described example of FIG. To do. W1 is a predetermined first ratio, for example, 100%.

  On the other hand, when it is determined in step S607 that the high-frequency component amount H exceeds the predetermined value H0, the width length WL of the overlapping region is obtained according to the following equation (2) (step S609).

WL = Nx · W2 (2)
Here, W2 is a second ratio smaller than the first ratio W1, and is, for example, 5%.

  Returning to FIG. 10, a description will be given after the data LD for one line in the horizontal direction for the overlapping portion OL1 has been extracted. Thereafter, for each data for one line, it is determined in step S607 whether the high-frequency component amount H is large or small. Based on the determination result, the width length WL of the overlapping region is set to Nx by S608 and S609. -It will be decided to W1 or Nx * W2.

  Returning to FIG. 7, on the other hand, if it is determined in step S606 that the [Smooth overlapping portion] check box CB1 is not checked, the process proceeds to step S609, and the above equation (2) is satisfied. The width length WL of the overlapping area is determined.

  After executing step S608 or S609, the CPU 11 advances the process to step S610. In step S610, the CPU 11 performs a process of combining the image data Dpi as an object to be joined on the j-th line determined by the variable j, with the overlapping region of the width WL obtained in step S608 or S609. .

  FIG. 11 is an explanatory diagram showing the synthesis process executed in step S610. The two graphs in the figure show the relationship between the position of the image and the composition ratio of the two image data, and (A) shows the case where the overlap region width length WL = Nx · W1 is set. (B) is a case where the width length WL is set to Nx · W2.

  In the figure, the area between the two broken lines in (A) and (B) is an overlapping area when connecting two image data, and the width length WL of the overlapping area in the case of (A) is: Assuming that the first ratio W1 is 100%, the overlapping portion OL1 is full of the horizontal length Nx. Within the range of this overlapping area, the composition ratio of the first image data Dp1 and the second image data Dp2 changes. As shown in the figure, in the overlapping region, the first image data Dp1 gradually decreases from 100% to 0% as the second image data Dp2 moves toward the second image data Dp2, and the second image data Dp2 The composite ratio gradually decreases from 100% to 0% toward the image data Dp1 of 1, and the sum of the composite ratios of both the image data Dp1 and Dp2 is 100%.

  On the other hand, the width WL of the overlap region in the case of (B) is 1/20 that in the case of (A) because the second ratio W2 is 5%. In the drawing, it is drawn larger than 1/20 for convenience of viewing. Similarly to the case of (A) above, the composition ratio of the first image data Dp1 and the second image data Dp2 changes within the range of the overlapping region. As shown in the figure, in the overlapping region, the first image data Dp1 gradually decreases from 100% to 0% as the second image data Dp2 moves toward the second image data Dp2, and the second image data Dp2 The composite ratio gradually decreases from 100% to 0% toward the image data Dp1 of 1, and the sum of the composite ratios of both the image data Dp1 and Dp2 is 100%.

  In step S610, one of the graphs (A) and (B) is selectively determined based on the width length WL of the overlapping region, and the j-th line determined by the variable j is shown in the graph. The first image data Dp1 and the second image data Dp2 are combined at a composition ratio determined from the position of the image to be recorded. Note that the above-described composition ratio is used by multiplying each color data of R, G, and B constituting each image data Dp1, Dp2.

  Returning to FIG. 7, after executing step S610, the CPU 11 increments the variable j by a value of 1 (step S611), and determines whether the variable j is larger than the number of vertical pixels Ny of the overlapping portion. (Step S612).

  If it is determined in step S612 that the variable j is less than or equal to the number Ny, the process returns to step S604 and the processes of steps S604 to S612 are repeatedly executed, assuming that the synthesis for the overlapping portion OL1 has not been completed. To do. On the other hand, if it is determined in step S612 that the variable j exceeds the number Ny, it is determined that the joining for the overlapping portion OL1 has been completed, and the process returns to “RETURN”, and the joining process is temporarily ended.

  According to the panoramic image generation process configured as described above, when the [Smooth overlapping portion] check box CB1 provided in the dialog box DB for stitching processing is checked, the following is performed. Works.

  For example, as shown in FIG. 10, if the overlapping portion OL1 of the first image data Dp1 is “sky” in the upper part and “forest” in the lower part, the high-frequency component amount H is small in the “sky” part. The “forest” portion has a high amount of high frequency component H. For this reason, as shown in the figure, the width length WL of the overlapping region continues to be a large value such as Nx · W1 (W1 is 100%) from top to bottom, and becomes a part of “forest”. However, it changes to a small value such as Nx · W2 (W2 is 5%), and the small value continues.

  FIG. 12 is an explanatory diagram showing an overlap area (composite area) DA set for the overlapping portion OL of the first image data Dp1 and the second image data Dp2. In the figure, the hatched portion is the overlapping area DA. The overlapping area DA initially has a wide width (first length) full of the horizontal length Nx of the overlapping portion OL from the top to the bottom of the overlapping portion OL. After that, at a certain point (where it becomes the “forest” portion), it becomes a narrow width (second length) that is 1/20 of Nx. In other words, the width length WL of the overlapping region is determined for each place in the vertical direction.

  On the other hand, when the check box CB1 for smoothing the overlapping portion provided in the stitching process dialog box DB is not checked, the overlapping area is as follows.

  FIG. 13 is an explanatory diagram showing the overlapping area DA when the [Smooth overlapping portion] check box CB1 is not checked. In the figure, the hatched portion is the overlapping area DA. The overlapping area DA is fixed to a narrow width (second length) such as 1/20 of the horizontal length Nx of the overlapping portion OL.

  Steps S604, S605, and S607 of the panorama image generation process configured as described above are performed by the analysis unit 31 (FIG. 1), steps S608 and S609 are performed by the width length control unit 32 (FIG. 1), and step S610 is performed by the synthesis unit 33 (FIG. 1). Correspond to each. The analysis unit 31 determines the complexity of the pattern according to the location of the overlapped portion from whether or not the high-frequency component amount H for each line in the horizontal direction is equal to or less than a predetermined value H0. From the above determination result, the unit 32 determines the width length WL of the overlapping region for each location in the vertical direction as described above. The synthesizer 33 synthesizes the first image data Dp1 and the second image data Dp2 with the overlapping region (synthesized region) having the width L.

C. Action / Effect:
According to the computer system of this embodiment configured as described above, as described with reference to FIGS. 10 to 12, the distribution state of the high-frequency component for the overlapping portion OL1 of the first image data Dp1 is detected. Then, the width length WL of the overlapping region of the first length is set in the portion where the high frequency component amount H is small, and the second length shorter than the first length is set in the portion where the high frequency component amount H is large. The width length WL of the length overlapping region is set. Then, the first image data Dp1 and the second image data Dp2 are synthesized with the overlapping region having the width L.

  For this reason, in the image data Dpi to be combined, a portion with a small amount of high-frequency component such as “sky” is combined so that the color changes smoothly. On the other hand, for parts with a large amount of high-frequency components such as “forest”, even if the alignment of both image data is insufficient, a complicated pattern is displayed because the width length WL of the overlapping area is narrow. As a result, the image is not over-smoothed, and as a result, the image can be synthesized with less blurring and the image is less unnatural. Therefore, it is possible to prevent the deterioration of the image quality at the joint portion between the two image data Dpi.

  Further, in this embodiment, when the check box CB1 for smoothing the overlapping portion provided in the dialog box DB for joining processing is not checked, the overlapping region is independent of the amount of high frequency components. The width length WL is fixed to the second length.

  FIG. 14 is an explanatory diagram illustrating an example of a composite image in which a double image is generated in a portion where the amount of high-frequency components is small when the alignment of two image data is insufficient. This is a case where the [Smooth overlapping portion] check box CB1 is checked. As shown in the figure, in the composite image, an object that appears in the shape of a white line is a double image. This part is a part with a small amount of high-frequency components, but when the [Smooth the overlapping part] check box CB1 is checked, as described above, by combining in the overlapping region having a wide width WL, Multiple shots such as double shots occur.

  On the other hand, in this embodiment, when the operator confirms the composite image by the display in step S394 and is concerned about the double image, check the “Smooth overlap” check box CB1. Just remove it. FIG. 15 is an explanatory diagram showing an effect when the “Smooth overlapping portion” check box CB1 is switched from the on state to the off state. As shown in FIG. 15A, when the check box CB1 is in the on state, the object LN that appears in the shape of a white line is a double image. On the other hand, as shown in FIG. 15B, when the check box CB1 is in the OFF state, the width L WL of the overlapping region is narrowed, so that the object LN that appears as a white line is cut off. It becomes. Since it is better in terms of image quality that the object LN is slightly displaced due to the image position displacement than the double image, the operator checks the “Smooth overlap” check box CB1. By removing, it is possible to avoid the occurrence of multiple shots as described above.

3. Other embodiments:
The present invention is not limited to the above-described embodiments and modifications, and can be carried out in various modes without departing from the gist of the present invention. For example, the following modifications are possible. is there.

(1) In the above embodiment, each line in the horizontal direction is sequentially extracted from the overlapping portion of the first image data Dp1, but this is because the direction along the width direction of the overlapping region is horizontal. This is because it is a direction. In other words, it is also because the arrangement of images to be joined is in the horizontal direction. On the other hand, when the arrangement of the images to be joined is the vertical direction, the width direction of the overlapping area is the vertical direction, and therefore one line in the vertical direction is sequentially applied from the overlapping portion of the image data Dpi. What is necessary is just to be the structure to extract.

(2) In the above embodiment, one line extracted from the overlapped portion has a vertical width of one pixel. However, it is not necessarily one pixel, for example, a plurality of pixels such as two pixels, three pixels, and five pixels. These pixels may have a vertical width.

(3) In the above-described embodiment, the arrangement of the images to be joined is a single horizontal row, but instead of this, it may be a single vertical row or a plurality of vertical and horizontal tiles. . If the images are arranged in a single vertical row, the two images may be connected in the vertical direction, and if they are tiled, the images connected in the horizontal direction and the vertical direction may be connected.

(4) In the previous embodiment, it was configured to analyze the complexity of the picture depending on the location of the overlapping part of either one of the image data to be joined. It is good also as a structure which analyzes about the duplication part. This is effective when there is a moving object at the joint. In an ideal panoramic photo (landscape only, no people), the overlapping part should have the same high-frequency component, but if a human enters the overlapping part at the moment the picture is taken, Since the person moves before taking one of the photos, the situation of the high frequency component changes. Therefore, it is possible to employ a configuration using both image data, such as taking an average of overlapping portions of both image data and analyzing them. Alternatively, the analysis of overlapping portions of both image data may be performed, and if there is a large difference in the situation of high frequency components, the width of the synthesis region may be set to the shorter one.

(5) In the above embodiment, the image data Dpi used for creating the panoramic image is taken by the digital camera 26, but instead of this, an image of a silver salt photograph acquired using a color scanner or the like. It may be data. In short, any configuration may be used as long as a captured image obtained by capturing with some imaging device (for example, a video camera having a still image capturing function) is acquired. For example, instead of a device prepared in advance in a storage device such as the HDD 15, a device imported from the outside via a network may be used. Further, it is not always necessary to use color image data, and can be applied to monochrome image data.

It is explanatory drawing which shows schematic structure of the computer system to which one Example of this invention is applied. It is explanatory drawing which shows an example of the some picked-up image image | photographed with the digital camera. It is a flowchart which shows the whole flow of the control processing according to a computer program. It is explanatory drawing which shows the application window WD. It is a flowchart which shows a panorama image generation process. It is explanatory drawing which shows dialog box DB for a joining process. It is a flowchart which shows the detail of the joining process performed by step S380 of a panoramic image generation process. It is explanatory drawing which shows the 1st image data Dp1. It is explanatory drawing which shows the 2nd image data Dp2. It is explanatory drawing which shows how the width length WL of a superimposition area | region is determined by the joining process. It is explanatory drawing which shows the synthetic | combination process of 1st image data Dp1 and 2nd image data Dp2. It is explanatory drawing which shows the overlap area | region DA in case the check box CB1 is checked in the "Smooth the overlap part." It is explanatory drawing which shows the overlap area | region DA when the check box CB1 is not checked in the "Smooth the overlap part." It is explanatory drawing which shows an example of the synthetic | combination image in which the double copy has arisen in the part with few high frequency component amounts when alignment of two image data is inadequate. It is explanatory drawing which shows that the said double image can be eliminated. It is explanatory drawing which shows the synthetic | combination method as a prior art example.

Explanation of symbols

10 ... Personal computer 11 ... CPU
12 ... Bus 13 ... Memory 14 ... Display image memory 15 ... Hard disk drive 16 ... Input control unit 17 ... Display control unit 18 ... Output control unit 20 ... Display 22 ... Keyboard 24 ... Mouse 26 ... Digital camera 28 ... CD drive 29 ... Printer 31 ... Analysis unit 32 ... Width control unit 33 ... Composition unit BR1 ... Menu bar BT1 to BT4 ... button Dpi ... image data Dp1 ... first image data Dp2 ... second image data FDW ... work field MN ... processing menu field Pr ... Computer program WD ... Application window DB ... Dialog box FD1 ... Processing instruction field FD2 ... Working field R ... Area FD1 ... Processing display field CB1 ... [Smooth overlapping parts] ] Check box BT10 ... [Create] button L1 ... overlapping portion OL2 ... overlapping portion H ... high frequency component amount W1 ... first ratio W2 ... second ratio Nx ... number of pixels in the horizontal direction of the overlapping portion OL1 Ny. .. Number of pixels in the vertical direction of overlapping part OL1 DA ... Overlapping area
WL ... Width of overlap area

Claims (13)

An image stitching device that creates image data of continuous images by stitching first image data and second image data captured so that part of an image overlaps,
Analyzing means for analyzing the complexity of the picture depending on the location of the overlapping part of at least one of the two image data;
A width control unit that determines a width of a synthesis area for combining the two image data when the two image data are joined at the overlapping portion, based on an analysis result by the analysis unit;
An image stitching unit comprising: a synthesis area having a width defined by the width length control means for the overlapping portion of the two image data; and a synthesis means for synthesizing the two image data in the synthesis area. Alignment device. The image stitching device according to claim 1,
The width control means is
Upon receiving a specific command from the input device operated by the operator, the width of the synthetic region is fixed to the shortest length that can be determined by the width length control means regardless of the analysis result. An image stitching device comprising fixing means. The image stitching device according to claim 2,
The width length fixing means is:
A switch that is displayed on a display device and is turned on and off by a pointing device as the input device;
Image stitching device. An image stitching device according to any one of claims 1 to 3,
The analysis means includes
An image stitching apparatus comprising high frequency component distribution status detection means for analyzing a spatial frequency of the overlapping portion and detecting a distribution status of high frequency components as a parameter indicating the complexity of the pattern. The image stitching device according to claim 4,
The high-frequency component distribution status detecting means is
Line extraction means for sequentially extracting each one line along the width direction of the composite region from the overlapping portion;
High-frequency component amount detection means for detecting the amount of the high-frequency component for each extracted line;
High-frequency component amount determination means for determining whether or not the amount of the high-frequency component of each line detected by the high-frequency component amount detection means is equal to or less than a predetermined threshold;
The width control means is
The line for which the amount of the high-frequency component is determined to be less than or equal to the threshold is set to the first length by the high-frequency component amount determination unit, and the line that has been determined that the amount of the high-frequency component is not less than or equal to the threshold. An image stitching apparatus comprising setting means for setting a second length shorter than the first length as the width length. An image stitching device according to any one of claims 1 to 5,
The synthesis means includes
The two image data are combined so that the combination ratio gradually decreases as it goes to the image on the other side in the width direction of the combination area.
Image stitching device. An image stitching method for creating image data of continuous images by stitching first image data and second image data captured so that a part of an image overlaps,
(A) a process of analyzing the complexity of the pattern according to the location of the overlapping part of at least one of the two image data;
(B) a step of determining, for each location, a width of a synthesis region for synthesizing the two image data when the two image data are joined at the overlapping portion, based on an analysis result by the analysis unit;
(C) An image including a step of setting a combined area having a width defined in the step (b) for the overlapping portion of the two image data and combining the two images with the combined region. How to join. The image stitching method according to claim 7,
The step (b) is:
(B1) Upon receiving a specific command from the input device operated by the operator, the width of the synthesis region is fixed to the shortest length that can be determined by the step (b) regardless of the analysis result. A method for stitching together images. A computer program for generating image data of continuous images by connecting first image data and second image data captured so that a part of an image overlaps,
(A) a function of analyzing the complexity of the pattern according to the location of the overlapping part of at least one of the two image data;
(B) a function of determining, for each location, a width of a synthesis area for synthesizing both image data when the two image data are joined at the overlapping portion, based on an analysis result by the analysis unit;
(C) a function of setting a composite region having a width defined by the function (b) for the overlapping portion of the two image data, and combining the two image data in the composite region; A computer program for realizing this. A computer program according to claim 9,
The function (b) is
(B1) Upon receiving a specific command from the input device operated by the operator, the width of the synthesis area is fixed to the shortest length that can be determined by the function (b) regardless of the analysis result. A computer program that causes a computer to realize further functions. A computer program according to claim 9 or 10,
The function (a) is:
(A1) A computer program comprising a function of analyzing a spatial frequency of the overlapping portion and detecting a distribution state of a high frequency component as a parameter indicating the complexity of the picture. A computer program according to claim 11,
The function (a1) is
(A11) A function of sequentially extracting each one line along the width direction of the composite region from the overlapping portion;
(A12) a function of detecting the amount of the high-frequency component for each extracted line;
(A13) a function of determining whether or not the amount of the high-frequency component of each line detected by the function (a12) is equal to or less than a predetermined threshold;
The function (b) is
(B2) It is determined by the function (a13) that the line for which the amount of the high-frequency component is determined to be less than or equal to the threshold is determined to have a first length and the amount of the high-frequency component is not equal to or less than the threshold. A computer program comprising a function of setting a second length, which is shorter than the first length, as the width length in the line.   A computer-readable recording medium in which the computer program according to claim 9 is recorded.

Images