JPH0990530A - Compositing device and forming method for panoramic image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To process a panoramic image in a short time. SOLUTION: Correspondence points where plural kinds of image data transmitted from an electronic camera 1 correspond to each other are extracted by a correspondence point extracting means 19 and then, a composting parameter calculating means 20 calculates a composing parameter for forming the panoramic image, based on the correspondence points, to store the composting parameter. Then, an image composting means 21 composting-process image based on the composting parameter, at the time of reproducing and displaying the panoramic image on a display 4.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a panoramic image synthesizing apparatus and a method for creating a panoramic image, and more specifically, a plurality of images which are partially overlapped from each other are picked up by an image pickup apparatus such as an electronic camera. The present invention relates to a panoramic image synthesizing apparatus capable of synthesizing images to obtain one image and a panoramic image creating method.

[0002]

2. Description of the Related Art In recent years, a panoramic image synthesizing technique capable of synthesizing a plurality of images in which a part of the images overlap each other in order to meet the demand for photographing a wide image into one image. Development is actively done.

In particular, it has been pointed out that an electronic camera has a lower resolution than a silver halide camera or a scanner due to the small number of pixels, and the panoramic image synthesis technique is applied to an image taken by such an electronic camera. It has become important not only for capturing wide images but also for capturing high resolution images.

That is, in such a panoramic image composition technique, one manuscript, magazine, etc. are divided into a plurality of images to obtain high resolution data equivalent to a scanner, or a landscape is divided into a plurality of parts to obtain a wide and high resolution. It is very effective for shooting.

A conventional panoramic image synthesizing apparatus using the above panoramic image synthesizing technique includes corresponding point extracting means for finding corresponding points which are overlapping positions of images to be synthesized, and parameters for synthesizing images from the corresponding points. The electronic camera includes a combination parameter calculating unit for calculating, and a combination processing unit for combining a plurality of images into one image based on the parameters calculated by the combination parameter calculating unit. This is done when capturing an image from the computer to. That is, when a picture is taken with an electronic camera, the image data and the attribute data of the image data are stored in the memory built in the electronic camera. When taking a panoramic image, the electronic camera is set in the panoramic image photographing mode at the time of photographing. According to the panoramic image shooting mode, an identifier indicating a set of panoramic images is automatically recorded in the attribute data of the shot image. Then, when the electronic camera is connected to the computer and the image data and the attribute data stored in the memory built in the electronic camera are registered in a database or the like inside the computer, the attribute data is checked via the application software. To do. Next, one set of images is automatically extracted from the attribute data having the identifier in the panoramic image shooting mode, and the panoramic image combination is performed. That is, the corresponding point extraction means, the synthesis parameter calculation means, and the synthesis processing means are sequentially executed to perform panoramic image synthesis.

[0006]

However, in the above-mentioned conventional panoramic image synthesizing apparatus, the processing amount of the corresponding point extracting means, the synthesizing parameter calculating means, and the synthesizing means is large, and the processing time becomes long. There was a problem that it would end up. That is, in the conventional panoramic image synthesizing device, all of the above-mentioned processes are performed at the time of loading into the computer. Therefore, if the image data includes panoramic image data, the image is transferred from the electronic camera to the computer. There is a problem that it takes a long time to perform a series of processes of capturing and registering the image in a database or the like.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a panoramic image synthesizing apparatus and a panoramic image creating method capable of performing panoramic image processing in a short time.

[0008]

In order to achieve the above object, among the present invention, the invention according to claim 1 is a combination parameter calculating means for calculating a combination parameter for combining images, and the combination parameter calculation. A panorama image synthesizing device for synthesizing images based on the synthesizing parameter calculated by the synthesizing device, and synthesizing a plurality of images in which a part of the images overlap to obtain one image. A storage means for storing the synthesis parameter calculated by the synthesis parameter calculation means, and a separation independent execution means for separately and independently executing the synthesis processing means and the synthesis parameter processing means at different times. It is characterized by being.

According to a second aspect of the present invention, in the first aspect of the present invention, the image pickup device has an image pickup device and an information processing device for fetching image data picked up by the image pickup device, and the synthesis parameter calculation means is the It is characterized in that it is executed at the time of capturing image data from the image pickup device to the information processing device, and the composition processing means is executed at the time of image reproduction.

[0010] Further, the invention according to claim 3 is based on claim 1.
In the invention described above, the parameter synthesizing means is executed when the image capturing by the image pickup apparatus is completed, and the synthesizing processing means is executed when the image is reproduced by the image pickup apparatus.

According to a fourth aspect of the present invention, in the first aspect of the invention, the parameter synthesizing means is executed when the image pickup by the image pickup device is completed, and
The combination processing means is executed when a panoramic image is output from the imaging device to an external device, and the invention according to claim 5 is the invention according to claim 4, wherein the external device is an information processing device. It is characterized by being.

Further, the invention according to claim 6 is a method of creating a panoramic image in which a plurality of images in which a part of the images overlap each other are obtained to obtain one image. After calculating the synthesis parameter for creating the panorama image based on the corresponding points, the synthesis parameter is stored, and the image is calculated based on the stored synthesis parameter at a time different from the time when the synthesis parameter is calculated. The feature is that a panoramic image is created by executing a combining process.

According to the invention of claim 7, in the invention of claim 6, when the image data for the captured panoramic image is transferred from the image pickup device to the information processing device, the calculation of the synthesis parameter is performed by the image pickup. It is characterized in that it is executed when the image data is loaded from the device to the information processing device, while the image combining process is executed when the image is reproduced.

Further, the invention of claim 8 is the same as claim 6.
In the invention described above, the synthesis parameter is calculated at the end of image capturing by the image pickup apparatus, and the image synthesis processing is executed at the time of image reproduction by the image pickup apparatus.

According to a ninth aspect of the present invention, in the sixth aspect of the invention, the composite parameter is calculated at the end of image capturing by the image pickup device, and the image is output at the time of outputting the panoramic image from the image pickup device to the external device. The invention according to claim 10 is characterized in that a synthesizing process is executed.
In the invention described above, the external device is an information processing device.

[0016]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is an overall configuration diagram showing an embodiment of a panoramic image synthesizing apparatus according to the present invention, in which an electronic camera 1 as an image pickup apparatus is connected to a personal computer (hereinafter referred to as "personal computer") via a connection cable 3. Called) 2. Further, the personal computer 2 has a display 4 for displaying image data and the like, a mouse 6 as a pointing device having a mouse button 5, a keyboard 7, and a system control unit 8 for controlling each of these components. Further, a hard disk (HD) 9 as an external storage device is connected to the system control unit 8.

FIG. 2 is a system configuration diagram of the panoramic image synthesizing apparatus. The system control unit 8 has a system memory (not shown) and a CPU (not shown), and the system memory has an operation system (hereinafter, referred to as " An OS 10) and an application program (hereinafter simply abbreviated as “application”) 11 are stored. The OS 10 and the application 11 are CPs as required.
It is loaded into U and executed by the CPU.

Specifically, the OS 10 includes an input device management system 12 that receives various user inputs, a drawing management system 13 that manages the drawing displayed on the display 4, and a file system that controls the input / output of files. 14 and.

The application 11 also comprises an image data management system 15 and a panoramic image composition system 16.

Specifically, the image data management system 15 includes a data management means 17 for managing the attribute data of the image data, keywords input by the user, and the like, and searches the image data on the display 4 with these attribute data and keywords. The data display means 18 for displaying is provided. The panoramic image composition system 16 also includes corresponding point extracting means 19 for extracting corresponding points between a plurality of image data.
A combination parameter calculation unit 20 that calculates a combination parameter for combining images based on the corresponding points, and an image combination unit 21 that combines a plurality of image data based on the combination parameters to obtain one panoramic image. I have it.

In this panoramic image synthesizing device, the OS
The input device management system 12 of 10 receives an input from the keyboard 7 via the keyboard interface 22 or an input from the mouse 6 via the mouse interface 23, and further at a high speed such as a bidirectional parallel interface or a SCSI interface. Image data is exchanged with the electronic camera 1 via the general-purpose interface 24 capable of image transfer. Further, the panoramic image system 16 is the image data management system 15
Receives an image shot in the panorama shooting mode from and performs panorama image processing. Further, the synthesis parameter obtained by the synthesis parameter calculation means 20 of the panoramic image system 16 and the image data as a synthesis result synthesized by the image synthesis means 21 are registered in the image data management system 15. Then, the image data registered in the image data management system 15 is sent to the drawing management system 13 of the OS 10 via the data display means 18 and displayed on the display 4 via the video interface 25. The file system 14 is connected to the hard disk 9 via a disk input / output (I / O) interface 26,
The files and image data physically stored in the hard disk 9 are read and written, and the files and image data are exchanged with the image data management system 15 of the application 11.

FIG. 3 is a diagram showing a data structure of image data and attribute data stored in the built-in memory of the electronic camera 1.

That is, an image management table 27 as shown in FIG.
o. Image data 28a, attribute data 29a,
Image No. 2 for image data 28b and attribute data 2
9b, the image management table 27 stores the image data 28 and the attribute data 2 corresponding to the image number of the captured image.
9 is stored.

The image data 28 may be the format data unique to the electronic camera 1 (hereinafter referred to as "native data") or JPEG (Joint Photographic Codin).
Stores one of the general-purpose format data such as g Experts Group). Native data is, for example, an output signal (analog signal) from a CCD as an image sensor.
Is a data obtained by simply converting the data into a digital signal, and generally has a characteristic that the data size is large, but the time required for recording is short, while, for example, JPEG data, although the time required for recording is long, It has a characteristic that the size can be made small, and the user who is the photographer appropriately selects desired format data according to the photographing situation and stores the image data.

As shown in FIG. 3B, the attribute data 29 includes a file name 30 automatically assigned by the electronic camera 1, a native data format, and a JP name.
A file type 31 for identifying the EG data format or another general-purpose format such as TIFF (Tag Image File Format) supported by the electronic camera 1, and the electronic camera 1 via a calendar and a timer built in the electronic camera 1. The shooting date and time 32 in which the date and time when the shutter button (not shown) is pressed and the shooting mode 33 at the time of shooting selected from a plurality of shooting modes of the electronic camera 1 are stored. . Then, when the shooting mode name stored in the shooting mode 33 is the panoramic shooting mode, the identifier 34 is added as shown in FIG. That is, the identifier 34 corresponds to the electronic camera 1
Mode identifier (hereinafter, referred to as “mode ID”) 35 attached when is set to the panorama shooting mode, and a shooting number 36 indicating the number of shots in the mode ID 35.
And are stored. Therefore, in the panoramic photography mode, one set includes a plurality of images having the same mode ID 35. That is, like the image data shown in FIG.
, The mode ID 35a of the image data 28a and the mode ID 35b of the image data 28b have the same ID, and the same I
D will form one set of panoramic images. In this way, the image data and its attribute data are stored in the built-in memory of the electronic camera 1.

FIG. 4 is a display screen on the display 4 showing how the image data and the attribute data stored in the built-in memory of the electronic camera 1 are copied or moved inside the personal computer 2.

That is, the electronic camera 1 is connected to the connection cable 3
When connected to the personal computer 2 via the
Activates the image data management system 15, and the electronic camera 1
A first window (hereinafter referred to as "camera catalog") 37 for displaying data stored in the hard disk 9 and a second window for displaying an image database stored in the hard disk 9.
Window (hereinafter referred to as "user catalog") 38
And are displayed on the display 4.

Therefore, the camera catalog 37 is provided with a plurality of display frames 39 for displaying a selected image for copying the image data to the personal computer 2 and the like. A thumbnail image display section 40 for displaying a reduced image (hereinafter, this reduced image is referred to as a “thumbnail image”) and an attribute data display section 41 are provided.
The thumbnail image of the image data 28 built in the electronic camera 1 and the attribute data 29 are displayed in a predetermined display frame 39. The user can freely select whether to display a part or all of the attribute data stored in the built-in memory of the electronic camera 1 on the attribute data display unit 41. That is, the user selects only the file name and the file type of the attribute data from the attribute data display section 41.
Can be freely selected or displayed on the attribute data display section 41 or all attribute data stored in the built-in memory can be displayed.

Next, the user operates the mouse 6 to select the display frame 39, and as shown by the arrow A, the selected image is copied or moved onto the user catalog 38. The user can freely select whether to copy (in this case, the data is stored in the built-in memory of the electronic camera 1 as it is) or move (in this case, the moved data is deleted from the built-in memory). Can be switched by.
Then, in the present embodiment, during the copying (or moving) operation, that is, when the image data or the like stored in the electronic camera 1 is loaded into the personal computer 2, the native data is converted into a predetermined general-purpose format data, and the panoramic shooting is performed. When the image is captured in the mode, corresponding point extraction processing and synthesis parameter calculation processing are performed according to the captured image, and then image synthesis processing is performed during image reproduction to display the synthesized panoramic image on the display 4.

Hereinafter, these processes will be specifically described.

FIG. 5 is a diagram showing a data structure on the user catalog 38, and the user catalog 38 is managed by the image data management system 15. Specifically, the image data management system 15 has a desired number of user catalogs 38, and a catalog table 42 for each of the user catalogs 38. Further, the catalog table 42 has a function of allowing the user to divide the image data in the user catalog 38 into a plurality of images into one group for each category, whereby the inside of one catalog table 42 is hierarchized. Manage. Specifically, the catalog table 42 has a data identifier (data ID) 4 of image data to which the catalog belongs.
3 and a group identifier (group ID) 44 for identifying the group to which the catalog belongs are stored, and the group ID 44 is linked to the group attribute table 45. The group attribute table 45 stores the group ID data 47 in addition to the data ID 46 of the image data to which the group belongs. As the group attribute data 47, a group name 48 arbitrarily given by the user, a creation date / time 49 when the group is created, and a group type 50 are stored. As the group name 48, when a group is created as a set of panoramic images, “panoramic image” is added by default. Group type 5
0 stores the name of “user created” when created by the user, and “panorama image” when a group is created as a set of panoramic images. Then, when the group type 50 is “panoramic image”, an identifier is further added. That is, the identifier 34 stores a mode ID 51 given when “panorama image” is set in the group type 50 and a combination parameter 52 obtained by a combination parameter calculation process described later.

Further, the image data management system 15 has a data management table 53 built therein, and is stored with a data ID 54 which is a unique identifier for the image data. In the case of a panoramic image, each data I
D54 corresponds to each image No. of the image data and the attribute data stored in the built-in memory of the electronic camera 1.

FIG. 6 is a flowchart showing the processing procedure of the panoramic image processing. This program is executed by the electronic camera 1.
When the image data is transferred from the personal computer 2 to the personal computer 2, it is executed by the CPU (not shown) of the personal computer 2.

In step S1, it is determined whether or not the data processing has been completed. The determination in step S1 is initially negative (N
o), the process proceeds to step S2, and the image data and the attribute data attached to the image data are acquired. And
Whether or not the image data is native data is determined based on the file type 31 of the attribute data 29 (step S3). If the image data is not native data, the process proceeds to step S5. If the image data is native data, the native data is JPEG data or the like. , The file type 31 is updated after conversion into a predetermined general-purpose format, and the process proceeds to step S5.

In step S5, the data management table 53
The shooting mode 33 is checked based on the attribute data 29 stored in the item (1) to determine whether the shooting was performed in the panoramic shooting mode. If it is not a "panorama image", it is registered as normal image data. That is, the data management table 5
3 is assigned with a unique data ID 54 and is also registered in the catalog table 42 (step S
6) and returns to step S1.

In the case of "panorama image", it is judged whether or not a group for the corresponding panorama image has already been created (step S7), and the answer is affirmative (Yes).
If so, the process proceeds to step S9, while the answer is negative (N
In the case of o), after creating a group (step S8), the process proceeds to step S9. That is, by checking whether the mode ID 51 (FIG. 5) in the catalog table 42 and the mode ID 35a in the image management table are the same,
It is determined whether or not the group has been created. If not created, a new group ID 44 is registered in the catalog table 42, and the group name 48, creation date / time 49, group type 5 is registered.
A group attribute 45 such as 0 is created. In this case, the group type 50 is recorded as "panoramic image", and the same ID as the mode ID 35 in the image data is stored in the mode ID 51. Then, in step S9, a unique data ID 54 is added to the panorama image data and registered in the data management system 53, the same data ID as the data ID is registered in the data ID 46 of the group attribute table 45, and the process returns to step S1.

The above series of processing is performed for all image data to be copied, and when the processing is completed for all image data, that is, the answer to step S1 is affirmative (Yes.
If it becomes s), the process proceeds to step S10, and it is determined whether or not a group of panoramic images has been created from those copied so far. If the answer is negative (No), the process is directly performed. While ending, the answer is affirmative (Yes
In the case of s), the corresponding point extraction processing (step S11) and the synthesis parameter calculation processing (step S12) described later are sequentially executed using the images in the group, and then a thumbnail image is created (step S13). finish.

FIGS. 7A to 7C are diagrams showing thumbnail formats selected in the thumbnail creation processing.

FIG. 7A is for displaying only a panoramic image, and an image preset in the system is used. FIG. 7B shows the images included in one set of panoramic image groups (hereinafter referred to as “panoramic group”) in a reduced size and displayed. FIG. 7C uses the combined image as a thumbnail image. In this panorama image synthesizing device, the user selects which of the three thumbnail formats is to be used for creating.

However, since the present panorama image synthesizing device does not perform the synthesizing process when the image data is fetched from the electronic camera 1, the panoramic group is formed in order to create the thumbnail image in the format of FIG. 7C. The plurality of images to be processed are reduced, corresponding point extraction processing and synthesis parameter calculation processing described later are performed on the reduced plurality of images, and then the synthesis processing is performed. In this case, since the target image is small, the processing time is shortened, and the influence on the time required to read the image data from the series of electronic cameras 1 is insignificant.

Next, the corresponding point extraction processing executed in step S11 of FIG. 6 will be described with reference to the flowchart shown in FIG.

In step S21, the number of images in the group is 2
Judge whether it is a sheet or not. When the answer is negative (No), that is, when the number is two or more, the process proceeds to step S22 to execute the automatic corresponding point extraction process, and then it is determined whether the result is successful (step S23). Here, it is determined whether or not it is successful depending on whether or not the corresponding points between the images are sufficiently found. When the answer to step S23 is affirmative (Yes), the process is terminated and the process returns to the main routine (FIG. 6), while the answer to step S23 is negative (N).
In the case of o), the process proceeds to step S26, the semi-auto corresponding point extraction process is executed, and the process returns to the main routine (FIG. 6).

On the other hand, when it is determined in step S21 that the number of images in the group is two, the process proceeds to step S24, the full-auto corresponding point extraction process is executed, and then it is determined whether or not the result is successful. (Step S25). That is, similar to step S23 described above, it is determined whether or not the corresponding points between the images have been sufficiently found. If the answer to step S25 is affirmative (Yes), the process is terminated and the main routine (FIG. 6) is performed. While returning, step S25
When the answer is negative (No), the process proceeds to step S26, the semi-auto corresponding point extraction process is executed, and the process returns to the main routine (FIG. 6).

FIG. 9 is a diagram showing a user interface of the automatic corresponding point extraction processing executed in step S22 (FIG. 8), in which all the images belonging to the group of the panoramic image are resized to fit in the window. Is displayed on the display screen on the display 4, and the user operates the mouse 6 while observing the display screen to rearrange the layout correctly. That is, as shown in FIG. 9A, all the images belonging to the group are displayed on the display 4, and the mouse 6 is operated to display each of the upper left image and the lower right image as shown by arrow B. Swap the position, Figure 9
As shown in (b), the images are rearranged in the correct arrangement.
At this time, when a part of the image is out of the window when the images are rearranged, the image is resized so that it can fit in the window and is displayed on the display screen on the display 4.

FIG. 10 is a flowchart showing the processing procedure of the automatic corresponding point extraction processing.

In step S31, the positional relationship between the plurality of images rearranged by the user is acquired, and in the following step S32, a region to be searched for finding a matching corresponding point, that is, a matching range is set.

FIG. 11 is a diagram showing the setting of the matching range between the left image and the right image. As a rule when shooting as a panoramic image, for example, a minimum of 10 in the lateral direction is set in advance.
%, The maximum overlap of 50% and the vertical shift of 5% or less are set, the minimum overlapping range is the area C in FIG. 11A, and the maximum overlapping range is FIG. It becomes the area D of (b). Then, the point corresponding to the point P inside the area C of FIG. 11A exists inside the area F of FIG. 11B.
That is, the area F becomes the search area.

After setting the matching range in this way,
In step S33 of FIG. 10, the corresponding point extraction execution process is executed to search the search area F for corresponding points that match each other. Then, in step S34, the corresponding points are set to the predetermined value N.
Determine if more. When the corresponding points are equal to or less than the predetermined value N, it is determined that the corresponding points are not sufficiently found, that is, it is determined that the extraction result is unsuccessful, and the process proceeds to the semi-automatic corresponding point extracting process (step S25). On the other hand, when the corresponding points are equal to or more than the predetermined value N, sufficient corresponding points are found, the extraction result is judged to be successful, and the process returns to the main routine (FIG. 6).

FIG. 12 is a flowchart showing the processing procedure of the fully automatic corresponding point extraction processing executed in step S24 (FIG. 8).

In step S41, step S3 in FIG.
The matching range is set by the same method as in 2, and then the corresponding point extraction execution processing is performed four times. In the case of full auto, the number of images is 2, so the possible positional relationship is shown in FIG.
As shown in FIG. 3, there are four types: upper and lower (a), lower and upper (b), left and right (c), and right and left (d). Therefore, corresponding point extraction execution processing is performed for these four cases, and the number and average matching level extracted as corresponding points are held.
That is, in step S42, the image position moves up and down (see FIG.
Corresponding point extraction execution processing is performed in the case of (a)), and in step S43, the number and average matching level extracted as corresponding points are held. Next, in step S44, corresponding point extraction execution processing is performed when the image position is lower and upper (FIG. 13B), and in step S45, the number and average matching level extracted as corresponding points are held. Furthermore, in step S46, the image position is changed to the left and right (see FIG.
Corresponding point extraction execution processing is performed in the case of (c)), and in step S47, the number and average matching level extracted as corresponding points are held. In the following step S48, corresponding point extraction execution processing is performed for the case where the image position is right and left (FIG. 13D), and in step S49, the number extracted as corresponding points and the average matching level are held.
Then, it is determined from the processing results of steps S42 to S49 whether there is a corresponding point having a predetermined value N or more (step S50), and it is sufficient if there is no corresponding point having a predetermined value N or more. This is the case where no corresponding point is found, and the extraction result is judged to be unsuccessful, and the process moves to the semi-auto corresponding point extraction processing (step S25). On the other hand, when the corresponding points are equal to or more than the predetermined value N, the one with the highest average matching level is determined to be in the correct true positional relationship among them, the processing is terminated, and the main routine (FIG. 6) is executed. Return. In a normal image, if the corresponding point is equal to or more than the predetermined value N, it should be only one of the above four image positions. For example, when the original is divided and photographed, similar characters are lined up correctly, Even if there is no positional relationship, a predetermined value N or more may be extracted as a corresponding point, and in this embodiment, the most suitable one in step S51, that is, the one in the image positional relationship with the highest average matching level is selected. I am trying.

FIG. 14 is a diagram showing a user interface in the semi-auto corresponding point extraction processing executed in step S25 of FIG. 8, in which all the images belonging to the group of panoramic images are resized to fit in the window and displayed. 4 is displayed on the display screen, and the user operates the mouse 6 on the basis of the position data while visually checking the display screen to superimpose most overlap positions. The overlapped portion is displayed by performing a logical product operation for each bit in pixel units. As a result, both images can be seen through in the superimposed overlapping portion. That is, in the present embodiment, since the overlapped portion can be seen through by the logical product operation even while the mouse 6 is being operated, the general alignment can be easily performed. At this time, as in the case of the automatic corresponding point extraction processing (FIG. 9), the window is resized so as to fit in the window and displayed on the display screen of the display 4.

FIG. 15 is a flow chart showing the processing procedure of the semi-auto corresponding point extraction processing.

In step S61, the positional relationship of the plurality of images rearranged by the user is acquired, and in the following step S62, the matching range is set. Since the matching range in this case is the error range and margin of the overlapping part assumed by the user, it is smaller than the range at the time of automatic corresponding point extraction processing, and the calculation time is shortened compared to the case of automatic corresponding point extraction processing. The accuracy can be improved. Then, in a succeeding step S63, corresponding point extraction execution processing is performed, and the process returns to the main routine (FIG. 6).

FIG. 16 shows steps S33 and 12 of FIG.
Step S42, Step S44, Step S46
FIG. 16 is a conceptual diagram of corresponding point extraction execution processing executed in step S48 and step S63 in FIG. 15, showing a case where corresponding points are extracted from two images (left image and right image).

First, as shown in FIG. 16 (a), by the above-mentioned matching range setting, 90 pixels in the vertical direction in the left image.
%, An area C of 10% in the horizontal direction is set, and the search range is a range in which corresponding points may exist. Therefore, in the right image, an area D of 100% in the vertical direction and 50% in the horizontal direction is set. It is set. Next, a point P (x, y) whose edge is stronger than a predetermined value M is searched for in the region C, and a rectangle of ± n pixels vertically and horizontally centering on the edge is used as the template image I.
Cut out as. Then, the template image I is placed on the search area F of the right image, and the difference is calculated pixel by pixel. Then, the position where the difference becomes the minimum value is calculated by shifting the search area F pixel by pixel. As a result of searching the entire inside of the search area F, a point P '(x', y ') whose minimum value is equal to or less than a predetermined value L is a point P'.
It is held as a corresponding point of (x, y). When extracting the corresponding points from two or more images, the corresponding points can be extracted by repeating the above-described processing for the two images.

FIG. 17 is a flowchart showing the processing procedure of the corresponding point execution processing described above.

In step S71, an edge extraction image is created, and in step S72, an edge point P having a predetermined value M or more is generated.
(X, y) is searched for, and when a point having an edge of a predetermined value M or more is found, the process proceeds to step S73, and a rectangle of ± n pixels in each of the vertical and horizontal directions is cut out from the point P (x, y) as the template image I. Then, the search area F is set in the right image based on the position of the point P (x, y) (step S7).
4). Then, the image of the search region F and the template image I are overlapped, the absolute value of the pixel value difference is calculated in pixel units, and the total value thereof is calculated (step S75). In a succeeding step S76, the total value of the differences is compared with the total value up to the previous time, and it is determined whether or not it is the minimum value so far. When the answer is negative (No), the process proceeds to step S78, while when the answer is affirmative (Yes), the minimum value and the coordinate position of the search area F are held, and then the process proceeds to step S78. In step S78, it is determined whether or not all the search areas F have been searched. If the answer is negative (No), the procedure returns to step S75, while the answer is affirmative (Yes), that is, if all the search areas F have been searched. Is step S79
Proceed to. As a result, the most coincident point having the smallest difference value is detected. And step S7
At 9, whether or not the minimum value of the difference is a sufficiently small value is compared with the predetermined value L to determine. Then, when the minimum value is larger than the predetermined value L, the process proceeds to step S81, while when the minimum value is smaller than the predetermined value L, it is determined that they are corresponding points, and the point P (x, y) and the minimum value are determined. Point P '
(X ', y') and the minimum value are registered in the corresponding point list (not shown) (step S80), and the process proceeds to step S81. Then, these processes are performed for all the points in the area C, and when it is determined in step S81 that all are finished, the process proceeds to step S82 to calculate the average value from all the minimum values in the corresponding point list, The average value is held as the matching level, and the process ends.

Next, the synthesis parameter calculation process executed in step S12 of FIG. 6 will be described.

When two images are considered (when two or more images are combined, two images are repeated, so it is sufficient to consider only two images). Translations Δx, Δy, rotation angles θ,
It can be expressed by the magnification m, and therefore, the point P (x, y) and the point P ′ (x ′, y ′) corresponding to each other are expressed as in Expression (1).

[0061]

[Equation 1] Here, A, B, C, and D are synthesis parameters.

In the above-mentioned corresponding point extraction execution processing (FIG. 17), a plurality of sets of corresponding points (point P (x, y) and point P '(x',
Since y ')) has been obtained, the synthesis parameters A, B, C, D are calculated using the least squares method. That is, the synthesis parameters A, B, C, and D satisfying the formulas (3) to (6) are calculated so that the formula (2) becomes the minimum value.

[0063]

[Equation 2] Here, when p _{1 to} p ₈ are defined as in Expressions (7) to (14), the synthesis parameters A, B, C, and D are expressed by Expression (1).
It is represented by 5) to (18).

[0064]

(Equation 3)

[0065]

[Equation 4] That is, the composite parameters A, B, C, and D are calculated by substituting p _{1 to} p ₈ into the formulas (15) to (18) and stored in the composite parameter 52 of the group attribute table (FIG. 5). .

In the panoramic image synthesizing apparatus, the corresponding point extracting process (step S11) and the synthesizing parameter calculating process (step S12) are executed when the electronic camera 1 loads the personal computer 2 to obtain the image data. The image combining process is executed during the reproducing operation, that is, during the image reproducing process.

Next, the reproducing operation of the image data registered in this apparatus will be described.

FIG. 18 shows a display screen when the image data registered in the image data management system 15 is displayed on the display 4.

That is, in the present embodiment, each user catalog (user catalog 38a, user catalog 38b,
......), a window is displayed and the user catalog 38
A thumbnail image 60 and attribute data 61 of the thumbnail image 60 are displayed inside. It should be noted that the user can freely select and change, for example, only the file name and file type of the attribute data or all attribute data of the thumbnail image 60 by user designation. Then, when the user wants to display an image on the display 4, the user operates the arbitrary thumbnail image selected from the thumbnail format (see FIG. 7) described above with the mouse 6 to manage the thumbnail image. The displayed original image can be displayed on the display 4.

FIG. 19 is a flow chart showing the processing procedure of the image reproduction processing.

In step S91, it is determined whether the specified thumbnail image belongs to the panorama group,
When the answer is negative (No), it is determined that the image is a normal image and the image is displayed on the display 4 as it is.

On the other hand, the answer in step S91 is affirmative (Yes
If it is determined in step s) that the thumbnail image belongs to the panorama group, the process advances to step S92 to perform a combining process. That is, when managing a panoramic image with this panorama image synthesizing device, at the time of registration, corresponding point extraction processing (step S11 in FIG. 6) and synthesis parameter calculation processing (FIG. 6).
Since only the step S12) is executed, if it is desired to display the registered panoramic image on the display 4, the compositing process needs to be performed. Therefore, it is determined whether or not the thumbnail format designated in step S91 represents a group for panoramic images, that is, a panorama group. A panoramic image is created by synthesizing a plurality of belonging images. Then, in the subsequent step S93, when the thumbnail format expressing the panorama group is the format shown in FIGS. 7A and 7B, thumbnail images of the thumbnail format as shown in FIG. 7C are created. Then, in the following step S94, the specified panorama group is deleted from the image data management system 15, the newly created panorama image is registered in the image data management system 15, and the newly registered panorama image is displayed on the display 4 in step S95. Display on top and end the process.

Next, the combining process executed in step S92 will be described.

Since the composition parameters A, B, C and D have already been calculated at the time of registration and stored in the catalog table 42 as described above, these composition parameters A, B, and
It is calculated by substituting C and D into Equation (19).

[0075]

(Equation 5) Then, as shown in FIG. 20, the image is the left image (see FIG.
(A)) and the right image (FIG. 20 (b)), 2 of the right image
The double size is ensured as the composite image area K (FIG. 20 (c)). Here, first, the left image is directly used as it is in the composite image region K.
Copy to Next, the residual area O (x,
For y), O '(x', y ') is calculated based on the equation (19).
Is calculated, the pixels of O '(x', y ') of the right image are copied to the residual area O (x, y), and these are applied to all areas of the combined image to create a combined image.

FIG. 21 is a flow chart showing the processing procedure of the combining processing. In step S101, a double area of the first image (left image) is secured as a composite image area K, and in the following step S102, the first image is copied as it is to the composite image area K on the same coordinates. Then step S10
In step 3, O '(x', y ') is calculated for the remaining area O (x, y) of the composite image area K based on the equation (19).
In the following step S104, it is determined whether or not O '(x', y ') exists in the second image (right image). If not, the process proceeds to step S106, while if it exists, O' (x ', y'). The pixels of x ', y') are copied to the residual area O (x, y), and the process proceeds to step S106. Then, the above processing is repeatedly executed over the entire synthetic image area K, and when the processing is completed, the answer to step S106 is affirmative (Yes).
Then, the process is terminated and the process returns to the main routine (FIG. 19). As a result, the registered synthesis parameters A, B,
Image composition can be performed at the time of image reproduction processing using C and D.

The present invention is not limited to the above embodiment. In the above-described embodiment, the synthesis process is performed when the panoramic image is reproduced, but image data management is performed when, for example, an instruction from an operator at any time or when the personal computer 2 is in a no-load state for a certain time or more. It is also preferable to perform a combining process on the panoramic image data stored in the system 15.

In the above embodiment, the electronic camera 1 is connected to the personal computer 2 and a series of panoramic image processing is performed on the personal computer 2 side. However, the electronic camera 1 may perform the series of panoramic composition processing. In this case, the corresponding point extraction process and the parameter calculation process can be performed at the time when the required image is captured, and the combining process can be performed at the time of outputting the data.

[0079]

As described in detail above, according to the panoramic image synthesizing apparatus and the panoramic image creating method of the present invention, the process of calculating the synthesizing parameter when creating the panoramic image and the image using the synthesizing parameter are performed. Since the process of synthesizing can be performed independently and separately in time, the processing time for creating a panoramic image can be dispersed,
There is an effect that it is possible to reduce the time required to transfer image data from the imaging device to an external device such as an information processing device.

[Brief description of drawings]

FIG. 1 is an overall configuration diagram showing an embodiment of a panoramic image synthesizing apparatus according to the present invention.

FIG. 2 is a system configuration diagram of the panoramic image combination processing.

FIG. 3 is a diagram showing an image management table built in the electronic camera.

FIG. 4 is a display screen of a display showing a state when image data or the like built in an electronic camera is copied in a personal computer.

FIG. 5 is a data structure diagram of a user catalog.

FIG. 6 is a flowchart showing a processing routine of panoramic image processing.

FIG. 7 is a diagram showing a thumbnail format for a panoramic image.

FIG. 8 is a flowchart showing a processing procedure of corresponding point extraction processing.

FIG. 9 is a diagram showing a user interface of automatic corresponding point extraction processing.

FIG. 10 is a flowchart showing a processing procedure of automatic corresponding point extraction processing.

FIG. 11 is a diagram for explaining setting of a matching range.

FIG. 12 is a flowchart showing a processing procedure of full-auto corresponding point extraction processing.

FIG. 13 is a diagram illustrating a user interface of a fully automatic corresponding point extraction process.

FIG. 14 is a diagram showing a user interface of a semi-auto corresponding point extraction process.

FIG. 15 is a flowchart showing a processing procedure of semi-auto corresponding point extraction processing.

FIG. 16 is a conceptual diagram of corresponding point extraction execution processing.

FIG. 17 is a flowchart showing a processing procedure of corresponding point extraction execution processing.

FIG. 18 is a display screen when the image data registered in the image data management system is displayed on the display.

FIG. 19 is a flowchart showing a processing procedure of image reproduction processing.

FIG. 20 is a conceptual diagram showing an image synthesizing process.

FIG. 21 is a flowchart showing a processing procedure of a combining process.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Electronic camera 2 Personal computer 15 Image data management system 16 Panorama image composition system 20 Composition parameter calculation means 21 Image composition means

Front page continuation (72) Inventor Kotaro Yano 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor Tatsutsugu Katayama 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. Within

Claims (10)

[Claims] 1. A combination parameter calculation means for calculating a combination parameter for combining images, and a combination processing means for combining the images based on the combination parameters calculated by the combination parameter calculation means. A panorama image synthesizing apparatus for synthesizing a plurality of partially overlapping images to obtain a single image, comprising: storage means for storing the synthesizing parameter calculated by the synthesizing parameter calculating means, A panoramic image synthesizing apparatus, characterized in that it has a separation and independent execution means for separately and independently executing the means and the synthesis parameter processing means at different times. 2. An image pickup apparatus and an information processing apparatus for fetching image data picked up by the image pickup apparatus, wherein the synthesis parameter calculation means is executed when the image data is fetched from the image pickup apparatus to the information processing apparatus. The panoramic image synthesizing apparatus according to claim 1, wherein the synthesizing processing unit is executed at the time of image reproduction. 3. The parameter synthesizing means is executed when the image pickup by the image pickup apparatus is completed, and the synthesizing processing means is executed when the image pickup apparatus reproduces the image. Panoramic image synthesizer. 4. The parameter synthesizing means is executed when the image pickup by the image pickup device is completed, and the synthesizing processing means is executed when the panoramic image is output from the image pickup device to an external device. Claim 1 characterized by
The described panoramic image composition device. 5. The panoramic image synthesizing apparatus according to claim 4, wherein the external device is an information processing device. 6. A method of creating a panoramic image in which a plurality of images, which partially overlap each other, are combined to obtain a single image, wherein a panorama is created based on corresponding points of the plurality of images. After calculating the synthesis parameter for creating the image, the synthesis parameter is stored, and the image synthesis process is executed based on the stored synthesis parameter at a time different from the time when the synthesis parameter is calculated, and the panorama image is displayed. A method for creating a panoramic image, which is characterized by: 7. When transferring image data for a captured panoramic image from an image pickup device to an information processing device, while the calculation of the synthesis parameter is executed when the image data is loaded from the image pickup device to the information processing device, 7. The method of creating a panoramic image according to claim 6, wherein the image synthesizing process is executed during image reproduction. 8. The method for creating a panoramic image according to claim 6, wherein the synthesis parameter is calculated at the end of image capturing by the image pickup device, and the image synthesis process is executed at the time of image reproduction by the image pickup device. 9. The panorama according to claim 6, wherein the synthesis parameter is calculated at the end of image capturing by the image pickup device, and the image synthesis process is executed at the time of outputting the panoramic image from the image pickup device to the external device. How to create an image. 10. The method for creating a panoramic image according to claim 9, wherein the external device is an information processing device.