JP3832894B2 - Image synthesizer - Google Patents

Description

【００４６】
ここで、θは回転角、Δｘおよびｙは並進、ｍは倍率を示す。従ってパラメータＡ、Ｂ、Ｃ、Ｄを求めることによりこの座標変換を表すことができる。先の対応点抽出処理では、対応点（ｘ，ｙ）、（ｘ′，ｙ′）の複数の組を取得した。これを最小自乗法を用いてパラメータＡ、Ｂ、Ｃ、Ｄを求める。
【００４７】
【数２】

【００４８】
の条件で、
【００４９】
【数３】

【００５０】
を満たすパラメータＡ、Ｂ、Ｃ、Ｄを求める。
ここで、
【００５１】
【数４】

【００５２】
とすると、パラメータＡ、Ｂ、Ｃ、Ｄは次のように表すことができる。
【００５３】
【数５】

【００５４】
このパラメータｐ₁ からｐ₈ を求め、上式に代入することにより、パラメータＡ、Ｂ、Ｃ、Ｄを算出する。
【００５５】
最後に、画像合成処理について説明する。すでに、パラメータＡ、Ｂ、Ｃ、Ｄは求められているので、次の式
ｘ′＝Ａｘ＋Ｂｙ＋Ｃ
ｙ′＝−Ｂｘ＋Ａｙ＋Ｄ
に代入すればよい。
【００５６】
図１９にこれを図示したものを示す。画像が左、右画像の場合、左画像の２倍の大きさを合成画像２１０３として確保する。ここに、まず左画像をそのままコピーしてくる。次に合成画像の残りの領域（ｘ，ｙ）について、上式から対応する（ｘ′，ｙ′）を求める。そして右画像の（ｘ′，ｙ′）の画素を（ｘ，ｙ）にコピーする。これを合成画像の残りの領域全てに対して行う。
【００５７】
図２０に同様の内容をフローチャートに示す。ステップＳ２２０１で、第１の画像（２０での左画像）の２倍の領域を合成画像領域として確保する。ステップＳ２２０２で、第１の画像をこの合成画像領域に単純にコピーする。ステップＳ２２０３で、合成画像の残りの領域（ｘ，ｙ）について、上式から対応する（ｘ′，ｙ′）を求める。ステップ２２０４で、（ｘ′，ｙ′）は第２の画像（図１９での右画像）内にあるかどうかをチェックし、あればステップＳ２２０５で（ｘ′，ｙ′）の画素を（ｘ，ｙ）にコピーする。以上を合成画像の残りの領域全てに対して繰り返し、処理は終了する。
以上によって、最終的なパノラマ合成画像を作成することができる。
【００５８】
本実施の形態によれば、まず、自動的にパノラマ画像として合成する画像のセットを抽出してくれる。これによりユーザが合成したい画像のセットを指示する必要がない。また、合成する画像のセットが２枚のときはフルオートで、２枚より多いときもユーザは位置関係を指示するだけ（オート）で合成処理を行うことができ、ユーザの負担は非常に軽いものとなっている。さらにフルオート、オートのときに十分な対応点を検出できなかったときには、セミオート処理に切り替え、この場合もユーザは画像をドラッグして重ね合わせた簡単な位置を指定するだけである。このセミオート処理に切り替わる場合は少ないし、またそうなっても非常に簡単な操作で行えるので、これもユーザの負担は軽いと言える。
【００５９】
【発明の効果】
以上のように本発明による画像形成装置によれば、ユーザは、第１の表示領域に表示されている画像をＤｒａｇ＆Ｄｒｏｐ等により第２の表示領域に移動させて合成したい複数の画像を表示することによって、合成する画像を確認しながら容易な操作で合成画像を得ることができる。
【図面の簡単な説明】
【図１】本発明の実施の形態を示すブロック図である。
【図２】機器の構成例を示す構成図である。
【図３】フルオート処理のときに想定される２枚の画像の組み合わせを示す構成図である。
【図４】オート処理における相対的位置を指示するユーザインターフェースを示す構成図である。
【図５】セミオート処理におけるだいだいのオーバーラップ位置を指示するユーザインターフェースを示す構成図である。
【図６】カラメ内に記録された画像データの構成図である。
【図７】カメラ内の画像データをコンピュータ内へコピーする操作を示す構成図である。
【図８】コンピュータ内で管理するデータの構成図である。
【図９】カメラ内の画像データをコンピュータ内へコピーする際に行う処理のフローチャートである。
【図１０】パノラマ合成のフルオート、オート、セミオートの流れを示すフローチャートである。
【図１１】オート処理のユーザインターフェースを示す構成図である。
【図１２】オート処理のフローチャートである。
【図１３】合成時のマッチングが範囲を示す構成図である。
【図１４】フルオート処理のフローチャートである。
【図１５】セミオート処理のユーザインターフェースを示す構成図である。
【図１６】セミオート処理のフローチャートである。
【図１７】対応点抽出処理のフローチャートである。
【図１８】対応点抽出装置でのテンプレート画像とマッチング範囲を示す構成図である。
【図１９】合成処理を示す構成図である。
【図２０】合成処理のフローチャートである。
【図２１】従来の対応点指示を示す構成図である。
【符号の説明】
３０１ コンピュータシステム本体
３０２ ディスプレイ
３０３ マウス
３０５ キーボード
３０７ 電子カメラ
５０４ アプリケーションソフトウェア
５０５ オペレーティングシステム
５１８ ＣＵＰ[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image synthesizing apparatus suitable for use in a panorama synthesizing system or the like that synthesizes a plurality of images that partially overlap images on a computer.
[0002]
[Prior art]
The process of composing a plurality of images on which part of the image overlaps on a computer is generally called panoramic composition. This can be said to be processing from a request to capture a wide subject to make a single image. In addition, in electronic cameras, low resolution (small number of pixels) has been pointed out as a disadvantage compared to silver halide cameras and scanners. The panoramic image synthesis for an image taken with this electronic camera is important not only for taking a wide image but also as a means for taking a high-resolution image. Specifically, a single paper manuscript, magazine, etc. is shot in multiple parts to acquire high resolution data similar to that of a scanner, or a landscape is divided into multiple parts and shot in wide, high resolution. Demonstrates power in cases.
[0003]
In the panorama synthesis, the most important and difficult process is a process of finding an overlap position of a plurality of images. In other words, this is a process of finding the same point (hereinafter referred to as a corresponding point) from the two images (hereinafter referred to as a corresponding point extraction process). The difficulty (error rate) of the corresponding point extraction process varies depending on each image. If there are many unique feature shapes not found elsewhere in the overlapping area, it is possible to search for the corresponding points without making a mistake. However, if a similar pattern is present in other places in the overlap (for example, characters in the document), the corresponding points may be mistaken.
[0004]
Therefore, conventionally, it is common to have a user explicitly specify a corresponding point and perform fine adjustment based on the position to perform synthesis. FIG. 21 shows a conventional example. First, when the user designates a plurality of images to be synthesized, a window as shown in FIG. 21 is opened. The user designates corresponding points in the two images, and puts marks 21a, 21b, 22a, 22b. As processing, a pattern very close from the center of each mark is examined, and a positional relationship in which the two match best is obtained and used as a corresponding point. Based on the corresponding points, a parameter for synthesis is obtained and a synthesis process is performed.
[0005]
[Problems to be solved by the invention]
However, the conventional example has the following problems. First, the user must indicate the set of images that he wants to compose. This is not a problem when the total number of images managed on the computer is not so large, but an oversight or the like occurs when many images of thousands or more are managed. In addition, the user must designate the corresponding points almost accurately, and this also becomes a burden on the user as the number of operations increases.
[0006]
The present invention has been made in view of the above-described circumstances, and an object thereof is to obtain an image composition apparatus capable of performing panoramic image composition more easily.
[0007]
[Means for Solving the Problems]
In the image composition device according to the first aspect of the present invention, the first display area capable of displaying a plurality of images is the same as the second display area capable of displaying a plurality of images and different from the first display area. Display means for displaying on a screen; moving means for moving a plurality of images selected from images displayed in the first display area to a second display area according to a user's instruction; and the first Detecting means for detecting an overlap portion of a plurality of images moved from the display area to the second display area, and corresponding point extracting means for extracting corresponding points in the overlap portion detected by the detecting means; And combining processing means for creating a single image by joining the plurality of images based on the corresponding points extracted by the corresponding point extracting means, and the detecting means includes the moving means. The number of the plurality of images are moved within the case is less than the predetermined number of sheets, and detects the overlapping portion of the image automatically I.
[0008]
In the image composition device according to the second aspect of the present invention, a first display area capable of displaying a plurality of images, and a second display area capable of displaying a plurality of images and different from the first display area, Displaying means on the same screen, moving means for moving a plurality of images selected from images displayed in the first display area to the second display area according to a user instruction, Detection means for detecting an overlap portion of a plurality of images moved from the first display area to the second display area, and corresponding point extraction for extracting a corresponding point in the overlap portion detected by the detection means Means for combining the plurality of images based on the corresponding points extracted by the corresponding point extraction means to create one image, and the moving means comprises a predetermined number of sheets. When the above images are input, the images are moved on the second display area to designate the relative positions of the images, and the detecting means is configured to specify the overlap based on the designated relative positions. A part is detected.
[0009]
[Action]
According to the image composition device of the first aspect of the present invention, the first display area capable of displaying a plurality of images and the second display area capable of displaying a plurality of images and different from the first display area are provided. A plurality of images displayed on the same screen and selected from the images displayed in the first display area according to a user's instruction are moved to the second display area, and the first display area When the plurality of images moved to the second display area are less than the predetermined number, the overlap portion is automatically detected, and the corresponding points in the detected overlap portion are extracted and extracted. Based on the corresponding points, the plurality of images are connected to create one image.
[0010]
According to the image composition device of the second aspect of the present invention, the first display area capable of displaying a plurality of images and the second display area capable of displaying a plurality of images and different from the first display area are provided. A plurality of images displayed on the same screen and selected from the images displayed in the first display area according to a user's instruction are moved to the second display area, and the first display area When a plurality of images moved to the second display area are a predetermined number or more, the plurality of images are moved on the second display area to designate the relative positions of the images, Based on this relative position, an overlap portion is detected, corresponding points in the detected overlap portion are extracted, and a plurality of images are connected based on the extracted corresponding points to create one image. To do.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
The present embodiment performs simple and convenient panoramic image synthesis by the following procedure.
(1) First, when shooting an image with an electronic camera, the user sets the camera in the “panoramic image shooting mode” before shooting. In this mode, an identifier indicating a set of panoramic images is automatically recorded in the attribute information of the captured image.
(2) When this camera is connected to a computer and image + attribute information in the memory in the camera is copied to the HD of the computer, this attribute information is checked via application software. One set of images is automatically extracted from the attribute information having an identifier in the panoramic image shooting mode.
[0012]
Next, panorama image composition processing is started. In the image composition, full auto composition, which will be described later, is performed fully automatically only when there are two images. When there are three or more images, auto-composition described later is performed, in which the user designates only the relative positions of the top, bottom, left and right of the image. In addition, when the corresponding points are not sufficiently obtained in the full auto synthesis and auto synthesis check stages, or when the user wants to perform the synthesis processing in a shorter time by omitting the time required for detecting the corresponding points, the user Perform semi-auto synthesis, which will be described later, to specify the approximate overlap position.
[0013]
(3) “Full auto composition”: When one set of panoramic images is extracted in the above (2) and there are two images, full auto composition processing is started. There are four possible positions for combining the two sheets, as shown in FIGS. 3A to 3D. Accordingly, processing for obtaining corresponding points in these four cases is performed, and the most frequently obtained corresponding points among the upper, lower, left, and right are synthesized as correct synthesis positions. At this time, in any of the four cases, the certainty is low when the number of corresponding points is equal to or less than a predetermined amount. At this time, the full-auto synthesis process is discontinued and the process proceeds to a semi-automatic synthesis process. The only user operation in this mode is to copy the image from the camera to the computer, and the rest will be done automatically. Since it is generally considered that two sheets are combined except for special purposes, this combining process is most often performed.
[0014]
(4) “Automatic composition”: When one set of panoramic images is extracted in the above (2) and there are three or more images, this automatic composition processing is started. In this automatic composition, a set of images is displayed in a window. FIG. 4 shows a case where four images are displayed. The user drags these images and considers various arrangements, and instructs only the positional relationship between the top, bottom, left and right. Based on this, corresponding point extraction processing is performed. As a result, when the number of corresponding points that match at a predetermined level or more is greater than or equal to a predetermined amount, they are combined as a correct combining position. Otherwise, the certainty is low, so the auto synthesis process is terminated and the process proceeds to semi-auto synthesis process.
[0015]
(5) “Semi-automatic synthesis”: When the certainty of corresponding point extraction is low in the above (3) and (4), or when it is desired to obtain a faster synthesis result by saving the time required for extracting corresponding points, this semiautomatic Process. The user drags these images and designates the approximate position of overlap as shown in FIG. Based on this position information, corresponding point extraction processing is performed in a much narrower range than in the automatic processing. As a result, the best matching position is obtained and the synthesis process is performed.
[0016]
In the case of (4) and (5) above, the user performs an operation for extracting the corresponding points, both of which are only operations for dragging the image, and this is the simplest and common operation. The burden is small. In addition, in the semi-automatic synthesis of (5), since the image is only dragged to the approximate position, it is much easier and easier than the operation of explicitly specifying the points in the conventional example.
[0017]
FIG. 2 shows a configuration example of a personal computer system which is a platform on which the present invention can be implemented. In FIG. 2, 301 is a computer system main body, 302 is a display for displaying data, 303 is a mouse as a typical pointing device, 304 is a mouse button, and 305 is a keyboard. Reference numeral 307 denotes an electronic camera that can be connected to a computer, which is connected by a general-purpose interface capable of transferring images at high speed, such as a bidirectional parallel interface or a SCSI interface indicated by 306.
[0018]
FIG. 1 is a diagram showing a configuration of a panoramic image synthesis system including software and hardware. In FIG. 1, reference numeral 509 denotes hardware, which has a CPU 518. Reference numeral 505 denotes an operating system (OS) that operates on the hardware 509, and reference numeral 504 denotes application software that operates on the OS 505. Of the blocks constituting the hardware 509 and the OS 505, the blocks that are naturally included as constituent elements but are not directly necessary for describing the embodiment of the present invention are not shown. An example of such a block (not shown) includes a memory as hardware and a memory management system as an OS.
[0019]
In FIG. 1, 515 is a hard disk that physically stores files and data, 508 is a file system that constitutes the OS 505, and has a function that allows the application software 504 to input and output files without being aware of the hardware 509. is there. Reference numeral 514 denotes a disk IO interface for the file system 508 to read / write the hard disk 515. Reference numeral 507 denotes a drawing management system constituting the OS 505, which has a function that allows the application software 504 to perform drawing without being aware of the hardware 509.
[0020]
Reference numeral 513 denotes a video interface for the drawing management system 507 to draw on the display 302. Reference numeral 506 denotes an input device management system constituting the OS 505, which has a function that allows the application software 504 to receive user input without being aware of the hardware 509. Reference numeral 510 denotes a keyboard interface for the input device management system 506 to receive input from the keyboard 305. Reference numeral 512 denotes a mouse interface for the input device management system 506 to receive input from the mouse 303.
[0021]
Further, the electronic camera 307 is connected to a bi-directional interface or a SCSI interface 516, and can exchange image data and the like through the input device management system 506. Reference numeral 501 denotes an image data management system, and reference numeral 502 denotes a data management unit for managing image data with attribute information or keywords input by the user. Reference numeral 503 denotes a data display unit that searches and displays the managed image data using the attribute information or keywords input by the user. Reference numeral 517 denotes a panoramic image synthesis system, which receives an image shot in the panoramic shooting mode from the image data management system 501 and performs panoramic image synthesis processing. The synthesized image is registered in the image data management system 501.
[0022]
FIG. 6 shows a data structure of image data and attribute information stored in a built-in memory in the camera. First, an image management table 81 is placed in the memory, and corresponding image data 82 and attribute information 83 are referred to. The image data 82 is stored as either camera-specific format data (native data) or general-purpose format data such as JPEG. The native data is, for example, data obtained by simply A / D output from the CCD. In general, native data takes a short time to record, but the data size is large, and JPEG data takes a long time to record, but is different in that the data size can be reduced. The user selects and stores either one according to the shooting situation.
[0023]
In the attribute information 83, a file name 84, a filter type 85, a shooting date / time 86, and a shooting mode 87 are recorded. The file name 84 is a unique file name automatically assigned by the camera. The filter type 85 indicates whether it is a native data format or a JPEG format, or another general-purpose format supported by the camera. As for the shooting date and time 86, the camera has a calendar and a timer inside, and the date and time when the shutter button of the camera is pressed is recorded. The shooting mode 87 indicates a mode selected at the time of shooting among several types of shooting modes of the camera. If this is the “panoramic shooting mode”, an identifier 88 is further added. This identifier stores a mode ID 89 which is a unique number set when the panorama shooting mode is set, and data 90 indicating the number of pictures in that mode. Therefore, a plurality of images having the same mode ID 89 in the panorama shooting mode is one set. In the example of FIG. 6, since the landscape is photographed as two left and right images, 89a and 89b have the same mode ID.
As described above, the image data 82 and the attribute information 83 are stored in the camera.
[0024]
FIG. 7 shows a screen for copying data in the camera to a personal computer. The camera 307 in FIG. 1 is connected to the computer via the general-purpose interface 306, and the image data management system 502 is activated. The image data management system 502 displays the data in the camera in a window named camera catalog 91 in FIG. Here, 94 is a reduced image (thumbnail image) of the image data, and 95 is a file name, filter type, etc. in the attribute information. How much of the attribute information is displayed can be changed by user designation. A user catalog 92 displays a part of the user image database existing in the hard disk of the personal computer.
[0025]
When the user selects an image from the camera catalog 91 (93 is a frame indicating that it has been selected) and performs a Drag & Drop operation on the user catalog 92, copying is performed. At this time, whether it is a copy (data remains in the camera) or a move (data in the camera is deleted) can be switched to either by the user's designation. During this copy operation,
・ Native data is converted into a predetermined general-purpose format.
・ If there are images shot in the panorama shooting mode, combine them.
[0026]
The above operation is automatically detected and executed. FIG. 8 shows a data structure on the user catalog 91 at that time, and FIG. 9 shows a flowchart.
First, FIG. 8 will be described. First, the image data management system 508 manages the image data stored therein with a unique ID number. That is the data management table 1108. This corresponds to the data ID 1109 and the image data and attribute information linked thereto. This data ID is the basis of management. In the image data management system 502, a user can have an arbitrary number of user catalogs 92. Each catalog has a catalog table 1100. The image data in the catalog has a function for the user to categorize a plurality of images as one group. As a result, data can be managed in a hierarchy in one catalog.
[0027]
In the catalog table 1100, the data ID 1101 of the image belonging to this catalog and the group ID 1102 of the group to which it belongs are held. The group ID 1102 is linked to the group table 1103. The group attribute table 1104 is basically the same as the catalog table, and has a data ID of an image belonging to this group or a group ID belonging to this group. The difference is that it has group attribute data 1104 at the head. This stores a group name 1106, a creation date 1107, and a group type 1110. The group name 1106 is an arbitrary name given by the user. When a group is created as a set of panoramic images, this group name is given as “panoramic image” by default.
[0028]
The creation date / time stores the date / time when this group was created. The group type 1110 contains “user created” when created by the user, and “panoramic image” when a group is created as a set of panoramic images. In the case of a panoramic image, a mode ID 89a is stored by linking with an identifier. Actual image data and attribute information are stored in the same structure as in FIG. 6, and these are referenced from the data management table 1108.
[0029]
Next, FIG. 9 will be described. While processing of all image data to be copied is not completed in step 1000, in step 1001, one piece of image data and attribute information associated therewith are first acquired in the copying operation. In step 1002, it is determined from the filter type 85 in the attribute information whether this image data is native data. If so, the native data is converted into a general-purpose format (JPEG, TIFF, etc.) determined as a default in step 1003. When the conversion is completed, the filter type 85 is also updated. In step 1004, the shooting mode 87 is checked to check whether the image is shot in the panoramic shooting mode. If it is not a panoramic image, it is registered as normal image data in step 1008. Specifically, the data management table 1108 in FIG. 8 is registered with a unique data ID, and the data ID is registered in the catalog table 1100.
[0030]
If it is a panoramic image, it is checked whether a group for this panoramic image has already been created. This is performed by tracing the catalog table 1100 of FIG. 8 and checking whether the mode ID 89a of the group ID is the same as the mode ID 89a of the image. If there is no corresponding group, create a group. In this case, a group ID 1102 is newly registered in the catalog table 1100, and a group name 1106, a creation date 1107, and a group type 1110 are created. “Panorama shooting” is recorded in the group type 1110, and the mode ID 89a in the attribute information of the image is stored. In step 1007, the panoramic image data is registered with a unique data ID in the management table 1108, and the data ID 1105 is registered.
[0031]
The above series of processing is performed for all images to be copied. If the processing is completed for all the images, the process proceeds to step 1009. In step 1009, it is checked whether or not a group of panoramic images has been created among those copied so far. If so, panoramic image synthesis processing described later is performed in step 1010 using the images in the group. Otherwise, the process is terminated.
[0032]
FIG. 10 shows a flowchart of the panoramic image synthesis process in step 1010. First, in step 1200, it is checked whether there are two or more images in the group. When there are two sheets, full auto processing described later is entered in step 1202. If there are more than two sheets, an automatic composition process (described later) is entered in step 1201. In the processing of steps 1201 and 1202, it is checked in step 1203 and 1204 whether the result is success or failure. This success or failure is determined based on whether or not a predetermined number or more of corresponding points between images have been found. Therefore, since the determination is made at an early stage of the corresponding point extraction process in the entire synthesis process, even if the result is unsuccessful, the time for the user to wait until obtaining the result is short. If successful, the process ends. If unsuccessful, a semi-automatic process (to be described later) is performed in step 1205.
[0033]
FIG. 11 shows a user interface for the auto synthesis process. First, all images belonging to the group of panoramic images in step 1009 are resized and displayed on the screen so as to fit in the window. The user sees this and drags and rearranges them in the correct order. In the example of FIG. 11, three images are displayed, and the image located at the bottom of FIG. 11A is actually the rightmost, so when dragging to the right, three images are arranged side by side from that position. It is detected that the panorama is as shown in FIG. Then, it is resized and displayed so that it can enter the window.
[0034]
FIG. 12 shows a flowchart of the auto composition process. First, in step 1301, the positional relationship information rearranged by the user is acquired. In step 1302, a search range is set in order to find matching points. This will be described with reference to FIG. As a rule when shooting as a panoramic image, if the overlap is 10% minimum and 50% maximum, and the vertical displacement perpendicular to it is determined to be 5% or less, the overlap range is always shown in FIG. 1504. Moreover, the range which may overlap is a range shown by 1505 of FIG.5 (b). According to the above rule, the point 1503 in the range 1504 has a corresponding point in 1505 in the search range. In the corresponding point extraction processing described later, it is determined whether or not the search range 1505 is matched.
[0035]
Returning again to the flowchart of FIG. 12, in step 1302, the parameters used for setting the search range 1505 are set. In step 1303, processing for extracting corresponding points is performed. Details of this will be described later. In step 1304, it is determined whether or not the obtained number of corresponding points is equal to or greater than a predetermined value. If the number of corresponding points is equal to or smaller than the predetermined value, sufficient corresponding points cannot be automatically found, and the process proceeds to semi-automatic processing. If it is greater than the predetermined value, the process proceeds to the synthesis parameter processing in step 1305. Here, the movement, enlargement (reduction), and rotation parameters used in the synthesis are obtained from the coordinates of the corresponding points. Details thereof will be described later. In step 1306, an image is synthesized based on these parameters. Details thereof will also be described later.
[0036]
FIG. 14 shows a flowchart of the full auto process. First, in step 1601, a matching range which is a search range for extracting corresponding points is set. This is the same as step 1302 described above. Next, corresponding point extraction processing is performed four times. In the case of full auto, since the number of sheets is limited to two, there are four possible positional relationships of image 1 and image 2, that is, top and bottom, bottom and top, left and right, and right and left. Accordingly, corresponding point extraction processing is performed for these four cases, and the number of points that can be extracted as corresponding points and the average of matching levels as corresponding points are held. These processes are steps 1602 to 1609. In step 1610, for the above four cases, it is checked whether or not there is a corresponding point having a predetermined number or more corresponding points extracted. If there is none, proceed to semi-automatic processing.
[0037]
If the number of extracted corresponding points is greater than or equal to the predetermined number, in step 1611, the one with the highest average of the matching levels as the corresponding points is assumed to be the true positional relationship. In a normal image, if the corresponding point is greater than or equal to a predetermined value, it should be only one of the four. However, for example, when a document is divided and taken, similar characters are lined up and the positional relationship is incorrect. Even in this case, there is a case where a predetermined value or more is extracted as a corresponding point. Therefore, in this step 1611, the best fit (the highest average match level) is selected. The synthesizing parameter process in step 1612 and the image synthesizing process in step 1613 are the same as those in steps 1305 and 1306.
[0038]
FIG. 15 shows a user interface in the semi-auto process. First, all images belonging to the group of panoramic images in step 1009 are resized and displayed on the screen so as to fit in the window. This is overlapped with the overlap position that the user sees. The overlapping area is displayed by performing an AND operation for each bit in units of pixels. Therefore, both images can be seen through the overlapped part. Then it is resized and displayed again to fit into the window.
[0039]
The above operations are basically the same as the operations in the automatic processing, and the burden on the user is small. The difference is that the position where the image is dragged and released with a pointing device such as a mouse is rearranged and displayed using only the positional information (automatic processing) and the position information is used as it is and overlapped and displayed. (Semi-automatic processing). Also, since it can be seen through in the Drag by the previous AND operation, the approximate position can be easily adjusted.
[0040]
FIG. 16 shows a flowchart in the semi-auto process. The processing here is almost the same as in the automatic processing. In step 1701, overlap position information obtained by the user of the image is acquired. In step 1702, a matching range is set. The range here is a predetermined range (an error range of an assumed position matched by the user + margin). Accordingly, the range is much narrower than the range at the time of auto processing, and the calculation time can be shortened and the accuracy can be improved. The corresponding point extraction process, the synthesis parameter setting process, and the image synthesis process in steps 1703, 1704, and 1705 are all the same as in the auto process.
[0041]
FIG. 17 shows a corresponding point extraction process flowchart, and FIG. FIG. 18 shows an example of two left and right images. When the number of images is larger than two, the composition is basically the same because the composition of the two images may be repeated several times. First, in accordance with the shooting rules, the range 2005 for setting the template is set to a range of 90% vertically and 10% horizontally. In addition, the search range is set to a range 2006 of 100% vertical and 50% horizontal because there is a possibility that a corresponding point exists. A point whose edge is stronger than a predetermined value is searched from a range 2005 in the image, and a rectangle of vertical and horizontal n pixels is cut out as a template image centering on the point. The template image 2003 is placed on the search range 2006, and the difference is obtained in units of pixels. The position where the sum is minimized is obtained by shifting the search range 2006 by one pixel. If the minimum value of the search results on the search range 2006 is not more than a predetermined value, the points (x, y) and (x ′, y ′) are held as a pair of corresponding points.
[0042]
The above is the outline of the processing. This will be described once more along the flowchart of FIG. First, in step 1901, an edge extraction image is created. In step 1902, a point whose edge is stronger than a predetermined value is searched from a range 2005 for setting a template in the edge extracted image. If there is such a point, in step 1903, an image is cut out from the point in a rectangle of vertical and horizontal ± n pixels and used as a template image. In step 1904, a search range in the right image is set from the position of the point. In step 1905, the image in the search range and the template image are overlapped, and the absolute value of the difference between the pixel values is obtained in units of pixels to obtain the total. In step 1906, it is checked whether or not the sum of the differences is the previous minimum value. If so, in step 1907, the coordinates of the point in the search range and the minimum value are held.
[0043]
The above is repeated over the entire search range to find the point with the best match (with the smallest difference). In step 1908, it is checked whether the search has been performed over the entire search range. In step 1909, it is determined by comparing with a predetermined value L whether the minimum value obtained as a result is a sufficiently small value (a reliable corresponding point). If it is smaller than the predetermined value L, in step 1910, the coordinates (x, y) of the point at which the template image is cut out in the list of corresponding points, and the coordinates (x ′, y ′) of the point for which the minimum value has been obtained. The minimum value is registered. The above is performed for all the points where the edge of the template setting range is stronger than the predetermined value in step 1911, and when completed, the average value is obtained from all the minimum values in the list of corresponding points, and this is set as the matching level value in step 1912. Hold. The corresponding point extraction process is thus completed.
[0044]
Next, the synthesis parameter setting process will be described. When the number of images is two (when two or more images are combined, the combination of two images is repeated, so two images may be considered first), and the shift is the translation in the x and y directions, the rotation, and the enlargement ratio It can be expressed by the difference. Accordingly, the corresponding points (x, y) and (x ′, y ′) can be expressed as follows.
[0045]
[Expression 1]

[0046]
Here, θ is a rotation angle, Δx and y are translations, and m is a magnification. Therefore, the coordinate transformation can be expressed by obtaining the parameters A, B, C, and D. In the previous corresponding point extraction process, a plurality of sets of corresponding points (x, y) and (x ′, y ′) are acquired. The parameters A, B, C, and D are obtained by using the least square method.
[0047]
[Expression 2]

[0048]
In the condition of
[0049]
[Equation 3]

[0050]
Parameters A, B, C, and D that satisfy the above are obtained.
here,
[0051]
[Expression 4]

[0052]
Then, the parameters A, B, C, and D can be expressed as follows.
[0053]
[Equation 5]

[0054]
This parameter p ₁ To p ₈ And parameters A, B, C, and D are calculated.
[0055]
Finally, the image composition process will be described. Since the parameters A, B, C, and D have already been obtained, the following formula
x ′ = Ax + By + C
y ′ = − Bx + Ay + D
Can be substituted into.
[0056]
FIG. 19 illustrates this. When the images are the left and right images, the composite image 2103 is secured twice as large as the left image. First, the left image is copied as it is. Next, for the remaining area (x, y) of the composite image, the corresponding (x ′, y ′) is obtained from the above equation. Then, the pixel (x ′, y ′) in the right image is copied to (x, y). This is performed for all remaining areas of the composite image.
[0057]
FIG. 20 shows a similar content in the flowchart. In step S2201, a region twice as large as the first image (the left image in 20) is secured as a composite image region. In step S2202, the first image is simply copied to this composite image area. In step S2203, the corresponding region (x ′, y ′) is obtained from the above equation for the remaining region (x, y) of the composite image. In step 2204, it is checked whether (x ′, y ′) is in the second image (the right image in FIG. 19), and if there is (x ′, y ′) in step S2205, the pixel (x , Y). The above is repeated for all the remaining areas of the composite image, and the process ends.
As described above, a final panorama composite image can be created.
[0058]
According to this embodiment, first, a set of images to be automatically combined as a panoramic image is extracted. This eliminates the need for the user to designate a set of images to be synthesized. In addition, when there are two sets of images to be combined, it is fully automatic, and when there are more than two images, the user can perform combining processing only by instructing the positional relationship (auto), and the burden on the user is very light. It has become a thing. Further, when sufficient corresponding points cannot be detected in full auto and auto, the mode is switched to semi-auto processing, and in this case, the user simply designates a simple position by dragging and superimposing the images. There are few cases of switching to this semi-automatic process, and even if so, it can be done with a very simple operation, so it can be said that the burden on the user is also light.
[0059]
【The invention's effect】
As described above, according to the image forming apparatus of the present invention, the user moves the image displayed in the first display area to the second display area by Drag & Drop or the like and displays a plurality of images to be combined. Thus, a synthesized image can be obtained by an easy operation while confirming the image to be synthesized.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an embodiment of the present invention.
FIG. 2 is a configuration diagram illustrating a configuration example of a device.
FIG. 3 is a configuration diagram showing a combination of two images assumed in the full auto process.
FIG. 4 is a configuration diagram showing a user interface for designating a relative position in auto processing.
FIG. 5 is a block diagram showing a user interface for instructing a substantial overlap position in semi-automatic processing.
FIG. 6 is a configuration diagram of image data recorded in a caramel.
FIG. 7 is a configuration diagram showing an operation for copying image data in a camera into a computer.
FIG. 8 is a configuration diagram of data managed in a computer.
FIG. 9 is a flowchart of processing performed when copying image data in a camera into a computer.
FIG. 10 is a flowchart showing the flow of panoramic synthesis full-auto, auto, and semi-auto.
FIG. 11 is a block diagram showing a user interface for auto processing.
FIG. 12 is a flowchart of auto processing.
FIG. 13 is a configuration diagram showing a range of matching at the time of synthesis.
FIG. 14 is a flowchart of a full auto process.
FIG. 15 is a block diagram showing a user interface for semi-automatic processing.
FIG. 16 is a flowchart of a semi-auto process.
FIG. 17 is a flowchart of corresponding point extraction processing;
FIG. 18 is a configuration diagram showing a template image and a matching range in the corresponding point extraction device.
FIG. 19 is a configuration diagram showing a synthesis process.
FIG. 20 is a flowchart of a composition process.
FIG. 21 is a block diagram showing a conventional corresponding point instruction.
[Explanation of symbols]
301 Computer system body
302 display
303 mouse
305 keyboard
307 Electronic camera
504 Application software
505 operating system
518 CUP

Claims (4)

Display means for displaying a first display area capable of displaying a plurality of images and a second display area capable of displaying a plurality of images different from the first display area;
Moving means for moving a plurality of images selected from the images displayed in the first display area to the second display area in accordance with a user instruction;
Detecting means for detecting overlapping portions of a plurality of images moved from the first display area to the second display area;
Corresponding point extracting means for extracting corresponding points in the overlap portion detected by the detecting means;
Combining processing means for connecting the plurality of images based on the corresponding points extracted by the corresponding point extracting means to create one image;
With
The image synthesizing apparatus according to claim 1, wherein the detecting unit automatically detects an overlapping portion of the images when the number of the plurality of images moved by the moving unit is less than a predetermined number. Display means for displaying a first display area capable of displaying a plurality of images and a second display area capable of displaying a plurality of images different from the first display area;
Moving means for moving a plurality of images selected from the images displayed in the first display area to the second display area in accordance with a user instruction;
Detecting means for detecting overlapping portions of a plurality of images moved from the first display area to the second display area;
Corresponding point extracting means for extracting corresponding points in the overlap portion detected by the detecting means;
Combining processing means for connecting the plurality of images based on the corresponding points extracted by the corresponding point extracting means to create one image;
With
The moving means, when a predetermined number of images or more are input, moves these images on the second display area and designates the relative position of each image;
The image synthesizing apparatus, wherein the detecting means detects the overlap portion based on the designated relative position.   3. The image according to claim 2, wherein the detecting unit detects an overlap portion of each image from position information of each image designated by overlapping the images on the second display area by the moving unit. Synthesizer.   The plurality of images includes identification information in which each image represents a set of combined images, and the detection unit and the combination processing unit perform processing based on the identification information. Item 3. The image composition device according to Item 2.

Images