JP2000244814A - Image compositing device and recording medium where image compositing method is recorded - Google Patents

Abstract

PROBLEM TO BE SOLVED: To confirm a composition result in real time and to improve the usability by an operator by providing a means which generates indicator information for displaying a composite image, a means which composes a panoramic image by using a difference image and a movement quantity. SOLUTION: A user informs an arithmetic processor 116 of transition to a state wherein panoramic composition is performed in real time first by using an input device 120. The arithmetic processor 116 when entering the panoramic photography mode inputs the video signal outputted from a video camera part 100, frame by frame, finds the camera movement quantities of the inputted frame images, and gathers parameters and data needed for the panoramic composition. At this time, a the whole or partial area of a display 112 is set as a preview screen for confirming the composition state and the state of a composite image which is connected in timing to the movement of the camera is updated and displayed in real time.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a synthesizing apparatus for synthesizing a panoramic image with a wide viewing angle and a recording medium on which a synthesizing method is recorded, and more particularly to an image synthesizing apparatus for synthesizing a panoramic image from continuous moving images in real time. And a recording medium on which the synthesis method image is recorded.

[0002]

2. Description of the Related Art A technique of synthesizing a plurality of images taken from different visual fields to form a single wide-field panoramic image has attracted attention. Recently, a technique called Video Mosaic, which creates a panoramic image similarly from a continuous moving image captured while changing the field of view by panning a video camera, has been attracting attention. For example, Teodosio, L. of the Massachusetts Institute of Technology, USA. , Bender, W. Salien
t Video Stills: Content and Context Preserved, Pr
oc. ACM Multimedia 93, pp. 39-46 (1993). In addition to these, there are various other examples of research.Each method involves finding the common area of multiple images obtained in succession and pasting the images together so that the common areas overlap. I have a panoramic image. As you move the camera, the captured image will contain images of a new field of view, so they will be joined together,
Finally, a wide-field panoramic image is obtained.

However, in many research examples, emphasis is placed on improving the accuracy of a composite image, such as how to find such a common area with high accuracy and how to correct the shift of the peripheral portion of the image due to lens distortion. However, it is hard to say that a study from the viewpoint of usability of an operator who actually uses such a panorama composition technique is sufficient. Generally, in order to perform image recognition on a natural image in which the type of a target image cannot be specified, it is necessary to intervene in various aspects and give appropriate judgment. In this case, the high-speed response of the processing greatly affects the usability of the operator. Depending on the application, if it is desirable to use a superior usability method that can process the input video in real time, that is, without delay with respect to the input of the video, and can confirm the result immediately, as long as the image is connected without failure. Not a few.

[0004]

In the creation of a panoramic image, two types of processing must be performed. One is a camera motion estimation process for recognizing continuous input images and determining coordinates of a common area. the other one is,
This is a synthesis process for actually connecting images based on the obtained motion amount. From a usability standpoint, it is most desirable that these be done simultaneously and in real time. That is, at the same time as the camera is shaken, the shake amount is calculated, the images are automatically joined, and when the shooting is completed, it is ideal that a composite image has already been made and can be confirmed. This makes it easy to redo as many times as you like until you get the image you like, and you can find mistakes on the spot even if you take a picture that is essentially impossible to synthesize, such as missing information. , You can fundamentally modify the way you shoot.

At the same time as the camera is shaken, the shake amount is calculated and the images are automatically joined together. When the photographing is completed, a composite image has already been made and can be confirmed. All of the combining processes need to be performed in real time. The processing for estimating the amount of camera motion is described in, for example,
1-004398 "High-speed processing method" described in "Digital Wide Camera". However, it cannot be said that the synthesis processing has been sufficiently studied to a level that can be simultaneously executed in real time. Simply calculating the amount of motion in real time at the same time as shooting will shorten the amount of calculation of the amount of motion after shooting, but it will not be possible to check the synthesis result without waiting for the time for the bonding process between the images. Is not enough.

SUMMARY OF THE INVENTION An object of the present invention is to provide a method of synthesizing a panoramic image from a continuous moving image, in order to improve the usability of an operator, to realize a high-speed responsiveness of the synthesizing process in which the synthesis result can be confirmed in real time. is there.

[0007]

In order to achieve the above-mentioned object, a display means for displaying a moving image being photographed, and frame images sequentially taken from the moving image and having different capture times.
Means for comparing the two frame images to determine the amount of movement, means for recording the difference image between the two frame images and the amount of movement, and combining the stored amount of movement and the captured frame image. Means for creating indicator information symbolizing the synthesized image, and means for synthesizing a panoramic image using the difference image and the movement amount.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, one embodiment of the present invention will be described in detail. FIG. 1 is an example of a schematic block diagram of a system configuration for realizing the present invention. Reference numeral 100 denotes a general video camera image pickup unit, which converts a captured image into a video signal line 10.
The data is transmitted to the arithmetic processing unit 116 via 6. The camera unit is connected to an arithmetic processing unit 116 by a control signal line 108,
The camera can set camera parameters such as changing the zoom magnification of the video camera and adjusting the white balance as needed. If such setting is unnecessary, it can be omitted. Arithmetic processing unit 11
6 is the same as the system configuration of a digital computer currently used in general, particularly a personal computer, and its display output is supplied to a display 112 through a video signal line.
An instruction received from the user via the input device 120 is drawn to the arithmetic processing unit 116. The video signal output from the camera unit 100 is sequentially 102 A / D
The data is converted into digital image data by the converter, enters the memory 114 via the interface 104, and is processed by the CPU 110 according to a program stored at another address of the memory 114.

Here, the signal from the video camera is NTSC
Although it is explained in the example of analog video format such as
102 when using a digital output video camera.
A / D converter is unnecessary. In this case, an image format converter may be required instead. Further, various kinds of information can be stored in the external information storage device 118 as needed for processing.

Various data created by the processing described below are stored in the memory 114, and are referred to as needed when the CPU 110 performs the processing. In the following description, all the software will be described as being executed by the CPU, but a part or all of the software may be replaced with a hardware logic circuit and executed. Further, an intelligent video camera form in which the camera unit 100, the arithmetic processing unit 116, and the display 112 are integrated may be used.

Hereinafter, a software flow executed on the above-mentioned hardware for solving the above-mentioned problem will be described in detail. First, a method of creating a panoramic image using the software will be described. The user can use the input device 120 to execute the arithmetic processing unit 116 using the input device 120 to switch to a state where panorama synthesis is performed first in real time (hereinafter, referred to as a panorama shooting mode).
Tell The input device corresponds to a keyboard, a mouse, and the like when using a computer system, and corresponds to a switch such as a “panorama shooting start button” when the camera is integrated. Arithmetic processing unit 116
Enters a panorama shooting mode, inputs a video signal output from a video camera for each frame, obtains a camera motion amount of the input frame image, and collects parameters and data necessary for panorama synthesis. At this time, the whole or a part of the display 112 is set as a preview screen for checking the synthesis status,
The state of the synthesized image that is connected according to the movement of the camera is updated and displayed in real time.

The image is shown in the display 112 of FIG. The present invention is characterized in that a composite image 122 for preview display (hereinafter, referred to as a preview image) is always scaled and displayed on the preview screen so that the entire image fits into the maximum size. . This makes it possible to instantly grasp the overall synthesis state. At this time, if the preview screen is rectangular,
Blank part 1 where no image exists, depending on the shape of the composite image
There are 24, which are filled with an appropriate background color.
The position of the latest input frame in the composite image is highlighted by changing the color of the frame or by changing the color of the frame so as to provide a role of a compass that does not disturb the direction in which the camera should be moved (126).

When the user again instructs the end of the panoramic photographing mode using the input device 120, input of a new frame image is stopped, and the apparatus enters a standby state. The display 112 holds the preview image updated by the last frame input, and the user sees the image to determine whether or not a desired result is obtained. A composite image as an output (hereinafter, referred to as a final output composite image) is created and stored in the external information recording device 118 or the like.

FIG. 2 shows an example of a flowchart for realizing the above functions. This algorithm has the following three basic concepts to enable high-speed processing essential for the above functions.

(1) Apart from the final output composite image, a composite image of reduced size is created exclusively for preview.

(2) The image to be stored for the final composite image is limited to a minimum necessary area.

(3) The creation of a preview image of an arbitrary shape and an arbitrary size is compatible with memory efficiency.

(1) is a device for reducing the time required for the synthesizing process. Generally, it is not always necessary to use a large-sized image in order to determine whether a created image is a desired image. Therefore, at the time of photographing, a final output composite image is not created, and a composite image of reduced size is separately created for preview. Thus, the play-view image is completed almost at the same time as the shooting. Since the preview image is synthesized based on the obtained camera motion estimation amount, the preview image is very close to the actual synthesis result, and distortion and mistakes are reflected as they are. The user determines whether the desired image has been obtained by this preview,
If it is confirmed, a final output composite image of the actual size is created again. If it is determined to be unsuccessful, it is possible to immediately start over and save the time for synthesizing the actual size, so that efficient work can be performed.

(2) is a device for reducing the recording time of the original data for the final output composite image. When the final output composite image is created later by adopting (1), it is necessary to temporarily record a frame image as its original data. Recording successively new frame images that are input one after another is a factor that hinders high-speed processing, such as a long data transfer time and an increase in required memory capacity.

Here, as shown in FIG. 3, a panoramic composite image is obtained by superimposing successive frame images one by one. FIG. 3 is an example in which a panoramic image is created from a series of panned images. The upper side schematically shows an image composed of continuous frame images, and the lower side shows a panoramic image synthesized therefrom. A plurality of lines connecting the top and bottom represent the correspondence between the frame images. In the lower panoramic image, there is always a region (hereinafter, referred to as a common region) that is consistent in image between successive frames, and the redundancy as data is extremely high. Therefore, a panoramic image can be sufficiently reproduced by combining only the areas where the common area between the consecutive frames is omitted. Therefore, by recording only the area excluding the common area per frame, the amount of data to be recorded per frame is greatly reduced, the data transfer time is shortened, and the required memory capacity is reduced. Become. The entire image is needed just to overlay the last frame,
Since it is only one time and the last, even if the data transfer takes a little time, there is no problem that the next frame does not exist and the process for the next frame cannot be completed in time.

Regarding the last (3), the preview image is not only obtained at the same time as the end of shooting, but also updated during shooting in real time in response to the movement of the camera, that is, in synchronization with the movement of the camera. It is easier for the user to confirm and the usability is better. However, the panoramic image to be synthesized becomes monotonically larger as the camera is moved, and depending on how the camera is moved,
Their shapes and sizes vary. In order to display the preview image, the panorama image to be synthesized for the preview must always be created in the memory area, but preview images of all possible shapes and sizes are created in the memory area To do so, the memory area must be made very large in advance.

According to the guideline (1), even if a reduced frame image is used, the required capacity of the memory area is enormous.
In addition, there are many memory portions that are not actually used, and the memory use efficiency is extremely low. Of course, it is possible to first determine the direction in which the camera is to be moved, and to secure the necessary memory area for that. However, in this method, the user has to instruct the direction in which the camera should be moved in advance, and even if the user changes his mind on the way, he cannot cope at all. Also, there are many cases where the user cannot predict how large an image will be finally. It is desired to support a composite image of unlimited size and unlimited shape.

Here, the size of the preview screen for displaying the preview image on the monitor is fixed, and when the size of the preview image is larger than that of the fixed size preview screen, the size of the preview image is adjusted to fit the screen size. So that it is scaled down and displayed. Conversely,
The number of pixels exceeding the resolution of the preview screen is wasted when displaying the preview image.

Therefore, when the size of the preview image exceeds n times the size of the preview screen (ratio as a vertical or horizontal length; n × n times in area), the size of the preview image Force by 1 / m times (same area
1 / (mxm) times). When the size of the reduced preview image exceeds n times the preview screen again, the preview image is reduced by 1 / m times. As a result, the size of the memory area for the preview image can be limited to n times or less of the size of the preview screen, and the preview image displayed on the preview screen has a higher resolution than that having a larger memory area. The effect that there is no deterioration is obtained.

Based on the above, the description returns to the flowchart of FIG. First, various initializations are performed in processing 200. For example, as a working buffer for creating a preview image, a memory area (hereinafter, referred to as a preview buffer) having a size obtained by increasing the size of the preview screen by n times vertically and horizontally is secured. The preview image is synthesized in the preview buffer. An area called a preview display area is set in the preview buffer, and this area is always scaled, that is, enlarged / reduced so as to just fill the size of the preview screen, and the preview screen is displayed.

First, an area having the same size as the preview screen in the center of the buffer is set as a preview display area, and as the preview image becomes larger on the preview buffer, the entire preview image is always included. The preview display area is sequentially enlarged. Next, an initial value of a preview reduction ratio r, which is a variable for creating a preview image, is calculated (202). r is a ratio between the size of the frame image input from the camera and the size of the preview screen, and is a parameter for adjusting the input frame image to be displayed in the full preview screen in the initial state where the camera does not move. It is. For example, if the size of the input frame image is the same as the size of the preview screen, r = 1.

In a subsequent step 204, the latest one frame image of the video image captured by the video camera is input.
Then, the moving amount detection processing of the image is performed while comparing the frame images inputted one after another. Various methods such as optical flow are considered for the movement amount detection. Here, as a method capable of calculating at high speed, for example,
Use the method described in "Digital Wide Camera" in Japanese Patent Application No. 9-153303. This method enables very high-speed processing to obtain the movement amount from the projection distribution of the image.

In step 206, a horizontal projection distribution Px and a vertical projection distribution Py are created from the frame image. The horizontal projection distribution is obtained by averaging the color and luminance of each row of pixels constituting an image, and is a one-dimensional series of average values for the number of rows. The vertical projection distribution is a one-dimensional series in which the average value of the color and the luminance is similarly obtained for each column of the image, and the average value is equal to the number of columns. For example, the arrangement of values of the vertical projection distribution has a property of shifting forward or backward correspondingly when an image moves left and right. Similarly, the vertical projection distribution shifts in the forward and reverse directions in accordance with the vertical movement of the image. Therefore, by comparing the horizontal / vertical projection distribution created from a certain frame image with the horizontal / vertical projection distribution created from the next input frame image, it is possible to determine how much each of the series of average values has shifted. ,
The movement amount in the horizontal / vertical direction between the two images is obtained. Since the image changes, the arrangement of the values of the projection distribution does not completely match due to noise and slight deformation, etc.
One of the two projection distributions to be compared is sequentially shifted by δ pixels and compared (where δ is any integer in the range from −R to + R, where R is a value indicating a preset search range). Δ at the time of lowering is obtained as the moving amount. The difference is
The absolute sum of the differences between the average values of the colors and luminances of the pixels in the corresponding row or column is used (210-212).
However, in the case of the first image input, since there is no frame image to be compared in the past, the process returns to the processing 204 (20).
8).

In the process 214, a common area between the immediately preceding frame image and the current frame image is obtained from the obtained horizontal / vertical movement amount δ, and only the area of the immediately preceding frame other than the common area is recorded. I do. As a result, the storage capacity can be reduced as compared with the case where recording is performed for all areas.

In the process 216, the obtained value of the horizontal / vertical movement amount δ is multiplied by r to obtain the movement amount for preview synthesis, and the reduced frame image drawn on the preview buffer immediately before is calculated. The image reduced by r times is superimposed and drawn on the position shifted by δ × r obtained above. The preview display area updated to include the entire preview image enlarged in this way is displayed on the preview screen (218).

At this time, if the entire preview image is
When the preview buffer is almost full (220), the entire preview buffer is reduced to 1 / m times (222). At this time, the preview display area is also reduced to 1 / m times. Then, r is similarly multiplied by 1 / m.

FIG. 4 shows a schematic flow of the preview buffer reduction process. Reference numeral 400 denotes the entire preview buffer, and reference numeral 402 denotes a preview display area.
Reference numeral 404 denotes a preview image. The position and size of the preview display area change so as to always circumscribe the entire preview image. A shows a state immediately after the start of panorama shooting, and the input frame image is drawn in the center. As the camera is moved, the preview image expands, and the preview display area follows and becomes larger (B). Then, as shown in C, when the preview image reaches the end of the preview buffer, it runs off the buffer, so that the entire preview buffer is reduced by a factor of 1 / m here (D). At this time, the preview display area is also reduced at the same magnification. Then, when the preview image further extends, drawing of the image is continued in accordance with the extension.

Here, assuming that n = 3 and m = 2, the center of the preview buffer and the center of the display area are first aligned, and then the preview display area is extended according to the photographed image and reaches the end. Is always larger than twice the original size of the preview screen (length / width ratio is twice). Therefore, even if this is set to 倍 (the vertical / horizontal length ratio is 倍) for the entire preview buffer, the size of the preview display area will not be smaller than the original size of the preview screen. Note that n = 3,
When m = 2, when the preview display area has just reached the buffer, the preview display area becomes twice the original size of the preview screen. Therefore, if the preview buffer is halved, the size of the preview display area becomes the same as the original size of the preview screen.

Thus, a huge memory area is secured, the preview image is freely drawn, and the result is almost the same as that of the case where the preview image is thinned to the size of the preview screen. It can be obtained simply by preparing the memory capacity of. In the case of m = 2, since it is sufficient to simply thin out every other pixel, the reduction can be performed at extremely high speed.

Here, for example, 10 frames of the input frame image
Consider creating a panoramic image twice as large. Regardless of the above method, in a method of securing a memory area in advance, a preview of a panoramic image of this area requires about 400 times the memory area of the input frame image in advance. This is because it is not possible to predict in which direction the camera will move from the beginning.
This is because it is necessary to secure a rectangular area having a length of 0 times. However, according to the method of the present invention, when n = 3, the size may be at most 9 times. Since the present invention guarantees an unlimited size at 9 times the size, the difference in the memory area from the conventional method is that the size of the panoramic image to be synthesized is 20 times.
It becomes exponentially larger as the number increases by a factor of 30.

Further, the above method is more effective in the case of a high-resolution still image synthesizing process using a zoom image.
FIG. 5 shows the principle of high-resolution still image synthesis by zooming.
This basic idea is based on the previously mentioned document Salient Vide
o Stills: Content and Context Preserved. A large high-resolution image can be obtained by superimposing other frame images in accordance with the zoom ratio with the frame image at the time of zooming in as the reference size. In the high-resolution image, the pixels in the peripheral portion simply increase in size, but gradually increase in definition as the pixel approaches the central portion. When an image taken by a normal still camera is focused on the center, the peripheral portion is blurred, and the characteristics are close to those of a superimposed high-resolution image. That is, an image equivalent to that obtained by using an inexpensive low-resolution imaging element and an expensive high-resolution imaging element can be obtained.

However, in such a conventional method, the superimposition processing must be performed after the image at the time of zooming out is enlarged, and the enlargement processing time increases as the enlargement ratio increases. Real-time processing becomes more difficult than before. Still, in the case of a video that gradually zooms out monotonically, if a huge memory area can be prepared first, it is only necessary to draw the first frame at full size and then overlap it while enlarging the frame image . However, on the other hand, in the case of a video that zooms in, the composition of the high-resolution image is performed while increasing the resolution so that the last input frame image is always the original size. It is necessary to enlarge the image, and it takes a very long time to enlarge a huge memory area every frame. This is the subject of the real-time processing unique unpredictable there is advance what camera movements.

On the other hand, if the concept of the present invention is applied, a preview can be confirmed in real time in a small memory area. For a method to quickly determine the zoom ratio, for example,
Already filed by the inventors in Japanese Patent Application No. Hei 10-53100 "Super Resolution Camera"
Can be used. Here, it is confirmed that the synthesis of the high-resolution image is performed by superimposing the input frame images while adjusting the size so that the common area between the consecutive frame images has the same scale. is there. Therefore, as in the case of the panorama synthesis, the common area needs to be recorded only once in order to obtain a final synthesized image. It is sufficient to use only the outer frame portion having a U-shaped. In creating a preview image, it is only necessary to draw frame images reduced according to the zoom ratio one after another in the preview buffer.

Here, the zoom ratio is defined as a ratio when the sizes are adjusted so that the common areas match based on the frame image that is input first. 1.0 for no zoom, less than 1 for zoom in,
For zooming out, it is larger than 1. On the other hand, in the case of a zoomed-out image, since the latest frame shows a wider range, a square-shaped outer frame portion excluding a common area in the latest frame is recorded. When creating the preview image, the outer frame portion is reduced and drawn according to the zoom ratio. In the case of zoom-out, the size of the preview image exceeds the size of the preview buffer when the zoom ratio is increased. In such a case, the entire preview buffer is reduced to 1 / m as in the case of panorama synthesis. As described above, even in the case of zooming, the only enlargement required for creating the preview image is processing for enlarging only the square frame-shaped outer frame portion to n times or less, and the processing can be performed clearly at a high speed. As described above, the result of the high-resolution image synthesis can be confirmed in real time simultaneously with the photographing.

FIGS. 6 and 7 show an example of the original data recorded for the final composite image in this embodiment. The data is roughly divided into a panorama file structure 600 and a panorama clip structure 700. The former panorama file structure is a header that summarizes the entire original data,
The latter panoramic clip structure is image data for which recording is determined for each frame. The panorama file structure stores a file ID 602 as an identifier indicating that the subsequent data is the original data for image synthesis, and the actual size of the synthesized image in the order of width (604) and height (606). The final size of the composite image is determined from the amount of camera movement and can be expressed as the smallest rectangle circumscribing it even if it has a special shape. Therefore, only the width and height are stored. Also, the number of colors 608 of the original data (for example, 256 colors or 24-bit full color) and the image format 610 (for example, RGB or YUV) are stored.

The coordinates of the starting point of the rectangular area circumscribing the composite image are set to the panorama image x, y offset (612, 61).
4), and the coordinates of a panorama clip described later are calculated based on the coordinates. Following this header information, a panoramic clip structure 700 is arranged as data. The panoramic clip structure is data for each frame, and stores the position and size of the frame image in the composite image. First, reference numeral 702 denotes the total data size, which stores the amount of image data. Image start point x
The coordinates 702 correspond to the panorama image x offset 612 described above.
Represents the x-coordinate value of the start point of the frame image associated with this panorama clip structure when the coordinates indicated by are set as the start point coordinates of the composite image. Similarly, image start point y coordinate 7
04 represents the y coordinate value of the start point of the frame image. The image width 706 and the image height 708 represent the actual size of the frame image as it is input. This is multiplied by the zoom ratio 710 to calculate the size of the frame image on the composite image.
Reference numerals 714 to 720 designate a range of the common area in the frame image. Common area start point x coordinate 714
And the common area start point y coordinate 716 represent the start point coordinates of the common area in the frame, and the size is represented by the common area width 718 and the common area height 720. In the pixel data 722, all pixel data are arranged in order while scanning from the upper left to the lower right. When an appropriate value is included in 714 to 720, the pixel data relating to the common area is simply ignored.

Next, a method of creating a final composite image from the original data stored above will be described. First, the width and height of the composite image are checked with reference to the panorama file structure 600, and a rectangular image area of that size is secured in the memory area. Then, the subsequent panorama clip structure 70
With reference to 0, the position and size to be drawn are checked, the relative position in the secured memory area is obtained, and the image data 722 is written. However, if there is even one clip whose zoom ratio is lower than 1x, the smallest zoom ratio is set to 1 first, that is, the most zoomed-in frame image is drawn at 1: 1. Then, the zoom ratios of all the panorama clip structures are proportionally converted.
In this case, the size of the composite image also increases proportionally. As a result, the frame image is not reduced, and the synthesis is performed only in the direction in which the resolution is increased.

In order to make the image synthesis described above look more beautiful, a smoothing process can be performed. In general, when pasting images, the amount of motion cannot be determined correctly, distortion occurs at the time of the input image due to lens aberration, or the brightness at the time of shooting fluctuates. A subtle boundary line may occur at the joints. In human perception, this boundary line gives an unpleasant impression, and thus needs to be corrected. The easiest way to make such a correction is to leave a margin between the images to be pasted, and to make a transition from one image color to the other image step by step with a boundary line in between. It is to stick to. FIG. 8 shows a very simplified image of the concept. 800 shows a state where two images are simply pasted together. Here, for the sake of simplicity, it is assumed that the two images are uniform at a specific luminance. Looking at the change in the luminance in the horizontal direction at this time, at the boundary of the image, it changes instantaneously as shown in a graph 802, and this forms a boundary 808. On the other hand, as shown by reference numerals 804 and 806, when a margin is provided at the boundary between the images and the images are pasted so as to have a stepwise change from the luminance of one image to the luminance of the other image, the boundary line becomes inconspicuous. .

In order to realize the above-mentioned smoothing processing by the synthesizing method of the present invention, it is sufficient to record an area wider by the margin when recording the original data. As a result, smoothing can be performed with a minimum necessary increase in data amount. Then, at the time of the later synthesizing process, by performing the smoothing process on the amount of the paste margin, a smoothly connected synthesized image can be obtained. The area to be used as the margin is basically selected in such a manner that a common area between two consecutive images is narrowed. FIG. 9 shows an example of a case where two frame images 900 and 902 are arranged so that a common area overlaps. Assuming that 902 is a temporally new frame image, an L-shaped hatched area 904 in the upper left of the figure is recorded as original image data. Since the margin is only required to fill the boundary between the two images in a stepwise manner, a region whose width is increased in the direction to narrow the common region along the boundary is the minimum necessary region. In the example of FIG. 9, an area serving as a margin is an L-shaped hatched area 906 inside the area 904. FIG. 10 shows a list of typical camera movement amounts and combinations of margin areas at that time. In the figure, Δx and Δy indicate the amounts of movement in the horizontal / vertical directions, respectively. FIG. 11 shows a similar margin area pattern when zooming is further performed. The figure shows the case of zoom-in. In this way, even if there is a zoom, the margin area is determined by extending in the direction of the common area from the boundary line between images when the size of the common area is adjusted so that the size of the common area is the same. In the case of zoom-out, the relationship is basically the same except that the relationship between the frame images 900 and 902 is reversed.

In the case of creating a high-definition image, when pasting image data in accordance with the zoom ratio, instead of simple pixel enlargement, enlargement is performed by using a pixel interpolation technique such as bilinear interpolation. Needless to say, a better-looking image can be obtained. Then, by performing the above-described smoothing process on the image data enlarged by the pixel interpolation, a more beautiful image can be obtained.

In the above description, the method of creating and displaying a preview image in real time has been described. However, instead of the preview image, a simplified image is displayed using a symbol indicating the state of synthesis, thereby improving usability and higher speed. It can also be compatible with the conversion. For example, by displaying not the preview image itself but the silhouette, that is, the shape of the composite image with a line drawing, it is not necessary to reduce and paste the frame image, and at the same time, it is high speed because it is only a line drawing Can be drawn. FIG. 12 shows an example of a preview screen based on this. The input image is displayed as it is on the preview screen, and the line image 1200 is also displayed as an indicator. In addition, in order to indicate where the currently input frame falls in the composite image, as shown at 1202, the area occupied by the current frame image in the line drawing may be highlighted. As a result, the purpose of checking the synthesis status in real time is realized with lower computational performance.

[0047]

According to the present invention, in the creation of a panoramic image or a high-resolution image, the progress and results of the processing can be checked at the same time as the photographing, and the photographer can easily take the image as many times as desired until a favorite image is obtained. It can provide excellent usability that can be fixed.

[Brief description of the drawings]

FIG. 1 is an example of a system configuration diagram for realizing the present invention.

FIG. 2 is an example of a flowchart of a process for realizing the present invention.

FIG. 3 is an image diagram for synthesizing a panoramic image from a panning moving image.

FIG. 4 is an example of the contents of a preview buffer when the present invention is executed.

FIG. 5 is an image diagram for synthesizing a high-resolution image from a moving image to be zoomed.

FIG. 6 is an example of a data structure used in the processing of the present invention.

FIG. 7 is an example of a data structure used in the processing of the present invention.

FIG. 8 is an image diagram showing a method of bonding images.

FIG. 9 is a diagram illustrating each image area defined in the processing of the present invention.

FIG. 10 is an example of a margin area pattern used in the processing of the present invention.

FIG. 11 is an example of a margin area pattern used in the processing of the present invention.

FIG. 12 is an example of an indicator display used in the processing of the present invention.

[Explanation of symbols]

100 camera unit, 102 A / D converter, 104 interface, 106 video signal line, 108 control signal line, 110 CPU, 112 display, 114 memory, 116 processing unit, 118 external Information storage device, 120 ... input device.

Continued on the front page F term (reference) 5B057 AA20 BA02 CD05 CE10 DC22 DC25 DC30 5C023 AA02 AA11 AA21 AA38 BA01 BA11 BA15 CA01 CA03 CA08 DA04 DA08 EA02

Claims (10)

[Claims] 1. A display means for displaying a moving image being photographed, a means for sequentially taking in frame images from the moving image, a means for comparing two frame images having different fetching times to determine a moving amount, and Means for storing a difference image of the image and the movement amount, means for creating indicator information that symbolizes a combined image synthesized from the stored movement amount and the captured frame image, and the difference image and the movement amount Means for synthesizing a panoramic image using the amount. 2. The image synthesizing apparatus according to claim 1, wherein
The indicator information is obtained by reducing the composite image, and includes a reduced frame image obtained by reducing the captured frame image at a predetermined magnification and a moving amount obtained by converting the moving amount according to the magnification. An image synthesizing apparatus, wherein the display unit displays the reduced image and the panoramic image. 3. The image synthesizing apparatus according to claim 1, wherein
The indicator information is a symbol image that symbolizes the overall shape of the composite image, and includes a unit that creates a symbol image that indicates the shape of the composite image. The display unit includes the symbol image and the panoramic image. An image synthesizing apparatus characterized by displaying: 4. The image synthesizing apparatus according to claim 2, wherein
An image synthesizing apparatus characterized in that an area corresponding to a frame captured latest in the reduced image is highlighted. 5. The image synthesizing apparatus according to claim 3, wherein
An image synthesizing apparatus, wherein an area in the symbol image corresponding to the most recently captured frame is highlighted. 6. The image synthesizing apparatus according to claim 1, wherein
The difference image also includes a common image area of the two frame images, and the panoramic image synthesizing unit smoothly synthesizes a boundary between the two frame images using the common image area. Image synthesizing device. 7. The image synthesizing apparatus according to claim 1, wherein
An image synthesizing apparatus, wherein, when a composite image is created by enlarging the difference image, pixels of the enlarged difference image are calculated by interpolation. 8. A display means for displaying a captured moving image, a means for sequentially capturing frame images from the moving image, a means for comparing two frame images at different capturing times to determine a moving amount of the image, Means for obtaining a composite image by combining the frame images captured based on the movement amount, buffer means for storing the composite image or its reduced image, and preview display for displaying one area of the composite image stored in the buffer Means for reducing the combined image and the one area stored in the buffer when the combined image or its reduced image exceeds a predetermined area. 9. A step of displaying a moving image being photographed, a step of sequentially taking in frame images from the moving image, and comparing two frame images having different capturing times to determine a moving amount; Storing the difference image and the moving amount; creating indicator information that symbolizes a combined image synthesized from the stored moving amount and the captured frame image; and storing the difference image and the moving amount. Synthesizing a panoramic image by using a computer-readable recording medium on which an image synthesizing method is recorded. 10. A step of comparing two captured frame images at different times to determine a moving amount of the image, a step of combining the captured frame images based on the moving amount to obtain a composite image, Or a step of storing the reduced image in a buffer; a step of displaying a preview of displaying one area of the composite image stored in the buffer; and a step of displaying the preview image when the composite image or the reduced image exceeds a specified area range. Displaying the combined image and the one area stored in the buffer in a reduced size, the computer-readable recording medium recording the image combining method.