JP4148817B2 - Panoramic image photographing apparatus and panoramic image photographing method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a photographing technique for creating a panoramic image by photographing a plurality of images and performing digital image processing, and particularly assisting in photographing a panoramic image in a digital camera (including a mobile phone with a photographing function). It relates to technology to be performed
[0002]
[Prior art]
In order to capture a panoramic image neatly, it is necessary to rotate or translate while keeping the camera from tilting up, down, left, and right, and to capture an appropriate amount of joints.
[0003]
For example, when panoramic shooting is performed with a digital camera or the like, while repeating the operation of displaying a part of the previously captured image as a residual image on a part of the liquid crystal screen and superimposing a part of the next subject image on the residual image, There is a method of pasting together by capturing several consecutive images (see, for example, Patent Document 1).
[0004]
Also, by providing a plurality of electronic viewfinders arranged adjacent to each other, a method of comparing a target subject image corresponding to an imaging signal to be recorded next with an image corresponding to an already recorded imaging signal Has also been proposed (see, for example, Patent Document 2).
[0005]
Furthermore, a technique for facilitating panoramic composition by preparing an area for displaying an entire captured image and a composition work area on the same screen on a personal computer is also disclosed (see Patent Document 3).
[0006]
[Patent Document 1]
JP-A-5-142682 [Patent Document 2]
JP-A-3-262276 [Patent Document 3]
Japanese Patent Laid-Open No. 2000-90232
[Problems to be solved by the invention]
However, the method described in Patent Document 1 has a problem that it is difficult to understand the overall connection relationship between two images. In the method described in Patent Document 2, it is necessary to provide a dedicated electronic viewfinder. Otherwise, when displaying on one liquid crystal screen, it is necessary to reduce the display, and detailed alignment is difficult. Furthermore, Patent Document 3 is suitable for use when a large screen is supported, such as a personal computer, but for a device having a small display screen, such as a digital camera or a mobile phone with a camera. Not suitable.
[0008]
The present invention makes it easy to check the overall connection status, makes it easy to understand the difference in the connection portion, and can create a good panoramic picture by connecting cleanly even on a relatively small (small number of pixels) screen. It aims at providing photography technology.
[0009]
[Means for Solving the Problems]
According to one aspect of the present invention, an imaging unit, an image data storing unit for storing the photographed image image taken by the photographing unit, a display unit having a screen for displaying an image, the photographed An image composition processing unit that creates a composite image in which the image and the image to be photographed are partially overlapped and combined, and a composite image created by the image composition processing unit is displayed on the screen of the display unit In addition, for the partial area included in the overlapping portion of the captured image and the imaging target image in the composite image, the captured image and the imaging target image corresponding to the partial area are cut out, and the extracted image Based on the captured image and the partial image of the image to be captured, a difference image for examining the difference in the overlapping portion between the captured image and the image to be captured is created. , Imaging apparatus and a display control unit having a synthetic auxiliary function of displaying the difference image such created on a screen of the display unit together with the composite image is provided.
[0010]
In the above-described photographing apparatus, since the partial image is used as a standard for synthesis, it is not necessary to reduce the image as a standard for synthesis even when the display screen is small. In addition, with respect to a virtual composite image that is displayed side by side in a state where the recorded image and the shooting target image are partially overlapped, a partial area of the overlapping portion of the recorded image and the current shooting target image is provided. By displaying at least one cut out partial image at a position that does not overlap with the virtual image, other operations such as reduction and movement of the partial image can be minimized, and there is an advantage that the operation is simplified. .
[0011]
It is preferable that the synthesis auxiliary function has a feature part image extraction / display function for extracting a characteristic part from the recorded image in the overlapping part and displaying the extracted characteristic part as the partial image. By cutting out the characteristic part as a partial image, it is possible to enhance the function as a standard for synthesis even if it is a part.
[0012]
The first image data that has already been photographed and the second image data that is to be photographed are partially overlapped and displayed in a reduced manner, and a part that is cut out from the overlapped portion and displayed. May be.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
The panoramic image photographing technique according to the present invention is a panoramic image photographing technique that can be realized even on a relatively small screen (small number of pixels) such as a digital camera or a mobile phone with a camera.
[0014]
Hereinafter, a panoramic image photographing technique according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a panoramic image photographing technique according to the first embodiment of the present invention. The display screen 101 of a digital camera, a mobile phone with a camera, etc. (hereinafter collectively referred to as “digital camera”) is divided so that it is easy to check the overall connection status on some of the screens. The image 102 based on the first video signal that has already been captured and the image 103 based on the second video signal that is to be captured are partially overlapped with each other in the connection portion between the image 102 and the image 103. -Reduced display so that 103 fits within the display screen (image indicated by reference numeral 104). At this time, only one of the images 102 and 103 may be displayed in the overlapping area 104a, or difference processing and mixing processing (the image mixing ratio is set so that the difference between the two images is easily understood). (It is not limited to 1: 1), but may be displayed by performing a process of combining the brightness in red and green.
[0015]
On the screen other than the overlapping region 104a, the partial region 105 corresponding to the image 102 based on the first video signal is selected by selecting a partial region 105 from the overlapping portion so that the difference between the images 102 and 103 can be easily understood. The first partial image 106 of 105 and the second partial image 107 of the partial area 105 corresponding to the image 103 corresponding to the second video signal are cut out, and the difference image creating means 108 extracts the first partial image 106 and the first partial image 106. A difference image 109 based on the two-part image 107 is created and displayed on the display screen.
[0016]
At this time, the difference image 109 is subjected to difference processing and mixing processing (the mixing ratio of the images is not limited to 1: 1), and the luminance is red so that the difference between the first partial image 106 and the second partial image 107 can be easily understood. Represents green and performs composition. Since the difference image 109 is a partial area of the entire image, normal magnification or enlarged display can be performed without reducing the image, so that the difference image 109 can be easily confirmed even on a small screen.
[0017]
The difference image may be one, or a part 105 ′ is selected from the overlapping parts shown in FIG. 1, and a part 106 ′ corresponding to the first image 102 corresponding to the first video signal and the second video are selected. The portion 107 ′ corresponding to the second image 103 corresponding to the signal may be cut out, and the difference image 109 ′ may be generated by the difference image generation means 108 ′ and displayed as the second difference image on the display screen. . Of course, more partial images may be selected. Note that the position of the portion 105 selected from the overlapping portions may be a predetermined position, or a method of automatically selecting a position as will be described later.
[0018]
According to the panoramic image shooting technique according to the present embodiment, the overall connection status can be checked in detail, and a good panoramic picture can be created by connecting cleanly even on a relatively small (small number of pixels) screen. There is an advantage that you can.
[0019]
Next, a panoramic image photographing technique according to the second embodiment of the present invention will be described with reference to the drawings. The panoramic image photographing technique according to the present embodiment automatically determines the position for performing the cut-out display in the panoramic image photographing technique according to the first embodiment from the overlapping portion of the image based on the first video signal. It is characterized by detecting an unfavorable part. FIG. 2 is a diagram illustrating a method for automatically detecting characteristic portions from overlapping portions. First, a cutout process 201 is performed to cut out an overlapping portion from an image 204 based on a first video signal that has already been shot, and an image 205 of an overlapping portion with an image based on a second video signal to be shot is created. For example, in this case, only luminance information or G information may be used instead of the three colors RGB.
[0020]
Next, an edge detection process 202 is performed by applying a Laplacian filter or the like to the overlapped image 205 to create an edge image 2061. Next, processing 203 for calculating the variance, standard deviation, and the like of each region with respect to the edge image 2061 and determining a characteristic portion in the edge image 2061, that is, a difference image position is performed. For example, since the area surrounded by the rectangular area 207 in the edge image 206 (the same image as the edge image 2061) is a characteristic image portion in which one white building exists while there is a forest as a background, this building And an area including the forest in the vicinity thereof can be estimated as the difference image position, and an image corresponding to the rectangular area 207 can be acquired in the overlapping partial image 205 to obtain the difference image 208. Since the difference image 208 is a partial area of the entire image, normal magnification or enlargement display can be performed without reducing it, so that it is easy to check even on a small screen and a characteristic area is automatically selected. Therefore, it is possible to check the difference in the overlapping portion based on the area and to eliminate the need for complicated confirmation processing on the user side.
[0021]
Next, a panoramic image photographing technique according to the third embodiment of the present invention will be described with reference to FIGS. When panoramic image shooting technology according to each embodiment of the present invention is used and photographs are stitched together, if the photographs are stitched together, the images may be bent discontinuously at the combined portion of the photographs. . In order to eliminate discontinuous bends and stitch together photos smoothly, it is necessary to correct the shape of the images when they are combined. As correction methods, (1) large plane projection method, that is, a method of correcting as a photograph taken on one large plane, and (2) cylindrical surface projection method, that is, correction as photographed on a cylindrical surface. There are two ways to do this.
[0022]
FIG. 3 is a conceptual diagram of the shooting situation viewed from below. 4A to 4C are diagrams for explaining the converted image. A camera photographer 301 first captures a first image 302 based on a first video signal that has already been captured. Next, the camera photographer 301 rotates the shooting direction of the camera and takes a second image 303 corresponding to the second video signal to be taken.
[0023]
In the large plane projection method, a large plane 304 is assumed, and a line segment connecting the position of the photographer 301 with respect to a point on the plane 304 is considered, and the line segment and the first image 302 or the second image. The intersection with 303 is made to correspond. Specifically, in consideration of line segments L1 and L2 that connect the photographer 301 with one first end point P1 and the other second end point P2 of the first image 302, these line segments L1 and L2 and Each intersection P1 ′ and intersection P2 ′ with the plane 304 is obtained. Further, considering line segments L3 and L4 that connect the photographer 301 with one third end point P3 and the other fourth end point P4 of the second image 303, these line segments L3 and L4, and the plane 304 Each intersection P3 ′ and intersection P4 ′ is obtained.
[0024]
FIGS. 4A and 4B are conceptual diagrams of viewing an image from the direction of the photographer 301 (FIG. 3). As shown in FIG. 4A, for a large plane 401 (corresponding to the plane 304 in FIG. 3), the first image 402 (corresponding to the image 302 in FIG. 3) based on the first video signal is The second image 403 corresponding to the second video signal (corresponding to the image 303 in FIG. 3 ) corresponding to the second video signal has a short left side and a long right side. Shape figure).
[0025]
Returning to FIG. 3, the cylindrical surface projection method will be described. Assuming a cylindrical surface 305 shown in FIG. 3, consider a line segment connecting the position of the photographer 301 with respect to a point on the cylindrical surface 305, and the line segment and the first image 302 or the second image 303. Corresponding intersection point with. FIG. 4B is a view as seen from the photographer direction based on the cylindrical surface projection method. An image 405 corresponding to the first video signal (corresponding to the image 302 in FIG. 3) 405 is also an image corresponding to the second video signal (the image 303 in FIG. 3) with respect to the plane 404 showing the open state of the cylinder. (Corresponding to 406) is also converted into a bowl shape. In the first and second panoramic image photographing methods, by using a video signal in addition to the first and second video signals, the projection image conversion onto the large plane 304 or the cylindrical surface 303 performed at the time of panorama combination is performed. The shape of the image can be corrected when the images are combined, and the images can be smoothly joined together by aligning the images so that the images can be combined more accurately. As described above, the first and second video signals can be combined more accurately by performing image conversion during panorama combination.
[0026]
Next, a panoramic image shooting technique according to the fourth embodiment of the present invention will be described with reference to FIGS. 5 (A) and 5 (B). In the panoramic image shooting technique according to the third embodiment, since the video to be synthesized must be converted for panorama combining, there is a possibility that the load on the screen switching process may increase.
[0027]
In the large plane projection method, as shown in FIG. 5A, an image 403 ′ corresponding to the second video signal to be photographed can be rectangular. In this case, the image 402 ′ corresponding to the first video signal that has already been shot is converted into a trapezoid with greater distortion. FIG. 5 is a diagram for explaining the panoramic photo composition method according to the present embodiment, taking the case of the cylindrical surface projection method as an example. Even in the case of the cylindrical projection method, it is considered that the image corresponding to the second video signal is rectangular as shown in FIG.
[0028]
As shown in FIG. 5B, the photographer 501 first captures an image 502 corresponding to the first video signal. Next, the camera is rotated to capture an image 503 corresponding to the second video signal. Next, assuming a cylindrical surface 505, a line segment L11 connecting a point O corresponding to the photographer 501 to a point P0 on the cylindrical surface 505 is considered, and the line segment L11 and the first image 502 are considered. And the intersection P1 and the point P0. Thereby, a saddle type conversion is performed. On the other hand, a plane 504 in which the second image 503 is expanded is considered. Assuming that the intersection of the line segment L11 (O-P0) and the plane 504 is P2, the inverse saddle type conversion from the saddle type conversion to the plane 504 is performed. This is the same as the conversion shown in FIG. That is, the plane 402 ′ shown in FIG. 5A is the same as the plane 504 viewed from the photographer 501 side.
[0029]
As described above, in any of the conversions of the cylindrical projection method and the large plane projection method, as shown in FIG. 5A, only the image corresponding to the first video signal is converted, thereby synthesizing more accurately. It is possible to perform alignment so that In the method according to the present embodiment, since the second video signal is not projected at the time of panorama combination, the processing relating to the second video signal becomes faster, and the load on the overall processing can be reduced. There is an advantage.
[0030]
Next, a panoramic image shooting technique according to the fifth embodiment of the present invention will be described with reference to the drawings. The panoramic image photographing technique according to the present embodiment is characterized in that the panoramic image photographing technique according to the third or fourth embodiment is performed only on the cut-out display portion. FIG. 6 is a diagram showing an outline of the panoramic image photographing technique according to the present embodiment. The first image 601 based on the first video signal that has already been photographed and the second image 602 based on the second video signal that is to be photographed are partially overlapped and reduced to fit within the display screen 603. (See reference numeral 604). On the other hand, when processing by the large plane projection method is performed on the first image 601 based on the first video signal and the second image 602 based on the second video signal, the third image 606 and the fourth image 607 are obtained. For example, a characteristic part 608 a is selected from the overlapping part 608 and displayed as a difference image 605 in the display screen 603.
[0031]
In the above embodiment, the processing by the large plane projection method has been described, but the present invention can be similarly applied to the cylindrical surface projection method. By performing the above processing, the connection position can be determined by reflecting the influence of the large plane projection method and the cylindrical surface projection method at the time of connection, so that more accurate determination can be made. In FIG. 6, all regions of both signals are converted for the sake of explanation. However, in order to automatically select a characteristic part, only the overlapping part needs to be converted. Therefore, if a characteristic part is determined, only the corresponding region (of the second signal) needs to be converted. Also, the conversion of the first signal can be determined so that one signal, for example, the second signal, need not be converted as in the fourth embodiment. By using these methods, it is possible to increase the speed of panorama image generation processing.
[0032]
As described above, when the panoramic image shooting technique according to the first embodiment of the present invention is used, it is possible to check the overall connection status and to check the details of the shift between images to be synthesized. Also has the advantage of being easy to do. When the panoramic image photographing technique according to the second embodiment is used, a characteristic portion can be automatically detected, so that it is easy to determine the details of the deviation between the images to be synthesized. When the panoramic image photographing technique according to the third embodiment of the present invention is used, more accurate composition is possible. The use of the panoramic image photographing technique according to the fourth embodiment has an advantage that the processing of the second video signal is accelerated while enabling more accurate composition. According to the panoramic image photographing technique according to the fifth embodiment of the present invention, it is possible to increase the processing speed while maintaining accuracy with respect to detailed determination.
As described above, the present invention has been described according to the embodiment. However, the present invention is not limited to these examples, and various modifications can be made.
[0033]
【The invention's effect】
According to the panoramic image photographing technique of the present invention, it is possible to cut out a part of an overlapping portion with respect to an image to be combined and check the details of the deviation between the two, so that it is possible to cope with a small display screen, and a connection portion Can generate a smooth composite image. If a method for automatically detecting characteristic portions is used, there is an advantage that it is easy to determine the details of the shift between the two in the composite image.
[0034]
Further, more accurate composition is possible by performing image conversion at the time of panorama combination. Furthermore, by performing image conversion on only one image, it is possible to speed up processing of the other image while enabling accurate composition. As described above, if partial images cut out are compared, they can be accommodated in the display screen without performing reduced display, etc., and detailed and accurate determination can be made. There is an advantage that the image processing speed for confirmation is also increased.
[Brief description of the drawings]
FIG. 1 is a diagram showing an outline of processing of a panoramic image photographing technique according to a first embodiment of the present invention.
FIG. 2 is a diagram showing an overview of processing of the panoramic image photographing technique according to the first embodiment of the present invention, and is a diagram showing a method for automatically detecting characteristic portions from overlapping portions.
FIG. 3 is a diagram illustrating the principle of a panoramic image photographing technique according to a third embodiment of the present invention.
FIGS. 4A and 4B are diagrams illustrating a specific display example of FIG.
FIG. 5 (A) is a diagram showing the principle of a panoramic image photographing technique according to the fourth embodiment of the present invention, and FIG. 5 (B) is a specific display example of FIG. 5 (A). FIG.
FIG. 6 is a diagram showing an outline of processing of a panoramic image photographing technique according to a fifth embodiment of the present invention.
[Explanation of symbols]
101: Display screen, 102: Image based on already captured first video signal, 103: Image based on second video signal to be captured, 104: Reduced display portion, 105: Difference image area, 106 ... Image of the first video signal corresponding to the difference image area, 107 ... Image of the second video signal corresponding to the difference image area, 108 ... Difference image creating means, 109 ... Difference image, 105 '... Second difference image Area, 106 '... an image of the first video signal corresponding to the second difference image area, 107' ... an image of the second video signal corresponding to the second difference image area, 108 '... a second difference image Creation means 109′—second difference image 201—overlapping part cutout process 202—edge detection process 203—difference image position determination process 204—first video signal 205—overlapping part image 206—edge 207 ... difference image position, 208 ... difference image, 301 ... photographer, 302 ... corresponds to the first video signal Image 303: Image corresponding to the second video signal 304: Large plane, 305 Cylindrical surface 401: Large plane 402: Image corresponding to the first video signal 403: Corresponding to the second video signal 404: a plane with an open cylinder, 405: an image corresponding to the first video signal, 406: an image corresponding to the second video signal, 401 ′: a large plane, 402 ′: a first video signal Corresponding image, 403 '... image corresponding to the second video signal, 501 ... photographer, 502 ... image corresponding to the first video signal, 503 ... image corresponding to the second video signal, 504 ... second A plane including an image corresponding to the video signal of 505, a cylindrical surface, 601 a first video signal that has already been shot, 602 a second video signal that is to be shot, 603 a screen, 604 a reduced display, 605 ... Difference image, 606 ... Image obtained by projecting the first video signal on a large plane, 607 ... Image obtained by projecting the second video signal on a large plane, 608 ... Overlapping An area selected from a part.

Claims (11)

A shooting section;
An image data storing unit for storing the photographed image image taken by the photographing unit,
A display unit having a screen for displaying an image ;
An image composition processing unit that creates a composite image in which the captured image and the image to be captured are partially overlapped and combined ;
The combined image created by the image composition processing unit is displayed on the screen of the display unit, and the partial region included in the overlapping portion of the captured image and the captured image in the combined image In the overlapped portion of the captured image and the image to be captured based on the captured image and the partial image of the image to be captured corresponding to An imaging apparatus comprising: a display control unit having a synthesis auxiliary function that creates a difference image for examining a difference and displays the created difference image on the screen of the display unit together with the synthesized image . The synthetic auxiliary function, the photographing extracts a characteristic portion from the overlapping portions of the target image in the captured image, for a partial area of the overlapped portion including the extracted characteristic portions cut out the partial image The imaging apparatus according to claim 1, further comprising a feature partial image extraction function that determines the difference image position of the image . The feature portion image extraction function creates an edge image of an image portion corresponding to the overlapped portion of the captured image, and features from the overlapped portion based on variance and standard deviation of each region constituting the edge image. The photographing apparatus according to claim 2 , wherein a specific part is extracted . The image synthesizing processing unit is characterized by having an image conversion means for performing image conversion processing of the relative and photographed image and the tracking image, the image data projected respectively into a cylindrical surface or a large plane The imaging device according to any one of claims 1 to 3. The image synthesizing processing unit, the only relative photographed image, from claim 1, characterized in that it comprises an image conversion unit for performing image conversion processing to the image data projected to the cylindrical surface or a large plane to 3 The imaging device according to any one of the above. The imaging apparatus according to claim 4, wherein the image composition processing unit performs the image conversion only on an overlapping portion of the captured image with the imaging target image . When displaying the created difference image together with the synthesized image on the screen of the display unit, the display control unit compares the created difference image with a corresponding portion of the overlapped portion of the displayed synthesized image. The photographing apparatus according to any one of claims 1 to 6, wherein the photographing apparatus has a function of displaying an enlarged image.   The portable terminal provided with the imaging device of any one of Claim 1-7. A panoramic image photographing apparatus comprising the photographing apparatus according to any one of claims 1 to 7 . A photographing unit, a panoramic image photographing method by the photographing device includes an image data storing unit for storing the photographed image photographed by the photographing unit, and a display unit having a screen for displaying an image,
An image data storage step for storing a photographed image photographed by the photographing unit;
An image composition step for creating a composite image by combining the captured image and the image to be photographed by the photographing unit side by side in a partially overlapped state .
A composite image display step of displaying the generated composite image on the screen of the display unit;
For the partial area included in the overlapping portion of the captured image and the imaging target image in the composite image, the captured image corresponding to the partial area and the partial image of each imaging target image are cut out, and the extracted captured image A difference image creating step for creating a difference image for examining a difference in an overlapping portion between the photographed image and the photographing target image based on the image and the partial image of the photographing target image;
A panoramic image photographing step including a difference image display step of enlarging the created difference image as compared with the corresponding portion in the overlapped portion of the composite image and displaying it on the screen of the display unit together with the composite image. Method. A shooting section;
An image data storing unit for storing the photographed image image taken by the photographing unit,
A display unit having a screen for displaying an image ;
An image composition processing unit that creates a composite image in which the captured image and the image to be captured are partially overlapped and combined ;
The combined image created by the image composition processing unit is displayed on the screen of the display unit, and the partial region included in the overlapping portion of the captured image and the captured image in the combined image In the overlapped portion of the captured image and the image to be captured based on the captured image and the partial image of the image to be captured corresponding to A difference image for examining the difference is created, and the created difference image is enlarged as compared with the corresponding portion in the overlapping portion of the synthesized image, and is displayed on the screen of the display unit together with the synthesized image. An imaging apparatus having a display control unit having a function.

Images