JP2004312549A - Panoramic image photographing apparatus and panoramic image photographing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an auxiliary function easy to operate for photographing a sharp composite panoramic image. <P>SOLUTION: A screen 101 is divided to display 104 a photographed image 102 and a (camera) image 103 to be next photographed while partially overlapping them, the entire connection situation is checked, a characteristic partial image 105, for example, is cut out of the partially overlapped portion, and the partial image 105 is automatically displayed together with the images 102, 103 without being reduced in the area different from the display area for the images 102, 103, thereby confirming the detailed connection situation of the images 102, 103. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a photographing technique for creating a panoramic image by photographing a plurality of images and performing digital image processing, and in particular, assists in photographing a panoramic image in a digital camera (including a mobile phone with a photographing function). Regarding the technology to be performed.
[0002]
[Prior art]
In order to capture a panoramic image neatly, it is necessary to rotate or translate the camera while keeping the camera from tilting up, down, left and right, and to take an appropriate amount of joints before shooting.
[0003]
For example, when performing panoramic shooting with a digital camera or the like, a part of the previously captured image is displayed as an afterimage on a part of the liquid crystal screen, and an operation of superimposing a part of the next subject image on the afterimage is repeated. There is a method of combining several image-wise images by taking several images (for example, see Patent Document 1).
[0004]
A method of providing a plurality of electronic viewfinders arranged adjacent to each other to compare a target subject image corresponding to an image signal to be recorded next with an image corresponding to an already recorded image signal (For example, see Patent Document 2).
[0005]
Further, there is disclosed a technique for facilitating panorama composition by preparing an area for displaying an entire captured image and a composition work area on the same screen on a personal computer (see Patent Document 3).
[0006]
[Patent Document 1]
JP-A-5-142682 [Patent Document 2]
JP-A-3-262276 [Patent Document 3]
JP 2000-90232 A
[Problems to be solved by the invention]
However, the method described in Patent Document 1 has a problem that it is difficult to understand the overall connection relationship between two images. In the method described in Patent Document 2, it is necessary to provide a dedicated electronic view finder. Otherwise, when displaying on a single liquid crystal screen, it was necessary to reduce the display, and it was difficult to perform detailed alignment. Further, Patent Literature 3 is suitable for use in a case where a large screen is supported such as a personal computer, but is not suitable for a device having a small display screen such as a digital camera or a camera-equipped mobile phone. Not suitable.
[0008]
The present invention makes it easy to check the overall connection status, easily understand the difference in the connection portion, and make a good panorama image that can be connected well on a relatively small screen (with a small number of pixels) to create a good panorama image. The purpose is to provide photography technology.
[0009]
[Means for Solving the Problems]
According to one aspect of the present invention, a photographing unit, an image data storage unit that stores first image data relating to a photographed image photographed by the photographing unit, and a photographing target image based on second image data to be photographed And a display unit that displays at least one of a storage image based on the first image data stored in the storage unit, and an image combination that performs a process of creating a composite image in which the recorded image and the image to be captured are combined. A processing unit, for a virtual composite image in which the recorded image and the photographing target image are displayed side by side in a partially overlapped state, a part of an overlapping portion of the recorded image and the current photographing target image There is provided an imaging device having a display control unit having a composition assisting function of displaying at least one partial image obtained by cutting out an area at a position not overlapping with the virtual image.
[0010]
In the above-described photographing apparatus, in order to use the partial image as a guide for combination, even if the display screen is small, it is not necessary to reduce the image serving as a guide for combination. In addition, for a virtual composite image in which the recorded image and the photographing target image are displayed side by side in a partially overlapped state, a partial area of an overlapping portion between the recorded image and the current photographing target image is displayed. By displaying at least one cut-out partial image at a position that does not overlap with the virtual image, other operations such as reduction and movement of the partial image can be minimized, and the operation is advantageously simplified. .
[0011]
It is preferable that the combination assisting function has a characteristic partial image extracting / displaying function of extracting a characteristic part from the recorded image in the overlapping part and displaying the extracted characteristic part as the partial image. By cutting out the characteristic part as a partial image, even if it is a part, the function as an indication of synthesis can be enhanced.
[0012]
In addition, there are provided a means for partially overlapping and reducing the first image data already photographed and the second image data to be photographed, and a means for cutting out and displaying a part from the overlapping part of both. May be.
[0013]
BEST MODE FOR CARRYING OUT THE INVENTION
The panoramic image photographing technique according to the present invention is a panoramic image photographing technique that can be realized even on a relatively small screen (with a small number of pixels) such as a digital camera or a camera-equipped mobile phone.
[0014]
Hereinafter, a panoramic image photographing technique according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a panoramic image photographing technique according to the first embodiment of the present invention. A display screen 101 of a digital camera, a camera-equipped mobile phone, or the like (hereinafter, these are collectively abbreviated as “digital camera”) is divided, and a partial screen is provided so that the overall connection status can be easily checked. An image 102 based on a first video signal that has already been captured and an image 103 based on a second video signal that is to be captured will be described later by partially overlapping the connection between the image 102 and the image 103. Reduce and display 103 so that it fits within the display screen (image indicated by reference numeral 104). At this time, only one image of the image 102 and the image 103 may be displayed in the overlapping area 104a, or the difference processing and the mixing processing (the mixing ratio of the image is (It is not limited to 1: 1.) Alternatively, the display may be performed by performing processing such as expressing the luminance in red and green and combining the luminance.
[0015]
On the screen other than the overlapping region 104a, a certain partial region 105 is selected from the overlapping portions of both images 102 and 103 so that the difference between the two images 102 and 103 can be easily understood, and the partial region 105 corresponding to the image 102 based on the first video signal is selected. A first partial image 106 of the partial area 105 and a second partial image 107 of the partial area 105 corresponding to the image 103 corresponding to the second video signal are cut out. A difference image 109 based on the two partial images 107 is created and displayed on a display screen.
[0016]
At this time, the difference image 109 is subjected to difference processing and mixing processing (the mixing ratio of the images is not limited to 1: 1) so that the difference between the first partial image 106 and the second partial image 107 is easily understood, and the luminance is set to red. Performs composition such as display in green. Since the difference image 109 is a partial region of the entire image, normal magnification or enlarged display can be performed without reduction, so that it is easy to confirm even a small screen.
[0017]
The difference image may be one, and a portion 105 ′ is selected from the overlapping portion shown in FIG. 1 and the portion 106 ′ corresponding to the first image 102 corresponding to the first video signal and the second video A portion 107 'corresponding to the second image 103 corresponding to the signal may be cut out, a difference image 109' may be created by the difference image creation means 108 ', and this may be displayed as a second difference image on the display screen. . Of course, more partial images may be selected. The position of the portion 105 selected from the overlapping portion may be a predetermined position, or a method of automatically selecting a position as described later may be used.
[0018]
According to the panoramic image photographing technology according to the present embodiment, it is possible to check the overall connection status in detail, and to create a good panoramic photograph by connecting neatly even on a relatively small screen (with a small number of pixels). There is an advantage that can be.
[0019]
Next, a panoramic image capturing technique according to a second embodiment of the present invention will be described with reference to the drawings. The panoramic image photographing technique according to the present embodiment automatically sets a position for performing cutout display in the panoramic image photographing technique according to the first embodiment from an overlapping portion of an image based on a first video signal. It is characterized by detecting an undesired portion. FIG. 2 is a diagram illustrating a method of automatically detecting a characteristic portion from an overlapping portion. First, a clipping process 201 for cutting out an overlapping portion from an image 204 based on a first video signal that has already been captured is performed, and an image 205 of a portion overlapping with an image based on a second video signal to be captured is created. For example, in this case, only the luminance information or the G information may be used instead of the three colors of RGB.
[0020]
Next, an edge image 2061 is created by performing an edge detection process 202 by applying a Laplacian filter or the like to the image 205 of the overlapping portion. Next, the variance, standard deviation, and the like of each region are calculated for the edge image 2061, and a process 203 for determining a characteristic portion in the edge image 2061, that is, a difference image position is performed. For example, since the area surrounded by the rectangular area 207 in the edge image 206 (the same image as the edge image 2061) is a characteristic image portion in which one white building exists in a forest as a background, this building It can be estimated that the region including the forest and the vicinity thereof is the difference image position, and the difference image 208 can be obtained by acquiring an image corresponding to the rectangular region 207 in the overlapping partial image 205. Since the difference image 208 is a partial area of the entire image, normal magnification or enlarged display can be performed without reduction, so that it is easy to check even on a small screen, and a characteristic area is automatically set. Since the difference in the overlapping portion can be checked based on the region and based on the region, there is an advantage that complicated confirmation processing on the user side is not required.
[0021]
Next, a panoramic image photographing technique according to a third embodiment of the present invention will be described with reference to FIGS. In the case where the panoramic image photographing technology according to each embodiment of the present invention is used and pictures are connected, if the pictures are connected as they are, the pictures may be discontinuously bent at the combined portion of the pictures. . In order to eliminate discontinuous folds and smoothly connect photos, it is necessary to correct the shape of the images when combining them. There are two methods of correction: (1) a large plane projection method, that is, a method of correcting a photograph taken on one large plane, and (2) a cylindrical surface projection method, that is, a correction as if photographed on a cylindrical surface. And two methods.
[0022]
FIG. 3 is a conceptual diagram of the shooting situation viewed from below. FIGS. 4A to 4C are diagrams for explaining the converted image. First, a photographer 301 of a camera takes a first image 302 based on a first video signal already taken. Next, the photographer 301 of the camera rotates the photographing direction of the camera, and photographs a second image 303 corresponding to a second video signal to be photographed.
[0023]
In the large plane projection method, a large plane 304 is assumed, a line segment connecting the position of the photographer 301 is considered with respect to a point on the plane 304, and the line segment and the first image 302 or the second image are considered. The intersection with 303 is made to correspond. Specifically, considering the line segments L1 and L2 connecting each of the first end point P1 and the other second end point P2 of the first image 302 to the photographer 301, these line segments L1 and L2 are , The plane P 304, and the respective intersections P 1 ′ and P 2 ′. Further, taking into account the line segments L3 and L4 connecting each of the third end point P3 and the other fourth end point P4 of the second image 303 with the photographer 301, these line segments L3 and L4 and the plane 304 And the respective intersection points P3 'and P4' are obtained.
[0024]
FIGS. 4A and 4B are conceptual diagrams showing an image viewed from the direction of the photographer 301 (FIG. 3). As shown in FIG. 4A, a first image 402 (corresponding to the image 302 in FIG. 3) based on the first video signal corresponds to a large plane 401 (corresponding to the plane 304 in FIG. 3). The second image 403 (corresponding to the image 303 in FIG. 3) 403 corresponding to the second video signal is converted into a rectangle having a long left side and a short right side (a trapezoidal figure). (A trapezoidal figure).
[0025]
Returning to FIG. 3, the cylindrical projection method will be described. Assuming the cylindrical surface 305 shown in FIG. 3, a line segment connecting the position of the photographer 301 is considered with respect to a point on the cylindrical surface 305, and the line segment and the first image 302 or the second image 303 are considered. Corresponding to the intersection with FIG. 4B is a diagram viewed from the photographer direction based on the cylindrical surface projection method. An image 405 corresponding to the first video signal (corresponding to the image 302 in FIG. 3) and an image 405 corresponding to the second video signal (the image 303 in FIG. 406) is also converted to a bale-shaped shape. In the first and second panoramic image photographing methods, the image conversion is performed by using the video signal added to the first and second video signals to convert the projection image onto the large plane 304 or the cylindrical surface 303 performed at the time of panorama combination. The image shape can be corrected before combining the images, and the images can be smoothly aligned and aligned so that the images can be synthesized more accurately. As described above, by performing image conversion at the time of panorama combination on the first and second video signals, it is possible to perform more accurate synthesis.
[0026]
Next, a panoramic image capturing technique according to a fourth embodiment of the present invention will be described with reference to FIGS. 5 (A) and 5 (B). In the panoramic image photographing technique according to the third embodiment, the video to be synthesized must also be converted for panoramic combination, so that the load related to the screen switching process may increase.
[0027]
In the large plane projection method, as shown in FIG. 5A, it is possible to make the image 403 'corresponding to the second video signal to be shot from now a rectangle. In this case, the image 402 'corresponding to the already captured first video signal is converted into a trapezoid with greater distortion. FIG. 5 is a view for explaining the panoramic photograph synthesizing method according to the present embodiment, taking the case of the cylindrical surface projection method as an example. Even in the case of the cylindrical projection method, it is considered that an image corresponding to the second video signal is made rectangular as shown in FIG.
[0028]
As shown in FIG. 5B, the photographer 501 first captures an image 502 corresponding to the first video signal. Next, the camera is rotated to capture an image 503 corresponding to the second video signal. Next, assuming a cylindrical surface 505, a line segment L11 connecting a point O corresponding to the photographer 501 with respect to a point P0 on the cylindrical surface 505 is considered, and the line segment L11 and the first image 502 are considered. Is made to correspond to the intersection point P1 with the point P0. Thus, a bale-shaped conversion is performed. On the other hand, a plane 504 obtained by expanding the second image 503 is considered. Assuming that the intersection point of the line segment L11 (O-P0) and the plane 504 is P2, the conversion from the bale type to the reverse bale type to the plane 504 is performed. This is the same as the conversion shown in FIG. That is, the plane 402 ′ shown in FIG. 5A is the same as the plane 504 viewed from the photographer 501 side.
[0029]
As described above, in any case of performing the conversion of the cylindrical surface projection method or the large plane projection method, more accurate synthesis is performed by converting only the image corresponding to the first video signal as shown in FIG. Can be aligned so that In the method according to the present embodiment, since the second video signal is not projected at the time of panorama combination, the processing relating to the second video signal is performed at high speed, and the load in the overall processing can be reduced. There is an advantage.
[0030]
Next, a panoramic image photographing technique according to a fifth embodiment of the present invention will be described with reference to the drawings. The panoramic image photographing technique according to the present embodiment is characterized in that the panoramic image photographing technique according to the third or fourth embodiment is performed only on a cut-out display portion. FIG. 6 is a diagram schematically illustrating a panoramic image photographing technique according to the present embodiment. The first image 601 based on the first video signal already captured and the second image 602 based on the second video signal to be captured are partially overlapped and reduced and displayed so as to fit within the display screen 603. (See reference numeral 604). On the other hand, if the processing by the large plane projection method is performed on the first image 601 based on the first video signal and the second image 602 based on the second video signal, a third image 606 and a fourth image 607 are obtained, respectively. For example, a characteristic portion 608 a is selected from the overlapping portion 608, and this is displayed on the display screen 603 as a difference image 605.
[0031]
In the above embodiment, processing by the large plane projection method has been described, but the present invention can be similarly applied to the cylindrical surface projection method. By performing the above processing, the connection position can be determined by reflecting the influence of the large plane projection method and the cylindrical surface projection method at the time of connection, so that more accurate judgment can be made. Further, in FIG. 6, all regions of both signals are converted for the sake of explanation. However, in order to automatically select a characteristic portion, only the overlapping portion needs to be converted. Therefore, once the characteristic part is determined, only the corresponding area (of the second signal) needs to be converted. Also, as in the fourth embodiment, the conversion of the first signal can be determined so that one signal, for example, the second signal need not be converted. By using these methods, it is possible to increase the speed of the panorama image generation processing.
[0032]
As described above, when the panoramic image capturing technique according to the first embodiment of the present invention is used, the overall connection status can be checked, and the details of the shift between images to be combined can be checked. There is an advantage that it can be easily performed. When the panoramic image capturing technique according to the second embodiment is used, a characteristic portion can be automatically detected, so that it is easy to determine the details of the difference between the two images to be combined. The use of the panoramic image photographing technique according to the third embodiment of the present invention enables more accurate composition. The use of the panoramic image photographing technique according to the fourth embodiment has the advantage that the processing of the second video signal is quickened while enabling more accurate synthesis. According to the panoramic image photographing technique according to the fifth embodiment of the present invention, it is possible to increase the processing speed while maintaining the accuracy of the detailed judgment.
As described above, the present invention has been described in accordance with the embodiments. However, it is needless to say that the present invention is not limited to these examples, and various modifications are possible.
[0033]
【The invention's effect】
According to the panoramic image photographing technique of the present invention, it is also possible to cut out a part of the overlapping portion with respect to the image to be synthesized and confirm the details of the deviation between the two, so that it is possible to cope with a small display screen, Can generate a smooth composite image. The use of the method of automatically detecting a characteristic portion has an advantage that it is easy to determine the details of the difference between the two in the composite image.
[0034]
Further, by performing image conversion at the time of panorama combination, more accurate composition can be performed. Further, by performing image conversion on only one image, it is possible to speed up processing of the other image while enabling accurate synthesis. As described above, by comparing the partial images cut out, it is possible to fit in the display screen without reducing the display, etc. There is an advantage that the image processing speed for confirmation is increased.
[Brief description of the drawings]
FIG. 1 is a diagram showing an outline of processing of a panoramic image photographing technique according to a first embodiment of the present invention.
FIG. 2 is a diagram illustrating an outline of processing of a panoramic image capturing technique according to the first embodiment of the present invention, and is a diagram illustrating a method of automatically detecting a characteristic portion from an overlapping portion.
FIG. 3 is a diagram illustrating the principle of a panoramic image photographing technique according to a third embodiment of the present invention.
FIGS. 4A and 4B are diagrams showing specific display examples of FIG. 3;
FIG. 5A is a diagram illustrating the principle of a panoramic image capturing technique according to a fourth embodiment of the present invention, and FIG. 5B is a specific display example of FIG. FIG.
FIG. 6 is a diagram illustrating an outline of processing of a panoramic image photographing technique according to a fifth embodiment of the present invention.
[Explanation of symbols]
101: display screen, 102: image based on the first video signal already taken, 103: image based on the second video signal to be taken from now on, 104: reduced display portion, 105: difference image area, 106 ... Image of the first video signal corresponding to the difference image area, 107... Image of the second video signal corresponding to the difference image area, 108... Difference image creation means, 109... Difference image, 105 ′. Area 106 ′... Image of the first video signal corresponding to the second difference image area, 107 ′... Image of the second video signal corresponding to the second difference image area, 108 ′. Creation means, 109 ': second difference image, 201: overlapping part cutout processing, 202: edge detection processing, 203: difference image position determination processing, 204: first video signal, 205: overlapping partial image, 206: edge Image, 207: difference image position, 208: difference image, 301: photographer, 302: image corresponding to the first video signal, 303: image corresponding to the second video signal, 304: large plane, 305: cylinder Plane, 401: large plane, 402: image corresponding to the first video signal, 403: image corresponding to the second video signal, 404: plane having an open cylinder, 405: image corresponding to the first video signal , 406: image corresponding to the second video signal, 401 ′: large plane, 402 ′: image corresponding to the first video signal, 403 ′: image corresponding to the second video signal, 501: photographer, 502: an image corresponding to the first video signal; 503: an image corresponding to the second video signal; 504: a plane including an image corresponding to the second video signal; 505: a cylindrical surface; First video signal, 6 02: a second video signal to be photographed, 603: screen, 604: reduced display, 605: difference image, 606: image obtained by projecting the first video signal on a large plane, 607: second video signal is large Plane projected image, 608... Area selected from overlapping parts.

Claims (11)

A shooting unit,
An image data storage unit that stores first image data related to a photographed image photographed by the photographing unit;
A display unit that displays at least one of a shooting target image based on second image data to be shot and a storage image based on the first image data stored in the storage unit;
An image synthesis processing unit that performs a process of creating a synthesized image obtained by synthesizing the recorded image and the shooting target image
For a virtual composite image in which the recorded image and the shooting target image are displayed side by side in a partially overlapped state, a partial area of an overlapping portion between the recorded image and the current shooting target image is cut out. A photographing device comprising: a display control unit having a combination assisting function of displaying at least one partial image at a position not overlapping with the virtual image. The composition assisting function has a feature partial image extraction / display function of extracting a feature portion from the recorded image in the overlapping portion and displaying the extracted feature portion as the partial image. The photographing device according to claim 1. The feature portion image extraction / display function detects an edge of the recorded image in the overlap portion, and then extracts a region in the overlap portion where a difference between images before and after edge extraction is large as a feature portion. The photographing device according to claim 2, wherein The image synthesizing processing unit includes means for performing image conversion processing for adding image data projected onto a cylindrical surface or a large plane, respectively, to the first image data and the second image data. The photographing apparatus according to any one of claims 1 to 3, wherein: 4. The image synthesis processing unit according to claim 1, further comprising means for performing an image conversion process of adding image data projected onto a cylindrical surface or a large plane only to the first image data. The imaging device according to any one of the above. The photographing apparatus according to claim 4, wherein the image synthesis processing unit performs the image conversion only on an area of the partial image. The display control unit is configured to display the virtual composite image and the partial image so that the virtual composite image and the partial image are displayed in the display unit when the partial image is displayed. The photographing apparatus according to claim 1, further comprising a function of enlarging or reducing an image. A mobile terminal comprising the imaging device according to any one of claims 1 to 7. Means for partially overlapping and reducing the first image data that has already been shot and the second image data to be shot, and means for cutting out and displaying a portion of the overlapped portion between the two. A panoramic image photographing apparatus characterized by the above-mentioned. A panoramic image photographing method by a photographing device having a photographing unit, an image data storage unit that stores image data, and a display unit that displays an image,
Displaying a recorded image based on the first image data already recorded in the image data storage unit and a photographing target image based on the second image data to be photographed from now on in a partially overlapped state; When,
At least one partial image obtained by cutting out a partial area of an overlapping portion between the recorded image and the photographing target image while displaying the recorded image and the photographing target image side by side in a partially overlapped state and displaying the same on the display unit Is displayed at a position that does not overlap with the composite image;
Creating a combined image by combining the recorded image and the image to be captured with reference to the partial image. A shooting unit,
An image data storage unit that stores first image data related to a photographed image photographed by the photographing unit;
A display unit that displays at least one of a shooting target image based on second image data to be shot and a storage image based on the first image data stored in the storage unit;
An image synthesis processing unit that performs a process of creating a synthesized image obtained by synthesizing the recorded image and the shooting target image
A photographing apparatus comprising: a display control unit having a combination assisting function of displaying at least one partial image obtained by cutting out a partial region of an overlapping portion between the recorded image and a current photographing target image.

Images