JP2008167092A - Image synthesizer, image synthesis program and image synthesizing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prepare a panoramic image by combining photographed images with high accuracy irrespective of frame images and objects of the frame images. <P>SOLUTION: Frame images are selected from a DRAM in a storing order (S201), and a feature amount of the selected frame images is calculated (S202). A block having the calculated feature amount is searched within the next frame image X+1 (S203), and a motion vector is calculated (S204). Extraction width showing the width of a partial image to be extracted is set (S205), and an image is extracted with the extraction width from a center line in the current frame image X (S206). The extracted partial image is combined with an edge part in the opposite direction from a motion vector of a composite panoramic image already prepared, that is, an edge part in the turning direction of an electronic camera to prepare the panoramic image (S207). <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image composition device, an image composition program, and an image composition method for generating a panoramic image obtained by combining a plurality of frame images sequentially obtained by imaging.

2. Description of the Related Art Conventionally, there has been proposed an imaging apparatus that combines panoramic images by combining images of a plurality of shooting frames in order to obtain an angle of view that is greater than the angle of view that can be shot with one shooting frame. When synthesizing this panoramic image, the feature points of the captured images are extracted and their coordinates are detected, and then the coordinates of the same feature points of the adjacent images captured are detected so that these feature points match. A panoramic image is obtained by combining and synthesizing images adjacent to each other (see Patent Document 1).
JP-A-11-282100

  As described above, in the conventional imaging apparatus, a feature point of an image is extracted, and a panoramic image is obtained by matching the feature point with an adjacent image. At this time, in the imaging apparatus, inevitable distortion occurs on both sides of the image formed by the optical system. Therefore, in order to accurately detect the feature point, the center of the frame image without distortion in each frame image. It is necessary to detect feature points in the vicinity of the part.

  However, if only the vicinity of the center of the frame image is set as the feature point extraction region, it is difficult to extract the feature points from the subject near the center of the frame image. It will also be difficult to obtain. In other words, when it is difficult to extract feature points near the center of one adjacent frame image, for example, white walls or pillars, it is difficult to accurately synthesize the adjacent images. It becomes.

  In other words, in the conventional imaging device, panorama synthesis is performed only depending on the composition of one captured frame image, and therefore whether or not other adjacent images can be accurately synthesized is determined according to the frame. It will be influenced by the composition of the image.

  The present invention has been made in view of such a conventional problem, and is an image composition apparatus and image that can synthesize a panoramic image by accurately combining captured images without depending on the composition of the frame image. An object is to provide a composition program and an image composition method.

  In order to solve the above-mentioned problem, in the image composition device according to the first aspect of the present invention, an acquisition means for capturing and sequentially acquiring a plurality of frame images, and a predetermined in each frame image sequentially acquired by the acquisition means Based on the feature amount calculated by the feature amount calculation unit that calculates the feature amount of the region and the feature amount calculation unit, the feature amount calculation unit has a predetermined temporal correspondence with the first frame image in which the feature amount is calculated. Movement vector calculation means for sequentially calculating the movement vector of the second frame image, setting means for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation means, and this setting means The image extraction means for extracting the partial image having the extraction width set by the first and / or second frame image and the image extraction means Image synthesizing means for sequentially combining and synthesizing the partial images, the image extracting means corresponding to the movement vector from the predetermined reference line in the frame image the partial image of the extraction width set by the setting means It is characterized by extracting in the direction.

  In the image composition device according to the second aspect of the present invention, the image composition apparatus further includes a determination unit that determines whether or not the feature amount calculation unit can calculate the feature amount in the predetermined region. The movement vector calculation means includes the feature within a non-calculatable time during which the determination means determines that the feature quantity calculation means cannot calculate the feature quantity in the predetermined area. Based on the feature amount last calculated by the amount calculation unit, the movement vector of the frame image last acquired by the acquisition unit within the non-calculatable time is calculated.

  In the image composition device according to the third aspect of the present invention, an acquisition unit that sequentially captures and acquires a plurality of frame images, and a feature amount of a predetermined region in each frame image that is sequentially acquired by the acquisition unit And a second frame having a predetermined temporal correspondence with the first frame image in which the feature value is calculated based on the feature value calculated by the feature value calculation unit. The movement vector calculation means for sequentially calculating the movement vector of the image, the setting means for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation means, and the setting means Image extracting means for extracting the partial image of the extraction width from the first and / or second frame image, and the partial images extracted by the image extracting means in order Image synthesizing means for combining, and determining means for determining whether or not the feature amount calculating means can calculate a feature amount in the predetermined region, and the movement vector calculating means comprises The feature amount calculated last by the feature amount calculation unit within the non-calculation time in which it is determined by the determination unit that the feature amount calculation unit cannot calculate the feature amount in the predetermined area. Based on the amount, the movement vector of the frame image last acquired by the acquisition unit within the non-calculatable time is calculated.

  In the image composition device according to the invention of claim 4, the image extraction means converts a partial image having an extraction width set by the setting means from the predetermined reference line in the frame image to the movement vector. It is characterized by extracting in the corresponding direction.

  In the image composition device according to the invention described in claim 5, the reference line is set near the center of the frame image.

  In the image synthesizing apparatus according to the sixth aspect of the present invention, start operation control means for starting the operation of the image synthesizing means on condition that the first movement vector is calculated by the movement vector calculating means. It is characterized by providing.

  In the image composition device according to the invention of claim 7, the movement vector is calculated by the movement vector calculation means and then the storage means for storing the movement vector first calculated by the movement vector calculation means and then the movement vector calculation means. When the movement vector is in a direction different from the initial movement vector stored in the storage means, the movement vector is provided with an end operation control means for ending the operation of the image composition means.

  In the image composition device according to the invention described in claim 8, the predetermined area in which the feature quantity calculation means calculates the feature quantity is a central area in the frame image.

  Further, in the image composition device according to the invention of claim 2, the predetermined area where the feature quantity calculating means calculates the feature quantity is composed of a plurality of divided areas divided according to the priority order, The determination unit determines whether or not the feature amount can be calculated in the partitioned region having a high priority, and the feature amount calculating unit is characterized in the partitioned region having the high priority by the determining unit. When it is determined that the amount cannot be calculated, the feature amount is calculated from the segment area having a lower priority.

  In the image composition device according to the invention of claim 10, the movement vector integrating means may cause the feature quantity calculating means to calculate a feature quantity from any of the divided regions by the judging means. In the non-calculatable time determined to be impossible, it was lastly acquired by the acquiring means within the non-calculatable time based on the feature quantity last calculated by the feature quantity calculating means. The movement vector of the frame image is calculated.

  In the image composition program according to an eleventh aspect of the present invention, the acquisition unit includes an imaging unit that performs imaging at regular time intervals in response to an operation.

  In the image composition device according to the twelfth aspect of the present invention, the computer included in the image composition device captures images of a plurality of frame images sequentially, and obtains each of the frame images sequentially. A feature amount calculation unit that calculates a feature amount of a predetermined region in the frame image, and a first frame image in which the feature amount is calculated based on the feature amount calculated by the feature amount calculation unit and a predetermined temporal amount Movement vector calculation means for sequentially calculating movement vectors of the second frame images having a correspondence relationship; setting means for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation means; The image extraction means for extracting the partial image having the extraction width set by the setting means from the first and / or second frame images, and the image The image extraction unit functions as an image synthesis unit that sequentially combines and combines the partial images extracted by the output unit, and the image extraction unit converts the partial image having the extraction width set by the setting unit to a predetermined reference line in the frame image. To the direction according to the movement vector.

  In the image composition program according to the invention described in claim 13, an acquisition unit that captures images of a computer included in the image composition apparatus and sequentially acquires a plurality of frame images, and each of the acquisition units sequentially acquired by the acquisition unit A feature amount calculation unit that calculates a feature amount of a predetermined region in the frame image, and a first frame image in which the feature amount is calculated based on the feature amount calculated by the feature amount calculation unit and a predetermined temporal amount Movement vector calculation means for sequentially calculating movement vectors of the second frame images having a correspondence relationship; setting means for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation means; Image extracting means for extracting the partial image having the extraction width set by the setting means from the first and / or second frame images; Image combining means for sequentially combining and combining the partial images extracted by the image extracting means, and determination means for determining whether or not the feature amount calculating means can calculate a feature amount in the predetermined area And the movement vector calculation means within the non-calculatable time in which the determination means determines that the feature quantity calculation means cannot calculate the feature quantity in the predetermined area. Is characterized in that the movement vector of the frame image last acquired by the acquisition unit within the non-calculatable time is calculated based on the feature amount last calculated by the feature amount calculation unit.

  In the image composition method according to the fourteenth aspect of the present invention, an acquisition step of capturing and sequentially acquiring a plurality of frame images, and a feature amount of a predetermined region in each frame image sequentially acquired by the acquisition step And a second frame having a predetermined temporal correspondence with the first frame image for which the feature value is calculated based on the feature value calculated by the feature value calculation step. The movement vector calculation step for sequentially calculating the movement vector of the image, the setting step for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation step, and the setting step An image extraction step of extracting the partial image of the extraction width from the first and / or second frame image; An image synthesis step of sequentially combining and synthesizing the partial images extracted in the output step, wherein the image extraction step extracts a partial image having an extraction width set in the setting step from a predetermined reference line in the frame image. Extracting in a direction according to the movement vector.

  In the image composition method according to the fifteenth aspect of the present invention, an acquisition step of capturing and sequentially acquiring a plurality of frame images, and a feature amount of a predetermined region in each frame image sequentially acquired by the acquisition step And a second frame having a predetermined temporal correspondence with the first frame image for which the feature value is calculated based on the feature value calculated by the feature value calculation step. The movement vector calculation step for sequentially calculating the movement vector of the image, the setting step for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation step, and the setting step An image extraction step of extracting the partial image of the extraction width from the first and / or second frame image; An image synthesis step for sequentially combining and synthesizing the partial images extracted in the output step; a determination step for determining whether or not the feature amount calculation step can calculate a feature amount in the predetermined region; The movement vector calculation step includes a non-calculatable time in which it is determined by the determination step that it is impossible to calculate a feature amount in the predetermined region in the feature amount calculation step. Based on the feature amount last calculated by the feature amount calculation step, a movement vector of the frame image last acquired by the acquisition step is calculated within the uncalculatable time.

In the image composition method according to the sixteenth aspect of the present invention, the panorama is based on a plurality of frame images photographed so that a part of the subject image overlaps at least between temporally adjacent frame images. An image composition method for generating an image, wherein a predetermined region of a frame image is an image that is ineligible as a feature amount extraction region, while omitting the frame image from the panoramic image The partial images are sequentially extracted based on the movement vector calculated between the first frame image and the second frame image having a predetermined temporal correspondence, and the extracted partial images are combined. In this way, a panoramic image is generated.

  According to the first aspect of the present invention, the partial image having the set extraction width is extracted from the predetermined reference line in the frame image in the direction corresponding to the movement vector, and the extracted partial images are sequentially combined and combined. In addition, between the temporally adjacent frame images, the image at the end edge of the extracted image from the extracted previous frame image and the start edge of the extracted image from the next frame image can be accurately matched. Therefore, it is possible to combine the photographed images with high accuracy to synthesize a panoramic image.

  According to the invention of claim 3, the feature last calculated by the feature amount calculation means within the non-calculatable time during which it is determined that the feature amount can be calculated in the predetermined region. Based on the amount, since the movement vector of the frame image last acquired by the acquisition unit within the non-calculatable time is calculated, a frame image in which the feature amount cannot be calculated in the constant region is acquired. Even if it is a case, panorama composition can be performed without any trouble. Therefore, the panoramic image can be synthesized by accurately combining the captured images without being influenced by the composition of the frame image.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of an electronic camera 1 common to each embodiment of the present invention. The electronic camera 1 has not only a function of shooting a frame image having a predetermined angle of view, but also a panorama shooting function of combining each image portion of a plurality of frame images to synthesize a panoramic image. The CCD 2 is disposed on the optical axis of an optical system (not shown) and the DSP / CPU 3. The DSP / CPU 3 is a one-chip microcomputer that has various digital signal processing functions including image data compression / decompression according to the JPEG method and controls each part of the electronic camera 1. A TG (Timing Generator) 4 that drives the CCD 2 is connected to the DSP / CPU 3, and a unit circuit 5 that receives an analog imaging signal corresponding to the optical image of the subject output from the CCD 2 is connected to the TG 4. Has been. The unit circuit 5 includes a CDS that holds an input imaging signal, a gain adjustment amplifier (AGC) that amplifies the imaging signal, and an A / D converter (AD) that converts the amplified imaging signal into a digital imaging signal. The output signal of the CCD 2 is sent to the DSP / CPU 3 as a digital signal through the unit circuit 5.

  The DSP / CPU 3 is connected to a display device 6, a key input unit 7, and a clock unit 8, and is also connected to a DRAM 10, a built-in flash memory 11, and a card interface 12 through an address / data bus 9. . The card interface 12 is connected to a memory card 13 that is detachably attached to a card slot of a camera body (not shown).

  The DRAM 10 is a buffer for temporarily storing image data of a subject imaged and digitized by the CCD 2 when the shooting standby mode is set, and is also used as a working memory of the DSP / CPU 3. The image data temporarily stored in the DRAM 10 is compressed and finally recorded in the memory card 13. The built-in flash memory 11 stores an operation program of the DSP / CPU 3 and various data used for control of each unit, and the DSP / CPU 3 operates according to the operation program to obtain the acquisition means and feature amount calculation of the present invention. It functions as a means, a movement vector calculation means, an image extraction means, an image composition means, a judgment means, a movement vector integration means, and the like.

  The display device 6 includes a color LCD and its drive circuit, and displays the subject image captured by the CCD 2 as a through image when in the shooting standby state, and is read from the memory card 13 and expanded when the recorded image is reproduced. Display the recorded image. The key input unit 7 includes a plurality of operation keys such as a shutter key, a power key, a mode selection key, and a cross key, and outputs a key input signal corresponding to the key operation by the user to the DSP / CPU 3. The clock unit 8 has a calendar function that counts the date and time and a clock function, and sends date data and time data to the DSP / CPU 3 as necessary during a shooting operation or the like.

  In the electronic camera 1 having the above configuration, the DSP / CPU 3 executes processing as shown in the flowchart shown in FIG. 2 according to the program. That is, when the panorama shooting mode is selected by the user's operation of the mode selection key in the key input unit 7, the shooting process is executed (step S1), and then the image composition process is executed (step S1).

  FIG. 3 is a flowchart showing the procedure of the photographing process (step S1). First, display of a through image is started (step S101), and the through image is displayed on the display device 6 by the processing in step S101. Next, it is determined whether or not the shutter key has been pressed (step S102). If the shutter key is pressed, the photographing / recording process is executed to capture the frame image from the CCD 2 and store it in the DRAM 10 (step S103).

  Next, it is determined whether or not the pressing of the shutter key is released (step S104). If the pressing of the shutter key is not released, it is determined whether or not a predetermined time (1/16 second in the present embodiment) has elapsed (step S105). The processing from S103 is repeated. Therefore, in a state where the pressing of the shutter key is not released, the processing from step S105 is repeated at predetermined time (1/16 second) intervals.

  Therefore, if the user continues to press the shutter key without repeatedly operating the shutter key, the DRAM 10 can store a plurality of frame images having a predetermined frame rate.

  When the shutter key pressed by the user is released, step S104 is YES, and the process returns to the general flow shown in FIG. Therefore, the DRAM 10 stores a plurality of frame images taken at predetermined time intervals (1/16 seconds) from when the user presses the shutter key until the release. At this time, if the user performs the shooting while rotating the electronic camera 1 in a desired direction, for example, the right direction to perform panoramic shooting, the DRAM 10 has a subject in a direction opposite to the rotation direction of the electronic camera 1. A plurality of moved frame images are stored.

  FIG. 4 is a flowchart showing the processing procedure of the image composition processing (step S2). Here, a case will be described in which image composition processing is performed on a plurality of frame images obtained by photographing by rotating the electronic camera in the right direction. That is, a case will be described in which image composition processing is performed on a plurality of frame images that appear to move the subject image in the left direction as time elapses between the frame images.

  First, frame images are selected in the order stored from the DRAM 10 (step S201). That is, the frame images are selected according to the order in which they were taken.

  Next, the feature amount of the selected frame image is calculated (step S202). In calculating the feature amount, as shown in FIG. 5A, a plurality of blocks set near the center (center region) of the current frame image X selected in step S201 are set as the target image. A feature amount is calculated for each of the plurality of blocks, and a block having a feature amount equal to or greater than a predetermined value is extracted. In the calculation of the feature amount, a HARRIS operator is used as an arithmetic expression.

  Next, each block whose calculated feature quantity is greater than or equal to a predetermined block and a block having a common feature quantity are searched for in the next frame (rear frame) image X + 1 shown in FIG. 5B (step S203). Here, the next frame (rear frame) image X + 1 is a frame image stored (captured) next to the current frame image X. Then, in the current frame image X and the next frame image X + 1, each movement vector (X-axis component, Y-axis component) is detected for each corresponding block having each common feature amount. Further, by calculating, for example, an average value of the detected movement vectors, the movement vector of the entire frame is calculated (step S204). That is, the movement vector of the subject image from the current frame image X to the next frame image X + 1 is calculated.

  Next, an extraction width that is the width of the partial image to be extracted is set (step S205). The extraction width is set by extracting a preset movement direction component from the movement vector of the entire frame, and storing it in the built-in flash memory 11 shown in FIG. 6C based on the extracted movement direction component. This is done by reading from the conversion table 112 that has been set.

  In the case of the present embodiment, a panorama composite image in the left-right direction is generated, and a left-right direction component (X-axis direction component), that is, an amount of movement of the subject image in the left direction, for example, is extracted from the movement vector of the entire frame. Then, the extraction width W corresponding to this movement amount is read out.

  Subsequently, an image is extracted from the current frame image X with the extraction width with reference to a predetermined center line of the current frame image X (step S206). That is, as shown in FIG. 6B, first, a center line C that is a straight line passing through the center point of the frame and runs in a direction orthogonal to the extracted moving direction component (here, the Y-axis direction) is the current frame. Set on image X. Then, the partial image XP of the image corresponding to the extraction width W is extracted from the center line C in the current frame image X in the rotation direction (here, the right direction) of the electronic camera 1 which is the direction opposite to the movement vector. That is, when a partial image is extracted from the current frame image, the partial image XP is extracted in the direction opposite to the movement vector (the rotation direction of the electronic camera 1) with the center line C as a reference.

  Thereafter, as shown in FIG. 6 (A), the extracted partial image XP is rearward in the direction opposite to the movement vector of the panoramic composite image XS synthesized up to this point, that is, the time of the electronic camera 1. It is combined with the rear end portion (here, the right side) in the moving direction and synthesized (step S207).

  Subsequently, it is determined whether or not the above processing for calculating the feature amount has been performed up to the last −1 frame image stored in the DRAM 10 (step S208), and the processing for the last −1 frame image is completed. Until this, the processing from step S201 is repeated.

  By performing such panoramic image synthesis, in the image formed by the optical system, it is possible to extract and synthesize the central image portion without distortion without extracting the portions with distortion on both sides. it can. As a result, it is possible to make the panorama composite image XS free from distortion over the entire area.

  If each frame image is extracted from the same center line C as a reference with an extraction width corresponding to the magnitude of the movement vector, extraction from the previous frame image extracted between temporally adjacent frame images is performed. It is possible to accurately match the image at the end edge of the image and the start edge of the extracted image from the next frame image. Therefore, it is possible to combine panoramic images by accurately combining images taken adjacent in time.

  In the image composition in step S207, the vertical alignment when combining and synthesizing the end of the partial image XP extracted to the panorama composite image XS is performed by moving the partial image XP up and down according to the vertical component of the movement vector. This is done by adjusting the movement. Therefore, even when the user holds the electronic camera 1 and rotates it to the right, if the electronic camera 1 inevitably undergoes vertical vibration, the panorama composite image XS is as shown in FIG. It becomes a combined body of partial images shifted vertically.

  If the determination in step S208 is YES by executing steps S201 to 207 up to the last −1 frame image, the cutout / storage process is executed in step S209 following step S208. In this cutout / storing process, as shown in FIG. 7, an image portion having a predetermined width along the joining direction is cut out from the panorama composite image XS combined and synthesized up to the last step S207. The portion is recorded on the memory card 13 as the final panorama composite image XE.

  That is, as described above, even when the user holds the electronic camera 1 and rotates it to the right, the electronic camera 1 inevitably generates vertical vibrations. As shown in FIG. 7, it becomes a combined body of partial images shifted vertically. Therefore, an image portion having a predetermined width along the joining direction is cut out, and the cut out image portion is recorded as a final panorama composite image XE in the memory card 13, so that the panorama composite image XE having no unevenness in the peripheral portion is recorded. Can be recorded and saved.

In the present embodiment, the partial image to be synthesized is extracted based on the center line C that is a straight line passing through the center point of the frame. However, the reference line is not limited to the center line C, and the center portion If it is near, the position may be slightly shifted to the side. In particular, the direction in which the user rotates the electronic camera is determined in advance. Further, as described above, a part from the current frame image based on the movement vector searched for in the next frame using the feature amount in the current frame image. When extracting an image, the reference line is preferably set in advance at a position shifted in the direction opposite to the rotation direction of the electronic camera. That is, when the partial image is extracted from the current frame image, the partial image XP is extracted in the direction opposite to the movement vector (the rotation direction of the electronic camera 1) with the center line C as a reference. If a position shifted in the direction opposite to the camera rotation direction is set in advance, it is possible to extract the central region of the frame image so as to be included in the partial image.
Note that the direction in which the user rotates the electronic camera may be configured to be arbitrarily set in advance by the user using the key input unit 7.

  Further, in the present embodiment, the partial image to be synthesized from the current frame image X is extracted, but it may be extracted from the next frame image X + 1 with the extraction width. When extracting from the next frame image X + 1 with the extraction width, an image is extracted with the extraction width from the next frame image X + 1 with reference to a predetermined center line of the next frame image X + 1. In the image X + 1, the partial image XP of the image corresponding to the extraction width W is extracted from the center line C in the direction of the movement vector, that is, the direction opposite to the rotation direction of the electronic camera 1 (here, the left direction). In particular, when the direction in which the user rotates the electronic camera is determined in advance, the reference line is preferably set in advance at a position shifted in the rotation direction of the electronic camera. That is, when extracting the partial image from the next frame image, the partial image XP is extracted in the direction of the movement vector (the direction opposite to the rotation direction of the electronic camera 1) with the center line C as a reference. Can be extracted so as to include the center region of the frame image in the partial image. Note that the direction in which the user rotates the electronic camera may be configured to be arbitrarily set in advance by the user using the key input unit 7.

(Second Embodiment)
FIG. 8 is a diagram showing a priority table 111 stored in the built-in flash memory 11 together with the program and the like in the second embodiment of the present invention. In the case of the present embodiment, a panorama composite image in the left-right direction is generated, and in this priority table 111, a later-described movement vector is the left direction (the movement vector includes a left-direction component). Data (A) in the case of shooting and data (B) in the case of panoramic shooting in which the movement vector is in the right direction (the movement vector includes a component in the right direction).

  As the data (A), when the motion vector to be detected and predicted is the left direction, that is, when the rotation direction of the electronic camera is the right direction, the feature amount is extracted in the current frame X selected this time. For this reason, areas (feature amount extraction areas) having priorities 1 to 4 are stored. With regard to the priorities 1 to 4, the central region is prioritized 1. Further, the priority order is set so as to decrease from the central area to the left, and the priority order 4 area is originally intended to be used for feature quantity extraction because of the distortion of the image formed by the optical system. There is no area.

  In this embodiment, as will be described later in step S302, the central area having priority 1 is given the highest priority, and feature amounts are extracted from the central area as much as possible. Therefore, an image area of a non-distorted portion of the image formed by the optical system can be set as a preferential area for calculating the feature quantity, whereby the feature quantity can be calculated appropriately.

  Further, as data (B), when the motion vector to be detected and predicted is the right direction, that is, when the rotation direction of the electronic camera is the left direction, the feature amount in the present frame PX selected this time is A region (feature amount extraction region) having priorities 1 to 4 for extraction is stored. The priorities 1 to 4 are set such that the central area has the priority 1 and the priority decreases as it goes to the right from the central area. The priority 4 area is imaged by the optical system. Since the image is distorted, it is an area that is not originally desired for feature amount extraction.

  In the present embodiment, only the data (A) and (B) when the movement vector is in the left direction and in the right direction are stored. Similar data in the case of the above may be stored.

  FIG. 9 is a flowchart showing a processing procedure of the image composition processing (step S2) in the second embodiment. Here, a case will be described in which the rotation direction of the electronic camera is the right direction among the left and right directions. First, frame images are selected in the order stored from the DRAM 10 (step S301). Next, in the selected frame image, a feature amount is calculated according to the above-described priority order (step S302).

  In the process of step S3, an initial value “1” is set as the value of the counter N for defining the priority order, and it is determined whether or not the feature amount can be calculated in the priority order N region in the previous frame. . That is, if N = 1, as shown in FIG. 8A, the priority order 1 area, which is the central area in the current frame image X, is divided into a plurality of blocks in the same manner as the processing in step S202 described above. Divide into Then, based on the result of the calculation of the feature values in each of the plurality of blocks, it is determined whether the feature value can be calculated (or whether there is a block whose feature value is equal to or greater than a predetermined value). .

  At this time, if, for example, a wide white column or wall is photographed in the priority order 1 area in the frame image X, it is impossible to calculate the feature amount in any block in the priority order 1 area. It becomes. Therefore, in this case, the value of the counter N is incremented and the same processing is performed for the priority order 2 area. In addition, when it is impossible to calculate the feature amount in any block in the priority order 2 area, the same processing is sequentially performed on the priority order 3 and 4 areas.

  Therefore, even if it is impossible to calculate the feature amount in any block in the priority order 1 region, the feature amount can be calculated using another region.

  Next, it is determined whether or not a feature amount greater than or equal to a predetermined value has been calculated in any of the priorities 1 to 4 (step S303). If it can be calculated, it is determined whether or not an update movement vector to be described later is stored (step S303). If not stored, in steps S305 to S310, processing similar to that in steps S203 to 208 in the first embodiment described above is executed.

  That is, each block whose feature value calculated in step S303 is equal to or greater than a predetermined block and a block having a common feature value are searched in the next frame (rear frame) image X + 1 shown in FIG. 10B (step S305). . That is, if the current frame image X is “(1)” and the feature amount is “a” in FIG. 10A, the feature in the frame image (2) in the next frame X + 1 in FIG. Search for the quantity “a”.

  Then, in the current frame image X (1) and the next frame image X + 1 (2), each movement vector (X-axis component, Y-axis component) is detected for each corresponding block having each common feature amount a. Further, by calculating, for example, an average value of the detected movement vectors, the movement vector (A) of the entire frame is calculated (step S306).

  Next, an extraction width that is the width of the partial image to be extracted is set (step S307). The extraction width is set by extracting a preset movement direction component from the movement vector of the entire frame, and storing it in the built-in flash memory 11 shown in FIG. 6C based on the extracted movement direction component. This is done by reading from the conversion table 112 that has been set.

  In the case of the present embodiment, a panorama composite image in the left-right direction is generated, and a left-right direction component (X-axis direction component), that is, an amount of movement of the subject image in the left direction, for example, is extracted from the movement vector of the entire frame. Then, the extraction width W corresponding to this movement amount is read out.

  Subsequently, an image is extracted from the current frame image X with the extraction width with reference to a predetermined center line of the current frame image X (step S308). That is, as shown in FIG. 10C, first, a reference line L that runs in a direction (in this case, the Y-axis direction) that is a straight line passing through the vicinity of the center of the frame and that is orthogonal to the extracted movement direction component is set to the current frame. Set on image X. Then, the partial image XP of the image corresponding to the extraction width W is extracted from the reference line L in the current frame image X in the rotation direction of the electronic camera 1 which is the direction opposite to the movement vector (here, the right direction). That is, when a partial image is extracted from the current frame image, the partial image XP is extracted in the direction opposite to the movement vector (the rotation direction of the electronic camera 1) with reference to the reference line L.

  Thereafter, as shown in FIG. 6 (A), the extracted partial image XP is rearward in the direction opposite to the movement vector of the panoramic composite image XS synthesized up to this point, that is, the time of the electronic camera 1. It is combined with the rear end portion (here, the right side) in the moving direction and synthesized (step S309).

  Subsequently, it is determined whether or not the above processing has been performed up to the last -1 frame image-1 stored in the DRAM 10 (step S310), and the steps until the processing for the last -1 frame image is completed. The processing from S301 is repeated.

  That is, as shown in (1) to (6) of FIG. 10A, when a subject other than the “feature feature extraction inappropriate subject” is photographed in any of the priority order 1 to 4 regions. The determination in step S303 is YES, and the determination in step S304 is NO. Therefore, the loop of steps S301 to S310 is repeated, and the processing in step S205 results in the next frame images (2) to (7) in FIG. 10B corresponding to the current frame images (1) to (6). The feature quantities a, b, c, d, e, and f are respectively searched, and the movement vectors A, B, C, D, E, and F are calculated by the processing in step S206.

  Then, by the processing in step S307 and step S308, as shown in FIG. 10 (c), from the current frame images (1) to (6), according to the movement vectors A, B, C, D, E, and F, respectively. The partial images WA, WB, WC, WD, WE, and WF are extracted from the reference line L with the extracted width. The partial images WA, WB, WC, WD, and WE are combined and combined by the processing in step S309, and the partial image WF described later is also combined and combined at this point.

  However, as shown in the current frame image (7) in FIG. 10 (a), when “feature feature extraction ineligible subjects” are photographed in all of the priority order 1 to 4 areas, the priority orders 1 to 4 are taken. In any of the regions, a feature amount greater than a predetermined value cannot be calculated, and the determination in step S303 is NO. Accordingly, the process proceeds from step S303 to step S311, and the next frame image X + 1 (the next frame image of the current frame image X selected in step S301) is searched based on the feature amount calculated last (step S311).

  That is, since the feature amount calculated last is the frame image (6) in FIG. 10 (a), the next frame image X + 1 in FIG. 10 (b) is based on the feature amount f of the frame image (6). ) In the frame image (8). Then, similarly to step S306 described above, the movement vector of the entire next frame image X + 1 (frame image (8)) is calculated (step S306). The calculated movement vector is stored in the DRAM 10 while being updated (step S313), and the process returns to step S301. Therefore, if the movement vector calculated at this time is “G” as shown in (8) of FIG. 10B, “movement vector G” is stored in the DRAM 10 by the processing in step S313. It will be.

  When the processing from step S301 is executed again, the frame image (8) following the frame image (7) in FIG. 10 (a) is selected as the current frame image X in step S301. However, since the frame image X (8) also has “feature feature extraction ineligible subjects” in all of the priority order 1 to 4 areas, the predetermined or higher characteristics are obtained in any of the priority order 1 to 4 areas. The amount cannot be calculated, and the determination in step S303 is NO.

  Accordingly, the process proceeds from step S303 to step S311, and the next frame image X + 1 (the next frame image of the current frame image X selected in step S301) is searched based on the feature amount calculated last (step S311). At this time, since the feature amount calculated last is from the frame image (6) of FIG. 10 (a) as described above, the next frame image X + 1 is based on the feature amount f of the frame image (6). A search is made in the frame image (9) of FIG. Then, similarly to step S306 described above, the movement vector of the entire next frame image X + 1 (frame image (9)) is calculated (step S306). The calculated movement vector is stored in the DRAM 10 while being updated (step S313), and the process returns to step S301. Accordingly, if the movement vector calculated at this time is “H” as shown in (9) of FIG. 10B, “movement” is substituted for the “movement vector G” by the process in step S313. The vector H ”is stored in the DRAM 10.

  When the processing from step S301 is executed again, the frame image (9) following the frame image (8) in FIG. 10 (a) is selected as the current frame image X in step S301. Since this frame image X (9) can calculate the “feature amount g” in the priority order 4 region, the determination in step S303 is YES. Therefore, the process proceeds from step S303 to step S304, and it is determined whether or not there is an updated movement vector, that is, whether or not there is an updated movement vector that is the movement vector stored in the DRAM 10 in step S313.

  In this example, the process of step S303 is performed twice, and the DRAM 10 stores the second updated movement vector H. Accordingly, the determination in step S304 is YES, and the process advances from step S304 to step S314 to read the stored updated movement vector H (step S314). Next, an extraction width that is the width of the partial image to be extracted is set (step S315). This extraction width is set from the conversion table 112 stored in advance in the built-in flash memory 11 shown in FIG. 6C by the size of the movement direction component of the updated movement vector H (in this example, the left direction component). This is performed by reading the extraction width W corresponding to

  Subsequently, in the frame image from which the partial image is finally extracted, an image is extracted with the extraction width from the reference line L (step S316). That is, in this example, as shown in FIG. 10C, the frame image from which the partial image is extracted last is the frame image (6) from which the partial image F is extracted. Therefore, as shown in FIG. 10 (d), first, a reference line L that is a straight line passing through the vicinity of the center of the frame and orthogonal to the movement vector is set on the frame image (6). Then, a partial image WH of the image corresponding to the extraction width W is extracted from the reference line L in the frame image (6) in the rotation direction of the electronic camera 1 which is the direction opposite to the movement vector.

  Thereafter, this partial image WH is overwritten and synthesized on the last synthesized partial image of the panorama synthesized image synthesized up to this point (step S317). That is, as described above, in the case of the frame images (1) to (6) in FIG. 10 (a), the movement vectors A, B, C, D, Partial images WA, WB, WC, WD, WE, and WF are extracted from the reference line L with an extraction width corresponding to E and F, and the partial images WA, WB, WC, WD, and WE are extracted by processing in step S309. , WF are combined and synthesized. Of the combined images composed of these partial images WA, WB, WC, WD, WE, and WF, the last combined partial image is a partial image WF. Therefore, when the partial image WH is overwritten and combined with the partial image WF, the partial image WH is combined with the partial image WE as shown in FIG.

  Thereafter, the process proceeds from step S317 to step S305, and the processes after step S305 described above are executed. Therefore, in step S305, the next frame image X + 1, which is the frame image (10) of FIG. 10 (b), is searched based on the feature amount g of the frame image (9) of FIG. 10 (b). The movement vector I is calculated, a width W corresponding to the movement vector is set in step S307, and in step S308, the partial image WI is extracted from the reference line L in the frame image (9) of FIG. In step S309, the partial image WI is combined and combined with the combined image having the partial image WH at the end.

  Further, as shown in (10) to (14) of FIG. 10 (a), when a subject other than the “feature feature extraction ineligible subject” is photographed in any of the priority order 1-4 regions. The determination in step S303 is YES, and the determination in step S304 is NO. Therefore, the loop of steps S301 to S310 is repeated.

  When the 15 frame images (1) to (15) shown in FIG. 10A are stored in the DRAM 10, when the last frame image (14) is selected in step S301, The determination in S310 is YES. Therefore, the process proceeds from step S310 to step S318, and the cutout / storage process, which is the same process as step S209 described above, is executed. As a result, as shown in FIG. 10E, the final panorama obtained by combining the partial images of “WA-WB-WC-WD-WE-WH-WI-WJ-WK-WL-WM-WN”. The image is stored in the memory card 13.

  That is, in this embodiment, when generating a panoramic image based on a plurality of frame images that are captured so that a part of the subject image overlaps at least between temporally adjacent frame images, When the predetermined region set in advance is an image that is ineligible as a feature amount extraction region, the frame image is omitted from the panorama synthesized image, and sequentially corresponds to the first frame image for a predetermined time. A partial image is extracted based on a movement vector calculated between the second frame images having a relationship, and a panoramic image is generated by combining the extracted partial images.

  In the present embodiment, the partial image is extracted from the current frame image X. However, as shown in FIG. 11, the partial image may be extracted from the next frame image X + 1. In this case, as shown in FIG. 11C, each partial image WA, WB, WC, WD... Is a movement vector in the next frame image (2) (3) (4) (5). What is necessary is just to cut out in the direction (the direction opposite to the moving direction of the electronic camera 1).

  In the present embodiment, the partial image is synthesized with the synthesized image immediately after the partial image is extracted, but the extracted partial image is stored and the next feature amount is detected. Thus, the stored partial image may be combined with the combined image. In this way, as shown in step S317, the panorama image is combined and synthesized without overwriting the partial image synthesized at the end of the panorama synthesized image synthesized up to this point, so that the panorama is combined. A composite image can be obtained.

  Further, in each of the above-described embodiments, it has been described that a panoramic composite image in the left-right direction is generated. However, the present invention can also be applied to a case where a panoramic composite image in the vertical direction is generated.

  In each embodiment, as shown in the flowcharts of FIGS. 4 and 9, the image synthesis process is started as a synthesis target from the first frame image stored in the DRAM 10. However, at the beginning of the image compositing process, the calculation of the feature amount and the calculation of the movement vector are executed, and the process after step S201 or the process from step S301 is performed on condition that a predetermined or more movement vector is calculated first. You may make it perform.

  This makes it possible to detect whether or not the user has rotated the electronic camera 1, that is, whether or not the user has started to rotate the electronic camera 1 with the intention of starting panoramic photography. Therefore, it is possible to prevent inconvenience that the panoramic image synthesis process or the like is performed using an unnecessary image before the user starts to rotate the electronic camera 1 with the intention of starting panoramic shooting.

  At this time, based on the detected direction of the predetermined or larger movement vector, it is detected whether the rotation direction of the electronic camera is the up / down / left / right direction, and the detected rotation direction of the electronic camera is detected. It may be configured to perform a corresponding panorama composite image generation process. That is, for example, when the rotation direction of the electronic camera is the left direction or the right direction, a panoramic composite image in the left-right direction is generated, and when it is upward or downward, a panoramic composite image in the vertical direction is generated. It is good also as a structure. For example, when it is detected that the rotation direction of the electronic camera is the right direction, for example, the priority order of the feature amount extraction region is set in FIG. 8A described above, and the left direction is detected. In some cases, for example, the priority order of the feature amount extraction regions may be set in the above-described FIG.

  In each embodiment, as shown in the flowcharts of FIGS. 4 and 9, the image composition process is terminated when the last −1 frame image is selected. However, the movement vector calculated first may be stored, and the image synthesis process may be terminated when a movement vector in a direction different from the movement vector (for example, the opposite direction) is detected. .

  As a result, the electronic camera 1 that has been rotated to perform panoramic photography can be rotated substantially in a direction different from the rotation direction without releasing the pressing of the shutter key described in step S104. Thus, panoramic shooting can be terminated.

  Further, in each embodiment, the image processing is performed and a plurality of frame images are stored in the DRAM, and then the image composition processing is performed. However, each time a frame image is acquired by the image capturing processing, the image processing is sequentially performed. An image composition process may be executed.

It is a block diagram of the electronic camera which concerns on one embodiment of this invention. 3 is a general flowchart common to the first and second embodiments of the present invention. It is a flowchart which shows the process sequence of an imaging | photography process. 3 is a flowchart illustrating a processing procedure of image composition processing in the first embodiment. It is explanatory drawing which shows the calculation procedure of a movement vector. It is explanatory drawing which shows the process of extracting the partial image to synthesize | combine from a now frame image, and the procedure of a synthesis | combination process. It is explanatory drawing which shows the trimming area | region of a panoramic composite image. It is explanatory drawing which shows the memory content of a priority table. It is a flowchart which shows the process sequence of the image composition process in 2nd Embodiment. It is explanatory drawing which shows the process sequence of the image composition process in the embodiment. It is explanatory drawing in the case of extracting a partial image from the next frame image.

Explanation of symbols

1 Electronic camera 2 CCD
3 DSP / CPU
4 TG
5 Unit Circuit 6 Display Device 7 Key Input Unit 9 Data Bus 10 DRAM
11 Built-in flash memory 111 Priority table 112 Conversion table

Claims (16)

Acquisition means for capturing and sequentially acquiring a plurality of frame images;
A feature amount calculating means for calculating a feature amount of a predetermined area in each frame image sequentially acquired by the acquiring means;
Based on the feature amount calculated by the feature amount calculation means, a movement for sequentially calculating a movement vector of a second frame image having a predetermined temporal correspondence with the first frame image for which the feature amount has been calculated Vector calculation means;
Setting means for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation means;
Image extraction means for extracting the partial image having the extraction width set by the setting means from the first and / or second frame images;
Image combining means for sequentially combining and combining the partial images extracted by the image extracting means,
The image synthesizing apparatus, wherein the image extracting unit extracts a partial image having an extraction width set by the setting unit from a predetermined reference line in the frame image in a direction according to the movement vector. A determination unit that determines whether the feature amount calculation unit can calculate the feature amount in the predetermined region;
The movement vector calculation means includes the feature quantity within a non-calculatable time during which it is determined by the judgment means that the feature quantity calculation means cannot calculate the feature quantity in the predetermined area. 2. The image composition according to claim 1, wherein a movement vector of a frame image last acquired by the acquisition unit within the uncalculatable time is calculated based on a feature amount calculated last by the calculation unit. apparatus. Acquisition means for capturing and sequentially acquiring a plurality of frame images;
A feature amount calculating means for calculating a feature amount of a predetermined area in each frame image sequentially acquired by the acquiring means;
Based on the feature amount calculated by the feature amount calculation means, a movement for sequentially calculating a movement vector of a second frame image having a predetermined temporal correspondence with the first frame image for which the feature amount has been calculated Vector calculation means;
Setting means for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation means;
Image extraction means for extracting the partial image having the extraction width set by the setting means from the first and / or second frame images;
Image combining means for sequentially combining and combining the partial images extracted by the image extracting means;
A determination unit that determines whether the feature amount calculation unit can calculate a feature amount in the predetermined region;
The movement vector calculation means includes the feature quantity within a non-calculatable time during which it is determined by the judgment means that the feature quantity calculation means cannot calculate the feature quantity in the predetermined area. An image synthesizing apparatus that calculates a movement vector of a frame image last acquired by the acquisition unit within the non-calculatable time based on a feature amount calculated last by the calculation unit.   4. The image composition according to claim 3, wherein the image extraction unit extracts a partial image having an extraction width set by the setting unit in a direction corresponding to the movement vector from a predetermined reference line in the frame image. apparatus.   5. The image synthesizing apparatus according to claim 1, wherein the reference line is set near a center portion of a frame image.   6. The image according to claim 1, further comprising start operation control means for starting the operation of the image composition means on condition that the first movement vector is calculated by the movement vector calculation means. Synthesizer. Storage means for storing a movement vector first calculated by the movement vector calculation means;
Thereafter, when the movement vector calculated by the movement vector calculation means is in a direction different from the initial movement vector stored in the storage means, an end operation control means for ending the operation of the image synthesizing means is provided. The image composition device according to any one of claims 1 to 6.   The image synthesizing apparatus according to any one of claims 1 to 7, wherein the predetermined area in which the feature quantity calculating unit calculates the feature quantity is a central area of the frame image. The predetermined area where the feature quantity calculating means calculates the feature quantity is composed of a plurality of divided areas divided according to the priority order,
The determination means determines whether or not the feature amount can be calculated in the divided region having a high priority,
The feature amount calculation unit calculates the feature amount from the segment region with a lower priority when the determination unit determines that the feature amount cannot be calculated in the segment region with the higher priority. 8. The image composition apparatus according to claim 1, wherein   The movement vector accumulating unit is configured so that the determination unit determines that the feature amount calculation unit is unable to calculate a feature amount from any of the divided regions. The movement vector of the frame image last acquired by the acquisition unit within the non-calculatable time is calculated based on the feature amount calculated last by the feature amount calculation unit. Image composition device.   The image synthesizing apparatus according to claim 1, wherein the acquisition unit includes an imaging unit that performs imaging at a constant time interval in response to an operation. A computer provided in the image composition device,
Acquisition means for capturing and sequentially acquiring a plurality of frame images;
A feature amount calculating means for calculating a feature amount of a predetermined area in each frame image sequentially acquired by the acquiring means;
Based on the feature amount calculated by the feature amount calculation means, a movement for sequentially calculating a movement vector of a second frame image having a predetermined temporal correspondence with the first frame image for which the feature amount has been calculated Vector calculation means;
Setting means for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation means;
Image extraction means for extracting the partial image having the extraction width set by the setting means from the first and / or second frame images;
While functioning as an image composition means for combining the partial images extracted by this image extraction means in sequence,
The image extracting program extracts a partial image having an extraction width set by the setting unit in a direction corresponding to the movement vector from a predetermined reference line in the frame image. A computer provided in the image composition device,
Acquisition means for capturing and sequentially acquiring a plurality of frame images;
A feature amount calculating means for calculating a feature amount of a predetermined area in each frame image sequentially acquired by the acquiring means;
Based on the feature amount calculated by the feature amount calculation means, a movement for sequentially calculating a movement vector of a second frame image having a predetermined temporal correspondence with the first frame image for which the feature amount has been calculated Vector calculation means;
Setting means for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation means;
Image extraction means for extracting the partial image having the extraction width set by the setting means from the first and / or second frame images;
Image combining means for sequentially combining and combining the partial images extracted by the image extracting means;
The feature amount calculation means functions as a determination means for determining whether or not the feature amount can be calculated in the predetermined area, and
The movement vector calculation means includes the feature quantity within a non-calculatable time during which it is determined by the judgment means that the feature quantity calculation means cannot calculate the feature quantity in the predetermined area. An image composition program for calculating a movement vector of a frame image last acquired by the acquisition unit within the non-calculatable time based on a feature amount calculated last by the calculation unit. An acquisition step of capturing and sequentially acquiring a plurality of frame images;
A feature amount calculating step for calculating a feature amount of a predetermined region in each frame image sequentially acquired by the acquiring step;
Based on the feature amount calculated in the feature amount calculation step, a movement for sequentially calculating a movement vector of a second frame image having a predetermined temporal correspondence with the first frame image for which the feature amount has been calculated A vector calculation step;
A setting step for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation step;
An image extraction step of extracting the partial image having the extraction width set by the setting step from the first and / or second frame images;
An image synthesis step of sequentially combining and synthesizing the partial images extracted by the image extraction step,
The image extracting method includes extracting a partial image having an extraction width set in the setting step from a predetermined reference line in the frame image in a direction according to the movement vector. An acquisition step of capturing and sequentially acquiring a plurality of frame images;
A feature amount calculating step for calculating a feature amount of a predetermined region in each frame image sequentially acquired by the acquiring step;
Based on the feature amount calculated in the feature amount calculation step, a movement for sequentially calculating a movement vector of a second frame image having a predetermined temporal correspondence with the first frame image for which the feature amount has been calculated A vector calculation step;
A setting step for setting the extraction width of the image according to the magnitude of the movement vector calculated by the movement vector calculation step;
An image extraction step of extracting the partial image having the extraction width set by the setting step from the first and / or second frame images;
An image synthesis step of sequentially combining and synthesizing the partial images extracted by the image extraction step;
A step of determining whether the feature amount calculating step can calculate a feature amount in the predetermined region;
In the movement vector calculation step, the feature amount is calculated within the non-calculatable time in which it is determined by the determination step that the feature amount cannot be calculated in the predetermined region in the feature amount calculation step. An image synthesizing method comprising: calculating a movement vector of a frame image last acquired by the acquisition step within the non-calculatable time based on a feature amount calculated last by the calculation step. An image synthesis method for generating a panoramic image based on a plurality of frame images photographed so that a part of a subject image overlaps at least between temporally adjacent frame images,
When a predetermined region set in advance in the frame image is an image that is ineligible as a feature amount extraction region, the first frame image and a predetermined amount are sequentially deleted while omitting the frame image from the panoramic image. A partial image is extracted based on a movement vector calculated with respect to a second frame image having a temporal correspondence relationship, and a panoramic image is generated by combining the extracted partial images. Image synthesis method.

Images