JP2009278432A - Imaging apparatus, and imaging method in imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of reducing a motion detection error between images and determining the amount of movement of an image with a sufficient precision. <P>SOLUTION: A motion detection portion 15 detects motion between the current frame image and a previous frame image just before the current one to compute the detected result with the previous frame, while the portion 15 detects motion between the current frame image and a reference frame image to compute the detected result with the reference frame. A reliability evaluation portion 151 at the motion detection portion 15 evaluates reliability for the detected result with the previous frame and the detected result with the reference frame. Based on the evaluated result by the reliability evaluation portion, a movement amount determination portion at the motion detection portion 15 determines the amount of image movement from the reference frame to the current frame using the detected result with the previous frames, if the detected result with the previous frames is significant. When the detected result with the previous frame is significant but the detected result with the reference frame is insignificant, the movement amount determination portion determines the amount of image movement, according to the detected result with the previous frame and the last value of the amount of image movement. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method in the imaging apparatus.

  Conventionally, an imaging device such as a digital camera is known which has a panoramic shooting mode for generating a panoramic image by synthesizing a plurality of captured images, and the subject is photographed for each part while panning the imaging device. By synthesizing the photographed image data, a panoramic image in which the photographed images for each part are connected can be obtained. In digital cameras, the output from the image sensor is displayed on the display unit at a predetermined frame rate and used as an electronic viewfinder. In the panorama shooting mode, the motion between live view images is detected and integrated for each frame. Thus, the amount of movement of the image is obtained, and the joint between the captured images is detected. According to this, it is possible to perform the photographing process at an appropriate timing that does not cause a sense of incongruity at the joint. For example, Patent Literature 1 discloses a technique for obtaining a motion (motion vector) between images accompanying panning and controlling a width when stitched images are joined based on the motion vector.

Japanese Patent No. 3421859

  By the way, when the movement between the live view images is detected for each frame to obtain the moving amount of the live view image, for example, when the frame rate is 30 (frames / second), from the first shooting to the second shooting. If the time required for the panning operation is 3 seconds, motion detection is performed 90 times. Here, in motion detection, a detection error with respect to the actual movement amount may occur. For this reason, when the movement amount of the image is calculated by integrating the motion detection results, the error generated each time the motion is detected is integrated, and the movement amount of the live view image cannot be calculated accurately. there were. As a result, there is a case where a gap occurs in the joint at the time of synthesis, and the generated panoramic image is uncomfortable.

  The present invention has been made in view of the above, and provides an imaging apparatus capable of reducing motion detection errors between images and accurately determining the amount of image movement, and an imaging method in the imaging apparatus. With the goal.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes an imaging unit that images a subject and generates image data, and performs a plurality of imaging processes to generate a plurality of captured images. An image pickup apparatus, wherein the image data picked up by the image pickup unit is displayed by switching in a predetermined frame unit, and an image of the current frame and an image of an adjacent frame adjacent to the current frame. Detection of motion between adjacent frames, detection of motion between reference frames, and detection of motion between adjacent frames, and detection of motion between adjacent frames Based on the result and the detection result of the reference inter-frame motion detection unit, based on the reliability evaluation unit that evaluates the reliability of each detection result, and the evaluation result of the reliability evaluation unit, A movement amount determination unit that determines an image movement amount from the reference frame to the current frame using a detection result of the inter-frame motion detection unit and / or a detection result of the reference inter-frame motion detection unit, and the movement amount determination A photographing start instruction unit that instructs the start of the photographing process based on an image movement amount determined by the unit.

  In the image pickup apparatus according to the present invention, in the above invention, a moving mark that moves according to an image moving amount determined by the moving amount determining unit and a movement target position of the moving mark are displayed on the image of the current frame. A display control unit that performs control to display a movement target mark, and the imaging start instruction unit instructs to start the imaging process when the movement mark moves to the movement target position indicated by the movement target mark It is characterized by doing.

  In the imaging device according to the present invention, in the above invention, the reliability evaluation unit determines whether or not each detection result is reliable based on a reliability evaluation result for each detection result. It is characterized by that.

  Further, in the imaging apparatus according to the present invention, in the above invention, the movement amount determination unit is at least when both the detection results are determined to be reliable as a result of the evaluation by the reliability evaluation unit. The image movement amount is determined using a detection result of the reference inter-frame motion detection unit.

  Further, in the imaging device according to the present invention, in the above invention, when the movement amount determination unit determines that both of the detection results are reliable as a result of the evaluation by the reliability evaluation unit, The average of the image movement amount from the reference frame to the current frame determined by the detection result of the motion detection unit between adjacent frames and the image movement amount from the reference frame to the current frame determined by the detection result of the motion detection unit between the reference frames A value is determined as the image movement amount.

  Further, in the imaging device according to the present invention, in the above invention, when the movement amount determination unit determines that both of the detection results are reliable as a result of the evaluation by the reliability evaluation unit, The image movement amount is determined by performing corresponding weighting on the detection result of the inter-adjacent frame motion detection unit and the detection result of the reference inter-frame motion detection unit.

  In the imaging device according to the present invention, in the above invention, the movement amount determination unit determines that one of the detection results is unreliable as a result of the evaluation by the reliability evaluation unit. In this case, the image movement amount is determined using the other detection result determined to be reliable.

  In the imaging device according to the present invention, in the above invention, the movement amount determination unit determines that the detection result of the inter-frame motion detection unit is reliable as a result of the evaluation by the reliability evaluation unit. In this case, the image movement amount is determined using the detection result of the reference inter-frame motion detection unit, and when the detection result of the reference inter-frame motion detection unit is determined to be unreliable, the inter-adjacent frame motion is determined. The image movement amount is determined using a detection result of a detection unit.

  Further, in the imaging apparatus according to the present invention, in the above invention, the movement amount determination unit is configured to display a current state when both of the detection results are determined to be unreliable as a result of the evaluation by the reliability evaluation unit. The image movement amount is determined by estimating a motion between an image of a frame and an image of an adjacent frame adjacent to the current frame.

  In the imaging device according to the present invention, in the above invention, when the detection result of the inter-frame motion detection unit is determined to be unreliable as a result of the evaluation by the reliability evaluation unit, A reference frame changing unit for changing to the current frame is provided.

  An imaging method in an imaging apparatus according to the present invention is an imaging method in an imaging apparatus that includes an imaging unit that images a subject and generates image data, and generates a plurality of captured images by performing a plurality of imaging processes. And detecting a motion between the display step of switching and displaying the image data picked up by the image pickup unit in units of a predetermined frame and the image of the current frame and the image of an adjacent frame adjacent to the current frame. A motion detection step between adjacent frames, a motion detection step between reference frames for detecting a motion between an image of the current frame and an image of a predetermined reference frame, a detection result in the motion detection step between adjacent frames, and the reference Based on the detection result in the inter-frame motion detection step, a reliability evaluation step for evaluating the reliability of each detection result; Based on the evaluation result in the reliability evaluation step, the detection result in the inter-adjacent frame motion detection step and / or the detection result in the reference inter-frame motion detection step is used to determine from the reference frame to the current frame. A movement amount determination step for determining an image movement amount; and a photographing start instruction step for instructing the start of the photographing process based on the image movement amount determined in the movement amount determination step.

  According to the present invention, the motion between adjacent frames between the current frame image and the adjacent frame image is detected, and the motion between the reference frames between the current frame image and the predetermined reference frame image is detected. Detecting and determining the amount of image movement from the reference frame to the current frame using the detection result between adjacent frames and / or the detection result between the reference frames based on the reliability evaluation result for each detection result Can do. For example, if the reliability evaluation of the detection result between the reference frames is high, the detection result between the reference frames can be used when determining the image movement amount, so that the motion detection error between images can be reduced. This is advantageous in that the amount of movement of the image can be determined with high accuracy.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In this embodiment, a case where the imaging device of the present invention is applied to a digital camera will be described as an example.

(Embodiment)
FIG. 1 is a rear view of the digital camera 1. As shown in FIG. 1, the digital camera 1 includes a shutter button (release button) 3 for instructing photographing timing provided on the upper surface of the camera body 2, and a power button 4 provided on the back surface of the camera body 2. And menu button 5, cross button 6 having up / down / left / right direction buttons (up button, down button, left button and right button), OK button 7 for confirming operation details, and display unit for displaying various screens 22 etc. Although not shown, a finder, an imaging lens, and the like are disposed on the front surface of the camera body 2. When the user presses the power button 4 to turn on the power, the digital camera 1 is in a state where it can shoot (shooting mode). In this shooting mode, a subject image formed on the image sensor 12 (see FIG. 2) through the imaging lens is captured every frame (for example, 1/30 second) and displayed on the display unit 22 in real time as a live view image. It has come to be. For example, electronic photographing is performed at the timing of pressing the shutter button 3 or the like.

  First, the configuration of the digital camera 1 will be described. FIG. 2 is a schematic block diagram illustrating a configuration example of the digital camera 1. As shown in FIG. 2, the digital camera 1 includes an imaging optical system 11, an imaging device 12 as an imaging unit, an AFE (Analog Front End) 13, a frame memory 14, a motion detection unit 15, an image processing unit 16, and a recording medium I. / F17, recording medium holding unit 18, recording medium 19, video encoder 20, display driver 21, display unit 22, video signal output terminal 23, operation unit 24, RAM 25, ROM 26, controller 27, and the like.

  The imaging optical system 11 includes an imaging lens, a diaphragm, a shutter, and the like, and forms an incident subject image on the imaging element 12. The image pickup device 12 is a solid-state image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), for example, and receives a light beam from a subject via the image pickup optical system 11 and photoelectrically converts it into frame units. Image data (analog electrical signal) is obtained. The AFE 13 performs analog signal processing such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the image data obtained by the image sensor 12, and then performs A / D conversion processing to convert it into a digital electrical signal. To do. The image data digitized by the AFE 13 is output to the frame memory 14 and the motion detection unit 15 and is temporarily stored in the RAM 25.

  The frame memory 14 is used as a working memory by the motion detector 15 and includes an area for storing image data for two frames. FIG. 3 is a schematic diagram showing an internal configuration of the frame memory 14. The frame memory 14 includes the current frame data 141, which is image data of the current frame image (current frame image), and the previous frame image (previous frame). The immediately preceding frame data 143 that is image data and the reference frame data 145 that is image data of a predetermined reference frame image (reference frame image) are stored.

  The motion detection unit 15 is for detecting the motion between frames based on the image data from the AFE 13. For example, the motion between images captured from the image sensor 12 and output from the AFE 13 for each frame is output. The motion between frames is detected by obtaining a vector. FIG. 4 is an explanatory diagram for explaining an outline of motion detection performed by the motion detection unit 15. In the present embodiment, a frame immediately preceding the current frame is set as the immediately preceding frame captured immediately before the current frame. Then, the motion detection unit 15 performs pattern matching between the current frame data 141 input from the AFE 13 as needed and stored in the frame memory 14 and the previous frame data 143, and obtains a motion vector representing the amount of movement of the image as the previous frame. By calculating as an inter-detection result, it functions as a motion detection unit between adjacent frames. In addition, the motion detection unit 15 performs pattern matching between the current frame data 141 and the reference frame data 145, and calculates a motion vector representing the amount of movement of the image as a reference inter-frame detection result, thereby moving the reference inter-frame motion. Functions as a detection unit.

  FIG. 5 is a schematic diagram showing a functional configuration of the motion detection unit 15. As shown in FIG. 5, the motion detection unit 15 includes a reliability evaluation unit 151 that evaluates the reliability of the immediately preceding inter-frame detection result and the reference inter-frame detection result detected as described above, and a reliability evaluation unit 151. A movement amount determination unit 153 that determines an image movement amount from the reference frame where motion detection is started to the current frame using the detection result between the immediately preceding frames and / or the detection result between the reference frames based on the evaluation result; A reference frame setting is updated based on the evaluation result by the evaluation unit 151, and a motion detection area setting unit 155 that functions as a reference frame changing unit is included. Although not shown, the motion detection unit 15 temporarily stores a detection result between previous frames, a detection result between reference frames, a reliability evaluation result of each detection result, setting information of a motion detection area, and the like. This RAM is provided inside.

  As shown in FIG. 2, the image processing unit 16 reads image data once stored in the RAM 25, performs various image processing on the image data, and is suitable for recording, display, panorama synthesis, and the like. To convert the image data into For example, when recording image data of a captured image or a panoramic image, or when displaying recorded image data, image data compression processing or decompression processing based on a JPEG (Joint Photographic Experts Group) method or the like is performed. . In addition, a process of generating a single panoramic image by combining image data of a plurality of continuous captured images shot during the panoramic shooting mode is performed. The image data image-processed by the image processing unit 16 is output to the recording medium I / F 17 and recorded on the recording medium 19 or is output to the video encoder 20 and displayed on the display unit 22.

  The video encoder 20 sends the image data converted for display to the display driver 21. For example, in the shooting mode, an image captured from the image sensor 12 for each frame and subjected to image processing by the image processing unit 16 is switched and displayed on the display unit 22 in units of frames to display a live view image. On the other hand, in the reproduction mode, a captured image or a panoramic image read from the recording medium 19 and subjected to image processing by the image processing unit 16 is displayed on the display unit 22. Further, the video encoder 20 outputs image data for display to an external device connected to the video signal output terminal 23 as necessary. The display unit 22 displays various setting information of the digital camera 1 in addition to the captured image and live view image, and is realized by a display device such as an LCD (Liquid Crystal Display) or an EL display (Electroluminescence Display). Is done.

  The recording medium I / F 17 performs writing of image data converted for recording, reading of recorded image data, and the like with respect to the recording medium 19 that is detachably held by the recording medium holding unit 18. . The recording medium 19 is a memory card such as an xD-picture card (registered trademark) or a compact flash (registered trademark) card.

  The operation unit 24 accepts various operations by the user, such as a shooting timing instruction, a mode setting operation such as various shooting modes including a panoramic shooting mode and a playback mode, and a shooting condition setting operation, and notifies the controller 27 of an operation signal. This is realized by a button switch to which various functions are assigned. The operation unit 24 includes the shutter button 3, the power button 4, the menu button 5, the cross button 6, and the OK button 7 in FIG. 1.

  The ROM 26 stores in advance various camera programs for operating the digital camera 1 and realizing various functions of the digital camera 1, data used during the execution of the camera program, and the like. The RAM 25 is used as a working memory for the image processing unit 16 and the controller 27. For example, image data or the like from the AFE 13 is temporarily stored, for work when generating live view image data to be displayed on the display unit 22, for work when generating a panoramic image, a captured image, or a panoramic image. Is used for work when recording the image on the recording medium 19.

  The controller 27 reads out and executes a camera program from the ROM 26 in accordance with an operation signal from the operation unit 24 and performs operation control and memory control of each unit constituting the digital camera 1 to control the entire operation of the digital camera 1. To control. In addition, processing such as AF (automatic focus), AE (automatic exposure), and AWB (automatic white balance) is performed. The controller 27 includes a shooting start instruction unit 271 that instructs the start timing of shooting processing in the panoramic shooting mode.

  Next, a panoramic shooting mode that is one of the shooting modes of the digital camera 1 configured as described above will be described. FIG. 6 is a diagram for explaining an overview of the panoramic shooting mode. In the panorama shooting mode, a plurality of shooting processes are performed to generate image data of a plurality of images (hereinafter, a shot image shot for a panorama image is referred to as a “partial image”). For example, the user presses the shutter button 3 and captures the first partial image. Thereafter, the camera body 2 is panned in a predetermined panoramic shooting direction, and when the second partial image shooting position is reached, the second partial image is shot. Thereafter, the photographing of an arbitrary number of partial images is repeated to photograph the subject for each portion. The panning operation refers to an operation of moving the camera body 2 while rotating it along the panoramic shooting direction. FIG. 6 shows a case where the user pans the camera body 2 in the horizontal direction from the left side to the right side after pressing the shutter button 3 and, for example, three partial images I1, I3, and I5 are photographed. Here, the photographing process of each of the partial images I1, I3, and I5 is performed at the timing when the shutter button 3 is pressed for the first partial image I1. The second and subsequent partial images I3 and I5 are automatically performed during the panning operation of the camera body 2 by the user. Then, the partial images I1, I3, and I5 generated for each part are connected and combined so that the positional relationships between the adjacent partial images I1, I3, and I5 are matched, and one panoramic image I7 is combined. Is generated. Specifically, an overlapping portion reflected in the rightmost region A1 of the partial image I1 and the leftmost region A3 of the partial image I3 is searched, and the partial image I1 and the partial image I3 are searched based on the searched overlapping portion. Is synthesized. Similarly, the partial image I3 and the partial image I5 are synthesized based on the overlapping portion shown in the area A5 on the right side of the partial image I3 and the area A7 on the left side of the partial image I5.

  As described above, in order to generate a panoramic image, the predetermined range of each partial image needs to overlap. On the other hand, if the overlapping portion deviates from the predetermined range, it takes time to search for a synthesis location. The synthesis processing time increases. Further, if the overlapping portion is significantly deviated from the predetermined range, the synthesized portion cannot be detected, and as a result of the synthesis, a panoramic image having a sense of incongruity is formed at the joint. For this reason, it is necessary to perform the imaging processing of the second and subsequent partial images at a timing such that the overlapping portion falls within an appropriate range. In the present embodiment, a pointer as a movement mark that moves according to an image movement amount determined as a result of the motion detection process performed by the motion detection unit 15 after the first image capturing process, and a movement target position of this pointer are shown. The target as a moving target mark is displayed. Specifically, the next shooting position is determined so that the predetermined ranges of the partial images overlap. Then, a panning operation is performed in which the pointer and the movement target position of the pointer (target position) are set according to the next shooting position, and the pointer is moved onto the target by placing the pointer and the target on the live view image. Prompt.

  FIG. 7 is a diagram for explaining the operation of the digital camera 1 when performing panoramic shooting, and shows a transition example of the display screen displayed on the display unit 22 in the panoramic shooting mode. In the panoramic shooting mode, first, display of a live view image is started (FIG. 7A). And it will be in a standby state until the shutter button 3 is pressed down. The user presses the shutter button 3 while viewing the live view image, and instructs the photographing timing of the first partial image. When the photographing timing is instructed, the photographing process is started and a first partial image is generated.

  When the first image is taken, the pointer PM and the target TM are displayed on the live view image (FIG. 7B), and the panning operation to the target TM for taking the second partial image is performed. Prompt. Here, FIG. 7 shows an operation example when the panoramic shooting direction is “right”, and the pointer PM is arranged on the right side and the target TM is arranged on the left side in FIG. 7B. The panorama shooting direction is determined, for example, when the user selects either “right” or “left”. When the panorama shooting direction is “left”, the pointer is arranged at the initial position on the left side of the screen and the target is arranged on the right side of the screen. When the user pans the camera body 2 in the right direction and horizontally, the pointer PM moves to the left side where the target TM is arranged together with the live view image (FIG. 7C). Then, when the pointer PM moves within the frame of the target TM (FIG. 7D), the photographing process is automatically started.

  Next, the principle of the motion detection process will be described. In this motion detection process, motion detection by the motion detection unit 15, reliability evaluation by the reliability evaluation unit 151, image movement amount determination by the movement amount determination unit 153, and motion detection area by the motion detection area setting unit 155. And setting. Here, a case where the panorama shooting direction is “right” will be described as an example.

  First, motion detection performed by the motion detection unit 15 will be described. As shown in FIG. 4, the motion detection unit 15 detects a motion between the current frame image and the previous frame image (between the previous frame), and between the current frame image and the reference frame image (the reference frame image). Motion between frames) is detected. At this time, motion detection is performed on motion detection areas that are individually set for the field angle range (subject range). FIG. 8 is a diagram illustrating an example of a motion detection area. FIG. 9 is a diagram for explaining motion detection between the immediately preceding frames, and shows an example of an angle of view range (preceding frame image) of the immediately preceding frame. FIG. 10 is a diagram for explaining motion detection between reference frames, and shows an example of an angle of view range (reference frame image) of the reference frame.

  In the motion detection between the immediately preceding frames, as shown in FIG. 8, the motion detecting unit 15 performs the motion detection by setting the immediately preceding frame detection area Eb indicated by the alternate long and short dash line in almost the entire field angle range Ev of the immediately preceding frame. . First, as shown in FIG. 9, the motion detection unit 15 arranges a plurality of representative points B11 in the immediately preceding frame detection area Eb in the immediately preceding frame image. The positions and the number of representative points to be arranged are set in advance, for example. In addition, a search range for each representative point arranged in the immediately preceding frame image is set in the current frame image. Then, pattern matching is performed between each representative point of the immediately preceding frame and the corresponding search range, and the most matching position (high correlation) is searched from each search range in the current frame image. Then, based on the motion vector of each representative point obtained by pattern matching, a representative motion vector indicating the motion of the inter-frame detection area Eb is obtained.

  On the other hand, in motion detection between the reference frames, the motion detection unit 15 includes a reference inter-frame detection area Ed that is narrower than the previous inter-frame detection area Eb in the reference frame image, as indicated by a two-dot chain line in FIG. Set to to perform motion detection. This reference inter-frame detection area Ed is set to the right end position of the angle of view range Ev when the panoramic shooting direction is “right”. When the panorama shooting direction is “left”, the left end position of the field angle range Ev is set. First, as shown in FIG. 10, the motion detection unit 15 sets a reference inter-frame detection area Ed at a predetermined position (right end position) in the reference frame image. Then, the motion detection unit 15 arranges a plurality of representative points B21 in the reference inter-frame detection area Ed. The positions and the number of representative points to be arranged are set in advance, for example. In addition, a search range for each representative point arranged in the reference frame image is set in the current frame image. Although details will be described later, in the present embodiment, the position of the search range set in the current frame image moves with a predetermined position (right end position) in the current frame image as an initial position. Yes. Then, similarly to the motion detection between the immediately preceding frames, pattern matching is performed between each representative point of the reference frame and the search range corresponding thereto, and the most matching position is searched from each search range in the current frame. Then, based on the motion vector of each representative point obtained by pattern matching, a representative motion vector indicating the motion in the reference inter-frame detection area Ed is obtained.

  Next, the reliability evaluation performed by the reliability evaluation unit 151 will be described. The reliability evaluation unit 151 calculates the reliability of the immediately preceding inter-frame detection result and the reference inter-frame detection result, and determines the presence or absence of reliability. For example, when the number of matching failures is large, the direction of each motion vector with respect to each obtained representative point is not uniform, or the image of each frame to be motion-detected is a repeated image such as a striped pattern or a lattice pattern In the case of a case or an image with a low contrast such as a plain wall, it is considered that the reliability of the detection result is low. Therefore, the number of successful matching with respect to the number of representative points is calculated as the first evaluation parameter. Further, the motion vectors that have been successfully matched are grouped in each direction to select the maximum value of the number of elements, and the maximum value of the number of elements with respect to the number of successful matching is calculated as a second evaluation parameter. In addition, when each frame to be motion-detected is a repeated image, a plurality of motion vectors with high correlation may be searched during pattern matching. However, the number of motion vector positions searched for increases as the number of motion vector positions increases. The value of the evaluation parameter 3 is calculated. Further, a histogram of correlation calculation results obtained by pattern matching for the current frame image is created, and the fourth evaluation parameter is calculated so that the value becomes smaller as the distribution width is narrower. Specifically, the reliability evaluation unit 151 calculates each evaluation parameter for each detection result of the immediately preceding inter-frame detection result and the reference inter-frame detection result, and, for example, each of the total values is used for the reliability of the corresponding detection result. The degree value. And the reliability evaluation part 151 determines the presence or absence of the reliability of each detection result by whether the obtained reliability exceeds the predetermined threshold value set beforehand.

  Next, calculation of the image movement amount performed by the movement amount determination unit 153 and setting of the motion detection area by the motion detection area setting unit 155 will be described. The movement amount determination unit 153 determines the amount of image movement from the reference frame from which the motion detection is started to the current frame. At this time, if the detection result between the reference frames is reliable, the detection result between the reference frames. Is used to determine the amount of image movement. When the reference inter-frame detection result is not reliable and the previous inter-frame detection result is reliable, the image movement amount is determined using the previous inter-frame detection result. On the other hand, after capturing the first partial image, the motion detection area setting unit 155 sets the immediately preceding frame detection area in the immediately preceding frame image and arranges the representative points as described with reference to FIG. . In addition, the motion detection area setting unit 155 sets the current frame as a reference frame, and initializes the reference movement amount to “0”. Then, as described with reference to FIG. 10, the motion detection area setting unit 155 sets the reference inter-frame detection area in the reference frame image and arranges the representative points, and also includes the representative points in the current frame image for each representative point. Set the search range to the initial position (right end position). If the detection result between the reference frames of the motion detection unit 15 is reliable, the set position of the search range in the current frame image is moved according to the image movement amount determined by the movement amount determination unit 153. If the detection result between the reference frames is not reliable, the reference frame is changed to the current frame, and the representative point for detection between the reference frames is set to the initial position (right end position). Further, along with the change of the reference frame, the search range set in the current frame image at the next motion detection is reset to the initial position (right end position). At this time, the reference movement amount is updated to the current image movement amount value.

  FIG. 11 and FIG. 12 are explanatory diagrams for explaining the principle of motion detection according to the present embodiment, and show the displacement of the field angle range Ev accompanying the panning operation of the camera body 2 with a landscape as a subject. 11 and FIG. 12, the inter-frame motion detection target area Es1 and the reference inter-frame motion detection target area Es2 are shown together with the angle of view range Ev. Here, the immediately preceding inter-frame motion detection target area Es1 indicates an area in the current frame image in which a search range for motion detection between the immediately preceding frames is set. Motion detection is performed on almost the entire area of the image. In the actual processing, as described with reference to FIG. 9, the search range for each representative point arranged in the immediately preceding frame detection area in the immediately preceding frame image is the motion detection target area between the immediately preceding frames shown in FIG. Each is set in Es1. On the other hand, the reference frame inter-frame motion detection target area Es2 indicates an area in the current frame image in which a search range for motion detection between the reference frames is set. Motion detection is performed on a part of the area. In the actual processing, as described with reference to FIG. 10, the search range for each representative point arranged in the reference interframe detection area in the reference frame image is the reference interframe motion detection target area shown in FIG. It will be set at each position in Es2. When it is determined that the reference inter-frame detection result is reliable, the reference inter-frame motion detection target area Es2 in which the search range is set is displaced according to the reference inter-frame detection result.

  More specifically, FIG. 11 illustrates the displacement of the reference inter-frame motion detection target area Es2 when the detection result between the reference frames is continuously determined to be reliable. An angle of view range Ev shown in FIG. 11A indicates the angle of view range of the current frame after the first partial image is captured, and the angle of view range is the angle of view shown in FIG. The reference inter-frame motion detection target area Es2 in the range Ev is located at the right end of the field angle range Ev. With the subsequent panning operation to the right, which is the panoramic shooting direction, while the reliability evaluation unit 151 determines that the reference interframe detection result is reliable, the reference interframe motion detection target area Es2 is 11 (b) and 11 (c), it is displaced toward the left end side according to the image movement amount. For example, the reliability evaluation unit 151 determines that there is reliability for the detection result between the reference frames performed by the motion detection unit 15 when the field angle range of the current frame is displaced to the field angle range Ev illustrated in FIG. Suppose that In this case, the movement amount determination unit 153 determines the image movement amount from the obtained reference interframe detection result. On the other hand, the motion detection area setting unit 155 moves the set position of the search range in the current frame image at the time of motion detection between the next reference frames based on the image movement amount determined by the movement amount determination unit 153. .

  On the other hand, FIG. 12 shows the displacement of the reference inter-frame motion detection target region Es2 when it is determined that the reference inter-frame detection result is not reliable. For example, the reliability evaluation unit 151 determines that the reliability is not reliable with respect to the detection result between the reference frames performed by the motion detection unit 15 when the field angle range of the current frame is displaced to the field angle range Ev illustrated in FIG. Suppose that In this case, the reference inter-frame motion detection target area Es2 is located at the right end of the field angle range Ev, which is the initial position, as shown in FIG. Specifically, the movement amount determination unit 153 determines the image movement amount from the previous inter-frame detection result performed between the previous frame and the previous value of the image movement amount. On the other hand, the motion detection area setting unit 155 resets the search range in the current frame image to the right end position, which is the initial position, in the next motion detection between reference frames. At this time, the motion detection area setting unit 155 sets the reference movement amount as the image movement amount determined by the movement amount determination unit 153.

Next, the operation of the digital camera 1 will be described. FIG. 13 is a state transition diagram of the digital camera 1. As shown in FIG. 13, the digital camera 1, when the power supply button 4 is pressed is ON (a1), a transition from the camera power OFF state C ₁ to the still image shooting standby state C _3, and shooting mode Become. In the shooting mode, the image data of the subject image formed on the image sensor 12 is stored in the RAM 25, a series of processes for switching the image data in units of frames and displaying them on the display unit 22 are repeated, and the live view image is displayed. Display continuously. Further, when the power supply button 4 is pressed it is OFF (a3), the digital camera 1 makes a transition to the camera power OFF state C _1.

When the panoramic mode is selected (a5), the digital camera 1 makes a transition from the still image shooting standby state C ₃ to photographing state C _5. Here, when the first partial image is shot, the shooting process is started when the shutter button 3 is pressed. When completing the photographing (a7), the digital camera 1 makes a transition from the photographing state C ₅ to photographing direction detection state C _7. Here, the panning operation of the camera body 2 is detected by detecting the motion of the live view image, and the panorama shooting direction is determined. When the panoramic photographing direction is determined (a9), the digital camera 1 makes a transition to the displacement detection state C ₉ from the photographing direction detection state C _7. Here, first, a pointer and a target are displayed on the live view image. Then, the motion detection between the immediately preceding frames and the motion detection between the reference frames are performed, the amount of image movement from the reference frame to the current frame is determined, and the pointer is moved. Pointer has completed the movement of the pointer overlaps the target (a11), the digital camera 1, the transition to start the shooting process and from the movement amount detection state C ₉ to photographing state C _5. When completing the shooting (a13), the digital camera 1 transitions to the moving amount detecting state C ₉ from the photographing state C _5, it moves to shooting the next partial image. When a predetermined number of partial images have been photographed in this way, a panoramic image is generated by combining the generated partial images, and panoramic photographing is finished (a15). At this time, the digital camera 1 transitions from the shooting state C ₅ to the still image shooting standby state C ₃ .

  FIG. 14 is a flowchart showing the operation of the digital camera 1 in the shooting mode. In the shooting mode, first, the controller 27 determines the type of shooting mode. When a shooting mode other than the panoramic shooting mode such as the normal shooting mode is set (step b1: No), the process proceeds to a shooting operation according to the shooting mode (step b3), and the process is ended after the end.

  On the other hand, if the panoramic shooting mode is set (step b1: Yes), first, the first partial image is shot. That is, the shooting start instruction unit 271 instructs the start of shooting processing with the timing when the shutter button 3 is pressed as the shooting timing. As a result, the photographing process is started, and the operation of each part is controlled to photograph the still image in the photographing range at this time as the first partial image (step b5). When this shooting process is started, the display of the live view image is temporarily stopped. The display of the live view image is restored after the image data transfer processing and image processing are completed after exposure.

  Subsequently, the controller 27 performs control to display a shooting direction guidance OSD (On-Screen Display) that prompts an instruction of the panoramic shooting direction on the live view image, and moves to the panoramic shooting direction of the camera body 2 by the shooting direction guidance OSD. The panning operation is urged, and for example, an instruction of either “left” or “right” panorama shooting direction is accepted (step b7). Subsequently, in the motion detection unit 15, the motion detection area setting unit 155 performs a motion detection initialization process (step b9). At this time, the motion detection area setting unit 155 sets a previous inter-frame detection area, for example, in the entire area of the previous frame image, arranges a representative point in the previous inter-frame detection area, and presents a current point for each representative point. The position of the search range in the frame image is fixedly set. In addition, the motion detection area setting unit 155 performs initial setting of the reference frame. Specifically, the current frame is set as the reference frame, and the reference movement amount is initialized to “0”. Then, the reference inter-frame detection area is set to a predetermined position in the reference frame image (the right end position of the image if the panoramic shooting direction is the right direction, and the left end position of the image if the panoramic shooting direction is the left direction). A representative point is arranged at the right end in the middle, and the position of the search range in the current frame image with respect to each representative point is set with the right end in the image as an initial position. And the process of step b11-step b15 is repeatedly performed for every frame, and the panning operation of the camera main body 2 is detected. That is, the motion detection unit 15 performs a motion detection process and determines the amount of image movement (step b11). Then, the controller 27 performs a shooting direction determination process based on the image movement amount (step b13). At this time, if the user rotates the camera body 2 toward the left side in the horizontal direction according to the display of the shooting direction guidance OSD, the panoramic shooting direction is determined as the left direction. Similarly, when the user rotates the camera body 2 toward the right side in the horizontal direction, the panorama shooting direction is determined as the right direction. When the camera body 2 is not moved and the panoramic shooting direction is not determined (step b15: No), the process returns to step b11.

  If the panorama shooting direction is determined (step b15: Yes), the controller 27 then performs control to display the pointer and the target on the live view image (step b17). Specifically, the controller 27 sets the pointer and the movement target position of the pointer based on the image movement amount calculated in step b11, the panoramic shooting direction determined in step b13, and the pointer and target are set on the live view image. To place.

  Thereafter, the processing of step b17 to step b23 is repeated for each frame until the pointer moves to the target position. That is, the motion detection unit 15 performs a motion detection process and determines an image movement amount (step b19). This motion detection process is the same as the motion detection in step b11. Subsequently, the controller 27 moves the display position of the pointer according to the image movement amount determined by the motion detection process, and updates the display (step b21). Then, the controller 27 determines whether or not the display position of the pointer is within the target frame. If the position is not within the target frame (step b23: No), the controller 27 returns to step b19.

  When the pointer moves into the target frame (step b23: Yes), the shooting start instruction unit 271 instructs the start of the shooting process. As a result, the photographing process is started, and a still image in the photographing range at this time is photographed as a partial image (step b25).

  Subsequently, the controller 27 determines whether or not a predetermined number of images have been taken. For example, if a panoramic image is generated by shooting three partial images, it is determined whether or not three partial images have been shot. The number of partial images may be fixed, or may be configured so that the user can set as appropriate. If the predetermined number of images has not been taken (step b27: No), the process returns to step b15.

  When a predetermined number of images have been shot (step b27: Yes), the image processing unit 16 joins and combines the shot number of partial images according to the panorama shooting direction to generate a panoramic image (step b29). . And the controller 27 performs control which displays the produced | generated panoramic image on the display part 22 (step b31), and complete | finishes a process. At this time, the generated panoramic image is recorded in the recording medium 19.

  Next, the shooting direction determination process in step b13 and the motion detection process in steps b11 and b19 will be sequentially described.

  FIG. 15 is a flowchart illustrating a detailed processing procedure of the shooting direction determination process. In this photographing direction determination process, the panning operation of the camera body 2 is detected by comparing the absolute value of the horizontal component of the image movement amount calculated in step b9 in FIG. 14 with a predetermined direction threshold value set in advance. If the absolute value of the image movement amount (horizontal component) is equal to or smaller than the direction threshold value (step c1: No), the controller 27 determines that the shooting direction has not been determined (step c3). Then, the process returns to step b11 in FIG. On the other hand, when the absolute value of the image movement amount (horizontal component) is larger than the direction threshold (step c1: Yes), the controller 27 determines that the image movement amount (horizontal component) is larger than 0 (step c5: Yes). The panorama shooting direction is determined as “right” (step c7). Then, the process returns to step b11 in FIG. If the image movement amount (horizontal component) is less than 0 (step c5: No), the controller 27 determines the panoramic shooting direction as “left” (step c9). Then, the process returns to step b11 in FIG.

  FIG. 16 is a flowchart showing a detailed processing procedure of the motion detection process. In this motion detection process, the motion detection unit 15 performs motion detection between the current frame image and the previous frame image (step d1), and temporarily stores the previous frame detection result in the RAM of the motion detection unit 15. At the same time (step d3), motion detection is performed between the current frame image and the reference frame image (step d5), and the detection result between the reference frames is temporarily stored in the RAM of the motion detector 15 (step d7). Subsequently, the reliability evaluation unit 151 calculates and evaluates the reliability of the immediately preceding inter-frame detection result and the reference inter-frame detection result (step d8). If the reliability evaluation unit 151 determines that the detection result between the reference frames is reliable as a result of the evaluation here (step d9: Yes), the movement amount determination unit 153 then determines the reference movement amount at the present time. And the current reference frame detection result are determined as the amount of image movement (step d11). Then, the motion detection area setting unit 155 updates the set position of the search range in the current frame image with respect to the representative point arranged in the detection area between the reference frames of the reference frame according to the image movement amount determined in step d11. (Step d13), the process returns to step b19 in FIG.

  On the other hand, as a result of the evaluation at step d8, when the reliability evaluation unit 151 determines that the reference interframe detection result is not reliable (step d9: No), the reliability evaluation result for the reference interframe detection result Determine. When the reliability evaluation unit 151 determines that the detection result between the previous frames is reliable (step d15: Yes), the movement amount determination unit 153 subsequently detects the previous value of the image movement amount and the current previous frame detection. The total value with the result is determined as the amount of image movement (step d17). After the determination, the process proceeds to step d23. On the other hand, if the reliability evaluation unit 151 determines that the previous inter-frame detection result is not reliable (step d15: No), the movement amount determination unit 153 performs a motion amount estimation process to obtain the current frame image and The amount of motion between the immediately preceding frame image is estimated (step d19). For example, the detection result between immediately preceding frames for the most recent predetermined number of frames is held in the RAM of the motion detection unit 15. Then, based on the predetermined number of detection results between the previous frames, for example, an average value of these is obtained to obtain an estimated motion amount between the current frame image and the previous frame image. Then, the movement amount determination unit 153 determines the total value of the previous value of the image movement amount and the estimated motion amount as the image movement amount (step d21). After the determination, the process proceeds to step d23.

  In step d23, the motion detection area setting unit 155 resets the current frame as a reference frame, and subsequently sets a search range setting position in the current frame image with respect to a representative point arranged in the reference frame detection area of the reference frame. Is set as the initial position (step d25), and the reference movement amount is updated to the current image movement amount value (step d27). Thereafter, the process returns to step b19 in FIG.

  As described above, according to the present embodiment, the motion detection between the previous frames and the motion detection between the reference frames are performed, and when the detection result between the reference frames is reliable, the detection between the reference frames is performed. The result can be used to determine the amount of image movement from the reference frame to the current frame. On the other hand, when the reference inter-frame detection result is not reliable, if the previous inter-frame detection result is reliable, the image movement amount can be determined using the previous inter-frame detection result. If both the reference inter-frame detection result and the previous inter-frame detection result are not reliable, the image movement amount can be determined by estimating the motion between the previous frames. In other words, while the detection result between the reference frames is reliable, the amount of image movement is determined by motion detection between the current frame image and the reference frame image, so that detection errors caused by motion detection may be integrated. Absent. If it is determined that the detection result between the reference frames is not reliable, if the detection result between the previous frames is reliable, the image movement amount is calculated from the detection result between the immediately preceding frames and the previous value of the image movement amount. Can be determined. Therefore, since the motion detection error between images can be reduced, there is an effect that the moving amount of the image can be determined with high accuracy. According to this, when synthesizing panoramic images, it is possible to appropriately connect the partial images and generate a natural panoramic image without a sense of incongruity.

  The preferred embodiment of the present invention has been described above, but the present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the invention.

  For example, in the above-described embodiment, the case where the image movement amount is determined using the reference interframe detection result when the reference interframe detection result is determined to be reliable has been described. However, the present invention is not limited to this. It is not something. FIG. 17 is a flowchart illustrating a procedure of motion detection processing according to the modification. In FIG. 17, the same steps as those in FIG. 16 are denoted by the same reference numerals. Moreover, in the following description, the same code | symbol is attached | subjected about the structure similar to above-described embodiment.

In this modification, as shown in FIG. 17, when the reliability evaluation unit 151 determines that the reference interframe detection result is reliable (step d9: Yes), the evaluation result of the immediately previous interframe detection result is subsequently displayed. judge. If there is reliability (step e29: Yes), the movement amount determination unit 153 performs a weighted movement amount calculation process (step e31). For example, the movement amount determination unit 153 calculates the weighted movement amount Mw according to the following equation (1). Here, Mb is the detection result between the previous frames, Md is the detection result between the current reference frames, Me is the detection result between the previous reference frames, Cb is the reliability of the detection result between the previous frames, and Cd is the current detection result between the frames. The reliability of the detection result between the reference frames is shown.
Mw = Mb × Cb / (Cb + Cd) + (Md−Me) × Cd / (Cb + Cd) (1)

  Then, the movement amount determination unit 153 determines the total value of the reference movement amount and the weighted movement amount Mw calculated in step e31 as the image movement amount (step e33). Thereafter, the process proceeds to step d13.

  On the other hand, if the previous inter-frame detection result is not reliable (step e29: No), the process proceeds to step e35, and the movement amount determination unit 153 determines the current reference movement amount and the current reference inter-frame detection result. Is determined as the amount of image movement. Thereafter, the process proceeds to step d13. That is, the motion detection area setting unit 155 updates the set position of the search range in the current frame image with respect to the representative points arranged in the reference interframe detection area of the reference frame.

  In the embodiment described above, the case where either “right” or “left” is set as the panorama shooting direction has been described, but directions other than the left and right can also be set as the panorama shooting direction. For example, when performing panoramic shooting with “upper” or “lower” as the panoramic shooting direction, the absolute value of the vertical component of the image movement amount is set in advance in the shooting direction determination process shown in FIG. It is configured to compare with the direction threshold value, and the positive / negative of the vertical component of the image movement amount is determined, and if it is a positive value, the panorama shooting direction is determined as “up”, and if it is a negative value, “down”.

  In the above embodiment, when the panoramic shooting direction is “right”, the reference inter-frame detection area is set with the right end position of the angle of view range as the initial position, and when the panoramic shooting direction is “left”. Although it has been described that the reference inter-frame detection area is set with the left end position of the view angle range as the initial position, for example, the center of the view angle range may be set as the initial position. Further, when performing panoramic shooting with “upper” or “lower” as the panoramic shooting direction, the detection between the reference frames may be set with the upper end position and the lower end position of the field angle range as the initial positions.

It is a rear view of a digital camera. It is a schematic block diagram which shows the structural example of a digital camera. It is a schematic diagram which shows the internal structure of a frame memory. It is explanatory drawing explaining the outline | summary of the motion detection which a motion detection part performs. It is a schematic diagram which shows the function structure of a motion detection part. It is explanatory drawing explaining the outline | summary of a panoramic imaging mode. It is a figure which shows the example of a transition of the display screen displayed on a display part in panorama imaging | photography mode. It is a figure which shows an example of a motion detection area. It is a figure for demonstrating the motion detection between immediately preceding frames. It is a figure for demonstrating the motion detection between reference | standard frames. It is explanatory drawing for demonstrating the principle of a motion detection. It is explanatory drawing for demonstrating the principle of a motion detection. It is a state transition diagram of a digital camera. It is a flowchart which shows operation | movement of the digital camera in imaging | photography mode. It is a flowchart which shows the detailed process sequence of an imaging | photography direction determination process. It is a flowchart which shows the detailed process sequence of a motion detection process. It is a flowchart which shows the process sequence of the motion detection process in a modification.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Digital camera 2 Camera body 11 Imaging optical system 12 Imaging element 13 AFE
14 frame memory 141 current frame data 143 previous frame data 145 reference frame data 15 motion detection unit 151 reliability evaluation unit 153 movement amount determination unit 155 motion detection area setting unit 16 image processing unit 17 recording medium I / F
DESCRIPTION OF SYMBOLS 18 Recording medium holding part 19 Recording medium 20 Video encoder 21 Display driver 22 Display part 23 Video signal output terminal 24 Operation part 3 Shutter button 4 Power button 25 RAM
26 ROM
27 Controller 271 Shooting start instruction section

Claims (11)

An imaging apparatus that includes an imaging unit that captures an image of a subject and generates image data, and that performs a plurality of imaging processes to generate a plurality of captured images,
A display unit that switches and displays the image data captured by the imaging unit in predetermined frame units;
An inter-adjacent frame motion detector that detects motion between the current frame image and the adjacent frame image adjacent to the current frame;
A reference inter-frame motion detector that detects a motion between the current frame image and a predetermined reference frame image;
Based on the detection result of the inter-frame motion detection unit and the detection result of the reference inter-frame motion detection unit, a reliability evaluation unit that evaluates the reliability of each detection result;
Based on the evaluation result of the reliability evaluation unit, the amount of image movement from the reference frame to the current frame using the detection result of the motion detection unit between adjacent frames and / or the detection result of the motion detection unit between reference frames A movement amount determination unit for determining
A shooting start instruction unit for instructing start of the shooting process based on an image movement amount determined by the movement amount determination unit;
An imaging apparatus comprising: A display control unit that performs control to display on the current frame image a movement mark that moves according to the image movement amount determined by the movement amount determination unit and a movement target mark that indicates a movement target position of the movement mark; ,
The imaging apparatus according to claim 1, wherein the imaging start instruction unit instructs the start of the imaging process when the movement mark moves to the movement target position indicated by the movement target mark.   The imaging apparatus according to claim 1, wherein the reliability evaluation unit determines whether or not each detection result is reliable based on a reliability evaluation result for each detection result. .   The movement amount determination unit uses at least the detection result of the reference inter-frame motion detection unit when both of the detection results are determined to be reliable as a result of the evaluation by the reliability evaluation unit. The imaging apparatus according to claim 3, wherein a movement amount is determined.   The movement amount determination unit determines from the reference frame determined by the detection result of the inter-adjacent frame motion detection unit when both the detection results are determined to be reliable as a result of the evaluation by the reliability evaluation unit. An average value of an image movement amount up to the current frame and an image movement amount from the reference frame to the current frame determined by a detection result of the reference interframe motion detection unit is determined as the image movement amount. Item 4. The imaging device according to Item 3.   The movement amount determination unit determines the detection result of the inter-adjacent frame motion detection unit and the reference inter-frame motion when both of the detection results are determined to be reliable as a result of the evaluation by the reliability evaluation unit. The imaging apparatus according to claim 3, wherein the image movement amount is determined by weighting a corresponding reliability with respect to a detection result of the detection unit.   The movement amount determination unit is the other detection result determined to be reliable when one of the detection results is determined to be unreliable as a result of the evaluation by the reliability evaluation unit. The imaging apparatus according to any one of claims 3 to 6, wherein the image movement amount is determined using an image.   The movement amount determination unit uses the detection result of the reference inter-frame motion detection unit when the detection result of the reference inter-frame motion detection unit is determined to be reliable as a result of the evaluation by the reliability evaluation unit. The image movement amount is determined, and when it is determined that the detection result of the reference interframe motion detection unit is unreliable, the image movement amount is determined using the detection result of the adjacent interframe motion detection unit. The imaging apparatus according to claim 3.   The movement amount determination unit, as a result of the evaluation by the reliability evaluation unit, when both of the detection results are determined to be unreliable, the current frame image and the adjacent frame image adjacent to the current frame, The imaging apparatus according to claim 3, wherein the movement amount of the image is estimated to determine the amount of movement of the image.   As a result of the evaluation by the reliability evaluation unit, when it is determined that the detection result of the motion detection unit between the reference frames is not reliable, a reference frame changing unit is provided that changes the reference frame to a current frame. The imaging device according to any one of claims 3 to 9. An imaging method in an imaging apparatus that includes an imaging unit that images a subject and generates image data, and that performs a plurality of imaging processes to generate a plurality of captured images,
A display step of switching and displaying the image data captured by the imaging unit in predetermined frame units;
A motion detection step between adjacent frames for detecting a motion between an image of the current frame and an image of an adjacent frame adjacent to the current frame;
A reference inter-frame motion detection step for detecting a motion between the current frame image and a predetermined reference frame image;
A reliability evaluation step for evaluating the reliability of each detection result based on the detection result in the motion detection step between adjacent frames and the detection result in the motion detection step between the reference frames;
Based on the evaluation result in the reliability evaluation step, the detection result in the inter-adjacent frame motion detection step and / or the detection result in the reference inter-frame motion detection step is used to determine from the reference frame to the current frame. A moving amount determining step for determining an image moving amount;
A shooting start instruction step for instructing the start of the shooting process based on the image movement amount determined in the movement amount determination step;
An imaging method for an imaging apparatus, comprising:

Images