JP2009044329A - Program, image processing method, and image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate an interpolated image with less sense of incongruity when an imaging range moves between two images. <P>SOLUTION: In a first step, data about a first image and data about a second image picked up after the first image at a state where the imaging range is moved so as to be partially overlapped with an imaging range of the first image are read in an operation part. In a second step, the operation part specifies an overlapped part and a non-overlapped part of the imaging ranges in the first and the second images, respectively. In a third step, the operation part generates an image in a first area corresponding to the overlapped part of the interpolated image for interpolating variation of an object between the first and the second images on the basis of variation of the image with the overlapped part between the first and the second images. In a fourth step, the operation part generates an image in a second area located at a moving destination of the imaging range to the first area of the interpolated image by segmenting the non-overlapped part of the second image. In a fifth step, the operation part generates the interpolated image by connecting the image in the second area to the image in the first area. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing technique for generating an interpolated image for interpolating the motion of a subject from two images captured discretely.

2. Description of the Related Art Conventionally, a technique for generating an interpolated image that interpolates the movement of a subject between two images serving as key frames is known in order to smoothly reproduce a plurality of discretely captured images as a moving image. For example, Patent Document 1 discloses an example of an image processing technique for generating the above-described interpolation image.
JP 2003-150972 A

  However, with the conventional technology, when the imaging range moves between two images that become key frames, or when there is a large change in the direction of movement of the subject between the two images that become key frames, the image is viewed as a movie. There is room for improvement in that an interpolated image with a great sense of discomfort can sometimes be generated.

  The present invention is to solve the above-mentioned problems of the prior art. An object of the present invention is to provide a means for generating an interpolated image with less discomfort even when the imaging range moves between two images.

  Another object of the present invention is to provide means for generating an interpolated image with less discomfort even when there is a large change in the direction of movement of the subject between two images.

  A first aspect of the present invention is a computer program having a data reading unit for reading image data and a calculation unit. This program causes the calculation unit to execute the following first to fourth steps. In the first step, the first image data captured by the imaging device is captured after the first image in a state where the imaging range of the imaging device is moved so as to partially overlap the imaging range of the first image. The data of the second image is read into the calculation unit by the data reading unit. In the second step, the calculation unit specifies an overlapping portion and a non-overlapping portion of the imaging range in the first image and the second image, respectively. In the third step, the arithmetic unit interpolates the change of the subject between the first image and the second image, and the first region image corresponding to the overlapping portion is replaced with the overlapping portion of the first image. It generates based on the amount of change of the image with the overlapping part of two images. In the fourth step, the calculation unit generates an image of the second area located at the movement destination of the imaging range with respect to the first area in the interpolated image by cutting out a non-overlapping portion of the second image. In the fifth step, the calculation unit generates an interpolation image by connecting the image of the second area to the image of the first area.

  In the first embodiment, the calculation unit may further execute the following sixth step. In the sixth step, the calculation unit generates an image of the third area located at the movement source of the imaging range with respect to the first area in the interpolated image by cutting out a non-overlapping portion of the first image. In the fifth step, the calculation unit may generate an interpolation image by further connecting the image of the third region to the image of the first region.

  In the first embodiment, the calculation unit may further execute the following seventh step. In the seventh step, the arithmetic unit is included in at least one of the second region and the third region when there is a common subject straddling the overlapping portion and the non-overlapping portion in at least one of the first image and the second image. The common subject is deformed based on the change amount of the common subject obtained from the overlapping portions of the first image and the second image.

  In the first embodiment, the arithmetic unit may further execute an eighth step of continuously reproducing the first image, the interpolation image, and the second image.

  The second aspect of the present invention is a computer program having a data reading unit for reading image data and a calculation unit. This program causes the calculation unit to execute the following first to fifth steps. In the first step, a plurality of frames of image data groups obtained by continuously capturing the subject of interest are read into the calculation unit by the data reading unit. In the second step, the calculation unit obtains the motion vector of the subject of interest for each frame section in the plurality of frames of the image data group, using two frames at which the imaging time points are adjacent. In the third step, the calculation unit extracts a first frame section and a second frame section, each of which is temporally continuous, and the motion of the subsequent second frame section with respect to the motion vector of the previous first frame section. It is determined whether the change in vector direction is less than a threshold value. In the fourth step, when it is determined that the change is less than the threshold value in the third step, the calculation unit interpolates the change of the subject of interest in the second frame section by interpolation using the motion vector of the second frame section. Generate an interpolated image. In the fifth step, when it is determined that the change is equal to or greater than the threshold value in the third step, the calculation unit performs an extrapolation using a motion vector of at least one frame section before and after the second frame section, thereby performing the second frame section. To generate an interpolated image for interpolating the change of the subject of interest.

  In the second embodiment, the calculation unit may further execute a sixth step of inserting an interpolated image between two frames in the second frame section and continuously reproducing the image.

  In addition, what is expressed by converting the configuration related to the first and second embodiments into an image processing method, an image processing apparatus, a storage medium storing the above program, and the like is also effective as a specific aspect of the present invention. It is.

  In one embodiment of the present invention, when an interpolated image is generated from the first image and the second image captured by moving the imaging range, the overlapping portion of the interpolated image is based on the amount of change in the first image and the second image. On the other hand, one non-overlapping portion of the interpolated image is generated by cutting out the non-overlapping portion of the second image. Therefore, when the imaging range moves between two images, it is possible to generate an interpolated image with less discomfort.

  In another embodiment of the present invention, when the change in the motion vector obtained for each frame section is equal to or greater than the threshold value, the interpolated image of the predetermined frame section is represented as the motion vector of the other frame section before and after the frame section. Generate by extrapolation used. Therefore, when there is a large change in the movement direction of the subject between the two images, it is possible to generate an interpolated image with less discomfort.

<Description of First Embodiment>
FIG. 1 is a block diagram illustrating a configuration of an electronic camera to which the image processing apparatus according to the first embodiment is applied. The electronic camera includes an imaging optical system 11, a lens driving unit 12, an imaging element 13, an imaging element driving circuit 14, a signal processing circuit 15, a data processing circuit 16, a first memory 17, and a display control circuit 18. The monitor 19, the compression / expansion circuit 20, the recording I / F 21, the communication I / F 22, the operation member 23, the release button 24, the vibration sensor 25, the second memory 26, the control circuit 27, and the bus. 28.

  Here, the data processing circuit 16, the first memory 17, the compression / expansion circuit 20, the second memory 26 and the control circuit 27 are connected to each other via a bus 28. The lens driving unit 12, the image sensor driving circuit 14, the signal processing circuit 15, the display control circuit 18, the recording I / F 21, the communication I / F 22, the operation member 23, the release button 24, and the vibration sensor 25 are each a control circuit 27. (In FIG. 1, signal lines connecting the signal processing circuit 15, the display control circuit 18, and the control circuit 27 are omitted for simplicity).

  The imaging optical system 11 includes a plurality of lens groups including a zoom lens and a focusing lens, and serves to form a subject image on the imaging surface of the imaging element 13. For simplicity, the imaging optical system 11 is illustrated as a single lens in FIG.

  Each lens position of the imaging optical system 11 is adjusted in the optical axis direction by the lens driving unit 12. The lens driving unit 12 includes a lens driving mechanism and adjusts the lens position in accordance with a lens driving command from the control circuit 27. For example, the focus adjustment of the imaging optical system 11 is performed by the lens driving unit 12 driving the focus lens back and forth in the optical axis direction. Further, zoom adjustment of the imaging optical system 11 is performed by the lens driving unit 12 driving the zoom lens to advance and retreat in the optical axis direction.

  The image sensor 13 photoelectrically converts a subject image by a light beam that has passed through the imaging optical system 11 to generate an analog image signal. The image sensor 13 in the present embodiment is capable of single-shot imaging of still images, continuous shooting imaging of still images, and imaging of moving images. The output of the image sensor 13 is connected to the signal processing circuit 15. The image sensor 13 is configured by a CCD image sensor or a CMOS image sensor.

  The image sensor drive circuit 14 generates a drive signal at a predetermined timing in accordance with a command from the control circuit 27 and supplies the drive signal to the image sensor 13. Then, the image sensor driving circuit 14 controls charge accumulation (imaging) and reading of the accumulated charges of the image sensor 13 by the drive signal.

  The signal processing circuit 15 is an ASIC that performs various types of signal processing on the output of the image sensor 13. Specifically, the signal processing circuit 15 performs correlated double sampling, gain adjustment, DC regeneration, A / D conversion, and the like. Parameters such as gain adjustment in the signal processing circuit 15 are determined in accordance with a command from the control circuit 27. The signal processing circuit 15 is connected to the data processing circuit 16, and the data after the signal processing is output to the data processing circuit 16.

  The data processing circuit 16 is a circuit that performs digital signal processing on the image data output from the signal processing circuit 15. In the data processing circuit 16, for example, image processing such as color interpolation processing, gradation conversion processing, contour enhancement processing, and white balance adjustment is executed. The data processing circuit 16 is connected to the display control circuit 18 and the compression / decompression circuit 20, respectively. Then, the data processing circuit 16 outputs the recorded image data after the image processing to the compression / decompression circuit 20 in accordance with a command from the control circuit 27.

  Further, the data processing circuit 16 executes image resolution conversion (pixel number conversion) processing in response to a command from the control circuit 27. As an example, when displaying a reproduced image on the monitor 19, the data processing circuit 16 executes resolution conversion (pixel number conversion) processing for matching the number of pixels of the monitor 19 with respect to the data of the image to be reproduced and displayed. (Note that unless otherwise specified, when an image is displayed on the monitor 19 in this specification, the number of pixels of the display image is adjusted by the data processing circuit 16). Then, the data processing circuit 16 outputs the reproduced image data after the resolution conversion to the display control circuit 18. When performing the electronic zoom processing, the data processing circuit 16 executes resolution conversion (pixel number conversion) processing on the input image data, and compresses / decompresses the image data after the resolution conversion. Each is output to the display control circuit 18.

  The first memory 17 is a buffer memory that temporarily stores image data in the pre-process and post-process of the processing by the data processing circuit 16 or the compression / decompression circuit 20.

  In response to a command from the control circuit 27, the display control circuit 18 performs predetermined signal processing on the image data input from the data processing circuit 16 and outputs the image data to the monitor 19. The display control circuit 18 further performs processing for superimposing overlay image data such as a shooting menu and a cursor on the image data. By such control of the control circuit 27 and the display control circuit 18, the subject image on which the overlay image is superimposed can be displayed on the monitor 19. Note that the monitor 19 in the present embodiment may be configured with any of an electronic viewfinder having an eyepiece and a liquid crystal display panel provided on the back surface of the camera housing.

  The compression / decompression circuit 20 performs predetermined compression processing on the image data input from the data processing circuit 16 in response to a command from the control circuit 27. The compression / decompression circuit 20 is connected to the recording I / F 21, and the compressed image data is output to the recording I / F 21. The compression / decompression circuit 20 executes a decoding process that is a reverse process of the compression process on the compressed image data. Note that the compression / decompression circuit 20 of the present embodiment has a configuration capable of performing lossless compression (so-called lossless encoding).

  The recording I / F 21 has a connector for connecting the storage medium 29. The recording I / F 21 writes / reads data to / from the storage medium 29 connected to the connector. The storage medium 29 includes a small hard disk, a memory card incorporating a semiconductor memory, an optical disk such as a DVD, and the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 29. The storage medium 29 may be built in the electronic camera or may be detachably attached to the electronic camera. An external storage medium electrically connected via the communication I / F 22 may be used as a storage medium for reading and writing image data.

  Here, when recording data of an image captured in a photographing mode which is one of the operation modes of the electronic camera, the control circuit 27 displays a reproduced image corresponding to the recorded image on the monitor 19. The recorded image in this specification means a still image corresponding to still image data obtained by imaging and finally recorded on the storage medium 29 (or recorded on the storage medium 29). . When non-compressed recording of a recorded image is instructed to the control circuit 27 by the user's operation using the operation member 23, the compression / decompression circuit 20 does not perform compression processing and records the recorded image data as a recording I / F 21. Output to. Also in the case of the above uncompressed recording, the control circuit 27 causes the monitor 19 to display a reproduced image corresponding to the recorded image.

  In the playback mode, which is one of the operation modes of the electronic camera, the control circuit 27 causes the monitor 19 to display a playback image based on the image data stored in the storage medium 29. In this playback mode, the recording I / F 21 reads data of an image to be played back from the storage medium 29 in response to a command from the control circuit 27. Then, the compression / decompression circuit 20 performs a decoding process on the image data to be reproduced, and sends the decoded image data to the data processing circuit 16. Thereafter, the data processing circuit 16 and the display control circuit 18 execute the above-described processing on the decoded image data, so that the reproduced image is displayed on the monitor 19. When uncompressed image data is read from the storage medium 29, the compression / decompression circuit 20 sends the image data to the data processing circuit 16 without performing the decoding process.

  The communication I / F 22 controls data transmission / reception with an external device 30 (for example, a personal computer or an external storage medium) in accordance with the specifications of a known communication standard such as wired or wireless. Communication between the electronic camera and the external device 30 is performed via a wired or wireless communication line.

  The operation member 23 includes, for example, a command dial, a cross-shaped cursor key, a zoom operation button, a determination button, and the like. The operation member 23 receives various inputs of the electronic camera from the user. The operation member 23 may be provided with a setting member for setting the number of frames for generating an interpolation image, which will be described later.

  As an example, when receiving an input from the zoom operation button, the control circuit 27 outputs a lens drive command for the zoom lens and causes the lens drive unit 12 to drive the zoom lens forward and backward. As a result, the subject image formed on the imaging surface of the imaging device 13 is enlarged or reduced, and optical zoom adjustment is performed by the imaging optical system 11.

  When the control circuit 27 further receives an input from the zoom operation button, the control circuit 27 outputs a command to the data processing circuit 16 to change the conversion ratio of the resolution conversion processing for the image data in accordance with the user's operation. Thereby, the image displayed on the monitor 19 is enlarged or reduced, and electronic zoom adjustment is performed (electronic zoom). The conversion ratio of the resolution conversion process corresponds to the electronic zoom magnification. When the data processing circuit 16 changes the conversion ratio in the direction of increasing the electronic zoom magnification, a part of the reproduced image is enlarged and displayed on the monitor 19 (while the enlargement ratio is increased, the display range of the reproduced image is narrowed). . On the other hand, when the data processing circuit 16 changes the conversion ratio in the direction of decreasing the electronic zoom magnification, the enlargement ratio of the reproduced image displayed on the monitor 19 is lowered, but the display range of the reproduced image is widened. In the above-described shooting mode, captured image data corresponding to the display image on the monitor 19 can be recorded in the storage medium 29.

  The release button 24 receives an instruction input for starting an autofocus (AF) operation by a pressing operation and an instruction input for starting an imaging operation from a user in a continuous shooting mode described later.

  The control circuit 27 executes a known contrast detection AF operation in response to the pressing operation of the release button 24. In this AF, a signal corresponding to a focus detection area set in advance in the shooting screen is used among image signals read from the image sensor 13. Specifically, the control circuit is configured to maximize the integrated value (so-called focus evaluation value) of the high frequency component for the data corresponding to the focus detection area among the data of the image signal processed by the signal processing circuit 15. 27 sends a lens drive command for the focus lens to the lens drive unit 12. The position of the focus lens that maximizes the focus evaluation value is a focus position that eliminates blurring of the edge of the subject image captured by the image sensor 13 and maximizes the contrast of the image (increases sharpness).

  The vibration sensor 25 detects the shake of the casing of the electronic camera in two directions orthogonal to each other. The vibration sensor 25 is composed of, for example, an angular velocity sensor, a gyro sensor, or the like, and is arranged in the housing of the electronic camera. The vibration sensor 25 detects a shake applied to the electronic camera in the photographing mode, and supplies the control circuit 27 with shake amount data (data including a shake direction and a shake magnitude (a shake speed or acceleration)) in two orthogonal directions. Output. The control circuit 27 performs camera shake correction based on the shake amount data. For example, when the image pickup optical system 11 has a shake correction lens, the control circuit 27 sets the shake correction lens via the lens driving unit 12 so that the movement of the subject on the image pickup surface due to the shake of the housing is canceled. Camera shake correction is performed by driving.

  The second memory 26 is a non-volatile storage medium such as a flash memory. The second memory 26 stores various setting data and the like.

  The control circuit 27 is a processor that comprehensively controls the operation of the electronic camera. For example, the control circuit 27 obtains the brightness of the object scene from the signal output from the image sensor 13. Then, the control unit executes a known AE calculation based on the brightness information described above, and performs imaging conditions (charge accumulation time of the image sensor 13, aperture value of the aperture (not shown), aperture value of the image signal) in the imaging mode. Amplification) is determined.

  Further, the control circuit 27 executes a program stored in the second memory 26 and the like, thereby executing an interpolation image generation process to be described later.

  Hereinafter, operations in the continuous shooting mode and playback mode of the electronic camera of the first embodiment will be described.

(Explanation of continuous shooting mode)
The continuous shooting mode is one of the above-described shooting modes, in which the electronic camera continuously executes still image capturing operations at predetermined time intervals while the release button 24 is pressed. Note that the control circuit 27 in the continuous shooting mode determines imaging conditions so that each recorded image (continuous shooting image) that is shot continuously can withstand appreciation as a still image. For example, the control circuit 27 reduces the aperture to increase the depth of field.

  Specifically, when detecting the pressing operation of the release button 24, the control circuit 27 continuously performs the AF operation described above and performs a continuous shooting operation of the recorded image.

  Here, regarding the AF operation in the continuous shooting mode, the control circuit 27 may perform the AF operation by tracking a specific subject moving within the screen. Alternatively, the control circuit 27 may perform the AF operation so that an object at a predetermined shooting distance is always focused. Note that the control circuit 27 may perform focus adjustment by a user's manual operation via the operation member 23.

  Regarding the continuous shooting operation in the continuous shooting mode, the control circuit 27 instructs the image sensor drive circuit 14 to output a drive signal for executing the continuous shooting operation. The image sensor 13 receives the drive signal and outputs an image signal at a frame rate of 10 fps, for example. The image signal of each frame output from the image sensor 13 is subjected to predetermined processing by the signal processing circuit 15 and the data processing circuit 16. Thereafter, the control circuit 27 temporarily stores recorded image data corresponding to each frame in the first memory 17.

  The compression / decompression circuit 20 compresses the data of each recorded image that has been continuously shot in accordance with a command from the control circuit 27. The compression rate of the compression / expansion circuit 20 is changed by the control circuit 27 according to the operation of the operation member 23. When the non-compressed state is selected by operating the operation member 23, the compression / decompression circuit 20 omits the above-described compression processing in response to a command from the control circuit 27.

  Thereafter, the recording I / F 21 records the compressed recording image data in the storage medium 29 in accordance with a command from the control circuit 27. At this time, the data of each recorded image is associated by the control circuit 27 with supplementary information indicating the imaging conditions of each recorded image (focal length of the imaging optical system 11, exposure conditions, shake amount data, etc.). To be recorded. As an example, the control circuit 27, in conformity with Exif (Exchangeable Image File Format), generates a single image file for each recording image by combining the header including the accompanying information and the recording image data. This image file is recorded in the storage medium 29.

  At the time of continuous shooting operation, the data processing circuit 16 performs resolution conversion processing on the recorded image data, and generates view image data in accordance with the number of pixels of the monitor 19. The data processing circuit 16 sequentially supplies view image data of each frame to the monitor 19 via the display control circuit 18. During the continuous shooting operation, view images corresponding to the respective frames are sequentially displayed on the monitor 19 under the control of the display control circuit 18. Thereby, the user can confirm the state of the object scene and the composition of the recorded image by viewing the view image on the monitor 19.

(Description of playback mode)
FIG. 2 is a flowchart showing the operation in the reproduction mode in the first embodiment. In the example of FIG. 2, the description will be made on the premise that two image files having adjacent shooting times among the image files of the recorded images generated in the above-described continuous shooting mode are to be reproduced. Here, the recorded images included in each image file to be reproduced are images having the same number of pixels.

  Step S101: The control circuit 27 reads two image files to be reproduced (recorded image data) from the storage medium 29 via the recording I / F 21, respectively. In the following description, among the recorded images to be reproduced, the previously recorded image is referred to as a first image. A recorded image captured later is referred to as a second image.

  Step S102: The control circuit 27 determines whether or not the recorded image to be reproduced is based on panning shooting (shooting that shifts the imaging range by shaking the camera in the left-right direction) based on the supplementary information of the image file.

  Specifically, the control circuit 27 refers to the shake amount data of the incidental information corresponding to the first image or the second image. Then, the control circuit 27 determines whether or not the panning shooting is performed based on the frequency characteristics (vibration frequency band, shake angle peak, etc.) of vibration with respect to the electronic camera.

  In the case of panning shooting (YES side), the process proceeds to S103. On the other hand, if it is not panning shooting (NO side), the process proceeds to S107.

  Step S103: The control circuit 27 specifies an overlapping portion and a non-overlapping portion of the imaging range in the first image and the second image, respectively. Specifically, the control circuit 27 executes a matching process between the first image and the second image. Then, the control circuit 27 specifies a range in which the subject correlation between the two images is equal to or greater than the threshold as an overlapping portion of the imaging range. In the following description, the overlapping portion of the imaging range in the first image is referred to as a first region S1. In addition, the overlapping portion of the imaging range in the second image is referred to as a first region S2.

  And the control circuit 27 specifies the remaining area | region except the overlap part (1st area | region S1, S2) of an imaging range as a non-overlapping part in a 1st image and a 2nd image, respectively. In the following description, a non-overlapping portion of the imaging range in the second image is referred to as a second region X. A non-overlapping portion of the imaging range in the first image is referred to as a third region Y.

  Here, the control circuit 27 in S103 may determine a range in which the matching process is performed based on the direction and magnitude of the shake indicated by the shake amount data. As an example, the control circuit 27 obtains a subject movement amount on the image by converting the direction and magnitude of the shake with a predetermined conversion formula, and determines a range for performing the matching process based on the subject movement amount.

  Step S104: The control circuit 27 performs feature point extraction processing on each of the first region S1 included in the first image and the first region S2 included in the second image.

  Specifically, the control circuit 27 extracts edge components from each of the first region S1 of the first image and the first region S2 of the second image, and features, for example, the position of the intersection (corner) of the extracted edges. Find as a point. Then, the control circuit 27 records the position of the feature point in the first image and the second image in the first memory 17 or the second memory 26. Note that the control circuit 27 may extract points designated by the user on the monitor 19 as feature points.

  Step S105: The control circuit 27 partially generates an interpolated image in a range corresponding to the first area among the interpolated images for interpolating the change of the subject between the first image and the second image. Here, the number of pixels of the interpolation image is set to the same number of pixels as the first image and the second image. In addition, the control circuit 27 according to the present embodiment generates an interpolation image by performing a geometric morphing process involving deformation of the subject image or a morphing process in which the subject image moves without being deformed.

  Specifically, the control circuit 27 generates an interpolated image corresponding to the first region in the following procedures (a) to (f). Further, the number of frames of the interpolated image inserted between the first image and the second image is changed by the control circuit 27 according to user input. In the following example, for the sake of simplicity, a case will be described in which an interpolation image for two frames is inserted between the first image and the second image.

  (A) The control circuit 27 associates feature points included in the first region S1 of the first image and the first region S2 of the second image, respectively. For example, the control circuit 27 executes a matching process of the feature points between the two images, and obtains a correspondence relationship between the feature points.

  (B) The control circuit 27 obtains a function (movement trajectory of the feature point in the morphing operation) that connects the positions of the pair of feature points associated in (a) in the spatial direction. Here, the above function may connect a pair of feature points with a straight line, or may connect a pair of feature points with a spline curve or the like. Note that the control circuit 27 obtains the above function for all feature points associated with each other between the first image and the second image.

  (C) The control circuit 27 determines the position of the feature point in the interpolated image corresponding to the first region by the function obtained in (b) above. Specifically, the control circuit 27 internally divides a section defined by the above function and a pair of feature points according to the number of interpolated images inserted, and the position of this internal dividing point corresponds to the first region. The position of the feature point in the interpolated image.

  FIG. 3 is an explanatory diagram illustrating an example of a method for determining feature points in an interpolated image corresponding to the first region. In the example of FIG. 3, the feature points D, E, and F of the second image correspond to the feature points A, B, and C of the first image, respectively. The position of the point (G, J) that internally divides the section of the linear function connecting the first feature points A and D into three becomes the position of the first feature point in each interpolated image. Note that the position (H, K) of the second feature point and the position (I, L) of the third feature point in the interpolated image can be obtained in the same procedure as in the case of the first feature point.

  (D) The control circuit 27 obtains the position of the pixel of interest other than the feature point in the interpolated image corresponding to the first region. Specifically, the control circuit 27 defines the position of the pixel of interest in the first image (or the second image) with a vector connecting feature points. Then, the control circuit 27 obtains the position of the target image on the interpolated image based on the above vector definition.

  FIG. 4 is an explanatory diagram illustrating an example of a method of determining a target pixel in the interpolation image corresponding to the first region.

  Therefore, the control circuit 27 can obtain the position of each pixel of interest in the interpolated image corresponding to the first region by the above method.

  (E) The control circuit 27 obtains a change in the gradation value of each pixel in the interpolated image corresponding to the first region. Specifically, first, the control circuit 27 sets the gradation value of a pixel (a target pixel) having a correspondence relationship between the first region S1 of the first image and the first region S2 of the second image. Ask for each. Second, the control circuit 27 internally divides the interval between the two gradation values according to the number of interpolation image insertions, and obtains the gradation value of the target pixel in the first region of the interpolation image. As an example, when the gradation value of the target pixel is 130 and 136 in the first image and the second image, the control circuit 27 obtains a point that internally divides the interval of the gradation values 130 to 136 into three. . Then, the control circuit 27 sets the gradation value (132, 134) corresponding to the above internal dividing point as the gradation value of the target pixel in the first region of each interpolated image. Note that the control circuit 27 obtains the gradation value for each value of RGB or YCbCr.

  (F) The control circuit 27 records the data of the first area of the interpolated image generated in the above process in the first memory 17 or the second memory 26, respectively.

  Step S106: The control circuit 27 is located in the second area image located at the movement destination of the imaging range with respect to the first area and the movement source of the imaging range with respect to the first area among the respective interpolation images. A third region image is generated. Note that after the processing of S106, the control circuit 27 proceeds to S109.

  Hereinafter, a method for generating an interpolation image corresponding to the second region and the third region will be described with reference to FIG. Here, in the example of FIG. 5, a case will be described in which continuous shooting is performed by panning from the right to the left. In the example of FIG. 5, similarly to S <b> 105, two frames of interpolated images are inserted between the first image and the second image.

  First, a method for generating an interpolation image corresponding to the second region will be described. The control circuit 27 cuts out the second area X of the second image and generates an interpolation image corresponding to the second area. Specifically, the control circuit 27 cuts out a range from the right side of the image to 1/3 in the horizontal direction from the second region X of the second image to generate an interpolated image of the second region of the first frame. Further, the control circuit 27 cuts out a range from the right side of the image to 2/3 in the horizontal direction in the second region X of the second image to generate an interpolation image of the second region of the second frame.

  Next, a method for generating an interpolation image corresponding to the third region will be described. The control circuit 27 cuts out the third area Y of the first image and generates an interpolation image corresponding to the third area. Specifically, the control circuit 27 cuts out a range from the left side of the image to 2/3 in the horizontal direction from the third region Y of the first image to generate an interpolation image of the third region of the first frame. Further, the control circuit 27 cuts out a range from the right side of the image to 1/3 in the horizontal direction in the third region Y of the first image to generate an interpolation image of the third region of the second frame.

  Then, the control circuit 27 connects the interpolation image of the second region of the first frame (the image of the right third of the second region X) to the left of the interpolation image of the first region of the first frame. Further, the control circuit 27 connects the interpolation image of the third region (the image of the left 2/3 of the third region Y) to the right side of the interpolation image of the first region of the first frame.

  The control circuit 27 connects the interpolation image of the second area of the second frame (the image of the right side 2/3 of the second area X) of the second frame to the left side of the interpolation image of the first area of the second frame. Further, the control circuit 27 connects the interpolation image of the third region (the image of the left third of the third region Y) to the right side of the interpolation image of the first region of the second frame.

  With the above method, the control circuit 27 generates an interpolation image for two frames with the same size as the first image and the second image. As shown in FIG. 5, when comparing the first image, the interpolated image for two frames, and the second image, it can be seen that the range corresponding to the first region is shifted by a predetermined amount in the left direction. For this reason, when a plurality of images including an interpolation image are reproduced as a moving image, a situation in which the imaging range pans leftward on the screen is reproduced.

  Here, the ratio of the cutout amount of the image in the second area and the third area changes according to the number of interpolation images inserted. As an example, when an n-frame interpolated image is inserted, the control circuit 27 cuts out a range from the right side of the image to the 1 / (n + 1) in the second region X of the second image in the horizontal direction. An interpolated image of the second region of the eye is generated. Further, the control circuit 27 cuts out a range from the left side of the image to n / (n + 1) in the horizontal direction from the third region Y of the first image to generate an interpolation image of the third region of the first frame.

  In S106, when there is a common subject straddling the boundary between the first region S2 and the second region X of the second image, the control circuit 27 determines the common subject included in the second region of the interpolation image as follows. You may make it deform | transform.

  First, the control circuit 27 specifies a feature point corresponding to the common subject from the first region S1 of the first image and the first region S2 of the second image. Next, the control circuit 27 obtains the amount of change of the common subject from the first image to the second image from the function connecting the corresponding feature points (obtained in S105). Then, the control circuit 27 changes the size of the common subject in the second region in each interpolated image based on the above change amount.

  As an example, the case of the common subject Z straddling the boundary between the first region S2 and the second region X of the second image will be specifically described in FIG. Here, the feature point BD of the common subject Z is included in the first region S1 of the first image. In addition, the feature point B'D 'of the common subject Z is included in the first region S2 of the second image. Further, the second region X of the second image includes the feature point A′C ′ of the common subject Z.

  (1) First, the control circuit 27 obtains the amount of change between the distance between the feature points BD of the first image and the distance between the feature points B′D ′ of the second image using the above function. (2) Next, the control circuit 27 considers that the positions of all feature points change between the first image and the second image in accordance with the amount of change described above, and the feature point A ′ in each interpolated image. The position of the virtual feature point corresponding to C ′ is estimated. (3) Then, the control circuit 27 generates an interpolated image corresponding to the second region by morphing based on the position of the feature point. (4) After that, the control circuit 27 cuts out a predetermined amount of the interpolated image of the second area generated in the above (3) and connects it to the interpolated image of the first area.

  When the common subject is deformed as described above, it is less likely that the common subject is shifted and connected at the boundary between the first region and the second region of the interpolated image, so that an interpolated image with less discomfort is generated. become able to. In particular, the effect is great when optical or electronic zooming is performed between the first image and the second image, or when a common subject approaches or separates from the camera during continuous shooting.

  Even when there is a common subject straddling the boundary between the first region S1 and the third region Y of the first image, the control circuit 27 selects the common subject included in the third region of the interpolated image as described above. Needless to say, it can be deformed (illustration is omitted in this case).

  Step S107: The control circuit 27 executes a feature point extraction process for the entire area of each of the first image and the second image. Note that the feature point extraction process in S107 is the same as that in S104, and thus a duplicate description is omitted.

  Step S108: The control circuit 27 generates an interpolated image for interpolating the change of the subject between the first image and the second image. Note that the interpolation image generation process in S108 is the same as that in S105 except that the entire area of the first image and the second image is used, and therefore a duplicate description is omitted.

  Step S109: The control circuit 27 buffers each interpolated image data (generated in S106 or S108) in the first memory 17. At this time, the control circuit 27 may record the generated interpolation image on the storage medium 29 via the recording I / F 21.

  Step S110: The control circuit 27 continuously reproduces the first image, the interpolated image, and the second image on the monitor 19 in time series via the data processing circuit 16 and the display control circuit 18. Above, description of the flowchart of FIG. 2 is complete | finished.

  Hereinafter, the operational effects of the first embodiment will be described. In the electronic camera according to the first embodiment, when the interpolation image is generated from the first image and the second image generated by the continuous shooting with panning, the overlapping portion of the interpolation image is the first image and the second image. Is generated based on the amount of change in the overlapping portion (S1, S2). On the other hand, the non-overlapping part of the interpolation image is generated by cutting out the non-overlapping part (X) of the second image and the non-overlapping part (Y) of the first image. For this reason, in an area where it is difficult to associate feature points between two images, an original image is cut out and an interpolated image is generated, so that an interpolated image with less discomfort can be generated.

<Description of Second Embodiment>
FIG. 6 is a flowchart showing the operation of the playback mode of the electronic camera of the second embodiment. The configuration of the electronic camera in the second embodiment is the same as that of the electronic camera of the first embodiment shown in FIG.

  Here, in the example of FIG. 6, the description will be made on the assumption that among the image files of the recorded images generated in the continuous shooting mode, a plurality of image files having adjacent shooting times are targeted for reproduction. The recorded images included in each image file to be reproduced are images having the same number of pixels. In the following description, recorded images to be reproduced are expressed as t0, t1, t2,.

  Step S201: The control circuit 27 determines whether or not an image file reproduction instruction has been received via the operation member 23. If there is a reproduction instruction (YES side), the process proceeds to S202. On the other hand, when there is no reproduction instruction (NO side), the control circuit 27 stands by until there is a reproduction instruction.

  Step S202: The control circuit 27 sets the number of interpolated image frames to be inserted between the respective recorded images based on the user input from the operation member 23. In the following example, a case where an interpolated image for two frames is inserted between two recorded images will be described.

  Step S203: The control circuit 27 reads an image file to be reproduced (recorded image data) from the storage medium 29 via the recording I / F 21.

  Step S204: The control circuit 27 obtains the motion vector of the subject between the two frames for two frames of the recorded images adjacent in the time axis direction. Note that the control circuit 27 in S204 obtains the motion vector of the subject in each of the frame sections from t0 to tn.

  Specifically, first, the control circuit 27 executes feature point extraction processing for the entire area of each recorded image (t0 to tn). Since this feature point extraction process is common to S104 of the first embodiment, redundant description is omitted. Next, the control circuit 27 associates each feature point with respect to two recorded images adjacent in the time axis direction. Then, the control circuit 27 obtains the motion vector of the subject in the frame section from the motion of the feature points between the frames of the two recorded images. Note that the motion vector calculation process described above may be obtained by pattern matching.

  Step S205: The control circuit 27 designates a frame section for generating an interpolation image along the time axis.

  Step S206: The control circuit 27 compares the direction of the motion vector of the subject of interest in the designated frame section (S205) with the direction of the motion vector of the subject of interest in the immediately preceding frame section. Then, the control circuit 27 determines whether or not a change greater than or equal to the threshold value has occurred in the direction of both motion vectors. Note that the threshold value in the present embodiment is set when there is a change of 30 ° or more in the direction of the motion vector, for example.

  If the change in the direction of the motion vector is greater than or equal to the threshold (YES side), the process proceeds to S207. On the other hand, if the change in the direction of the motion vector is less than the threshold (NO side), the process proceeds to S209. When the designated frame section includes the first recorded image t0 (that is, when there is no immediately preceding frame section), the control circuit 27 determines NO in S206 and proceeds to S209.

  Step S207: The control circuit 27 compares the direction of the motion vector of the subject of interest in the designated frame section (S205) with the direction of the motion vector of the subject of interest in the immediately following frame section. Then, the control circuit 27 determines whether or not a change greater than or equal to the threshold value has occurred in the direction of both motion vectors.

  If the change in the direction of the motion vector is greater than or equal to the threshold (YES side), the process proceeds to S208. On the other hand, if the change in the direction of the motion vector is less than the threshold (NO side), the process proceeds to S209. When the designated frame section includes the last recorded image tn (that is, when there is no immediately following frame section), the control circuit 27 determines NO in S207 and moves the process to S209.

  Step S208: The control circuit 27 generates an interpolated image for interpolating subject changes in the designated frame section (S205) by extrapolation using the motion vectors of the preceding and following frame sections. Note that after the processing of S208, the control circuit 27 proceeds to S210.

  FIG. 7 is a diagram schematically illustrating the interpolation image generation method in S208. FIG. 7 illustrates an example in which a subject of interest (ball) rebounds by colliding with a wall in a designated frame section (t2t3). In FIG. 7, for the sake of convenience, the spatial position change of the subject of interest in each frame is represented on one screen.

  First, the control circuit 27 obtains a function that connects the positions of a pair of feature points in the spatial direction in the immediately preceding frame interval (t1t2) with respect to the designated frame interval (t2t3). In addition, the control circuit 27 obtains a function that connects the positions of the pair of feature points in the spatial direction in the immediately following frame section (t3t4) with respect to the designated frame section (t2t3). Then, the control circuit 27 determines the position of the feature point of the interpolation image in the designated frame section (t2t3) by extrapolation using the above two functions. As an example, the control circuit 27 obtains a section from t2 to t3 through an intersection of a function obtained from t1t2 and a function obtained from t3t4. Then, the control circuit 27 internally divides the above interval by the number of interpolation image insertions, and determines the position of the feature point in each interpolation image. Thereby, among the interpolation images inserted in the frame interval t2t3, the position of the interpolation image t2a (the first frame interpolation image inserted in the frame interval t2t3) is determined according to the motion vector of the immediately preceding frame interval t1t2. . Further, the position of the interpolated image t2b (the interpolated image of the second frame inserted into the frame section t2t3) is determined according to the motion vector of the immediately subsequent frame section t3t4.

  Further, the control circuit 27 in S208 obtains the position of the pixel of interest other than the feature point in the interpolated image and the change in the gradation value of each pixel using the method in S105 of the first embodiment.

  Step S209: The control circuit 27 generates an interpolated image for interpolating the change of the subject in the designated frame section (S205) by interpolation using the motion vector of the frame section. Note that the interpolation image generation processing in S209 is substantially the same as S105 in the first embodiment, and thus a duplicate description is omitted.

  Step S210: The control circuit 27 buffers each interpolation image data (generated in S208 or S209) in the first memory 17. At this time, the control circuit 27 may record the generated interpolation image on the storage medium 29 via the recording I / F 21.

  Step S211: The control circuit 27 determines whether or not the designated frame section (S205) is the last frame section. If the above requirement is satisfied (YES side), the process proceeds to S212. On the other hand, if the above requirement is not satisfied (NO side), the process returns to S205, and the control circuit 27 repeats the above operation. Thereby, an interpolation image is sequentially generated in each frame section.

  Step S212: The control circuit 27 continuously reproduces the recorded image and the interpolated image on the monitor 19 in time series via the data processing circuit 16 and the display control circuit 18. Above, description of the flowchart of FIG. 7 is complete | finished.

  Hereinafter, the operational effects of the second embodiment will be described. In the electronic camera of the second embodiment, when the movement of the subject of interest changes abruptly between recorded images, an interpolation image is generated by extrapolation using the motion vectors of the previous and subsequent frames. As an example, in FIG. 7, the interpolation of the motion vector in the frame interval t2t3 generates an interpolated image in which the ball suddenly turns back in front of the wall, but in the interpolated image of this embodiment, the ball has t2 and t3. Since the object is closer to the wall than the position, an interpolation image in which the behavior of the subject of interest is more natural can be generated.

(Supplementary items of the embodiment)
(1) In the electronic camera of the above-described embodiment, a phase difference AF module may be provided separately from the imaging device 13 and a phase difference AF operation by a known pupil division may be performed in addition to a contrast type AF operation. In the above-described embodiment, an example in which AE calculation is performed based on the output of the image sensor 13 has been described. However, a photometric element for AE calculation may be provided separately from the image sensor 13.

  (2) In the above embodiment, the example in which the interpolation image is generated from the data of the recording image has been described. However, in the above-described embodiment, for example, when the recorded image data and the view image data set to be smaller than the display size of the monitor 19 are grouped and recorded in the storage medium 29, the control circuit 27 The view image data may be preferentially selected, and an interpolation image may be generated from the view image data.

  (3) In the first embodiment described above, an example in which an image file generated by continuous shooting with panning is used has been described. However, the present invention can be similarly applied to tilt shooting in which the imaging range is shifted by shaking the camera in the vertical direction, or in the case where the imaging range is shifted by moving the camera horizontally (so-called track operation).

  (4) In the above embodiment, an example in which the control circuit 27 of the electronic camera generates a composite image has been described. For example, an external device 30 (such as a personal computer) connected to the electronic camera controls the control of the above embodiment. The processing of the circuit 27 may be executed. In addition, in the above embodiment, a part of the processing executed by the control circuit 27 of the electronic camera is borne by the external device 30, and the electronic camera and the external device 30 cooperate with each other to perform the first embodiment or the second embodiment. You may implement | achieve the algorithm demonstrated by the form.

  It should be noted that the present invention can be implemented in various other forms without departing from the spirit or main features thereof. Therefore, the above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. The present invention is defined by the claims, and the present invention is not limited to the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

The block diagram which shows the structure of the electronic camera of 1st Embodiment. Flow chart showing the operation in the playback mode in the first embodiment Explanatory drawing which shows an example of the determination method of the feature point in the interpolation image corresponding to a 1st area | region. Explanatory drawing which shows an example of the determination method of the attention pixel in the interpolation image corresponding to a 1st area | region. The figure which shows typically the production | generation method of the interpolation image corresponding to a 2nd area | region and a 3rd area | region. Flowchart showing the operation of the playback mode of the electronic camera of the second embodiment The figure which shows typically the production | generation method of the interpolation image in S208

Explanation of symbols

DESCRIPTION OF SYMBOLS 19 ... Monitor, 21 ... Recording I / F, 22 ... Communication I / F, 23 ... Operation member, 27 ... Control circuit, 29 ... Storage medium, 30 ... External device

Claims (10)

For a computer having a data reading unit for reading image data and a calculation unit,
The second image captured after the first image in a state where the imaging range of the imaging device is moved so as to partially overlap the imaging range of the first image and the data of the first image captured by the imaging device A first step of reading image data into the calculation unit by the data reading unit;
A second step of identifying each of an overlapping portion and a non-overlapping portion of the imaging range in the first image and the second image;
Of the interpolated images that interpolate the change of the subject between the first image and the second image, the image of the first region corresponding to the overlapping portion is represented by the overlapping portion of the first image and the second image. A third step of generating based on the amount of change in the image with the overlapping portion;
A fourth step of generating, in the interpolated image, an image of a second region located at a destination of the imaging range with respect to the first region by cutting out the non-overlapping portion of the second image;
A fifth step of generating the interpolated image by connecting the image of the second region to the image of the first region;
Is executed by the calculation unit. The program according to claim 1,
Sixth step of generating an image of a third region located at a source of movement of the imaging range with respect to the first region in the interpolated image by cutting out the non-overlapping portion of the first image. Runs further,
In the fifth step, the interpolation image is generated by further connecting the image of the third region to the image of the first region. In the program according to claim 1 or 2,
The common subject included in at least one of the second region and the third region when there is a common subject straddling the overlapping portion and the non-overlapping portion in at least one of the first image and the second image The calculation unit further executes a seventh step of deforming the image based on the amount of change of the common subject obtained from the overlapping portion of each of the first image and the second image. In the program according to any one of claims 1 to 3,
The computer program further includes an eighth step of continuously reproducing the first image, the interpolated image, and the second image. An image processing method using at least one computer,
The second image captured after the first image in a state where the imaging range of the imaging device is moved so as to partially overlap the imaging range of the first image and the data of the first image captured by the imaging device A first step for the computer to read image data;
A second step in which the computer specifies an overlapping portion and a non-overlapping portion of the imaging range in the first image and the second image,
Of the interpolated images that interpolate the change of the subject between the first image and the second image, the image of the first region corresponding to the overlapping portion is represented by the overlapping portion of the first image and the second image. A third step generated by the computer based on the amount of change in the image with the overlapping portion;
A fourth step in which the computer generates an image of a second area located at a destination of the imaging range with respect to the first area in the interpolated image by cutting out the non-overlapping portion of the second image; ,
A fifth step in which the computer generates the interpolated image by connecting the image of the second region to the image of the first region;
An image processing method comprising: The second image captured after the first image in a state where the imaging range of the imaging device is moved so as to partially overlap the imaging range of the first image and the data of the first image captured by the imaging device A data reading unit for reading image data;
An image analysis unit that respectively identifies an overlapping portion and a non-overlapping portion of the imaging range in the first image and the second image;
Of the interpolated images that interpolate the change of the subject between the first image and the second image, the image of the first region corresponding to the overlapping portion is represented by the overlapping portion of the first image and the second image. A first calculation unit that generates based on an image change amount with the overlapping portion;
A second calculation unit that generates an image of a second region located at a movement destination of the imaging range with respect to the first region of the interpolated image by cutting out the non-overlapping portion of the second image;
An image connecting unit that connects the image of the second region to the image of the first region to generate the interpolated image;
An image processing apparatus comprising: For a computer having a data reading unit for reading image data and a calculation unit,
A first step of reading, by the data reading unit, a plurality of frames of image data groups obtained by continuously capturing the subject of interest into the calculation unit;
A second step of obtaining a motion vector of the subject of interest for each frame section in a plurality of frames of the image data group using two frames at which the imaging time points are adjacent;
A first frame section and a second frame section, each of which is temporally continuous, are extracted, and a change in the direction of the motion vector in the subsequent second frame section with respect to the motion vector in the previous first frame section A third step of determining whether is less than a threshold;
When it is determined in the third step that the change is less than the threshold value, an interpolation image is generated that interpolates the change of the subject of interest in the second frame section by interpolation using the motion vector of the second frame section. And a fourth step
When it is determined in the third step that the change is greater than or equal to the threshold, the subject of interest in the second frame section is extrapolated using the motion vector of at least one of the frame sections preceding and following the second frame section. A fifth step of generating an interpolated image for interpolating the change of
Is executed by the calculation unit. The program according to claim 7,
The program, wherein the arithmetic unit further executes a sixth step of inserting the interpolated image between two frames of the second frame section and continuously reproducing the image. An image processing method using at least one computer,
A first step in which the computer reads a plurality of frames of image data obtained by continuously capturing a target object;
A second step in which the computer obtains a motion vector of the subject of interest for each frame section in a plurality of frames of the image data group, using two frames at which imaging points are adjacent;
A first frame section and a second frame section, each of which is temporally continuous, are extracted, and a change in the direction of the motion vector in the subsequent second frame section with respect to the motion vector in the previous first frame section A third step by which the computer determines if is less than a threshold;
When it is determined that the change is less than the threshold in the third step, an interpolation image that interpolates the change of the subject of interest in the second frame section is obtained by interpolation using the motion vector of the second frame section. A fourth step generated by a computer;
When it is determined in the third step that the change is equal to or greater than the threshold value, the subject of interest in the second frame section is extrapolated using the motion vector of at least one of the frame sections preceding and following the second frame section. A fifth step in which the computer generates an interpolated image for interpolating the change of
An image processing method comprising: A data reading unit that reads a plurality of frames of image data obtained by continuously capturing the subject of interest;
A motion vector computing unit that obtains a motion vector of the subject of interest for each frame section in a plurality of frames of the image data group using two frames at which imaging points are adjacent;
A first frame section and a second frame section, each of which is temporally continuous, are extracted, and a change in the direction of the motion vector in the subsequent second frame section with respect to the motion vector in the previous first frame section A determination unit that determines whether is less than a threshold value;
An image processing unit that generates an interpolated image for interpolating the change of the subject of interest,
The image processing unit generates the interpolated image of the second frame section by interpolation using the motion vector of the second frame section when the determination unit determines that the change is less than a threshold, When the determination unit determines that the change is greater than or equal to the threshold, the interpolated image of the second frame section is generated by extrapolation using the motion vector of at least one of the frame sections before and after the second frame section. An image processing apparatus.

Images