JP2018060549A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately select which region of a still picture to express a part in the still picture in a case where the part of the still picture is expressed dynamically.SOLUTION: An image processing device 1 comprises: an instruction unit 16 for instructing a first change region to be changed with time on an image of one frame among images of a plurality of time-sequentially photographed frames; a position information acquisition unit 16 for acquiring position information on the first change region instructed by the instruction unit 16 in each of the images of the plurality of frames; a time series image acquisition unit 16 for acquiring a time series image for the first change region, on the basis of the position information acquired by the position information acquisition unit 16 and the images of the plurality of frames; and an image composition unit 16 for sequentially replacing and compositing the time series image for the first change region acquired by the time series image acquisition unit 16 with respect to a still picture generated on the basis of the image of one frame among the images of the plurality of frames.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus.

  A technique for dynamically expressing a part of a still image is known (see Patent Document 1). This technology expresses a heterogeneous world where time is progressing only in some areas in a still-timed world of still images in which noise does not change over time or things that should move originally do not move It is attracting attention.

JP 2010-212781 A

  In the related art, when a part of a still image is dynamically expressed, it is not possible to appropriately select which region of the still image is dynamically expressed.

An image processing apparatus according to a first aspect of the present invention is based on an instruction unit that indicates a first change area that changes with time on at least one of a plurality of captured images, and the plurality of images. A specifying unit that specifies a second change region that changes in synchronization with a change in the first change region, the first change region specified by the instruction unit, and the second change region specified by the specifying unit; Corresponding to the first change area in the plurality of images based on the position information acquired by the position information acquisition unit that acquires the position information on the plurality of images and the position information acquisition unit. An image acquisition unit that acquires an image to be acquired and an image corresponding to the second change area, and an image synthesis unit that combines the image acquired by the image acquisition unit with a reference image of the plurality of images, Is provided.
An image processing apparatus according to a second aspect of the present invention is based on an instruction unit that indicates a first change area that changes with time on at least one of a plurality of captured images, and the plurality of images. A specifying unit that specifies a second change region that changes in accordance with a change in the first change region, a first change region that is specified by the instruction unit, and a second change region that is specified by the specifying unit. Corresponding to the first change region in the plurality of images based on the position information acquisition unit that acquires the position information on the plurality of images and the position information acquired by the position information acquisition unit An image acquisition unit that acquires an image and an image corresponding to the second change area; and an image combination unit that combines the image acquired by the image acquisition unit with a reference image of the plurality of images. Prepare.
An image processing apparatus according to a third aspect of the present invention is directed by an instruction unit that indicates a first change area that changes with time on at least one of a plurality of captured images, and the instruction unit Based on the position information acquisition unit that acquires position information on the plurality of images corresponding to the first change area, the position information acquired by the position information acquisition unit, and the plurality of images, An image acquisition unit that acquires an image at a position corresponding to one change area; and an image combination unit that combines an image acquired by the image acquisition unit with a reference image of the plurality of images, The position information acquisition unit further converts the position information of the first change region into position information based on an image used for generating the reference image, and the image acquisition unit includes the converted position information and the plurality of position information. Based on the image and Acquires an image corresponding to the serial first change area, the image synthesizing unit, based on the position information of the converted relative to the reference image, to synthesize the image acquired by the image acquisition unit.
An image processing apparatus according to a fourth aspect of the present invention is instructed by an instruction unit that indicates a first change area that changes with time on at least one of a plurality of captured images, and the instruction unit. Based on the position information acquisition unit that acquires position information on the plurality of images corresponding to the first change area, the position information acquired by the position information acquisition unit, and the plurality of images, An image acquisition unit that acquires an image at a position corresponding to one change region; and a size information acquisition unit that acquires size information of the first change region instructed by the instruction unit in each of the plurality of images. The image acquired by the image acquisition unit is combined with one of the plurality of images determined by the size information of the first change area acquired by the size information acquisition unit. An image composition unit That.
An image processing apparatus according to a fifth aspect of the present invention is directed by an instruction unit that indicates a first change area that changes with time on at least one of a plurality of captured images, and by the instruction unit. Based on the position information acquisition unit that acquires position information on the plurality of images corresponding to the first change area, the position information acquired by the position information acquisition unit, and the plurality of images, An image acquisition unit that acquires an image at a position corresponding to one change area; an image synthesis unit that combines the image acquired by the image acquisition unit with a reference image of the plurality of images; A setting unit that aligns the positions of the plurality of images with reference to a background region other than the first change region, and sets a region including a moving range of the first change region as a first change region in the plurality of images; With Serial image combining unit an image corresponding to the first change area in the plurality of images set by the setting unit, synthesized as an image for the first change area that is acquired by the image acquisition unit.
An image processing program according to a sixth aspect of the present invention is based on an instruction process for instructing a first change area that changes with time on at least one of a plurality of captured images, and the plurality of images. A specifying process for specifying a second changing area that changes in synchronization with a change in the first changing area, the first changing area specified in the instruction process, and a second changing area specified by the specifying process; Corresponding to the first change region in the plurality of images based on the position information acquisition processing for acquiring the position information on the plurality of images and the position information acquired in the position information acquisition processing. An image acquisition process for acquiring an image to be acquired and an image corresponding to the second change area; and an image synthesis process for combining the image acquired in the image acquisition process with a reference image among the plurality of images; To be executed by a computer.

  According to the present invention, when a part of a still image is dynamically expressed, it is possible to appropriately select which region of the still image is dynamically expressed.

It is a block diagram which illustrates the composition of the digital camera by one embodiment of the invention. It is a figure explaining the acquisition timing of the image in "pre-shooting photography mode". It is a figure which illustrates the display of a thumbnail image. It is a figure which illustrates image display of frame S. (a) is a diagram illustrating an image of a scene in which an object moves from left to right on the screen, (b) is a diagram illustrating a moving object region and a non-moving object region in frame S, and (c) is an extracted moving object region. FIG. It is a flowchart explaining the flow of the process which CPU performs. It is a flowchart explaining the flow of the process which CPU performs. It is a figure which illustrates the image which comprises the flame | frame S, the flame | frame P, and the flame | frame Q which CPU selected based on the time-sequential frame image which image | photographed the hitting scene by the baseball batter. FIG. 9 is a diagram illustrating an image in which a ball and a bat are sequentially replaced and synthesized at corresponding positions in the frame S based on the frame S, the frame P, and the frame Q in FIG. It is a figure which illustrates the image by which the sequential replacement composition of the modification 5 was carried out. It is a figure which illustrates the image by which the sequential replacement composition of the modification 6 was carried out. It is a figure which illustrates a computer.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram illustrating the configuration of a digital camera 1 according to the first embodiment of the invention. The digital camera 1 appropriately selects a region to be used for “motion expression” that dynamically represents a partial region of a still image based on a plurality of frames of still images. "Motion expression" expresses a heterogeneous world as if time has progressed in some areas in the still-time world of still images.

  Details of the digital camera 1 will be described below. In FIG. 1, the photographic lens 11 forms a subject image on the imaging surface of the imaging element 12. 1 illustrates the digital camera 1 in which the photographing lens 11 is formed integrally with the camera (the photographing lens cannot be attached / detached), but the camera includes a camera body and an interchangeable lens that can be attached to and detached from the camera body. You may comprise by a system. In that case, the configuration consisting of reference numerals 11 and 21 in FIG. 1 is included on the interchangeable lens side, and the configuration consisting of other reference numerals is included on the camera body side.

  When the half-press switch 20a is turned on in conjunction with a half-press operation of a shutter button (not shown), the CPU 16 performs an AE (automatic exposure) calculation and an autofocus (AF) process to form a focusing lens (a focusing lens constituting the photographing lens 11). (Not shown) is moved forward and backward in the optical axis direction (arrow direction in FIG. 1). Thereby, the focal position of the photographic lens 11 is automatically adjusted. The driving of the focusing lens (not shown) is performed by the lens driving unit 21 that receives an instruction from the CPU 16.

  The image sensor 12 is configured by a CMOS image sensor or the like. The image sensor 12 captures a subject image formed on the imaging surface. The image signal output from the image sensor 12 is converted from an analog signal to a digital signal by the A / D converter 13. The image processing unit 14 performs predetermined image processing on the digital image signal.

  The liquid crystal monitor 15 displays an image captured by the image sensor 12 and an image stored in a buffer memory 18 or a memory card 50 described later, and displays an image, an icon, an operation menu, and the like according to an instruction from the CPU 16. indicate. A touch operation member 15 a is laminated on the display surface of the liquid crystal monitor 15. The touch operation member 15a sends a signal indicating the touch position on the display screen to the CPU 16. The acoustic processing circuit 23 amplifies the acoustic signal collected by the microphone 22 and converts the amplified signal into digital acoustic data by an A / D conversion circuit (not shown).

  The flash memory 17 stores programs executed by the CPU 16 and data necessary for execution processing. The contents of the program and data stored in the flash memory 17 can be added and changed by an instruction from the CPU 16.

  The CPU 16 executes a control program using the buffer memory 18 as a work area, for example, and performs various controls on each part of the camera. The buffer memory 18 is also used when image data is temporarily stored. In the digital camera 1 according to the present embodiment, images of a plurality of frames (referred to as pre-taken images) acquired at a predetermined frame rate by the image sensor 12 before a shooting instruction (full pressing operation of the shutter button), or after a shooting instruction is taken. It is also used for temporarily storing a plurality of frames of images (referred to as post-taken images) acquired by the element 12 at a predetermined frame rate.

  In addition to the image processing on the digital image signal, the image processing unit 14 generates an image file (for example, an Exif file) in a predetermined format storing the image signal. The recording / reproducing unit 19 records (saves) the image file on the memory card 50 based on an instruction from the CPU 16 and reads out the image file recorded (saved) on the memory card 50. The recording / playback unit 19 further records (saves) the sound file storing the sound data in the memory card 50 and reads out the sound file recorded (saved) in the memory card 50.

  The memory card 50 is a recording medium that is detachably attached to a card slot (not shown). The CPU 16 reproduces and displays the captured image on the liquid crystal monitor 15 based on the image file read from the memory card 50 by the recording / reproducing unit 19 or the image data stored in the post-capture buffer memory 18. Further, the CPU 16 can also reproduce sound by a speaker (not shown) based on the acoustic file read from the memory card 50 by the recording / reproducing unit 19.

  The operation member 20 includes the half-press switch 20a, a full-press switch 20b that is turned on in response to a full-press operation of the shutter button, a mode switching switch, and the like, and sends an operation signal associated with the operation of each member to the CPU 16.

  The ON signal (half-press operation signal) from the half-press switch 20a is output when the shutter button is pressed down to about half of the normal stroke, and the output is canceled when the half-stroke press operation is canceled. The ON signal (full push operation signal) from the full push switch 20b is output when the shutter button is pushed down to the normal stroke, and the output is released when the push operation of the normal stroke is released.

<Shooting mode>
The digital camera 1 has a “normal shooting mode” and a “first shooting mode”. The “normal shooting mode” is a shooting mode in which a shot image is acquired frame by frame in response to the full-press operation signal and recorded in the memory card 50. In “pre-shooting shooting mode”, a still image is acquired at a predetermined frame rate (for example, 120 frames per second), and when the full-press operation signal is received, a plurality of frames acquired before and after the full-press operation signal is received. This is a shooting mode in which a predetermined image among images (the above-mentioned pre-taken image and post-taken image) is recorded on the memory card 50. Each photographing mode is configured to be switchable according to an operation signal from the operation member 20 or the touch operation member 15a.

<Playback mode>
The digital camera 1 in the reproduction mode reproduces and displays the image recorded in the “normal photographing mode” or the “pre-shooting photographing mode” on the liquid crystal monitor 15 for each frame or every predetermined number of frames. In addition, in the case of “motion expression”, the liquid crystal monitor 15 is configured to dynamically express an object that changes in a still image based on a plurality of frames of images acquired in the “pre-shooting shooting mode”. Perform playback display.

<Motion expression>
Since the digital camera 1 according to the present embodiment is characterized by the above-described “motion expression” process, the following description will be focused on this point. ON / OFF of the “motion expression” process is configured to be switchable according to an operation signal from the operation member 20 or the touch operation member 15a. The digital camera 1 that is in the playback mode and the “motion expression” process is turned on selects a frame suitable for “motion expression” based on the images of a plurality of frames acquired in the “first shooting mode”. The selected frame image is reproduced and displayed on the liquid crystal monitor 15. Hereinafter, acquisition of a plurality of frame images, selection of a frame suitable for “motion expression”, and image generation for “motion expression” will be described in order.

-Acquisition of multiple frame images-
The user obtains a plurality of frames of images, for example, by holding the digital camera 1 or fixing the digital camera 1 to a tripod. FIG. 2 is a diagram for explaining image acquisition timing in the “pre-shooting shooting mode”. In FIG. 2, when the “pre-shooting shooting mode” is activated at time t0, the CPU 16 starts shooting standby processing. In the shooting standby process, the subject image is captured at the predetermined frame rate, exposure calculation and focus adjustment are performed, and the acquired image data is sequentially stored in the buffer memory 18.

  In the “pre-shooting shooting mode”, a sufficient memory capacity of the buffer memory 18 used for storing the pre-shooting image and the post-shooting image is secured in advance. When the number of frame images stored in the buffer memory 18 after time t0 reaches a predetermined number (for example, A), the CPU 16 overwrites and erases the oldest frame images in order. Thereby, the memory capacity of the buffer memory 18 used for storing the pre-taken image and the post-taken image in the “pre-photographing shooting mode” can be limited.

  When a full-press operation signal is input at time t1, the CPU 16 starts shooting processing. In the photographing process, (A + B) frame images obtained by combining A frame images pre-taken before time t1 and B frame images taken after time t1 are recorded on the memory card 50. An image. A black band in FIG. 2 represents a section in which (A + B) frame images that are recording candidate images are acquired. The hatched band represents a section in which a frame image that has been temporarily stored in the buffer memory 18 but has been overwritten and erased is acquired.

  The CPU 16 records a predetermined frame image among the recording candidate images on the memory card 50 based on the set recording method. In the digital camera 1 according to the present embodiment, all (A + B) frame images are recorded on the memory card 50.

―Selecting frames suitable for motion expression―
The CPU 16 selects a frame suitable for “motion expression” based on a plurality of frames of images recorded on the memory card 50. Specifically, a frame S used as a still region (background), a frame P acquired before the frame S, and a frame Q acquired after the frame S are (A + B) frames. Select from the frame images.

―Frame S―
The CPU 16 of the digital camera 1 according to the present embodiment uses the frame image designated by the user operation as the frame S. Specifically, the CPU 16 sends an instruction to the recording / reproducing unit 19, and based on the (A + B) frame images recorded in the memory card 50, the thumbnail images of the respective frames are illustrated in FIG. Display them side by side on 15 screens. When (A + B) thumbnail images do not fit on one screen, the CPU 16 causes the liquid crystal monitor 15 to display the thumbnail images intended by the user by scrolling the screen according to the user operation.

  When a plurality of sets of (A + B) frame images are recorded on the memory card 50, the CPU 16 causes the liquid crystal monitor 15 to display a set of thumbnail images indicated by an operation signal from the touch operation member 15a.

  On the screen of the liquid crystal monitor 15 illustrated in FIG. 3, the user selects a frame to be used as a still area (background) for “motion expression” by a touch operation. The CPU 16 sets a frame image corresponding to the thumbnail image X indicated by the operation signal from the touch operation member 15a as a frame S. When determining the frame S, the CPU 16 sends an instruction to the recording / reproducing unit 19 to display an image of the frame S read from the memory card 50 on the screen of the liquid crystal monitor 15 as illustrated in FIG.

  The user selects an object to be moved on the stationary region (background) during “motion expression” by touch operation on the screen of the liquid crystal monitor 15. The CPU 16 uses the operation signal from the touch operation member 15 a in the frame S. A mark m indicating a corresponding rectangular area is displayed on the image of the frame S, and a range surrounded by the mark m is set as a moving object area.

-Frame P, Frame Q-
The CPU 16 selects the frame P from the frame image acquired before the frame S in time. Further, the CPU 16 selects the frame Q from the frame image acquired after the frame S in terms of time. FIG. 5A shows a frame f1, a frame f2,..., A frame fA, a frame f (A + 1), a frame f (A + 2),. It is a figure which illustrates the image of frame f (A + B). In this example, the frame f (A + 1) corresponds to the frame S. Therefore, the frame f1 to the frame fA correspond to the frame P candidates, and the frame f (A + 2) to the frame f (A + B) correspond to the frame Q candidates.

  The moving object region m (A + 1) in the frame f (A + 1) (that is, the frame S) in FIG. 5A corresponds to the moving object region determined in FIG. CPU16 extracts a moving body area | region by a template matching method from other frame images other than the frame f (A + 1) (namely, frame S) of Fig.5 (a), for example using the moving body area | region m (A + 1) as a template. In FIG. 5A, m1, m2,... MA, m (A + 1),..., M (A + B) represent moving object regions in each frame.

  FIG. 5B is a diagram illustrating a moving object area m (A + 1) and a non-moving object area (that is, a stationary area) in the frame f (A + 1) (that is, the frame S). A hatched area represents a non-moving object area. Next, the CPU 16 extracts two or more feature points in the non-moving object region from which each moving object region is excluded from each frame image of FIG.

  In FIG. 5, a plurality of black dots represent feature points. For example, Lowe, David G. (1999). "Object recognition from local scale-invariant features". Proceedings of the International Conference on Computer Vision. 2. pp.1150 Known techniques disclosed in -1157. And the like can be used.

  The CPU 16 associates feature points between the frame f1, the frame f2,..., The frame fA, the frame f (A + 1), the frame f (A + 2)..., And the frame f (A + B), and a non-moving object region (that is, a stationary region). The image is aligned between frames with reference to. Further, the CPU 16 extracts two or more feature points in the moving object region. Then, based on the positional relationship between the feature points of the non-moving object region and the feature points of the moving object region in each frame, the position of the moving object region in each frame is changed to, for example, the characteristics of the non-moving object region in the frame f (A + 1) (ie, frame S). Represents a point.

  In the case of FIG. 5 (a), in the frame that is a candidate for the frame P, a difference is generated between each adjacent frame. The CPU 16 selects a frame fA in which the relative distance between the object and the background feature point is within a predetermined range as the frame P. A plurality of frames P may be selected. For example, when the relative distance between the object and the background feature point in the frame f2 is within a predetermined range, the frame f2 and the frame fA are selected as the frames P, respectively. In general, when the number of frames constituting the frame P is increased, the “motion expression” can be smoothed. Note that if the relative distance between the target object and the background feature point exceeds a predetermined range, the target object may come off the screen of the frame S, and thus is not suitable as the frame P. This will be described below.

  For example, in FIG. 5 (a), the X coordinate in the screen of the feature point of the background (left of the top of the mountain) of the frame f (A + 1) used as the frame S is XA1, and the corresponding feature point of the frame f1 Let the X coordinate in the screen be X1. Here, assuming that the X coordinate reference (origin) is the left edge of the screen of each frame, it is assumed here that the coordinate value of X1 is larger than the coordinate value XA1. When the object in the frame f1 is combined with the background of the frame f (A + 1), the relative positional relationship between the object to be combined and the feature point of the background of the frame f (A + 1) is the object in the frame f1. The composition processing is performed so as to be equal to the relative positional relationship between the object and the background feature point in the frame f1. Therefore, for example, when the coordinate value of X1 is larger than the coordinate value XA1, and the X coordinate in the screen of the object in the frame f1 is small (when it is present near the left end of the screen), the frame f (A + 1) The object (m1) in () protrudes from the frame f (A + 1). Whether or not such a protrusion occurs when the object is combined with the frame S is that the difference between the X coordinate values of the background feature point and the object in the frame f1 is larger than the coordinate value XA1. It can be judged by whether or not. That is, when the difference between the X coordinate values of the background feature point and the object in the frame f1 is larger than the coordinate value XA1 of the corresponding feature point on the frame f (Al + 1), the object in the frame f (A + 1) Things will protrude from the screen. In the above example, the protrusion of the left direction of the screen has been described.

  In the case of FIG. 5 (a), even in a frame that is a candidate for the frame Q, a difference is generated between adjacent frames. In addition, since the objects in the frames other than the frame S used as the still region (background) are the same aircraft and have the same shape and the same size, the degree of similarity between the objects is high and becomes a predetermined value or more. . The CPU 16 selects frames f (A + 2) and f (A + 3) whose relative distance between the object and the background feature point is within a predetermined range as the frame Q. As in the case of the frame P, if the number of frames constituting the frame Q is increased, the “motion expression” can be smoothed. Similarly to the description of the frame P described above, if the relative distance between the target object and the background feature point exceeds a predetermined range, the target object may be off the screen of the frame S. Not suitable as Q. When such a frame is selected as the frame Q or the above-described frame P, a warning can be given to the operator. In addition, after the frame S and the object are designated, the CPU 16 automatically determines whether the frame P and the frame Q are appropriate, and then the CPU 16 automatically selects the frame P and the frame Q. Good.

  The CPU 16 combines the frame S, the frame P (frame fA), and the frame Q (frame f (A + 2) and frame f (A + 3)) selected as described above, and displays the moving object region on the still image (frame S). Express dynamically. Specifically, in place of the moving object region constituting a part of the frame S, the moving object region of another frame is inserted and combined while sequentially replacing the corresponding positions in the frame S at predetermined time intervals (hereinafter referred to as sequential replacement combining). Called). For this purpose, the CPU 16 first extracts a moving object region from the image of the frame S and the selected plurality of frame images by a template matching method. FIG. 5C is a diagram illustrating the extracted moving object region. Since these moving object regions have already been extracted when performing a frame selection suitable for the “motion expression” described above, this information can be used.

-Image generation for "motion expression"-
The CPU 16 replaces the moving object area m (A + 1) in the frame S (f (A + 1)) illustrated in FIG. 5B with the moving object area mA in the frame P illustrated in FIG. The region m (A + 2) and the moving object region m (A + 3) are sequentially replaced and synthesized at corresponding positions in the frame S every predetermined time. The corresponding position in the frame S is the position of the moving object region obtained when the frame suitable for “motion expression” is selected, and is the position expressed with reference to the non-moving object region in the frame S. As a result, images corresponding to the frame fA, the frame f (A + 1), the frame f (A + 2), and the frame f (A + 3) in FIG. 5A are obtained in order. When these images are reproduced and displayed in order on the liquid crystal monitor 15, it seems as if the time is moving only in the moving object region (aircraft) in the world where the time of still images (= f (A + 1)) has stopped. Can express a heterogeneous world.

  Here, when the object moves within the screen, the position of the moving object region that is sequentially replaced and synthesized with the frame S (f (A + 1)) is different from the position of the moving object region m (A + 1) of the frame S. In the frame S, replacement synthesis is sequentially performed at two locations. In the first synthesis, an image of the moving body region of the frame P or the frame Q is sequentially replaced and synthesized at a position corresponding to the moving body region of the frame P or the frame Q in the frame S.

  In the second synthesis, the non-moving object region (background) image is sequentially replaced and synthesized at the position of the moving object region m (A + 1) in the frame S. For example, a background image corresponding to the position of the moving object area m (A + 1) of the frame S is extracted from an image of a frame other than the frame S (for example, the frame f1 or the frame f (A + B)). Are sequentially replaced and synthesized at the position where the moving object region m (A + 1) is present. The CPU 16 generates an image for “motion expression” by the sequential replacement synthesis described above.

  The CPU 16 dynamically displays the moving object region on the still image (frame S) by reproducing and displaying the generated image for “motion expression” on the liquid crystal monitor 15 every predetermined time. If the time interval for switching the position of the moving object region from mA → m (A + 1) → m (A + 2) → m (A + 3) → mA → m (A + 1). Longer times result in slower “motion expressions”.

―Recording data necessary for motion expression―
The CPU 16 records data necessary for “motion expression” on the memory card 50 in accordance with the recording instruction. The data to be recorded includes, for example, each image of frame S, frame P, and frame Q, the range of the moving object region in these images, the coordinate position, and the control (shift) value for aligning from frame P and frame Q to frame S Information on whether or not loop reproduction (repetitive reproduction) is necessary.

―Explanation of flowchart―
FIG. 6 is a flowchart for explaining the flow of processing executed by the CPU 16 during the above-described “motion expression”. The CPU 16 activates the process shown in FIG. 6 when the “pre-shooting shooting mode” is set and the “motion expression” process is turned on. In step S1 of FIG. 6, the CPU 16 starts acquisition of a pre-taken image (that is, shooting standby processing), and proceeds to step S2. As a result, the pre-taken frame image is accumulated in the buffer memory 18 and is sequentially displayed on the liquid crystal monitor 15.

  In step S2, the CPU 16 performs AE (automatic exposure) calculation and AF (automatic focus adjustment) processing based on the pre-captured image data, and proceeds to step S3. In step S3, the CPU 16 determines whether or not the release button has been fully pressed. The CPU 16 makes an affirmative decision in step S3 when an on signal from the full push switch 20b is input, and proceeds to step S4. If an on signal from the full push switch 20b is not inputted, the CPU 16 makes a negative decision in step S3. To return to step S2. When returning to step S2, the above-described processing is repeated.

  In step S4, the CPU 16 starts acquisition of a later shot image and proceeds to step S5. The post-taken images are one sheet acquired in response to the full press operation and (B-1) sheets acquired thereafter. In step S5, the CPU 16 records (A + B) frame images stored in the buffer memory 18 on the memory card 50, and then proceeds to step S6.

  In step S6, the CPU 16 determines whether or not an end instruction has been issued. When an operation signal for switching to the “normal photographing mode” is input from the operation member 20 or the touch operation member 15a, the CPU 16 makes a positive determination in step S6 and ends the process of FIG. If the switching operation signal for switching to the “normal shooting mode” is not input from the operation member 20 or the touch operation member 15a, the CPU 16 makes a negative determination in step S6 and returns to step S1. When returning to step S1, the above-described processing is repeated.

  A flow of processing executed by the CPU 16 for “movement expression” will be described with reference to a flowchart illustrated in FIG. 7. When the CPU 16 is in the reproduction mode and the “motion expression” process is turned on, the CPU 16 starts the process shown in FIG.

  In step S51 of FIG. 7, the CPU 16 determines a still image frame (frame S) from the (A + B) frame images, and proceeds to step S52. In step S52, the CPU 16 determines a moving object region to be dynamically expressed on a still image (frame S) during reproduction, and proceeds to step S53.

  In step S53, the CPU 16 aligns each frame image shown in FIG. 5A and proceeds to step S54. The alignment is performed based on the positions of two or more feature points extracted in the non-moving object region excluding the moving object region from the frame image.

  In step S54, the CPU 16 extracts a moving object region from each frame image by the template matching method, and proceeds to step S55. In step S55, the CPU 16 obtains the position of the moving object region of each frame based on the position of the non-moving object region (background) in the still image (frame S), and proceeds to step S56.

  In step S56, the CPU 16 is a frame image acquired temporally before the still image (frame S), and the relative distance between the target object and the background feature point in the frame is within a predetermined range. The frame P is extracted and the process proceeds to step S57.

  In step S57, the CPU 16 is a frame image acquired temporally after the still image (frame S), and the relative distance between the target object and the background feature point in the frame is within a predetermined range, In addition, a frame Q in which the similarity between the object in the frame and the object in the frame P is equal to or greater than a predetermined value is extracted, and the process proceeds to step S58.

  In step S58, the CPU 16 determines whether or not a predetermined condition is satisfied. For example, if the frame P and the frame Q are substantially the same as the frame S and there is no change, and the moving object area cannot be expressed dynamically even if reproduced and displayed, the CPU 16 makes a negative determination in step S58 and returns to step S56. The frame P and the frame Q are extracted again. When the frames P and Q are moderately different from the frame S and the objects in the frames P and Q have a predetermined relative distance from the objects in the frame S, the CPU 16 makes a positive determination in step S58. Then, the process proceeds to step S59.

  In step S59, the CPU 16 sequentially substitutes and combines the moving object regions illustrated in FIG. 5C to the corresponding positions in the frame S at predetermined time intervals, and proceeds to step S60.

  In step S60, the CPU 16 determines whether an end instruction has been issued. When the end operation is performed, the CPU 16 makes an affirmative determination in step S60 and ends the process of FIG. If the end operation is not performed, the CPU 16 makes a negative determination in step S60 and returns to step S51.

According to the first embodiment described above, the following operational effects can be obtained.
(1) The digital camera 1 has a first moving body region m (A + 1) that changes with time on an image of one frame f (A + 1) among the images f1 to f (A + B) of a plurality of frames taken in time series. CPU 16 for obtaining the position information of the area corresponding to the first moving body area m (A + 1) designated by the CPU 16 in each of the images f1 to f (A + B) of the plurality of frames. CPU 16 that obtains time-series images mA-m (A + 3) for the first moving body region m (A + 1) based on the positional information and the images f1-f (A + B) of the plurality of frames, and images f1- With respect to the first moving body region m (A + 1) acquired by the CPU 16 with respect to the still image (shaded region in FIG. 5B) generated based on one frame image f (A + 1) of f (A + B). Time series Sequentially replacing the image mA~m (A + 3) comprises a CPU16 to synthesize, the. Thus, when a part of a still image is dynamically expressed, it is possible to appropriately select which region of the still image is to be expressed.

(2) In the digital camera 1 of the above (1), the one frame image f (A + 1) used for the instruction of the first moving object area m (A + 1) in the CPU 16 and the still image (FIG. ) Is the same as the one-frame image f (A + 1) used to generate the hatched area), and therefore, the process of dynamically expressing a part of the still image can be simplified as compared with the case where the image f (A + 1) is not the same.

(3) In the digital camera 1 of (1) above, the CPU 16 further uses the position information of the first moving body areas mA to m (A + 3) in each frame to generate a still image (shaded area in FIG. 5B). The one frame image f (A + 1) is converted into position information with reference, and the CPU 16 determines the time for the first moving body region m based on the converted position information and the plurality of frame images fA to f (A + 3). The series images mA to m (A + 3) are acquired, and the CPU 16 applies the time series images mA to the still images (shaded areas in FIG. 5B) acquired by the CPU 16 based on the converted position information. m (A + 3) is sequentially substituted and synthesized. Thereby, the processing time for dynamically expressing a part of a still image can be shortened.

(4) In the digital camera 1 of the above (1), the CPU 16 further includes a first moving object region m (on the one-frame image f (A + 1) used for generating a still image (shaded area in FIG. 5B). After aligning each of the images f1 to f (A + B) of a plurality of frames using the region excluding A + 1) as a reference, the first moving object region m of each of the images f1 to f (A + B) of the plurality of frames is aligned. The position information is acquired, and the CPU 16 determines the first moving object region m based on the position information from the image of the frame including the first moving object region m (A + 1) after alignment among the images f1 to f (A + B) of the plurality of frames. The time series images mA to m (A + 3) are obtained, and the CPU 16 obtains the time series images mA to m obtained by the CPU 16 for the still image (shaded area in FIG. 5B) based on the position information. (A + 3) is sequentially substituted and synthesized. Thereby, the processing time for dynamically expressing a part of a still image can be shortened.

(Modification 1)
In the above description, an example has been described in which the CPU 16 of the digital camera 1 determines an area (object) to be dynamically expressed on a still area (background) for “motion expression” based on a user's touch operation. . If the user accidentally touches an area where the target object does not exist, the CPU 16 may display a message to that effect on the screen of the liquid crystal monitor 15. In this case, whether or not the object (aircraft) is included in the rectangular area (moving object area m) based on the touch operation on the screen of the liquid crystal monitor 15 is determined based on the frame S and any frame image arranged in time series. Is determined based on whether or not there is a difference between the rectangular areas. That is, after aligning the background region between the frame images arranged in time series, the difference between both frames is taken, and if there is a difference, it is determined that the object is included, and the difference has occurred. If not, it is determined that the object is not included.

(Modification 2)
In the above description, when the CPU 16 of the digital camera 1 determines an area (object) to be dynamically expressed on the still area (background) for “motion expression” based on the touch operation of the user, When a touch operation is performed on the screen of the liquid crystal monitor 15, a rectangular area based on the touch operation is displayed with a mark m. Instead of this, the CPU 16 may present a candidate for an area (object) to be dynamically expressed on a static area (background) for “motion expression” before the touch operation by the user.

  The CPU 16 of Modification 2 automatically searches for an object (aircraft) in the frame S in a state where the reproduced image of the frame S is displayed on the liquid crystal monitor 15, and displays the area including the detected object with the mark m. . When the presented candidate (mark m) is acceptable, the user touches the mark m. When the position of the mark m is touch-operated, the CPU 16 determines the area surrounded by the mark m as an area (object) that is dynamically expressed on a stationary area (background) for “motion expression”.

  The search for the target object (aircraft) in the frame S may be performed by searching for an area where a difference occurs between the frame S and the frame images arranged in time series. In other words, after aligning the background area between time-series frame images and taking the difference between the two frames, it is determined that the object is included in the area where the difference occurs, and the difference occurs. If not, it is determined that the object is not included.

(Modification 3)
In the above-described second modification, an area including an object automatically detected by the CPU 16 without a touch operation by the user is dynamically expressed on a stationary area (background) for “motion expression”. (Object) may be determined. The CPU 16 of the modification 3 automatically searches for and detects an object (aircraft) in the frame S in a state where the reproduction image of the frame S is displayed on the liquid crystal monitor 15 by, for example, recognition processing using image data or the like. A region including the target object is displayed with a mark m. The CPU 16 determines an area (target object) that is dynamically expressed on a static area (background) for “motion expression” for the area surrounded by the mark m.

(Modification 4)
In the above description, an example has been described in which the CPU 16 of the digital camera 1 determines an area (object) to be dynamically expressed on a still area (background) for “motion expression” based on a user's touch operation. . It is good also as a structure which CPU16 extracts automatically the area | region (object) to express this dynamically. For example, assuming that four feature points near the screen edge of the frame S are included in the background area, the image of each frame is aligned based on the four feature points near the screen edge of the frame S. Thus, an area (for example, a rectangular area) including an area of the moving object (object moving area) in each frame image is extracted from each frame image. Here, the region of the moving object can be extracted by calculating a difference between adjacent frame images subjected to the above-described alignment and obtaining the sum region. The area including the area may include a background area. In other words, with such a configuration, the size of the area to be sequentially replaced and synthesized on the frame S is the same, and the position of the area to be sequentially replaced and synthesized on the frame S can be the same position. Further, the positions of the regions to be sequentially replaced and synthesized on the frame S can also be set to the same position in terms of the coordinate system with the frame S as a reference, and an image for “motion expression”. Is easy to generate.

  In this case, if there is a moving area other than the object in the area including the area of the moving object, the moving area also changes with time as the object changes. In order to avoid such a situation, a difference calculation is performed between areas including the moving object in each frame, and it is determined whether or not a difference of a predetermined value or more is generated in an area other than the object. To do. When a difference of a predetermined value or more has occurred, a region in which a difference of a predetermined value or more is excluded from the region including the region of the moving object, or notified to the operator. You can also. In addition, in the difference calculation between each frame area including the above moving object, alignment is not necessary.

(Modification 5)
In the above description, if the relative distance between the target object and the predetermined position in the background exceeds the predetermined range, the target object may come off the screen of the frame S, so the frame P or the frame Q As explained as not suitable. With such a configuration, an object is always present in an image generated for “motion expression”. However, for example, when this method is used to repeatedly reproduce an image generated for “motion expression”, from the position of the object in frame f (A + 3) in FIG. 5 to the position of the object in frame fA. The position of the object will change abruptly. That is, for example, when the above-described repeated reproduction is performed, it is preferable that the first image and the last image among images generated for “motion expression” are similar. By adopting such a configuration, it is possible to reduce a sense of discomfort during repeated reproduction.

  Here, a frame in which the relative distance between the target object and a predetermined position in the background exceeds a predetermined range is identified, and the frame is set as the first frame of the frame P and the last frame of the frame Q. In the image generated for “expression”, an image in which no object exists on the frame S can be obtained. When the image for “motion expression” generated in this way is repeatedly reproduced, for example, “motion expression” is started from a frame in which the object does not exist, and again after the object appears in the screen. When the frame in which the object does not exist is reached, the first reproduction by “motion expression” ends. Thereafter, the second “motion expression” is started again from the frame in which the target object does not exist, and it is possible to reduce a sense of discomfort felt by the viewer.

(Modification 6)
In the second modification, instead of the touch operation by the user, an area (object) to be dynamically expressed on the stationary area (background) may be determined for “motion expression” based on a voice instruction. The CPU 16 of Modification 4 performs voice recognition based on the acoustic data collected by the microphone 22 and converted by the acoustic processing circuit 23. For example, when the candidate of the object is sequentially changed by the operation member and the CPU 16 recognizes the utterance of “OK” by the operator by voice recognition, the CPU 16 moves the area surrounded by the mark “m” The area (object) to be dynamically expressed on the stationary area (background) is determined for “expression”. Further, for example, when an operator's utterance of “aircraft” is recognized in a state where the frame f (A + 1) is displayed on the liquid crystal monitor 15 as any frame or frame S, image data or the like is used. A configuration may be adopted in which an area of an aircraft is recognized and the area is determined as an area (object) that is dynamically expressed on a stationary area (background) for “motion expression”.

(Modification 7)
In the above embodiment, the example in which the frame S is selected by the touch operation from the thumbnail images displayed side by side on the screen of the liquid crystal monitor 15 has been described. However, as described with reference to FIG. The frame image acquired at the timing may be set as the frame S. With such a configuration, an image of “motion expression” is generated based on an image acquired at a timing when the photographer intends to shoot, so a frame image at a timing when the photographer intends to shoot is generated. It is possible to adopt a configuration that is always included in the “motion expression” image.

  Alternatively, a frame image with a characteristic highlight may be selected as the frame S from a plurality of frame images. As a highlight frame image, for example, a frame image acquired when the volume level of sound collected by the microphone 22 is maximum, or a climax frame image detected based on the amount of movement or posture of an object. Can be selected automatically.

  Furthermore, as the frame S, a frame image having the smallest object size may be selected as the frame S. For example, when a frame image having a large object size is selected as the frame S, when the object moves or when the object size decreases, not only the object, The background portion in the area to be sequentially replaced and synthesized is also replaced using the image data of the corresponding area from another frame image taken at a time different from that of the frame S. That is, in the image synthesized with the still area of the frame S in this way, backgrounds photographed at different times are mixed. In the above case, or in the case of the above-described modification example 4, the background area photographed particularly at different times increases, and the object generated on the finally generated “motion expression” image is displayed. There is a high possibility that movement will occur in the background area that is not desired to move. Therefore, if the frame image having the smallest size of the object is selected as the frame S, the possibility that backgrounds photographed at different times are mixed is reduced.

(Second embodiment)
In the second embodiment, two regions (objects) that are dynamically expressed on a static region (background) are provided for “motion expression”. FIG. 8 exemplifies images constituting the frames S, P, and Q selected by the CPU 16 in the same manner as in the first embodiment, based on a time-series frame image obtained by shooting a baseball batter hitting scene. It is a figure to do. In this example, the first object that changes in the screen is a ball.

  In FIG. 8, a frame f (A + 1) corresponds to the frame S determined by the CPU 16 based on the user operation. The mark m (A + 1) represents the first moving body region determined by the CPU 16 based on the operation signal output from the touch operation member 15a by the user's touch operation.

  The frame f (A-1) and the frame fA are images selected by the CPU 16 as the frame P from the frame images acquired temporally before the frame S. The frame f (A + 2) and the frame f (A + 3) are images selected by the CPU 16 as the frame Q from the frame images acquired later than the frame S.

  The CPU 16 extracts the first moving object region from the frame images other than the frame f (A + 1) (that is, the frame S) for the frame images in FIG. 8 arranged in time series. For example, the CPU 16 extracts the first moving body region by the template matching method using the moving body region m (A + 1) of the frame S as a template. In FIG. 8, m (A-1), mA, m (A + 1), m (A + 2), and m (A + 3) represent the first moving body region in each frame.

  The CPU 16 of the digital camera 1 according to the second embodiment further extracts a region that changes in association with the first object (ball) included in the first moving body region from the frame image, and extracts the extracted region as the first region. 2 moving body region. In FIG. 8, the movement direction of the ball is changed at the frame f (A + 1) (that is, the frame S). In the frame f (A + 1), the ball is in contact with the bat. Since the movement of the ball is changed due to the movement of the bat, the CPU 16 determines that the movement of the bat and the movement of the ball are related, and sets the area including the bat as the second moving body area. Note that the CPU 16 determines that both movements are related when the acceleration during movement changes even if the movement direction does not change.

  The CPU 16 extracts the second moving body region (bat) from the frame images other than the frame f (A + 1) (that is, the frame S) for the frame images of FIG. 8 arranged in time series. In the case of this example, since the shape and the position in the screen of the bat change for each frame, a difference detection method is used.

  For the frame images of FIG. 8 arranged in chronological order, the CPU 16 excludes the first moving body regions m (A−1), mA, m (A + 1), m (A + 2), and m (A + 3) of each frame. After performing the alignment in step 1, the bat in the frame image is extracted by taking a difference between adjacent frames. Specifically, a region including a bat is extracted by taking a logical product between difference images that are temporally adjacent and binarized with a predetermined threshold. Then, the CPU 16 specifies a bat by performing a known object recognition process in the extracted area.

  Here, a region where a difference between frames exceeds a predetermined value corresponds to a region including a bat, and a region where the difference between frames is less than a predetermined value corresponds to a region not including a bat. In the case of FIG. 8, in the region including the bat or batter's arm, a difference of a predetermined value or more is detected between frames. The CPU 16 extracts the bat from the region including the bat or batter's arm by object recognition processing.

  Next, the CPU 16 extracts two or more feature points in the non-moving object region excluding the first moving object region and the second moving object region from each frame image of FIG. Each feature point is associated with each other.

  The CPU 16 further extracts two or more feature points in each of the first moving body region and the second moving body region. Then, based on the positional relationship between the feature points of the non-moving object region and the feature points of each moving object region in each frame, the positions of the first moving object region and the second moving object region in each frame are represented by frame f (A + 1) (that is, frame S ) With reference to the non-moving object region.

  The CPU 16 combines the frame S (frame f (A + 1)), the frame P (frame f (A-1) and frame fA), and the frame Q (frame f (A + 2) and frame f (A + 3)) to make a still image. The first moving body area and the second moving body area are dynamically expressed on the image (frame S). Specifically, instead of the first moving body region and the second moving body region in the frame S, the first moving body region and the second moving body region of other frames are sequentially transferred to the corresponding positions in the frame S at predetermined intervals. Substitute synthesis.

  As described above, in the case of generating an image for “motion expression”, two areas (objects) that are dynamically expressed on a stationary area (background) for “motion expression” are provided. A technique for sequentially replacing and synthesizing the moving object region at each corresponding position in the frame S every predetermined time is performed in the same manner as in the first embodiment. The recording of data necessary for “motion expression” is performed in the same manner as in the first embodiment.

  9 corresponds to the first moving body region (ball) and the second moving body region (bat) in the frame S (= f (A + 1)) based on the frame S, the frame P, and the frame Q in FIG. FIG. 6 is a diagram illustrating composite images c (A−1), cA, c (A + 1), c (A + 2), and c (A + 3) that are sequentially replaced and synthesized at a predetermined position.

  The composite image c (A-1) is obtained by replacing the first moving object region (ball) and the second moving object region (bat) of the frame S (= f (A + 1)) with the first image of the frame f (A-1). It is the image which combined 1 moving body area | region (ball) and 2nd moving body area | region (bat). The composite image cA includes the first moving body region (ball) and the second moving body region (ball) and the second moving body region (ball) and the second moving body region (ball) of the frame S (= f (A + 1)). It is the image which synthesize | combined the moving body area | region (bat). The composite image c (A + 1) corresponding to the frame S is the same as the frame f (A + 1).

  In addition, the composite image c (A + 2) is the first image of the frame f (A + 2) instead of the first moving object region (ball) and the second moving object region (bat) of the frame S (= f (A + 1)). It is the image which synthesize | combined the moving body area | region (ball | bowl) and the 2nd moving body area | region (bat). The composite image c (A + 3) is the first moving body region of the frame f (A + 3) instead of the first moving body region (ball) and the second moving body region (bat) of the frame S (= f (A + 1)). It is the image which synthesize | combined (ball) and the 2nd moving body area | region (bat).

  When these composite images c (A-1), cA, c (A + 1), c (A + 2), and c (A + 3) are reproduced and displayed in order on the liquid crystal monitor 15, they are called still images (= f (A + 1)). In the world where time has stopped, it is possible to express a heterogeneous world as if time is progressing only by the first moving body region (ball) and the second moving body region (bat).

According to the second embodiment described above, the following operational effects can be obtained.
(1) The digital camera 1 includes a CPU 16 for instructing a first moving body region (ball) that changes with time on one frame f (A + 1) image of a plurality of frames taken in time series, and a plurality of CPUs 16. In each of the frame images, the CPU 16 that acquires the position information of the first moving body region (ball) instructed by the CPU 16, and the first moving body region (based on the position information acquired by the CPU 16 and the images of the plurality of frames). CPU 16 for obtaining time-series images m (A-1) A to m (A + 3) for the ball), and a still image generated based on one frame image f (A + 1) of the plurality of frames. , And a CPU 16 that sequentially replaces and synthesizes the time-series images m (A-1) to m (A + 3) for the first moving body region (ball) acquired by the CPU 16. Thus, when a part of a still image is dynamically expressed, it is possible to appropriately select which region of the still image is to be expressed.

(2) In the digital camera 1 of the above (1), the CPU 16 further extracts a region that changes in accordance with the change of the first moving body region (ball) based on the images of a plurality of frames, and the extracted region is used as the second moving body. Instructed as an area (bat), the CPU 16 further acquires position information of the second moving body area (bat) instructed by the CPU 16 in each of the images of the plurality of frames, and the CPU 16 acquires the position information acquired by the CPU 16 and A time-series image for the second moving body region (bat) is further acquired based on the images of the plurality of frames, and the CPU 16 is further generated based on the image f (A + 1) of one frame among the images of the plurality of frames. The time-series images for the second moving object region (bat) acquired by the CPU 16 are sequentially replaced with the still image and synthesized. Thereby, the motion of the bat having a causal relationship with the motion of the ball can be appropriately expressed in the still image.

(3) In the digital camera 1 of the above (1) and (2), the CPU 16 designates an area in contact with the first moving body area (ball) in one frame f (A + 1) of the plurality of frames as the second moving body area ( Therefore, the movement of the bat having a causal relationship with the movement of the ball can be appropriately expressed in the still image.

(Modification 8)
In image generation for “motion expression”, sequential replacement synthesis different from that illustrated in FIG. 9 may be performed. 10 corresponds to the first moving body region (ball) and the second moving body region (bat) in the frame S (= f (A + 1)) based on the frame S, the frame P, and the frame Q in FIG. FIG. 6 is a diagram illustrating composite images c (A−1) ′, cA ′, c (A + 1), c (A + 2) ′, and c (A + 3) ′ that are sequentially replaced and synthesized at a predetermined position.

  The composite image c (A-1) ′ includes the second moving object region (bat) of the frame f (A-1) instead of the second moving object region (butt) of the frame S (= f (A + 1)). This is a composite image. The synthesized image cA ′ is an image obtained by synthesizing the second moving object region (bat) of the frame fA instead of the second moving object region (bat) of the frame S (= f (A + 1)). The composite image c (A + 1) corresponding to the frame S is the same as the frame f (A + 1).

  The composite image c (A + 2) ′ is a composite of the first moving body region (ball) of the frame f (A + 2) instead of the first moving body region (ball) of the frame S (= f (A + 1)). It is an image. The synthesized image c (A + 3) ′ is an image obtained by synthesizing the first moving body region (ball) of the frame f (A + 3) instead of the first moving body region (ball) of the frame S (= f (A + 1)). is there.

  When these images c (A-1) ′, cA ′, c (A + 1), c (A + 2) ′, and c (A + 3) ′ are sequentially reproduced and displayed on the liquid crystal monitor 15, a still image (= f (A + 1) is obtained. )) In the world where time has stopped, the time is advanced by the second moving object region (bat) before the bat and the ball contact, and after the contact between the bat and the ball, the first moving object region (ball). It can express a heterogeneous world as if time only progressed.

(Modification 9)
In image generation for “motion expression”, sequential replacement synthesis different from that of the modification 8 may be performed. 11 corresponds to the first moving body region (ball) and the second moving body region (bat) in the frame S (= f (A + 1)) based on the frame S, the frame P, and the frame Q in FIG. FIG. 6 is a diagram illustrating composite images c (A−1) ′, cA ′, c (A + 1), c (A + 2), and c (A + 3) that are sequentially replaced and synthesized at a predetermined position.

  The composite image c (A-1) ′ includes the second moving object region (bat) of the frame f (A-1) instead of the second moving object region (butt) of the frame S (= f (A + 1)). This is a composite image. The synthesized image cA ′ is an image obtained by synthesizing the second moving object region (bat) of the frame fA instead of the second moving object region (bat) of the frame S (= f (A + 1)). The composite image c (A + 1) corresponding to the frame S is the same as the frame f (A + 1).

  In addition, the composite image c (A + 2) is the first image of the frame f (A + 2) instead of the first moving object region (ball) and the second moving object region (bat) of the frame S (= f (A + 1)). It is the image which synthesize | combined the moving body area | region (ball | bowl) and the 2nd moving body area | region (bat). The composite image c (A + 3) is the first moving body region of the frame f (A + 3) instead of the first moving body region (ball) and the second moving body region (bat) of the frame S (= f (A + 1)). It is the image which synthesize | combined (ball) and the 2nd moving body area | region (bat).

  When these images c (A-1) ′, cA ′, c (A + 1), c (A + 2), and c (A + 3) are sequentially reproduced and displayed on the liquid crystal monitor 15, a still image (= f (A + 1)) In the world where time has stopped, before the bat and the ball contact, the time advances by the second moving object region (bat), and after the bat and the ball contact, the first moving object region (ball) and the first moving region. It is possible to express a heterogeneous world as if time progressed by two moving object regions (bats).

(Modification 10)
In the second embodiment described above, it is determined that the movement of the bat and the movement of the ball are related based on the contact between the ball and the bat in the frame f (A + 1). In addition to or instead of such contact determination of two objects, it may be determined whether or not the movements of the two moving object regions are related based on the sound level indicated by the sound data.

  In the case of the first moving body region (ball) and the second moving body region (bat) described above, a striking sound is generated when the two contact each other. The CPU 16 of the modified example 7 determines the sound pressure level based on the sound data collected by the microphone 22 and converted by the sound processing circuit 23. The CPU 16 determines that the movement of the bat and the movement of the ball are related based on the contact between the ball and the bat when the sound pressure level equal to or higher than the predetermined value is determined at the time of the contact.

(Modification 11)
As an example of providing two regions (objects) that are dynamically expressed on a stationary region (background) for “motion expression”, a first object (ball) and a first object that contacts the first object Two objects (bats) were illustrated. In addition, the shadow of the first object (for example, a ball) may be used as the second object. The shadow moves along with the movement of the ball and has a contour shape that depends on the contour shape of the ball. From this, the CPU 16 determines that the movement of the ball and the movement of the shadow are synchronized, and sets the area including the shadow as the second moving object area.

(Modification 12)
As an example of providing two areas (objects) that are dynamically expressed on a stationary area (background) for “motion expression”, a first moving object area that includes a first object that changes and a first area that changes. The second moving object region including the two objects may be adjacent to each other. For example, when a specific ingredient boiled in a pan is shaking, the specific ingredient is set as a first object, and steam rising from the specific ingredient is set as a second object. CPU16 determines with the movement of both (a 1st target object and a 2nd target object) related, when the direction of the steam which stands | starts up by the direction of a specific material changing changes. In the case of the modified example 9, it is possible to express a heterogeneous world as if the time progressed only by the specific ingredient and the steam rising from the specific ingredient in the still-time world of still images.

(Modification 13)
In the above description, an example in which “motion expression” is performed based on a group of images of a plurality of frames arranged in time series acquired in the “pre-shooting shooting mode” has been described. However, a plurality of image groups are not necessarily in the “pre-shooting shooting mode”. It does not have to be a group of images taken in. For example, “motion expression” may be performed based on a group of images of a plurality of frames arranged in time series obtained by continuous shooting.

(Modification 14)
The frame rate for acquiring a group of images of a plurality of frames may not be 120 frames per second as described above, but may be 30 frames per second or 5 frames per second. This frame rate may be appropriately set according to the moving speed of the moving subject (target object). It should be noted that so-called slow playback can be performed by lowering the frame rate at the time of playback display than the frame rate at the time of acquisition.

(Modification 15)
Data necessary for “motion expression” may be recorded as a moving image file. In the moving image file, the fact that the video synthesized for “motion expression” can be repeatedly reproduced is also recorded. Specifically, as a moving image file, moving image compressed data of a moving object region, compressed image data of a non-moving object region (a still region as a background), and other necessary data (coordinates of moving region data to be sequentially replaced and synthesized into a stationary region) Position, necessity of repeated playback).

(Modification 16)
In the above description, an example in which the digital camera 1 is used as an electronic device has been described. However, a multi-function mobile phone or a tablet computer may be used.

(Modification 17)
In the above-described embodiment, the example in which the “motion expression” process is performed by the digital camera 1 based on the image group of a plurality of frames arranged in time series has been described. However, the posterior can be performed using a personal computer, a tablet computer, or the like. An image processing apparatus that performs “motion expression” processing may be configured. In this case, an image group of a plurality of frames arranged in time series is stored as image data acquired by the digital camera 1.

  Then, by causing the computer 200 shown in FIG. 12 to execute a program that performs the processing of the flowchart illustrated in FIG. 7, an image processing apparatus that dynamically expresses a subject (object) that changes in a still image is configured. . When a program is used by being loaded into the computer 200, the program is loaded into the data storage device of the computer 200 and then used as an image processing device by causing the program to be executed.

  The loading of the program to the computer 200 may be performed by setting the storage medium 204 such as a CD-ROM storing the program in the computer 200 or by loading the computer 200 by a method via the communication line 201 such as a network. Also good. When passing through the communication line 201, the program is stored in the hard disk device 203 of the server (computer) 202 connected to the communication line 201. The program can be supplied as various forms of computer program products such as provision via the storage medium 204 or the communication line 201.

  The above description is merely an example, and is not limited to the configuration of the above embodiment.

DESCRIPTION OF SYMBOLS 1 ... Digital camera 12 ... Image pick-up element 14 ... Image processing part 15 ... Liquid crystal monitor 15a ... Touch operation member 16 ... CPU
17 ... Flash memory 18 ... Buffer memory 19 ... Recording / reproducing unit 20 ... Operation member 50 ... Memory card 200 ... Computer 204 ... Storage medium

Claims (1)

An instruction unit for instructing a first change area to be changed with time on at least one of a plurality of captured images;
A specifying unit that specifies a second change region that changes in synchronization with a change in the first change region based on the plurality of images;
A position information acquisition unit that acquires position information on the plurality of images corresponding to the first change region specified by the instruction unit and the second change region specified by the specifying unit;
An image acquisition unit that acquires an image corresponding to the first change region and an image corresponding to the second change region in the plurality of images based on the position information acquired by the position information acquisition unit;
An image synthesis unit that synthesizes an image acquired by the image acquisition unit with respect to a reference image of the plurality of images;
An image processing apparatus comprising:

Images