JP4930905B2 - Imaging apparatus and program thereof - Google Patents

Description

  The present invention relates to a photographing apparatus and a program thereof, and more particularly, to a photographing apparatus having a function for supporting creation of an animation moving image by a live action and a program thereof.

  A camera that supports creation of an animation movie, and a technique for guiding and displaying the position of a moving object in the animation movie in a shooting frame when shooting each frame (frame) constituting the animation movie is known ( Patent Document 1).

JP Patent Publication No. 2003-37808

  However, according to the technique of Patent Document 1, since only the position of the object specified by the user or the position of the object in the immediately preceding frame (frame) is guided and displayed, the object is in a desired state over a plurality of frames (frames). In order to move smoothly, it was impossible to shoot with an object installed.

  Therefore, the present invention has been made in view of such conventional problems, and an object of the present invention is to provide a photographing apparatus and a program for assisting so that a live-action animation moving image in which a subject moves smoothly can be easily created. To do.

In order to achieve the above object, a photographing apparatus according to a first aspect of the present invention comprises photographing means and an electronic viewfinder that support creation of a live-action animation movie that expresses the movement of a predetermined subject using a plurality of photographed image data. A photographing device,
Generating means for generating a parameter indicating the state of the predetermined subject in correspondence with each frame constituting the live-action animation video;
Based on the parameters generated by the generating means, guidance display control means for guiding and displaying the planned shooting state of the subject corresponding to the frame to be shot on the electronic viewfinder when shooting each frame;
Moving image generation means for generating moving image data based on image data of each frame imaged by the imaging means corresponding to the guidance display by the guidance display control means;
It is provided with.

Further, for example, as described in claim 2, a specifying unit that specifies the movement of the predetermined subject in a three-dimensional space corresponding to a background portion of the live-action animation movie,
The generating means includes
The movement of the predetermined subject in the three-dimensional space specified by the specifying means is converted into the state of the predetermined subject corresponding to each frame in accordance with timing information at which each frame in the live-action animation movie is reproduced. By doing so, the parameter may be generated.

In addition, for example, as described in claim 3, an input unit for a user to input a parameter indicating a state of the predetermined subject captured in a part of the frames constituting the animation moving image. With
The specifying means is:
The movement of the predetermined subject in the three-dimensional space corresponding to the background portion of the live-action animation movie is specified based on the parameter indicating the state of the predetermined subject that is captured in a part of the frames input by the input unit You may make it do.

For example, as described in claim 4, the specifying unit includes:
3 corresponding to the background portion of the live-action animation video so that the movement of the predetermined subject is smoothed based on the parameter indicating the state of the predetermined subject that is captured in a part of the frames input by the input means. The movement of the predetermined subject in the dimensional space may be specified.

For example, as specified in claim 5, the specifying unit includes:
The parameter indicating the state of the predetermined subject captured in a part of the frames input by the input unit is corrected so that the movement of the predetermined subject is smooth, and the background corresponding to the background portion of the live-action animation moving image is corrected. The movement of the predetermined subject in the three-dimensional space may be specified.

Further, for example, as described in claim 6, based on actually captured image data, detection means for detecting a parameter indicating a photographing state of the predetermined subject corresponding to the actually photographed frame With
The specifying means is:
Based on the parameter detected by the detection means, the movement of the predetermined subject in the three-dimensional space corresponding to the background portion of the live-action animation movie may be newly specified.

For example, as described in claim 7, the specifying unit includes
Identifying a direction and speed of movement of the predetermined subject based on at least two frame states immediately before the current frame for generating a parameter detected by the detection means;
The generating means includes
The parameter of the current frame may be generated based on the detected direction and speed of movement.

For example, as described in claim 8, the specifying unit includes:
Identifying the direction and acceleration of movement of the subject based on at least the state of the three frames immediately before the current frame for generating the parameters detected by the detecting means;
The generating means includes
The parameter of the current frame may be generated based on the detected direction of movement and acceleration.

For example, as described in claim 9, the specifying means includes
Identifying a direction and a speed at which the direction of the predetermined subject changes based on at least two frame states immediately before the current frame for generating a parameter detected by the detection unit;
The generating means includes
The parameter of the current frame may be generated based on the direction and speed in which the detected direction changes.

Further, for example, as described in claim 10, an input unit for a user to input a parameter indicating a state of the predetermined subject captured in a part of the frames constituting the animation moving image. With
The generating means includes
A parameter indicating the state of the subject corresponding to each frame other than the part of the frame so that the movement of the subject is smooth based on the parameter corresponding to the part of the frame input by the input unit. May be generated.

For example, as described in claim 11, the generation unit includes:
The parameter input by the input unit may be corrected so that the predetermined subject moves smoothly, and a parameter indicating the state of the subject in each frame may be generated.

Further, for example, as described in claim 12, detection means for detecting a shooting state of the predetermined subject corresponding to the actually shot frame based on the actually shot image data;
Correction means for correcting a parameter of a frame portion that has not yet been photographed out of the parameters generated by the generation means based on the photographing state of the predetermined subject detected by the detection means;
You may make it provide.

Further, for example, as described in claim 13, a detection unit that detects a shooting state of the predetermined subject actually shot based on image data of each frame actually shot,
The generating means includes
A parameter of a frame portion that has not yet been generated may be generated based on the shooting state of the predetermined subject in each frame detected by the detection means.

For example, as described in claim 14, the generation unit includes:
The direction and speed of movement of the predetermined subject are detected based on the state of at least two frames immediately before the current frame for generating a parameter detected by the detecting means, and the direction and speed of the detected movement are detected. The parameters of the current frame may be generated based on

For example, as described in claim 15, the generation unit includes:
Based on at least the state of the three frames immediately before the current frame for generating the parameter detected by the detecting means, the direction and acceleration of the predetermined movement are detected, and the detected direction and acceleration of the movement are detected. The parameters of the current frame may be generated based on

For example, as described in claim 16, the generation unit includes:
Based on the state of at least two frames immediately before the current frame for generating the parameter detected by the detecting means, the direction and speed of the predetermined subject change are detected, and the detected direction changes. The parameters of the current frame may be generated based on the direction and speed of the current frame.

Further, for example, as described in claim 17, the detection unit includes:
The photographing state of the subject may be detected by detecting the feature point of the subject.

For example, as described in claim 18, the detection unit includes:
A plurality of feature points of the subject may be detected, and the centers of the detected feature points may be detected as the subject position.

For example, as described in claim 19, the detection means includes:
A plurality of feature points of the subject may be detected, and the orientation of the subject may be detected based on the relative positional relationship between the detected feature points.

In order to achieve the above object, a program according to the invention described in claim 20 includes a photographing means and an electronic finder for supporting creation of a live-action animation movie that expresses the movement of a predetermined subject using a plurality of photographed image data. A program for executing a photographing apparatus,
A generation process for generating a parameter indicating the state of the predetermined subject in correspondence with each frame constituting the live-action animation video;
Based on the parameters generated by the generation process, a guidance display control process that guides and displays on the electronic viewfinder the planned shooting state of the subject corresponding to the frame to be shot when shooting each frame;
A moving image generating process for generating moving image data based on the image data of each frame imaged on the imaging element by the imaging unit corresponding to the guidance display by the guidance display control process;
It is characterized by including.

  According to the present invention, it is possible to easily create a live-action animation movie in which a subject moves smoothly.

Hereinafter, the present embodiment will be described in detail with reference to the drawings as an example in which the photographing apparatus of the present invention is applied to a digital camera.
[Embodiment]
A. Configuration of Digital Camera FIG. 1 is a block diagram showing a schematic electrical configuration of a digital camera 1 that implements the photographing apparatus of the present invention.
The digital camera 1 includes a photographing lens 2, a lens driving block 3, an aperture 4, a CCD 5, a driver 6, a TG (timing generator) 7, a unit circuit 8, an image generation unit 9, a CPU 10, a key input unit 11, a memory 12, a DRAM 13, A flash memory 14, an image display unit 15, and a bus 16 are provided.

  The photographing lens 2 includes a focus lens, a zoom lens, and the like that are constituted by a plurality of lens groups (not shown). A lens driving block 3 is connected to the photographing lens 2. The lens driving block 3 includes a focus motor and a zoom motor that drive the focus lens and the zoom lens along the optical axis direction, respectively, and a focus motor driver that drives the focus motor and the zoom motor according to a control signal sent from the CPU 10. And a zoom motor driver (not shown).

The diaphragm 4 includes a drive circuit (not shown), and the drive circuit operates the diaphragm 4 in accordance with a control signal sent from the CPU 10.
The diaphragm 4 is a mechanism that controls the amount of light that enters from the photographing lens 2.

  The CCD 5 is driven by a driver 6 and photoelectrically converts the intensity of light of each color of the RGB value of the subject image at regular intervals and outputs it as a photographic signal to the unit circuit 8. The operation timing of the driver 6 and the unit circuit 8 is controlled by the CPU 10 via the TG 7. The CCD 5 has a Bayer color filter and also functions as an electronic shutter. The shutter speed of the electronic shutter is controlled by the CPU 10 via the driver 6 and TG7.

  A TG 7 is connected to the unit circuit 8, a CDS (Correlated Double Sampling) circuit that holds the photographic signal output from the CCD 5 by correlated double sampling, and an AGC that performs automatic gain adjustment of the photographic signal after the sampling. (Automatic Gain Control) circuit and an A / D converter for converting the analog image pickup signal after the automatic gain adjustment into a digital signal. The image pickup signal output from the CCD 5 passes through the unit circuit 8 as a digital signal. It is sent to the image generator 9.

  The image generation unit 9 performs processing such as γ correction processing and white balance processing on the image data sent from the unit circuit 8, generates a luminance color difference signal (YUV data), and generates the generated luminance color difference. The image data of the signal is stored in the DRAM 13 (buffer memory). That is, the image generation unit 9 performs image processing on the image data output from the CCD 5.

  The CPU 10 is a one-chip microcomputer that has functions of performing imaging control on the CCD 5, recording processing on the flash memory 14, and display processing of image data, and controls each part of the digital camera 1. Further, the CPU 10 includes a clock circuit and has a function as a timer.

  In particular, the CPU 10 has a function for the user to input and specify the movement of a predetermined subject (moving object) in a three-dimensional space corresponding to the background portion of the live-action animation, and the movement of the subject input and specified by the user to each frame. A function for converting to the state of the corresponding subject, a function for transparently displaying the photographing position of the subject of the frame on the image display unit 15 based on the value of each position parameter of the frame, and each of the images photographed corresponding to the transparent display It has a function of generating moving image data based on frame image data.

The input designation function and the conversion function also have a function of performing interpolation and correction so that the user can easily designate the movement of the object through a plurality of frames, and more natural and smooth movement of the object. Various methods can be considered as specific methods for performing interpolation and correction so that such natural and smooth data can be easily input. In this embodiment, a live-action animation movie is configured. When the user inputs a parameter indicating the state of the subject in correspondence with some of the plurality of frames, the parameters of other frames are based on the parameter values of any frame input by the user. Is calculated by a spline function to perform interpolation and correction.

  The key input unit 11 includes a plurality of operation keys such as a shutter button, a mode switching key, a cross key, and a SET key, and outputs an operation signal corresponding to the user's key operation to the CPU 10.

The memory 12 stores a control program and necessary data necessary for the CPU 10 to control each unit, and the CPU 10 operates according to the program. The memory 12 is a rewritable nonvolatile memory.
Further, the memory 12 is provided with a position parameter table for recording the position parameters of the subject captured in each frame constituting the moving image data.

The DRAM 13 is used as a buffer memory for temporarily storing image data sent to the CPU 10 after being photographed by the CCD 5 and also as a working memory for the CPU 10.
The flash memory 14 is a recording medium that stores image data.

  The image display unit 15 includes a color LCD and its driving circuit, and displays the subject photographed by the CCD 5 as a through image when in a photographing standby state, and is read from the flash memory 14 and decompressed when reproducing a recorded image. The recorded image is displayed.

B. Operation of Digital Camera 1 The operation of the digital camera 1 in the embodiment will be described separately for parameter registration and animation moving image shooting.
This parameter registration simply registers the movement of a predetermined subject (moving object) in a three-dimensional space corresponding to the background portion of a live-action animation movie. This refers to registering the movement of a subject using a (position) parameter that indicates the position and orientation of the subject corresponding to the frame. Animation video shooting is not a real-time video recording. Rather than sequentially shooting in real time, the same subject is shot separately for each frame, and moving image data is generated based on the image data obtained by the still image shooting. is there.

B-1. Parameter Registration Operation First, the parameter registration operation will be described with reference to the flowchart of FIG.
When the parameter registration mode is set by the user operating the mode switching key of the key input unit 11, the CPU 10 selects the first position parameter (step S1). Here, the position of the subject is shown in three dimensions (x, y, z), and the position parameters are an x position parameter indicating a position in the x direction, a position in the y direction as a y position parameter, and a position in the z direction. And a z-position parameter indicating the position. Here, since the order in which the position parameters are selected is the order of x, y, and z, the x position parameter is selected as the first position parameter in step S1.

FIG. 3A shows an example of a three-dimensional state represented by x, y, and z.
As can be seen from FIG. 3A, a certain position is (x, y, z) = (0, 0, 0), and the x, y, and z axes extend from the position. Here, there is no minus direction of the z-axis because it is assumed that the imaging surface of the CCD 5 is on a plane formed by the x-axis and the y-axis and is oriented in the plus direction of the z-axis. This is because the negative direction of the z-axis is not photographed. Further, the center of the photographing surface of the CCD 5 is (x, y, z) = (0, 0, 0). The positive direction of the x axis is the right direction, the negative direction is the left direction, the positive direction of the y axis is the upward direction, and the negative direction is the downward direction.

Next, the CPU 10 displays a position graph showing the relationship with the currently selected position parameter in each frame (step S2).
FIG. 3B is a diagram illustrating an example of a state of the displayed position graph.
As can be seen from FIG. 3B, the horizontal axis indicates the number of frames such as the first frame and the second frame, and the vertical axis indicates the value of the position parameter. Here, since the x position parameter is selected, the vertical axis indicates the value of the x position parameter.

  Then, the CPU 10 determines whether or not the input of the position parameter selected by the user has been completed (step S3). This determination is made based on whether or not an operation signal corresponding to the operation of the SET key is sent from the key input unit 11.

  Here, the user can arbitrarily select a frame by operating the key input unit 11, and can input a value of a position parameter indicating the position of the subject in the selected frame. At this time, the value of the input position parameter is displayed in the displayed relation graph. Further, it is not necessary to input the position parameter value for each frame, and the position parameter value may be input every 10 frames or 100 frames. That is, the value of the position parameter currently selected may be roughly determined so that the subject to be photographed has a desired movement. At this time, the position parameter of the first frame (first frame) and the last frame position parameter may be input without fail. This last frame refers to the last frame of the moving image that the user wants to shoot. For example, when 1000 frames are to be shot, the last frame is 1000 frames. In this way, it is possible to specify in advance the number of frames scheduled to be captured included in the moving image simultaneously with the registration of the position parameter.

  When it is determined in step S3 that the input of the currently selected position parameter is completed, the CPU 10 obtains an approximate line by a spline function based on the input position parameter value (step S4). The position parameter value of each frame is obtained by the spline function so that the position parameter becomes smooth. That is, the position parameter value of each frame is obtained so that the movement of the subject is smooth. The obtained approximate line is displayed on the relation graph.

  At this time, the position parameter value of each frame may be obtained by a spline function so as not to change the position parameter value of the frame input by the user. If the position parameter value of each frame does not become smooth due to the value of the frame position parameter input by the user, the value of the position parameter of the frame input by the user may be changed (corrected). Good. That is, the position parameter value of the frame input by the user is not absolute, and an approximate line is obtained by a spline function using the input position parameter value as a guide.

Here, FIG. 4A is a diagram illustrating an example of the state of the input x-position parameter when the input of the x-position parameter is completed, and FIG. 4B is a diagram illustrating the input position parameter. It is a figure which shows an example of a mode when the approximate line calculated | required by the spline function based on is displayed. Here, the last frame is the nth frame.
As shown in FIG. 4, an approximate line is obtained based on the position parameter of the subject roughly input by the user, and thereby the position parameter of each frame from the first frame to the nth frame is determined.
As can be seen from FIG. 4B, the position of the subject moves from the center (x = 0) to the right side, and then moves from the center to the left and right alternately, such as leftward and rightward. I understand.

Next, the CPU 10 determines whether or not to register the currently selected position parameter based on the obtained approximate line (step S5).
This determination is made when the operation signal corresponding to the user's operation of the SET key is sent from the key input unit 11, and is registered when the operation signal corresponding to the operation of the cancel key is sent. Judge.

  If it is determined in step S5 that parameter registration is not performed, the process returns to step S3, and it is determined again whether or not the input of the position parameter has been completed by the user. When the displayed approximate line is not the desired subject motion, the user can change the desired subject motion by re-inputting the position parameter again.

  On the other hand, if it is determined in step S5 that the parameter is to be registered, the CPU 10 registers the currently selected position parameter based on the obtained approximate line (step S6). More specifically, based on the approximate line, the value of the currently selected position parameter that is the planned position of the subject in each frame is registered. This registered value is recorded in the position parameter table of the memory 12.

  Then, the CPU 10 determines whether or not all position parameters have been selected (step S7). If the CPU 10 determines that all position parameters have not been selected, the CPU 10 selects the next parameter (step S8), and step S2 Return to. Since the position parameters are selected in the order of x, y, and z, if the current x position parameter is selected, the y position parameter is selected in step S8, and if the current y position parameter is selected, the z position parameter is selected. Will be selected.

FIG. 5A shows an example of the appearance of the approximate line that is the source of the registered y-position parameter, and FIG. 5B shows the appearance of the approximate line that is the source of the registered z-position parameter. An example is shown.
As can be seen from FIG. 5A, the height of the position of the subject does not change, and it can be seen that the position is lower than the center of the CCD 5. Further, as can be seen from FIG. 5B, it can be seen that the position of the subject is further away from the CCD 5 as the frame is captured later.

  FIG. 6 shows an example of the state of each position parameter of the subject recorded (registered) in the position parameter table of the memory 12, and the value of the x position parameter, y position, for each frame. Each value of the parameter value and the z position parameter value is recorded.

  The advantage of such a registration method is that it is very difficult for the user to accurately input parameters corresponding to all frames so that the motion is natural and smooth, but the parameters of some frames are input. Therefore, interpolation and correction of parameters of other frames are performed, so that natural and smooth motion can be easily registered. In other words, if 2 points are specified, a linear movement represented by a linear function passing through these 2 points, and if 3 points are specified, a constant acceleration motion represented by a cubic function passing through these 3 points. Thus, as the number of points to be specified increases, a complex motion can be specified.

  In this registration process, the parameters are registered in correspondence with the frames that are scheduled to be photographed. However, since it is only necessary that the movement of the object can be registered here, the parameters need not be registered in correspondence with the frames that are scheduled to be photographed. Also good. When actually starting shooting, a new frame rate for the movie is set, and a plurality of frames to be shot are newly determined according to the set frame rate. A parameter corresponding to each of the newly determined frames may be automatically generated based on the motion. Also, even if the video is not a movie that plays back at a constant frame rate, but a movie with a variable frame rate, if the timing at which each frame is played back can be specified, parameters corresponding to that frame can be generated. It is.

B-2. Next, the animation moving image shooting operation will be described with reference to the flowchart of FIG.
When the parameter registration mode is set by the user operating the mode switching key of the key input unit 11, the CPU 10 sets the actual size of an object that is a predetermined subject to be photographed and moves with respect to the background (step). S11). At this time, a screen for allowing the user to input the size of the subject is displayed, and the input size is set.

Next, the CPU 10 specifies the first frame (first frame) as a subject to be photographed (step S12). That is, the frame to be shot from now is specified.
Next, the CPU 10 starts photographing with the CCD 5 at a predetermined frame rate, and stores the frame image data (YUV data) of the luminance and color difference signals sequentially photographed by the CCD 5 and generated by the image generation unit 9 in the buffer memory (DRAM 13). Then, so-called through image display is started in which an image based on the stored frame image data is displayed on the image display unit 15 (step S13).

  Next, the CPU 10 determines the value of each position parameter of the currently specified frame shown in FIG. 6, the actual size of the predetermined subject set in step S1, and the set shooting angle of view ( Based on the information such as the zoom factor, a frame (scheduled shooting state frame) indicating the shooting schedule state of the subject in the specified frame is displayed transparently on the image display unit 15 (step S14). Here, the scheduled shooting state to be transparently displayed means the position and size of the subject on the image display unit 15 to be shot.

  At this time, a three-dimensional model is generated based on the value of each position parameter of the specified frame and the size of the subject set in step S1, and the three-dimensional model is configured from the x-axis and the y-axis. The state when viewed from the CCD 5 which is on the plane and is arranged so that (x, y, z) = (0, 0, 0) is in the center is obtained as a subject state by two-dimensional projection. Then, the generated frame of the subject is transparently displayed on the image display unit 15 as the scheduled shooting state frame.

  FIG. 9 shows an example of a three-dimensional model generated based on the value of each position parameter of the specified frame and the size of the subject set in step S1. A black dot 31 in FIG. 9 indicates the position of the subject determined by the value of each position parameter of the identified frame, and an ellipse 32 indicates the size of the subject set in step S1.

  A state of the generated three-dimensional model of the subject as viewed from the CCD 5 is generated by two-dimensional projection as a subject photographing scheduled state. At this time, since the size of the subject also changes depending on the focal length of the photographing lens of the digital camera 1, it is generated by two-dimensional projection in consideration of the focal length of the photographing lens. As described above, the size of the subject, the focal length, and the subject generated by the two-dimensional projection according to the distance from (x, y, z) = (0, 0, 0) to the subject position 31. The size will change.

FIG. 10A shows an example of a state of a shooting state frame of a subject generated by two-dimensionally projecting the three-dimensional model of FIG.
FIGS. 10B and 10C show the state of the scheduled shooting state frame that is transparently displayed based on the position parameter values of the second and third frames. As can be seen from FIG. 10, the scheduled shooting state frame of the subject gradually moves to the right and the position of the subject moves away (see FIGS. 4 and 5), so that it is transparently displayed as the scheduled shooting state frame. It can be seen that the size of the subject is smaller.

As a result, by displaying the position on the image display unit 15 where the subject is scheduled to be photographed and its size as the scheduled photographing state frame, the subject can be placed at a position where the value of each position parameter of the identified frame is obtained. You can shoot.
The technique for obtaining a two-dimensional object by two-dimensionally projecting from this three-dimensional model is a well-known technique and will not be described.

Returning to the description of the flowchart of FIG. 7, when the photographing schedule state frame of the subject is displayed, the CPU 10 determines whether or not the shutter button has been pressed by the user (step S <b> 15).
At this time, by changing the position of the digital camera 1, the user adjusts the position and size of the subject so that the subject to be photographed is the same as the scheduled shooting state frame that is transparently displayed. When it is determined that the size is the same as the scheduled shooting state frame that is transparently displayed, the shutter button is pressed.

  If it is determined in step S15 that the shutter button has not been pressed, the process stays in step S15 until the button is pressed. If it is determined that the shutter button has been pressed, the CPU 10 performs still image shooting and currently identifies the obtained image data. The image data of the frame that has been recorded is recorded in the flash memory 14 (step S16). At this time, information on the currently specified frame is recorded together. This frame information is information indicating whether or not the image data corresponds to what frame.

Next, the CPU 10 specifies the actual position and size of the subject based on the actually captured image data (step S17). The size is specified by extracting a plurality of feature points of the subject in the vicinity of the transparently displayed scheduled shooting state frame, specifying the contour of the subject by the extracted feature points, and specifying the position The center of the contour is specified as the position of the subject.
Next, the CPU 10 detects the three-dimensional coordinate position of the subject based on the identified actual subject position and size (step S18). Since the technique for detecting the three-dimensional one from the two-dimensional one is a well-known technique, the description is omitted.

Then, the CPU 10 compares the detected three-dimensional coordinate position with the value of each registered position parameter of the currently specified frame (step S19).
Next, the CPU 10 determines whether or not the difference between the compared values is equal to or greater than a predetermined value (step S20).

If it is determined in step S20 that the difference is greater than or equal to a predetermined value, the CPU 10 determines whether or not to retake the image.
At this time, the CPU 10 displays the message “Different from the planned shooting state of the subject. Do you want to retake the image?” And determines that the image is not retaken when the user operates the SET key. When the cancel key is operated, it is determined that the image is retaken.

If it is determined in step S21 that the image is retaken, the image data recorded in step S16 as the currently specified frame is deleted (step S22), and the process returns to step S14.
On the other hand, if it is determined in step S21 that the image is not retaken, the CPU 10 determines whether the position parameter of the frame after the currently specified frame is registered (step S23).

  If it is determined in step S23 that the position parameter of the frame after the currently specified frame is registered, the CPU 10 currently specifies based on the three-dimensional coordinate position of the subject detected in step S18. The registered position parameter of the frame after the frame is corrected (step S24), and the process proceeds to step S26. The corrected position parameter value is overwritten in the position parameter table shown in FIG.

  In addition, this correction is performed so that the value of each position parameter of each frame is smoothed by the spline function. This correction is performed with priority given to the value of each currently registered position parameter in a frame farther from the currently specified frame. That is, a frame closer to the currently specified frame is affected by the detected three-dimensional coordinate position of the actually photographed subject.

FIG. 11 shows an example of the relationship graph between the registered x-position parameter and the frame, and the modified relationship graph between the x-position parameter and the frame.
The upper relation graph of FIG. 11 shows the relation graph between the registered x-position parameter and the frame, and the lower relation graph shows the relation chart between the modified x-position parameter and the frame.

Also, the crosses in the upper relational graph of FIG. 11 indicate the positions of the actually photographed subjects. When the fourth frame is viewed, the registered x-position parameters and the actually photographed images are taken. It can be seen that the x-axis position of the subject is shifted.
As can be seen from the lower relation graph of FIG. 11, it can be seen that the value of the x-position parameter after the fourth frame is corrected. In addition, when viewing the relationship graph below, it can be seen that as the distance from the fourth frame increases, the value of the x-position parameter of the registered frame does not change. In the lower relation graph of FIG. 11, the part represented by a dotted line shows an approximate line of the position parameter value before correction.

  On the other hand, if it is determined in step S20 that the difference between the detected three-dimensional coordinate position of the actually photographed subject and the value of each registered position parameter of the currently specified frame is not greater than or equal to a predetermined value, The CPU 10 determines whether or not the position parameter of the frame next to the currently specified frame is registered (step S25).

If it is determined in step S25 that the position parameter of the frame next to the currently specified frame is registered, the process proceeds to step S26.
In step S26, the CPU 10 specifies a frame next to the currently specified frame as a subject to be photographed, and returns to step S13.
For example, when the currently specified frame is the first frame, the second frame is specified as the next frame, and the currently specified frame is the second frame. Specifies the third frame as the next frame.

On the other hand, if it is determined in step S23 that the position parameter of the frame after the currently specified frame is not registered, the position parameter of the frame next to the currently specified frame is not registered in step S25. If it is determined, the process proceeds to step S31 in FIG. 8, and the CPU 10 determines whether or not to capture the next frame.
Here, since the position parameter of the Nth frame is registered, if the currently specified frame is the Nth frame, in step S23 or step S25, the frame after the currently specified frame is registered. Alternatively, it is determined that the position parameter of the frame next to the currently specified frame is not registered.

At this time, the CPU 10 displays a question “Do you want to shoot the next frame” on the image display unit 15, determines that the next frame is shot when the user operates the SET key, and the user presses the cancel key. If the above operation is performed, it is determined that the next frame is not shot.
If it is determined in step S31 that the next frame is to be shot, the CPU 10 specifies the next frame as a shooting target (step S32).

  Next, the CPU 10 predicts each position parameter of the subject of the specified frame based on the registered position parameter of each frame (step S33). This prediction is predicted by a spline function. At this time, each position parameter of the next frame is predicted so that the value of each position parameter of the next frame and the value of each position parameter of each past frame already registered are smooth.

  Next, the CPU 10 starts to display a through image (step S34), and based on the predicted value of each position parameter of the subject and the size of the subject set in step S11, the CPU 10 A scheduled shooting state frame indicating the scheduled shooting state of the subject is displayed transparently on the image display unit 15 (step S35). Since this transmissive display is the same operation as step S14 described above, the description thereof is omitted here.

Next, the CPU 10 determines whether or not the shutter button has been pressed (step S36).
At this time, by changing the position of the digital camera 1, the user adjusts the position and size of the subject so that the subject to be photographed is the same as the scheduled shooting state frame that is transparently displayed. When it is determined that the size is the same as the scheduled shooting state frame that is transparently displayed, the shutter button is pressed.

  If it is determined in step S36 that the shutter button has not been pressed, the process stays in step S36 until it is determined that the shutter button has been pressed. If it is determined that the shutter button has been pressed, the CPU 10 performs still image shooting processing and obtains the obtained image data. Are recorded in the flash memory 14 as image data of the currently specified frame (step S37). At this time, information on the currently specified frame is recorded together. This frame information is information indicating whether or not the image data corresponds to what frame.

  Next, the CPU 10 specifies the actual position and size of the subject based on the actually captured image data (step S38). The size is specified by extracting a plurality of feature points of the subject in the vicinity of the transparently displayed scheduled shooting state frame, specifying the contour of the subject by the extracted feature points, and specifying the position The center of the contour is specified as the position of the subject.

Next, the CPU 10 detects the three-dimensional coordinate position of the subject based on the identified actual subject position and size (step S39). Since the technique for detecting the three-dimensional one from the two-dimensional one is a well-known technique, the description is omitted.
Then, the CPU 10 compares the detected three-dimensional coordinate position with the value of each registered position parameter of the currently specified frame (step S40).

Next, the CPU 10 determines whether or not the difference between the compared values is equal to or greater than a predetermined value (step S41).
If it is determined in step S40 that the difference is greater than or equal to a predetermined value, the CPU 10 determines whether or not to retake the image (step S42).
At this time, the CPU 10 displays the message “Different from the planned shooting state of the subject. Do you want to retake the image?” And determines that the image is not retaken when the user operates the SET key. When the cancel key is operated, it is determined that the image is retaken.

If it is determined in step S42 that the image is retaken, the CPU 10 deletes the image data recorded in step S37 as the currently specified frame (step S43), and the process returns to step S34.
On the other hand, if it is determined in step S42 that the image is not retaken, the CPU 10 determines the three-dimensional coordinate position of the actually photographed subject detected in step S39 as the position parameter of the subject of the currently specified frame. Is newly recorded (additionally registered) as a value in the position parameter table (step S44), and the process returns to step S31.

If it is determined in step S41 that the difference is not greater than or equal to the predetermined value, each position parameter predicted in step S33 is additionally recorded (additionally registered) as a position parameter of the currently specified frame (step S45). Return to step S31.
On the other hand, if it is determined in step S31 that the next frame is not shot, a moving image file is generated on the flash memory 14 based on the recorded image data (step S46).

FIG. 12 is a diagram illustrating a state of the x-position parameter of a newly additionally registered frame when it is determined in step S31 that a frame after the frame in which the position parameter is registered is captured.
As can be seen from FIG. 12, only the position parameters of the nth frame have been registered, but it can be seen that the position parameters of the nth and subsequent frames have been registered. Based on the position parameter of each past frame, the value of each position parameter of the specified frame is estimated, and a scheduled shooting state frame is displayed. If the difference between the three-dimensional coordinate position of the actually photographed subject and the estimated value of each position parameter is not equal to or greater than a predetermined value, the estimated position parameter value indicates the position of the specified frame. When the difference is greater than or equal to a predetermined value, the three-dimensional coordinate position of the actually photographed subject is registered as the value of each position parameter of the specified frame.

  As described above, in the embodiment, the planned shooting state of the subject is roughly input so that the user's desired movement is obtained in advance (the value of each position parameter is input for one frame) The CPU 10 calculates and registers the shooting schedule state of each frame based on the input shooting schedule state, and when performing animation moving image shooting, the shooting indicating the registered shooting schedule state for each frame shooting. By transparently displaying the scheduled state frame, the user can be prompted to shoot the subject in the same state as the registered scheduled shooting state, and an animated movie can be easily created.

Further, when the user inputs the value of each position parameter for one frame so that the movement of the subject becomes a desired movement, the user automatically determines the value of each position parameter of each frame. The position parameter value can be easily determined. In addition, since the position parameter value is determined so that the position parameter value of each frame becomes smooth based on the value of each position parameter of a part of the frame input by the user, the animation in which the movement of the subject is smooth You can create a video.
In addition, when the difference between the state of the actually photographed subject and the registered planned shooting state is greater than or equal to a predetermined value, the image can be retaken, so that an image that is in the registered planned shooting state can be obtained. it can. In addition, when not re-taking the image, the scheduled shooting state of the subsequent frames was corrected based on the state of the actually shot subject so that the movement of the subject was smooth. Even when the subject is shot in a state different from the scheduled shooting state, the subject can be shot so that the movement of the subject is smooth, and an animation movie with smooth movement can be created.

  Also, even when shooting a frame for which no position parameter is registered, the position parameter is registered because the value of each position parameter to be shot is estimated based on each registered position parameter of each frame. Even when a frame that has not been photographed is taken, it can be dealt with. For example, the position parameters of the frames are registered at first because it is assumed that such a number of images will be shot, but this can be handled even when the user actually wants to shoot a few more frames.

[Modification]
D. The above embodiment may be the following modes.

  (01) In the above embodiment, the CCD 5 is on the plane of the x-axis and the y-axis, and the center of the CCD 5 is (x, y, z) = (0, 0, 0). The CCD 5 may not be on the plane of the x-axis and the y-axis, and the center of the CCD 5 may not be (x, y, z) = (0, 0, 0). That is, a three-dimensional object composed of x, y, and z may be generated by two-dimensional projection when the CCD 5 is arranged at an arbitrary angle and distance.

  (02) In the above embodiment, each three-dimensional position parameter is registered for each frame. However, the position of the subject on the CCD 5 and its size may be registered for each frame.

(03) In the above embodiment, each of the three-dimensional position parameters is registered as the scheduled shooting state of the subject of each frame. However, a parameter indicating the direction of the subject for each frame is further registered. It may be.
At this time, the position and size of the subject to be photographed as well as the direction thereof may be displayed as the scheduled photographing state.
At this time, as a method of detecting the direction of the subject based on the actually captured image data, for example, when the subject has eyes such as a person or an animal, the distance between the eyes The direction of the subject may be detected according to the distance. This is because the distance between the eyes when facing the front is the longest, and the distance between the eyes becomes shorter as the direction of the subject becomes oblique. That is, the direction of the subject is detected based on the relative positional relationship between the feature points of the eyes.
Further, the subject to be photographed is photographed and registered from every angle of 360 degrees, and the image data of every 360 degrees of the registered subject is compared with the image data of the actually photographed subject. Thus, the direction of the actually photographed subject may be detected.
Also, in step S17 in FIG. 7 and step S38 in FIG. 8, the position and size of the actually photographed subject are also specified based on the image data of the registered subject taken from all angles of 360 degrees. May be.

  (04) In the above embodiment, the planned shooting state of the subject is displayed by displaying a frame. However, the shooting planned state is not limited to the frame and may be displayed by other methods. .

  (05) In the above-described embodiment, the position parameter of each frame is registered in advance and animation moving image shooting is performed. However, a plurality of frames are shot, and the subject image is captured based on the captured image data. A three-dimensional coordinate position may be detected, and each position parameter of a subsequent frame may be estimated based on the detected coordinate position and displayed transparently.

(06) In the above embodiment, when estimating the value of each position parameter of an unregistered frame, the speed of the subject and the moving direction are calculated based on at least the position parameters of the two immediately preceding frames. Then, the value of each position parameter when moving in the calculated direction at the same speed as the calculated subject speed may be calculated.
Further, when estimating the value of each position parameter of a non-registered frame, the acceleration of the subject and the moving direction are calculated based on at least the position parameters of the immediately preceding three frames, and the calculated acceleration and The value of each position parameter when moving in the calculated direction with the same acceleration may be calculated.
When estimating the value of a parameter indicating the direction of a subject in an unregistered frame, at least the direction in which the direction of the previous two frames changes and the speed of the change are calculated, and the same speed as the calculated speed Thus, the value of the parameter indicating the direction when the direction changes to the direction in which the calculated direction changes may be calculated.

  (07) In the above embodiment, by calculating the position parameters of other frames based on the position parameters indicating the state of the subject of a part of the frame input by the user, the movement of the object is simultaneously detected. Although specified, a series of movements of the subject is specified based on the approximate movement of the subject input by the user, and each position parameter of each frame is calculated based on the movement of the specified subject. It may be. This also makes it possible to easily create an animation movie.

As for the input of the approximate movement of the subject of the user, the approximate movement of the subject may be directly input by each position parameter indicating the state of the subject of some frames as in the above embodiment. Alternatively, in a three-dimensional space, the movement of the subject may be roughly input at this position at a certain time and at this time.
In this case, a series of subject movements specified based on the input schematic subject movement are specified so that the movement is smooth. Then, based on the identified movement, the object state corresponding to each frame is converted, and each position parameter of each frame is calculated. At this time, the movement of the subject may be specified so as not to change the value of the position parameter of the frame input by the user, or the movement of the subject may be determined according to the value of the position parameter of the frame input by the user. If the image does not become smooth, the value of the position parameter of the frame input by the user may be changed (corrected) to specify the movement of the subject.

In this case, in the above-described embodiment, when each position parameter to be photographed is different from the three-dimensional coordinate position of the actually photographed subject, each position parameter of the subsequent frame is corrected. Then, the movement of the specified subject is corrected (newly specified), and the position parameter of each frame is calculated based on the corrected movement of the subject. As a result, the subject can be imaged so that the movement of the subject is smooth.
Similarly, when imaging a frame in which no position parameter is registered, the movement of the subject is predicted, the movement of the subject is newly specified, and the movement of the frame is determined based on the movement of the specified subject. A position parameter is calculated. This motion prediction identifies the direction, speed, acceleration, direction, etc. of the subject's motion based on the position parameters of the frames that have been captured so far, and determines each frame's motion based on the identified subject's motion. Find the parameter value. As a result, the subject can be imaged so that the movement of the subject is smooth.

  (08) In the above embodiment, the case where the camera is specified with respect to the background has been described. However, the present invention may be applied to the case where the camera is not fixed with respect to the background. In this case, based on the registered position parameter of each frame and the position of the camera, the planned shooting state of the subject is transparently displayed.

  (09) A mode in which the above-described modifications (01) to (08) are arbitrarily combined within a consistent range may be used.

(10) The above embodiments of the present invention are merely examples as the best embodiments, and are described in order to better understand the principle and structure of the present invention. And is not intended to limit the scope of the appended claims.
Therefore, it should be understood that all the various variations and modifications that can be made to the above-described embodiments of the present invention are included in the scope of the present invention and protected by the appended claims.

  Finally, in the above-described embodiment, the case where the photographing apparatus of the present invention is applied to the digital camera 1 has been described. However, the present invention is not limited to the above-described embodiment. Any device that can be used is applicable.

1 is a block diagram of a digital camera according to an embodiment of the present invention. It is a flowchart which shows the operation | movement of parameter registration. It is a figure which shows an example of the state of the three-dimensional state represented by x, y, z, and the state of the displayed position graph. An example of the state of the input x-position parameter when the input of the x-position parameter is completed, and an example of the state when the approximate line obtained by the spline function based on the input position parameter is displayed FIG. An example of the appearance of the approximate line that is the origin of the registered y position parameter and an example of the appearance of the approximate line that is the origin of the registered z position parameter are shown. It is a figure which shows an example of the mode of each position parameter of the to-be-photographed object registered into the position parameter table of the memory. It is a flowchart which shows the operation | movement of animation moving image photography. It is a flowchart which shows the operation | movement of animation moving image photography. An example of a three-dimensional model generated based on the value of each position parameter of the specified frame and the size of the subject is shown. It is a figure which shows an example of the imaging | photography plan state frame of the produced | generated subject. It is a figure which shows an example of the mode of the relationship graph of the corrected x position parameter and a flame | frame. It is a figure which shows the mode of the parameter of the x position of the flame | frame newly newly registered.

Explanation of symbols

1 Digital Camera 2 Shooting Lens 3 Lens Drive Block 4 Aperture 5 CCD
6 Driver 7 TG
8 Unit circuit 9 Image generator 10 CPU
11 Key input section 12 Memory 13 DRAM
14 Flash memory 15 Image display unit 16 Bus

Claims (20)

An imaging device including an imaging unit and an electronic viewfinder for supporting the creation of a live-action animation movie that expresses the movement of a predetermined subject using a plurality of captured image data,
Generating means for generating a parameter indicating the state of the predetermined subject in correspondence with each frame constituting the live-action animation video;
Based on the parameters generated by the generating means, guidance display control means for guiding and displaying the planned shooting state of the subject corresponding to the frame to be shot on the electronic viewfinder when shooting each frame;
Moving image generation means for generating moving image data based on image data of each frame imaged by the imaging means corresponding to the guidance display by the guidance display control means;
An imaging apparatus comprising: Specifying means for specifying the movement of the predetermined subject in a three-dimensional space corresponding to the background portion of the live-action animation video;
The generating means includes
The movement of the predetermined subject in the three-dimensional space specified by the specifying means is converted into the state of the predetermined subject corresponding to each frame in accordance with timing information at which each frame in the live-action animation movie is reproduced. The imaging apparatus according to claim 1, wherein the parameter is generated. An input means for a user to input a parameter indicating a state of the predetermined subject in a part of the frames constituting the animation moving image;
The specifying means is:
The movement of the predetermined subject in the three-dimensional space corresponding to the background portion of the live-action animation movie is specified based on the parameter indicating the state of the predetermined subject that is captured in a part of the frames input by the input unit The photographing apparatus according to claim 2, wherein: The specifying means is:
3 corresponding to the background portion of the live-action animation video so that the movement of the predetermined subject is smoothed based on the parameter indicating the state of the predetermined subject that is captured in a part of the frames input by the input means. The imaging apparatus according to claim 3, wherein a movement of the predetermined subject in a dimensional space is specified. The specifying means is:
The parameter indicating the state of the predetermined subject captured in a part of the frames input by the input unit is corrected so that the movement of the predetermined subject is smooth, and the background corresponding to the background portion of the live-action animation moving image is corrected. The imaging apparatus according to claim 3, wherein the movement of the predetermined subject in a three-dimensional space is specified. A detection unit for detecting a parameter indicating a shooting state of the predetermined subject corresponding to the actually shot frame based on the actually shot image data;
The specifying means is:
6. The movement of the predetermined subject in the three-dimensional space corresponding to the background portion of the live-action animation moving image is newly specified based on the parameter detected by the detecting means. An imaging device described in any one of the above. The specifying means is:
Identifying a direction and speed of movement of the predetermined subject based on at least two frame states immediately before the current frame for generating a parameter detected by the detection means;
The generating means includes
7. The photographing apparatus according to claim 6, wherein the parameter of the current frame is generated based on the detected direction and speed of movement. The specifying means is:
Identifying the direction and acceleration of movement of the subject based on at least the state of the three frames immediately before the current frame for generating the parameters detected by the detecting means;
The generating means includes
8. The photographing apparatus according to claim 6, wherein the parameter of the current frame is generated based on the detected direction of movement and acceleration. The specifying means is:
Identifying a direction and a speed at which the direction of the predetermined subject changes based on at least two frame states immediately before the current frame for generating a parameter detected by the detection unit;
The generating means includes
9. The photographing apparatus according to claim 6, wherein a parameter of the current frame is generated based on a direction and speed at which the detected direction changes. An input means for a user to input a parameter indicating a state of the predetermined subject in a part of the frames constituting the animation moving image;
The generating means includes
A parameter indicating the state of the subject corresponding to each frame other than the part of the frame so that the movement of the subject is smooth based on the parameter corresponding to the part of the frame input by the input unit. The imaging apparatus according to claim 1, wherein: The generating means includes
11. The parameter indicating the state of the subject in each frame is generated by correcting the parameter input by the input means so that the predetermined subject moves smoothly. Shooting device. Detecting means for detecting a photographing state of the predetermined subject corresponding to the actually photographed frame based on the actually photographed image data;
Correction means for correcting a parameter of a frame portion that has not yet been photographed out of the parameters generated by the generation means based on the photographing state of the predetermined subject detected by the detection means;
The imaging apparatus according to claim 10 or 11, further comprising: Based on the image data of each frame that was actually shot, a detection unit that detects the shooting state of the predetermined subject that was actually shot,
The generating means includes
13. The shooting according to claim 10, wherein a parameter of a frame portion that has not yet been generated is generated based on a shooting state of the predetermined subject in each frame detected by the detection means. apparatus. The generating means includes
The direction and speed of movement of the predetermined subject are detected based on the state of at least two frames immediately before the current frame for generating a parameter detected by the detecting means, and the direction and speed of the detected movement are detected. 14. The photographing apparatus according to claim 13, wherein the parameter of the current frame is generated based on the information. The generating means includes
Based on at least the state of the three frames immediately before the current frame for generating the parameter detected by the detecting means, the direction and acceleration of the predetermined movement are detected, and the detected direction and acceleration of the movement are detected. 15. The photographing apparatus according to claim 13, wherein the parameter of the current frame is generated based on The generating means includes
Based on the state of at least two frames immediately before the current frame for generating the parameter detected by the detecting means, the direction and speed of the predetermined subject change are detected, and the detected direction changes. 16. The photographing apparatus according to claim 13, wherein the parameter of the current frame is generated based on a direction and a speed of the photographing. The detection means includes
The photographing apparatus according to any one of 6 to 9, and 12 to 16, wherein a photographing state of the subject is detected by detecting a feature point of the subject. The detection means includes
18. The photographing apparatus according to claim 17, wherein a plurality of feature points of the subject are detected, and the centers of the detected feature points are detected as subject positions. The detection means includes
The photographing apparatus according to claim 17 or 18, wherein a plurality of feature points of the subject are detected, and an orientation of the subject is detected based on a relative positional relationship between the detected feature points. A program for executing an imaging device including an imaging unit and an electronic viewfinder that supports creation of a live-action animation movie that expresses the movement of a predetermined subject using a plurality of captured image data,
A generation process for generating a parameter indicating the state of the predetermined subject in correspondence with each frame constituting the live-action animation video;
Based on the parameters generated by the generation process, a guidance display control process that guides and displays on the electronic viewfinder the planned shooting state of the subject corresponding to the frame to be shot when shooting each frame;
A moving image generating process for generating moving image data based on the image data of each frame imaged on the imaging element by the imaging unit corresponding to the guidance display by the guidance display control process;
The program characterized by including.

Images