JP2016010078A - Image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system that prevents unnatural animation effects even when animation effects are synthesized to a moving image with panning and zooming.SOLUTION: Synthesis is performed by calculating a synthetic position obtained by adding a movement amount generated by panning or zooming of a previous frame and a current frame to an image object synthetic position of a current frame specified by a user. Thus, with the synthetic position tracking the panning and the zooming, natural synthesis is performed.

Description

  The present invention relates to an image processing apparatus that combines an image object with a moving image and a control method thereof.

  There is a conventional technique to add an animation effect by synthesizing an image object to a video shot with a camera or video camera. However, the image object is synthesized at the position where the motion amount between frames is added to the motion amount of the specified composite object. And there was no technology to add a natural animation effect to follow the panning and zooming of the video.

  Patent Document 1 discloses a technique for tracing a touch panel for designating a motion of an image object to be combined, detecting a motion with a gyro sensor or the like by moving a device, and defining an animation.

  Patent Document 2 discloses a technique for detecting the tilt of a camera and determining the image composition speed accordingly.

  Japanese Patent Laid-Open No. 2004-228561 detects the camera tilt with a gyro sensor or the like in order to fix the display position of the composite object to the earth (that is, the fixed position with respect to the through image), or compares the amount of motion by comparing previous and next frame images. A technique for detecting the above is disclosed.

JP 2002-223389 A JP 2010-009575 A Japanese Patent Laid-Open No. 2001-243012

  However, in the technique of Patent Document 1, in a moving image whose screen changes due to panning or zooming, when an image object is simply synthesized with only a motion specified by the user, it does not follow the panning or zooming of the moving image. It becomes an animation effect.

  In the technique of Patent Document 2, processing according to the movement of the image itself, such as panning and zooming, is not considered.

  Further, in the technique of Patent Document 3, a process for moving a composite object is not considered because it is a technique for displaying at a fixed position.

  The configuration of the image processing apparatus according to claim 1 includes a storage unit that stores a plurality of composite objects to be combined, a tilt detection unit that detects a tilt of the image processing device, and a correlation calculation between two frames before and after the moving image, Based on the movement direction calculation means for calculating the movement amount between frames, the direction of gravity obtained from the detection result of the inclination detection means, and the movement amount between frames, the motion of the composite object to be combined with the moving image is calculated. A composite object trajectory calculating means for calculating and a compositing means for compositing the composite object with a moving image according to the trajectory calculated by the composite object trajectory calculating means are provided.

  An operation speed detecting means for detecting the speed of movement of the image processing apparatus according to claim 1 and a composition for adjusting a movement speed parameter of the combined object of the trajectory according to the movement speed of the image processing apparatus detected by the operation speed detecting means. And an object moving speed adjusting means.

  An operation speed detecting means for detecting the speed of movement of the image processing apparatus according to claim 1, and a composite amount adjustment for adjusting the amount of the composite object in the locus based on the moving speed of the image processing apparatus detected by the operation speed detecting means. And means.

  And a predetermined area avoiding unit that determines that the predetermined area is not on the second synthetic object trajectory or is not combined until it passes through the predetermined area.

  In the predetermined area avoiding means, the range of the face detected by the face detecting means is set as a predetermined area.

  A zooming amount calculation unit that calculates a zooming amount between frames by performing a correlation operation on two frames before and after a moving image, and a composite object resizing unit that resizes a composite object from a default size according to the zooming amount. And

  Preview means for converting the locus calculated by the synthesized object locus calculating means into a VRAM size coordinate system of a display device and superimposing or synthesizing the synthesized object to reproduce a preview on the display unit.

  A trajectory storage means for storing one or more kinds of trajectories calculated by the composite object trajectory calculation means, and a memory for selecting the second composite object trajectory stored in the trajectory storage means by a user's synthesis execution operation and combining it with a moving image. Trajectory synthesis means is provided.

  The configuration of the image processing device according to claim 9 is the inter-frame movement amount calculation means for calculating the movement amount between the frames by performing a correlation operation on the two frames before and after the moving image, and the top-and-bottom direction described in the shooting information of the moving image. Based on the gravitational direction obtained by the above and the amount of movement between frames, the synthesized object trajectory calculating means for calculating the motion of the synthesized object to be synthesized with the video, and the synthesized object according to the trajectory calculated by the synthesized object trajectory calculating means. And synthesizing means for synthesizing.

  The configuration of the image processing apparatus according to claim 10 includes a storage unit that stores a plurality of composite objects to be combined, a trajectory input unit that recognizes a touch on the screen of the display unit, and inputs a trajectory for combining the composite objects, and a moving image. Based on the inter-frame movement amount calculation means for calculating the movement amount between the frames, the trajectory obtained by the locus input means, and the inter-frame movement amount. A synthetic object trajectory calculating means for calculating a motion of the composite object to be performed; and a synthesizing means for synthesizing the composite object with a moving image in accordance with the trajectory calculated by the synthetic object trajectory calculating means.

The configuration of the image processing apparatus according to claim 11 correlates the storage means storing a plurality of composite objects to be combined, the inclination detection means for detecting the inclination of the image processing apparatus, and the two frames before and after the moving image, Inter-frame movement amount calculation means for calculating the movement amount between frames, and coordinate calculation means for calculating the coordinates on the screen of the display unit from the direction of gravity obtained from the detection result of the inclination detection means for a predetermined period. Coordinate display means for displaying the coordinates on a display unit, combined object trajectory calculation means for calculating a motion of a composite object to be combined with a moving image based on the coordinates and the inter-frame movement amount,
Synthesizing means for synthesizing the synthesized object with the moving image according to the trajectory calculated by the synthesized object trajectory calculating means.

  According to the present invention, a composite position is calculated by adding the motion amount by panning or zooming between the previous frame and the current frame to the image object composite position in the current frame of the motion designated by the user, and composites. As a result, the synthesis position follows natural panning and zooming, resulting in natural synthesis.

Hardware configuration diagram Flow chart of the first embodiment Explanation of moving pixel parameters Illustration of image object parameters Explanatory drawing of the screen of Example 1 Flow chart of embodiment 2 Flow chart of embodiment 3 Flow chart of embodiment 4 Explanatory drawing of the screen of Example 4

  The operation of each embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram illustrating a configuration example of the digital camera 100 according to the present embodiment. In FIG. 1, reference numeral 103 denotes a photographing lens including a focus lens, 101 denotes a shutter having a diaphragm function, and 22 denotes an image pickup unit including a CCD, a CMOS element, or the like that converts an optical image into an electric signal. Reference numeral 23 denotes an A / D converter that converts an analog signal into a digital signal. The A / D converter 23 is used to convert an analog signal output from the imaging unit 22 into a digital signal.

  A barrier 102 covers the imaging unit including the imaging lens 103 of the digital camera 100 to prevent the imaging system including the imaging lens 103, the shutter 101, and the imaging unit 22 from becoming dirty or damaged.

  An image processing unit 24 performs resize processing such as predetermined pixel interpolation and reduction and color conversion processing on the data from the A / D converter 23 or the data from the memory control unit 15. The image processing unit 24 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation result. Thereby, AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing of the TTL (through-the-lens) method are performed. The image processing unit 24 further performs predetermined calculation processing using the captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained calculation result.

  Output data from the A / D converter 23 is directly written into the memory 32 via the image processing unit 24 and the memory control unit 15 or via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23 and image data to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a moving image and sound for a predetermined time.

  The memory 32 also serves as an image display memory (video memory). A D / A converter 13 converts image display data stored in the memory 32 into an analog signal and supplies the analog signal to the display unit 28. Thus, the display image data written in the memory 32 is displayed on the display unit 28 via the D / A converter 13. The display unit 28 performs display according to the analog signal from the D / A converter 13 on a display such as an LCD.

  The nonvolatile memory 56 is an electrically erasable / recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 56 stores constants, programs, and the like for operating the system control unit 50. Here, the program is a program for executing various flowcharts described later in the present embodiment.

  Reference numeral 50 denotes a system control unit that controls the entire digital camera 100. By executing the program recorded in the non-volatile memory 56 described above, each process of the present embodiment to be described later is realized. A system memory 52 is a RAM. In the system memory 52, constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are expanded. The system control unit also performs display control by controlling the memory 32, the D / A converter 13, the display unit 28, and the like.

  The mode switch 60, the first shutter switch 62, the second shutter switch 64, and the operation unit 70 are operation means for inputting various operation instructions to the system control unit 50. The mode switch 60 switches the operation mode of the system control unit 50 to any one of a still image recording mode, a moving image recording mode, a reproduction mode, and the like. The first shutter switch 62 is turned on when the shutter button 61 provided in the digital camera 100 is being operated, so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1.

  In response to the first shutter switch signal SW1, operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-emission) processing are started.

  The second shutter switch 64 is turned ON when the operation of the shutter button 61 is completed, that is, when it is fully pressed (shooting instruction), and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processing operations from reading a signal from the imaging unit 22 to writing image data on the recording medium 200.

  Each operation member of the operation unit 70 is appropriately assigned a function for each scene by selecting and operating various function icons displayed on the display unit 28, and functions as various function buttons. Examples of the function buttons include an end button, a return button, an image advance button, a jump button, a narrowing button, and an attribute change button. For example, when a menu button is pressed, various setting menu screens are displayed on the display unit 28. The user can make various settings intuitively using the menu screen displayed on the display unit 28, the four-way button, and the SET button.

  The controller wheel 73 is a rotatable operation member included in the operation unit 70, and is used when a selection item is instructed together with a direction button. When the controller wheel 73 is rotated, an electrical pulse signal is generated according to the operation amount, and the system control unit 50 controls each unit of the digital camera 100 based on the pulse signal. From this pulse signal, it is possible to determine the angle at which the controller wheel 73 is rotated, how many rotations, and the like. The controller wheel 73 may be any operation member that can detect a rotation operation.

  For example, it may be a dial operation member that generates a pulse signal by rotating the controller wheel 73 itself according to the rotation operation of the user. Further, an operation member made of a touch sensor may detect the rotation operation of the user's finger on the controller wheel 73 without rotating the controller wheel 73 itself (so-called touch wheel).

  A power control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like, and detects whether or not a battery is attached, the type of battery, and the remaining battery level. Further, the power control unit 80 controls the DC-DC converter based on the detection result and an instruction from the system control unit 50, and supplies a necessary voltage to each unit including the recording medium 200 for a necessary period.

  A power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. Reference numeral 18 denotes an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card, and includes a semiconductor memory or a magnetic disk.

  Reference numeral 108 denotes an inclination detection unit, which includes a gyro sensor or the like and detects the inclination of the digital camera 100. Reference numeral 109 denotes an operation speed detection unit, which includes a gyro sensor or the like and detects the speed at which the digital camera 100 moves.

  In the digital camera 100 described above, it is possible to perform photographing using central one-point AF or face AF. Central one-point AF is to perform AF on one central position in the shooting screen. Face AF is to perform AF on the face in the shooting screen detected by the face detection function.

  The face detection function will be described. The system control unit 50 sends the face detection target image data to the image processing unit 24. The face detection unit 201 inside the image processing unit 24 applies a horizontal bandpass filter to the image data. Further, the face detection unit 201 applies a vertical band pass filter to the processed image data. By these horizontal and vertical band pass filters, edge components are detected from the image data.

  After that, the face detection unit 201 performs pattern matching on the detected edge component, and extracts a candidate group of eyes, nose, mouth, and ears. Then, the face detection unit 201 determines that an eye pair that satisfies a preset condition (for example, distance between two eyes, inclination, etc.) from the extracted eye candidate group is an eye pair. Narrow down the candidate group of only those with. Then, the face detection unit 201 associates the narrowed eye candidate group with other parts (nose, mouth, ears) that form the corresponding face, and passes a preset non-face condition filter. , Detect the face. The face detection unit 201 outputs the face information according to the face detection result, and ends the process. At this time, the system control unit 50 stores the feature amount such as the number of faces passed from the image processing unit 24 in the system memory 52.

  As described above, it is possible to analyze the image data displayed or reproduced and display the live view, extract the feature amount of the image data, and detect the subject information. In this embodiment, face information is taken as an example of subject information. However, subject information includes various information such as red-eye determination, eye detection, eye-brow detection, and smile detection.

  Note that the face AE, the face FE, and the face WB can be performed simultaneously with the face AF. The face AE is to optimize the exposure of the entire screen according to the brightness of the detected face. The face FE is to perform flash dimming around the detected face. The face WB is to optimize the WB of the entire screen according to the detected face color.

  As one of the operation units 70, a touch panel capable of detecting contact with the display unit 28 is provided. The system control unit 50 can detect the following operations on the touch panel. Touching the touch panel with a finger or a pen (hereinafter referred to as touchdown). The touch panel is touched with a finger or pen (hereinafter proof of touch-on). The touch panel is moved while being touched with a finger or a pen (hereinafter referred to as a move). The finger or pen that was touching the touch panel is released (hereinafter referred to as touch-up). A state where nothing is touched on the touch panel (hereinafter referred to as touch-off).

  These operations and the position coordinates of the finger or pen touching the touch panel are notified to the system control unit 50 through the internal bus, and the system control unit 50 performs any operation on the touch panel based on the notified information. Judge whether it was done. Regarding the move, the moving direction of the finger or pen moving on the touch panel can also be determined for each vertical component / horizontal component on the touch panel based on the change of the position coordinates. It is also assumed that a stroke is drawn when touch-up is performed on the touch panel through a certain move from touch-down. The operation of drawing a stroke quickly is called a flick.

  A flick is an operation of quickly moving a certain distance while touching a finger on the touch panel and then releasing it, in other words, an operation of quickly tracing a finger on the touch panel. If it is detected that the moving is performed at a predetermined speed or more over a predetermined distance, and a touch-up is detected as it is, it can be determined that a flick has been performed. In addition, when it is detected that the movement is performed at a predetermined distance or more and less than a predetermined speed, it is determined that the drag has been performed.

  The operation of each embodiment of the present invention will be described below with reference to FIGS.

<Example 1>
In the first embodiment, an example will be described in which image object parameters in the direction of gravity are calculated based on the result of detecting the tilt of the camera with a gyro sensor, and an image object is synthesized with a moving image.

  FIG. 2 is a flowchart of the first embodiment. Each process in the flowchart is realized by the system control unit 50 expanding and executing a program stored in the nonvolatile memory 56 in the memory 32.

  Example 1 will be described with reference to FIGS. In S100, the user selects a desired movie to be edited from the displayed movies (FIG. 3 (a)). In S101, the system control unit determines whether or not the edit moving image determination button has been pressed. If it is pressed, the process proceeds to S102, and if it is not pressed, the process proceeds to S100.

  In S102, the user selects a desired image object to be combined from the displayed image objects (FIG. 3B). In S103, the system control unit determines whether or not the composite object determination button has been pressed. If pressed, the process proceeds to parallel processing of S104 and S108, and if not pressed, the process proceeds to S102.

  S105 to S117 are processes for displaying a preview with reference to a parameter updated during the process of determining the composite object parameter. Therefore, the preview display process is performed in parallel with the composite object parameter determination process (S118 to S140). The composite objects to be determined are the number of composite objects, moving pixels in the X-axis direction, moving pixels in the Y-axis direction, and the width and height of each composite object (FIG. 4).

  In S105, the image processing unit 24 decodes the moving image stream. In S106, the image processing unit 24 resizes the frame to the VRAM size. VRAM is video memory. In S107, the image processing unit 24 calculates the amount of movement between two frames by performing a correlation operation on the decoded previous frame and the entire current frame. In S108, the image processing unit 24 calculates a zooming amount between two frames by performing a correlation operation on the decoded previous frame and the entire current frame. In S109, the image processing unit 24 resizes the composite object to a size proportionally calculated from the image object size for the VRAM size and the calculated zooming amount.

  In S110, the image processing unit 24 detects a face from the current frame of the VRAM size and notifies the system control unit 50 of the number, position, and size of the face. In S112, the system control unit 50 calculates a composite position obtained by adding the movement amount between frames to the composite position of the current frame for the VRAM size. In S113, the system control unit 50 determines whether or not the synthesis position of the synthesis object is within the face range. If it is within the face range (S113 YES), the process proceeds to S115, and if it is outside the face range (S113 NO), the process proceeds to S114.

  In S114, the image processing unit 24 synthesizes the synthesis object in the VRAM at the synthesis position. In S115, the system control unit 50 determines whether processing for the number of composite objects is completed. If processing for the number of composite objects is completed, the process proceeds to S134, and if not completed, the process proceeds to S112 for repetitive processing.

  In S116, the system control unit 50 controls to display the created VRAM on the display unit. In S117, the system control unit 50 determines whether all the composite object parameters have been confirmed. If all composite object parameters are confirmed, the process proceeds to S142, and if not confirmed, the process proceeds to S105.

  S118 to S140 are composite object parameter determination processing to be processed in parallel with the preview display. In S <b> 118, the user tilts the digital camera 100 to select the direction of gravity, and the tilt detection unit 108 detects the current tilt of the digital camera 100. In S119, the system control unit 50 calculates the equation of the gravity direction from the detected inclination, and determines the value of the moving pixel in the Y-axis direction and the moving pixel in the X-axis direction per frame. At this time, the initial value is used for the moving pixel of the axis in the gravity direction, and the other moving pixel parameter is determined.

  For example, when the digital camera 100 is tilted as shown in FIG. 3A, the equation y = x is calculated, and the gravity direction becomes the Y-axis direction. Therefore, a moving pixel in the Y-axis direction uses the initial value M, and a moving pixel in the X-axis direction is calculated as M due to the inclination.

  Similarly, when the digital camera 100 is tilted as shown in FIG. 3B, the equation y = 240 is calculated and the gravity direction becomes the X-axis direction. Therefore, the moving pixel in the axial direction uses the initial value N, and the moving pixel in the Y-axis direction is calculated as 0 because there is no inclination. Similarly, when the digital camera 100 is tilted as shown in FIG. 3C, the equation y = −1 / 2x + 400 is calculated, and the gravity direction becomes the Y-axis direction. For this reason, the initial value M is used for the moving pixel in the Y-axis direction, and the moving pixel in the X-axis direction is calculated as -1 / 2M due to the inclination.

  In S120, the system control unit 50 updates the value of the moving pixel in the Y-axis direction and the moving pixel in the X-axis direction of the composite object parameter for the VRAM size calculated in S119 (example of a preview screen at that time, FIG. 5C). . In S122, the system control unit determines whether the moving pixel parameter determination button or the cancel button of the composite object has been pressed. If the determination button is pressed, the process proceeds to S124, and if the cancel button is pressed, the process proceeds to S123.

  In S123, the system control unit 50 returns the moving pixel parameter for the VRAM size to the initial value. In S124, the operation speed detection unit 109 detects the moving direction and speed of the movement when the user shakes the camera. In S125, the system control unit 50 determines the increase / decrease direction of the number parameter according to the moving direction, and calculates the predetermined increase / decrease number in proportion to the predetermined speed. In S126, the system control unit 50 updates the number parameter for the VRAM size based on the calculated increase / decrease number (example of preview screen at that time, FIG. 5 (d)).

  In S127, the system control unit determines whether the composite object parameter determination button or cancel button of the composite object has been pressed. If the determination button is pressed, the process proceeds to S129, and if the cancel button is pressed, the process proceeds to S128. In S128, the system control unit 50 returns the composite number parameter for the VRAM size to the initial value. In S129, the system control unit 50 detects the number of times the zoom button has been pressed, and calculates a predetermined resizing rate according to the number of times the button has been pressed. When the tele side is pressed, the enlargement ratio is obtained, and when the wide side is pressed, the reduction ratio is obtained.

  In S130, the system control unit 50 calculates the size parameter of the image object for each VRAM size in proportion to the calculated resizing rate. In S131, the system control unit 50 updates the size parameter of the image object for the VRAM size calculated in S131 (preview screen example at that time (e) in FIG. 5). In S132, the system control unit 50 determines whether the size parameter determination button or the cancel button of the composite object has been pressed.

  If the determination button is pressed, the process proceeds to S134, and if the cancel button is pressed, the process proceeds to S133. In S133, the system control unit 50 returns the size parameter for the VRAM size to the initial value. In S134, the operation speed detection unit 109 detects the speed when the user shakes the camera. In S135, the system control unit 50 calculates a predetermined number of moving pixels based on the detected speed. The calculated number of pixels is added to the gravity direction parameter.

  For example, when the digital camera 100 is tilted as shown in FIG. 3A, the moving pixel parameter in the gravitational direction is M2, and the calculated moving pixel number is added to M2. Similarly, when the digital camera 100 is tilted as shown in FIG. 3B, the moving pixel parameter in the gravitational direction is N2, so the calculated number of moving pixels is added to N2. Similarly, when the digital camera 100 is tilted as shown in FIG. 3C, the moving pixel parameter in the gravitational direction is M2, and the calculated moving pixel number is added to M2.

  In s136, the current value is temporarily saved and saved in the memory, and the system control unit 50 updates the moving pixel parameter of the image object for the VRAM size calculated in S135 (preview screen example at that time (f)). In step S137, the system control unit 50 determines whether the moving pixel parameter determination button or the cancel button of the composite object has been pressed. If the determination button is pressed, the process proceeds to S139, and if the cancel button is pressed, the process proceeds to S138.

  In S138, the system control unit 50 restores the moving pixel parameter for the VRAM size to the value temporarily saved in S136. In S139, the system control unit 50 determines whether or not the composition execution button has been pressed. If the composition execution button is pressed, the process proceeds to S140, and if not, the process proceeds to S139. In S140, the system control unit 50 calculates and updates the composite object parameter for the recording size by performing proportional calculation based on the composite object parameter for the VRAM size.

  S141 to S153 are processes for combining the recorded moving image with the determined image object parameters, and are repeated until the final frame of the recorded moving image is reached. In S142, the image processing unit 24 decodes the stream. In S143, the image processing unit 24 calculates the amount of movement between the two frames by performing a correlation operation on the decoded previous frame and the entire current frame. In S144, the image processing unit 24 calculates a zooming amount between two frames by performing a correlation operation on the decoded previous frame and the entire current frame. In S145, the image processing unit 24 resizes the composite object to a size proportionally calculated from the image object size for the recording size and the calculated zooming amount.

  In S146, the image processing unit 24 detects a face from the current frame, and notifies the system control unit 50 of the number, position, and size of the face. In S148, the system control unit 50 calculates a composite position obtained by adding the amount of movement between frames to the composite position of the current frame for VRAM size. In S149, the system control unit 50 determines whether or not the synthesis position of the synthesis object is within the face range. If it is within the face range, the process proceeds to S151. If it is outside the face range, the process proceeds to S150. In S150, the image processing unit 24 combines the composite object with the current frame at the composite position.

  In S151, the system control unit 50 determines whether processing for the number of composite objects is completed. If processing for the number of composite objects is completed, the process proceeds to S152, and if not completed, the process proceeds to S148 for repetitive processing. In S152, the image processing unit 24 encodes the frame. In S153, the system control unit 50 determines whether it is the last frame of the moving image. If it is the last frame, the process proceeds to S154, and if the final frame is not output, the process proceeds to S142 for repetition.

<Example 2>
In the second embodiment, an example will be described in which a composite object parameter in the direction of gravity is calculated based on top-and-bottom direction information of an image at the time of shooting and an image object is combined with a moving image. FIG. 6 is a flowchart of the second embodiment. Each process in the flowchart is realized by the system control unit 50 expanding and executing a program stored in the nonvolatile memory 56 in the memory 32. 6 are the same as steps S100 to S117, S120 to S123, S126 to S133, and S136 to S154 of FIG. 2 described in the first embodiment. Because of this process, the description is omitted.

  In S219, the system control unit 50 reads the information on the vertical direction described in the moving image file, and determines the value of the moving pixel in the Y-axis direction and the moving pixel in the X-axis direction per frame based on the gravity direction. In S224, the system control unit 50 detects the number of times the zoom button has been pressed, and calculates a predetermined increase / decrease number corresponding to the number of times the zoom button has been pressed. When the tele side is pressed, the number of composite objects is increased, and when the wide side is pressed, the number of composite objects is decreased. In S225, the system control unit 50 calculates a composite object number parameter based on the increase / decrease number calculated in S224.

  In S234, the system control unit 50 detects the number of times the zoom button has been pressed, and calculates a predetermined moving pixel according to the number of times the button has been pressed. When the tele side is pressed, the moving pixels increase, and when the wide side is pressed, the moving pixels decrease. In S235, the system control unit 50 calculates the moving pixel parameter of the image object for each VRAM size based on the calculated number of moving pixels.

  As described above, according to the second embodiment, since the gravity direction of the fall is determined from the top and bottom information at the time of shooting, it is possible to save the user from specifying the gravity direction.

<Example 3>
In the third embodiment, an example will be described in which a composite object parameter is calculated from a trajectory moved by a touch panel and an image object is combined with a moving image.

  FIG. 7 is a flowchart of the third embodiment. Each process in the flowchart is realized by the system control unit 50 expanding and executing a program stored in the nonvolatile memory 56 in the memory 32.

  Steps S300 to S317 and S322 to S354 in FIG. 7 are the same as steps S200 to S217 and S222 to S254 in FIG.

  In S318, the system control unit 50 detects a move. In S319, the system control unit 50 records the coordinates sent by the operation unit 70 at predetermined intervals in the system memory 52. In S320, the system control unit 50 determines whether or not the locus addition button has been pressed. If the locus addition button is pressed (S320 YES), the process proceeds to S318, and if not (S320 NO), the process proceeds to S322.

  As described above, according to the third embodiment, the user can easily specify the locus of the curve since the designation is made by the locus moved by the touch panel.

<Example 4>
In the fourth embodiment, an example in which an image object parameter is calculated based on a result of detecting a camera tilt by a gyro sensor for a predetermined period and an image object is combined with a moving image will be described. FIG. 9 is a flowchart of the fourth embodiment. Each process in the flowchart is realized by the system control unit 50 expanding and executing a program stored in the nonvolatile memory 56 in the memory 32.

  Note that steps S400 to S417 and S420 to S454 in FIG. 8 are the same as steps S200 to S217 and S220 to S254 in FIG.

In S419, the tilt detection unit 108 detects the tilt of the camera at a predetermined interval for a predetermined period, and sends it to the system control unit 50. The system control unit 50 calculates coordinates from the inclination and stores them in the system memory 52. The system control unit 50 displays the calculated coordinate locus on the display unit 28.
For example, when the initial inclination is the inclination of (b) after the elapse of a predetermined time and the inclination of (c) after the elapse of a predetermined time, the locus is displayed as shown in FIG.

  As described above, according to the fourth embodiment, the user can specify the locus of a curve even in a model that does not include a touch panel.

  As described above, according to the present invention, it is possible to follow the panning or zooming of the moving image according to the parameters of the synthesized object specified by the user, and to naturally combine the recorded moving image.

  In each of the above-described embodiments, the case where the present invention is applied to a digital camera has been described as an example. However, this is not limited to this example. That is, the present invention may be applied to a playback device such as an image viewer. Furthermore, the present invention is applicable to any device that can display an image, such as a mobile phone terminal.

100 digital camera, 103 taking lens, 101 shutter, 22 image pickup unit,
23 A / D converter, 102 barrier

Claims (11)

Storage means for storing a plurality of composite objects to be combined, inclination detection means for detecting the inclination of the image processing apparatus, and interframe movement for calculating the amount of movement between frames by correlating the two frames before and after the moving image A composite object trajectory calculating unit that calculates a motion of a composite object to be combined with a moving image based on a gravitational direction obtained from a detection result of the tilt detection unit, and an inter-frame movement amount; An image processing apparatus comprising: combining means for combining a synthesized object with a moving image according to the locus calculated by the object locus calculating means. An operation speed detecting means for detecting the speed of movement of the image processing apparatus according to claim 1 and a composition for adjusting a movement speed parameter of the combined object of the trajectory according to the movement speed of the image processing apparatus detected by the operation speed detecting means. The image processing apparatus according to claim 1, further comprising an object moving speed adjusting unit. An operation speed detecting means for detecting the speed of movement of the image processing apparatus according to claim 1, and a composite amount adjustment for adjusting the amount of the composite object in the locus based on the moving speed of the image processing apparatus detected by the operation speed detecting means. The image processing apparatus according to claim 1, further comprising: means. The predetermined area avoiding means for judging to bypass the predetermined area when the predetermined area is on the second synthetic object trajectory or to pass until the predetermined area is passed is provided. Image processing device. The image processing apparatus according to claim 4, wherein the predetermined area avoiding unit sets a range of the face detected by the face detecting unit as a predetermined area. A zooming amount calculation unit that calculates a zooming amount between frames by performing a correlation operation on two frames before and after a moving image, and a composite object resizing unit that resizes a composite object from a default size according to the zooming amount. The image processing apparatus according to claim 1. And a preview unit that converts the trajectory calculated by the composite object trajectory calculation unit into a coordinate system of a VRAM size of a display device, and superimposes or combines the composite objects to reproduce a preview on the display unit. Item 8. The image processing apparatus according to Item 1. A trajectory storage means for storing one or more kinds of trajectories calculated by the composite object trajectory calculation means and the second composite object trajectory stored in the trajectory storage means by a user's synthesis execution operation are selected and synthesized into a moving image. The image processing apparatus according to claim 1, further comprising storage trajectory synthesis means. Inter-frame movement amount calculating means for calculating the amount of movement between frames by calculating the correlation between the two frames before and after the moving image, the gravitational direction obtained from the vertical direction described in the shooting information of the moving image, and the movement between the frames A synthetic object trajectory calculating means for calculating a motion of the composite object to be synthesized with the moving image based on the amount, and a synthesizing means for synthesizing the synthetic object with the video according to the trajectory calculated by the synthetic object trajectory calculating means. A featured image processing apparatus. A storage means for storing a plurality of composite objects to be combined, a trajectory input means for recognizing the touch of the screen of the display unit and inputting a trajectory for combining the composite objects, and a correlation operation between the two frames before and after the video, Based on the inter-frame movement amount calculating means for calculating the movement amount between frames, the locus obtained by the locus input means, and the inter-frame movement amount, the composite object locus for calculating the motion of the composite object to be combined with the moving image. An image processing apparatus comprising: a calculating unit; and a combining unit that combines a synthesized object with a moving image in accordance with the locus calculated by the combined object locus calculating unit. Storage means for storing a plurality of composite objects to be combined, inclination detection means for detecting the inclination of the image processing apparatus, and interframe movement for calculating the amount of movement between frames by correlating the two frames before and after the moving image A quantity calculating means, a coordinate calculating means for calculating as coordinates on the screen of the display section from a gravitational direction obtained from a detection result by the tilt detecting means for a predetermined period, and coordinates for displaying the coordinates on the display section Based on the display means, the coordinates, and the inter-frame movement amount, a synthesized object trajectory calculating means for calculating a motion of the synthesized object to be synthesized with the moving image;
An image processing apparatus comprising: combining means for combining a synthesized object with a moving image according to the locus calculated by the synthesized object locus calculating means.