JP2017059893A - Image processing apparatus, shooting guide device, imaging device, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem of a conventional image processing apparatus, built in an imaging device while included in a central control circuit, that the image effect cannot be confirmed easily in frame imaging in advance, although the travel from the previously captured image to the next image is known.SOLUTION: An image processing apparatus 114 includes a positional difference determination unit 114a, a time difference setting unit 114b, and an auxiliary image generation unit 114c. The positional difference determination unit determines the positional difference of a moving object from at least two images having common moving object and background, and the time difference setting unit sets a time difference in the frame advance of an image and a time difference of image reproduction corresponding to the positional difference. The auxiliary image generation unit generates at least one auxiliary image, becoming a frame image where the moving object moves on the background, in accordance with the positional difference determined in the positional difference determination unit, the difference in frame advance set in the time difference setting unit and the time difference of reproduction.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, an imaging guide device, an imaging device, an image processing method, and an image processing program, and in particular, an image processing device, an imaging guide device, an imaging device, an image processing method, and an image processing suitable for producing an animation movie. Regarding the program.

  In addition to digital cameras and digital video cameras, mobile information terminal devices such as mobile phones (including smartphones (including multi-function mobile phones)) and tablet terminals (including tablet PCs) also have shooting functions (camera functions). These are collectively referred to as an imaging device.

In an imaging apparatus such as a digital camera, a digital video camera, or a portable information terminal device, an image from a subject (subject image) is formed on an image sensor via a photographing lens, and is photoelectrically converted and digitized. The digitized image is output to the central control circuit, and necessary image processing such as color and gradation correction and compression processing is performed in the central control circuit. For example, JPEG format for still images and H.264 for moving images. It is converted into an image file in the H.264 format and recorded in a recording unit such as a flash memory.
Note that an imaging unit of the imaging apparatus is configured by the imaging lens, the imaging element, and the like.

In order to confirm the composition and timing of shooting prior to shooting, the image signal from the imaging unit is displayed as a live view image (through moving image) on the display unit. On the display unit, not only the live view image before photographing but also the image signal recorded in the recording unit is expanded and displayed (reproduced) by the central control circuit. The display unit is formed from a liquid crystal display (LCD) or the like. The central control circuit not only processes the image, but also controls the operation of the components of the imaging device, such as a recording unit, a display unit, and a series of switches (for example, a release switch, a power switch, a menu key, etc.).
The subject image is confirmed with the live view image, shot under the operation of the release switch, a digital image is continuously shot and a frame-by-frame compression is performed, and a moving image is obtained. It is done.

  If you take a character (moving object) as a subject and move it in the background little by little, you can create an animated movie (animated movie) that moves the character. An animation moving image is usually created by combining moving images of a plurality of scenes. The character is, for example, a clay molding formed from clay. However, it is not limited to a clay molding, and may be molded from a material other than clay.

  Japanese Patent Application Laid-Open No. 2008-060661 discloses an electronic camera (imaging device) that creates a file (moving image file) for an animation moving image and suggests a moving distance of a character (subject) for each time-lapse shooting when the moving image file is created. Have been described.

JP 2008-060661 A

  In Japanese Patent Laid-Open No. 2008-060661, an image of a previous shot frame (previous shot image) and a live view image are displayed superimposed on a liquid crystal display (display unit). For this reason, the positions of the previously captured image and the live view image are confirmed on the liquid crystal display, and the moving distance from the previously captured image to the live view image (the amount of movement for each time-lapse shooting) is known. If the subject is a character, an animated movie is shot.

  In this configuration, although the moving distance from the previous photographed image to the next image can be known, the image effect in the time-lapse photography cannot be easily confirmed in advance.

  An object of the present invention is to provide an image processing apparatus, an imaging guide apparatus, an imaging apparatus, an image processing method, and an image processing program that can easily check an image effect in time-lapse photography in advance.

In the present invention, an image assisting shooting (imaging) in frame advance is created.
That is, according to the first aspect of the present invention, the image processing apparatus includes a position difference determination unit that determines a position difference of a moving object from at least two images having a common moving object and a background, and a frame of the image. A time difference setting unit that sets a time difference in feeding and a time difference in reproduction of the image corresponding to the position difference, and a time difference in the frame feeding set in the time difference setting unit and the position difference determined in the position difference determining unit. And an auxiliary image creating unit that creates at least one auxiliary image that is a frame image in which the moving object moves on the background according to the time difference of reproduction.
According to the present invention, the photographing guide device includes a position difference determination unit that determines a position difference of a moving object from at least two images having a common moving object and a background, and frame advance of the image. A time difference setting unit that sets a time difference and a time difference of reproduction of the image corresponding to the position difference; and the time difference and reproduction in the position difference determined by the position difference determination unit and the frame advance set by the time difference setting unit According to the time difference, the moving object is displayed in a semi-transparent manner and includes an auxiliary image creating unit that creates at least one auxiliary image arranged in the screen.
According to the present invention described in claim 11, the imaging apparatus includes a display unit that displays an image, and an image processing device that processes the image and displays the processed image on the display unit. A processing apparatus determines a position difference of a moving object from at least two images having a common moving object and a background, and reproduces the image corresponding to the time difference and the position difference in frame advancement of the image. The moving object is the background according to the time difference setting unit for setting the time difference, the position difference determined by the position difference determination unit, and the time difference in the frame advance and the reproduction time difference set by the time difference setting unit. And an auxiliary image creation unit that creates at least one auxiliary image to be a frame image moving up.
According to the invention of claim 12, the image processing method includes a step of determining a position difference of a moving object from at least two images having a common moving object and a background, a time difference in frame advance of the image, and the A step of setting a reproduction time difference of the image corresponding to the position difference, and a frame image in which the moving object moves on the background according to the determined position difference, a time difference in the frame advance and a reproduction time difference, and And creating at least one auxiliary image.
According to the present invention of claim 13, the image processing program has a function of determining a position difference of a moving object from at least two images having a common moving object and a background, a time difference in frame advance of the image, and the A function of setting a reproduction time difference of the image corresponding to a position difference, and a frame image in which the moving object moves on the background according to the determined position difference, a time difference in the frame advance and a reproduction time difference, and The computer is configured to realize a function of creating at least one auxiliary image.

  In the present invention, since an auxiliary image that assists (guides) shooting (imaging) in frame advance is created, the image effect in frame advance can be easily confirmed.

1 is a conceptual block diagram of an image processing unit (image processing apparatus) according to an embodiment of the present invention. The schematic diagram of the imaging device which concerns on one Example of this invention which consists of a combination which wirelessly connected the lens type digital camera and the smart phone is shown. (A)-(D) show the schematic diagram of auxiliary image preparation by this invention. (A)-(C) show the screen of a smart phone, respectively. The screen of the smart phone when “smoothness” is selected on the screen of FIG. (A)-(D) show the screen of a smart phone when “time difference” is selected on the screen of FIG. (A)-(C) show the screen of the smart phone when “music selection” is selected on the screen of FIG. 6 (A). FIGS. 4A and 4B show smartphone screens when “positional difference” is selected on the screen of FIG. FIGS. 4A to 4D show smartphone screens when “Guide” is selected on the screen of FIG. The screen of the smart phone when “prediction” is selected on the screen of FIG. 6 shows a flowchart of creating a moving image for an animated moving image according to an embodiment of the present invention. The subroutine of auxiliary image creation is shown. (A) to (C) show a character image at the start point and end point in the creation of an interpolation image, and a live view image (background live view image) where no character exists, respectively.

  The image processing apparatus includes a position difference determination unit, a time difference setting unit, and an auxiliary image creation unit. The position difference determination unit determines the position difference of the moving object from at least two images having a common moving object and background, and the time difference setting unit determines the time difference in image frame advance and the time difference of image reproduction corresponding to the position difference. It is set. The auxiliary image creating unit has at least one frame image in which the moving object moves on the background according to the position difference determined by the position difference determination unit and the time difference in frame advance and the reproduction time difference set by the time difference setting unit. Two auxiliary images are created.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a conceptual block diagram of an image processing unit (image processing apparatus) according to an embodiment of the present invention. FIG. 2 is a schematic diagram of an imaging apparatus according to an embodiment of the present invention, which is a combination of a lens-type digital camera and a smartphone wirelessly connected.
Note that the image processing apparatus 10 creates an image that assists shooting (imaging) in frame advance and guides shooting, and can be called an imaging guide apparatus.

  The main part of the present invention is an image processing unit (image processing apparatus) that creates an auxiliary image as a material for shooting an animated moving image. For this reason, the description of the invention is simplified by separating the constituent members of the imaging apparatus other than the image processing unit, for example, the imaging unit. An imaging apparatus according to an embodiment of the present invention in which an image processing unit (image processing apparatus) is incorporated can be configured by combining, for example, a lens-type digital camera with a smartphone.

  In FIG. 2, the imaging device 10 is configured by wirelessly connecting a lens-type digital camera 10a and a smartphone 10b, and the smartphone functions as a main body of the imaging device and the digital camera functions as an imaging unit. Here, since the lens-type digital camera 10a is equipped with a lens mount, the lens-type digital camera may be integrated into the smartphone 10b using the lens mount.

  Smartphones can send and receive content using various communication lines, making it easy to upload created videos to external sites and to download viewing images and material images from external sites. Yes. Of course, tablet computers and computers can be substituted for the same reason. Also, as will be described later, services such as music data download can be used.

As shown in FIG. 1, the imaging device 10 includes an imaging unit 12, a central control circuit 14, an operation unit 16, a display unit 18, a recording unit 20, a communication unit 22, and a time measuring unit 24.
The imaging unit 12 captures a plurality of images, and in the embodiment shown in FIG. 2, the lens-type digital camera 10 a that is separate from the smartphone 10 b functions as the imaging unit 12, and the central control circuit 14 to the timing unit 24 are connected to the smartphone ( (Main body part) 10b of the image pickup apparatus. The central control circuit 14 includes an image processing unit (image processing device) 114 that creates an auxiliary image serving as an assist (guide) for shooting (imaging) in frame advance.

  In the following, the operation unit 16, the display unit 18, the recording unit 20, the communication unit 22, and the time measuring unit 24 are naturally equipped with the smartphone 10b. Device) 114 will be described in detail. The imaging unit 12 is provided in a lens-type digital camera, but since its configuration is the same as a known one, it will be briefly described here.

The imaging unit 12 includes an imaging element such as a photographing lens, a CCD, and a CMOS sensor, and an image from a subject is formed on the imaging element via the photographing lens, photoelectrically converted, digitized, and the central control circuit 14. Is output. The central control circuit 14 performs necessary image processing such as color and gradation correction and compression processing. For example, the JPEG format is used for still images, and H.264 is used for moving images. It is converted into an image file in the H.264 format and recorded in a recording unit (memory) 20 such as a flash memory.
The timekeeping unit (clock) 24 measures the shooting date and time, and the date and time information is recorded in the memory 20 together with the shot image (captured image), so that the captured image is recorded in order and can be easily searched.

  The display unit 18 displays an image, and an image signal from the imaging unit 12 is displayed as a live view image (through moving image) in order to confirm the composition and timing of the shooting prior to shooting (imaging). On the display unit 18, not only the live view image before photographing but also the image signal recorded in the memory 20 is expanded and displayed for reproduction by the central control circuit 14. The display unit 18 is formed of a liquid crystal display (LCD) or the like, and is provided on the upper surface of the smartphone 10b.

  The central control circuit 14 that processes the captured image from the imaging unit 12 includes an image processing unit (image processing apparatus) 114, and includes, for example, a CPU and an MPU. An image of the character photographed in the animation mode is recorded in the memory 20, reproduced by the central control circuit 14, displayed on the liquid crystal display (display unit) 18, and the photographed result is immediately confirmed.

  The communication unit 22 includes a transmission / reception unit that transmits and receives data by USB connection, wireless communication, and the like. The captured image data recorded in the memory 20 is transmitted to an external display unit such as a TV or another imaging device via the communication unit. Can be output from the smartphone 10b. Moreover, the data of the picked-up image recorded on another imaging device can be input into the smart phone 10b via the communication part 22. FIG.

Image data 24a, music data 24b, and the like are stored in, for example, an external database (DB) 24, and the communication unit 22 communicates with an external database via the Internet, a communication cable, or the like. Therefore, information such as image data 24 a and music data 24 b can be downloaded (saved) from the external database 24 to the smartphone 10 b via the communication unit 22. Also, photographed image data incorporating the music data is uploaded (saved) to an external database via the communication unit 22.
Of course, the database 24 such as the image data 24 a and the music data 24 b may be built in the imaging apparatus 10. In this case, the image data and the like are stored in the memory 20.

In the smartphone 10b, a layer (conductive layer) that recognizes a touch (tap) operation by a photographer's finger is disposed on the entire surface of the liquid crystal display 18, and switches such as a power switch, a release switch, and a mode key are, for example, on the liquid crystal display. Is displayed. Of course, a physical dedicated switch may be provided. For example, a frequently used release switch may be provided on the peripheral surface side of the smartphone 10b.
The operation unit 16 detects a photographer's switch operation by a power switch, a release switch, a mode key, and the like by touching an icon on the liquid crystal display, and the detection result is output to the central control circuit 14, and the central control circuit performs a photographing operation. Is controlled.

An image processing unit (image processing apparatus) 114 is included in the central control circuit 14 and incorporated in the imaging apparatus 10.
The image processing unit 114 includes a position difference determination unit 114a, a time difference setting unit 114b, and an auxiliary image creation unit 114c.
The position difference determination unit 114a makes a determination by comparing the position difference of the moving object from a plurality of images having a common moving object and background, and the time difference setting unit 114b determines a time difference between the plurality of images (time difference of frame advance and time difference). Playback time difference) is set. As will be described later, the position difference determination unit 114a can also determine (determine) the moving object and the background from the target image. The auxiliary image creation unit 114c has at least one auxiliary image having a moving object and a background in accordance with the position difference determined by the position difference determination unit 114a, the time difference in frame advance and the reproduction time difference set by the time difference setting unit 114b. Have created. Here, the moving object refers to a character as a subject.
For example, the auxiliary image creation unit 114c creates an auxiliary image after subjecting the moving object and the background to image processing from the target image as necessary.

The image processing unit 114 further includes an operation control unit 114d, a communication control unit 114e, and an image capturing unit 114f.
The operation control unit 114d controls the operation result detected by the operation unit 16 and output to the central control circuit 14, and outputs the result to the position difference determination unit 114a and the time difference setting unit 114b.
The communication control unit 114e controls settings and protocols of communication partners such as image data and music data input / output via the communication unit 22.

For example, music data 24b input from an external database 24 via the Internet via the communication unit 22 is acquired by controlling access to a server or the like that performs a specific service using the communication control unit 114e. The result of analyzing is obtained. Note that such a function can be provided by the time difference setting unit 114b, for example.
Image data from the external database 24 is captured by the image capturing unit 114f as controlled by the communication control unit 114e. Therefore, not only the image photographed by the imaging unit 12 but also an external image can be used to create a moving image frame that is the basis of the animated movie.

3A to 3D are schematic views of auxiliary image creation according to the present invention.
In FIG. 3, the imaging device 10 is integrated by incorporating a lens-type digital camera 10a into a smartphone 10b, and the smartphone is supported by a tripod 10c.

FIGS. 3A and 3B show a state where the character is placed at the start point and the end point, respectively.
First, as shown in FIG. 3A, the subject character Ca is placed at the start point (start point position) of the moving image, and for example, the release switch on the peripheral surface side of the smartphone is pressed to capture (capture) the start point image. The Then, the character Ca is moved from the start point to the end point. Then, as shown in FIG. 3B, the character Ca is placed at the end point (end point position) of the moving image, and the release switch on the peripheral surface side of the smartphone is pressed to take the end point image.

FIGS. 3C and 3D respectively show images in auxiliary image creation.
For example, the images of the captured start point image Im-s and end point image Im-f are taken, and the positional difference (the change in the position of the subject between the start point and the end point, that is, the distance between the start point and the end point) is Dm. To do. Then, it is assumed that the character moves the position difference Dm with a frame advance time difference of 2 seconds (= F) and a reproduction time difference (time from the start point to the end point) of 6 seconds (= Tm). Then, it is only necessary to move the character between the start point and the end point with Tm / F = 6/2 = 3 and finely divided into three parts.

  For example, in FIG. 3C, if the character moves a distance of 2/3 (= Dm) on the screen in the moving direction, the interval for frame advance from (2/3) / 3 = 2/9. ΔDm is 2/9 on the screen. Therefore, if two images (auxiliary images) serving as guides are arranged between the start point and the end point at this division position, a scene (scene 1) having a frame advance interval of ΔDm and a position difference of Dm is obtained.

  Instead of shooting a character at the start and end points, as shown in FIG. 3D, if a character is shot at two positions including the start point, an image that guides the movement in the direction of the character's movement is predicted and the scene is predicted. Can be created. In order to facilitate background setting, it is preferable to shoot so that two images do not overlap. Since the image to be guided (auxiliary image) can be created infinitely, the position of the image at the end point is determined from the time difference (the frame advance time difference and the reproduction time difference).

  For example, in FIG. 3 (D), as in FIG. 3 (C), the distance between the captured start point and the next position image (interval in frame advance) is ΔDm, and the position between the start point image and the end point image. It is assumed that the difference (position change of the image at the start point and end point) is Dm, the time difference of frame advance (time required for one frame) is 2 seconds, and the playback time (video (scene) playback time) takes 6 seconds. . Then, two images (including the image at the end point position) are predicted between the position after the start point and the end point position, and the distance ΔDm between the image at the start point and the next position is 2/9 on the screen in the direction in which the character moves. If so, the position difference Dm is 2/3 of the screen.

  In this manner, as a guide for creating animation animation, an interpolation guide shown in FIG. 3 (C) for capturing characters at the start and end points and interpolating between the images, and characters at the first two positions are used. A prediction guide shown in FIG. 3 (D) that captures an image up to the end point can be considered. Then, by determining the position difference between a plurality of images and setting the frame advance time difference and the reproduction time difference, an interpolation image is created in the interpolation guide and a prediction image is created in the prediction guide, respectively. For example, in the embodiment of FIGS. 3C and 3D, the position difference is Dm (2/3 of the screen), the frame advance time difference is F (2 seconds), and the playback time difference is Tm (6 seconds). A scene (one moving image) is determined and set.

Although not shown, the scenes 2... Following the scene 1 are sequentially created by the same method. Then, for example, a character is placed at the position of the auxiliary image (interpolated image, predicted image), or the captured image is enlarged and reduced and superimposed on the auxiliary image. Images (moving images) are sequentially created.
Auxiliary images (interpolated images, predicted images) are created and used as the material for video shooting, so you can easily know in advance the images in time-lapse shooting, and the video that is the basis of the animation video (scenes 1, 2,.・ ・) Can be created easily.

In the following, operations in the interpolation guide will be mainly described.
4A to 4C show smartphone screens, respectively.
The initial screen of the application of the smartphone 10b wirelessly connected to the imaging unit (lens type digital camera) 10a (FIG. 2) is, for example, the screen of FIG.
First, when “external camera” is touched on the screen of FIG. 4A, the screen of FIG. 4B is displayed on the smartphone 10b, and “camera” is selected. Then, for example, the start and end images are taken in advance in the interpolation guide, and the start and next images are taken in advance in the prediction guide.

  Of course, in the embodiment, a camera with a built-in smartphone may be used instead of the lens-type digital camera 10a. However, in order to keep the relationship between the camera and the subject constant, a camera (external camera, lens type digital camera) other than the camera built in the smartphone is easier to operate with the smartphone. Also, with an external camera, it is possible to select a dedicated optical system for macro photography, telephoto photography, etc., and there is an advantage that the application range is expanded.

  Then, returning to the screen of FIG. 4B and touching “animation mode”, an animation mode menu shown in FIG. 4C is displayed. In FIG. 4C, four menus “position difference”, “time difference”, “smoothness”, and “guide” are displayed.

FIG. 5 shows a smartphone screen when “smoothness” is selected on the screen of FIG.
In the embodiment, “smooth”, “normal”, and “parala” are set as the “smoothness” of the movement of the character, which is a moving image expression. “Smoothness” corresponds to the number of frames, and if the number of frames is small, the movement of the character is “ParaPara”, that is, it is a smooth (patter-like) frame-taking movement, and if the number of frames is large, it becomes “Smooth”. “Normal” moves between them. For example, if the user wants to express “feeling of happiness”, “smoothness” is set to “smoothness”.

In order to set “smoothness”, the image processing apparatus (image processing unit) 114 may include a “smoothness setting unit”. However, the position difference determination unit 114a and the time setting unit 114b may be configured to be able to set “smoothness” such as “smooth”, “normal”, and “parala” and also to function as a “smoothness setting unit”.
Alternatively, the rhythm, tempo, volume, and the like of the music data may be analyzed, and “smoothness” may be automatically set from the analysis result.

If you set “Smoothness”, you can get a smoother motion video (images of scenes 1, 2,...), But the “Smoothness” setting is not essential and you can skip that setting. May be.
When the preference of the user (photographer) is constant smoothness, or when the smoothness of a moving image is automatically adjusted to music, the “smoothness” setting by the user can be omitted. If the “smoothness” setting is omitted, the setting procedure is reduced and the burden on the user is reduced.

  Touch any of “Smooth”, “Normal” or “Parapara” on the screen of FIG. 5, set “Smoothness”, and then touch “Return” to save the set “Smoothness” and save FIG. C) screen is displayed.

  6A to 6D show smartphone screens when “time difference” is selected on the screen of FIG. 4C. 6 (C) and 6 (D), the horizontal axis indicates the frame feed time difference, and the vertical axis indicates the distance from the start point to the end point.

  When “Time difference” is touched and selected on the screen shown in FIG. 4C where the “smoothness” is saved and switched, the initial screen of “Time difference” shown in FIG. 6A is displayed. Two menus of “time difference” and “specify music” are displayed. When “time difference” is selected by touching, a screen shown in FIG. 6B is displayed. Here, 2 seconds is initially set as the time difference between frames (time required for one frame). However, if the upward triangle mark (Δ) or the downward triangle mark (▽) is touched, the time difference can be increased or decreased in 0.5 second increments, for example. When “Pattern input” is touched on the screen of FIG. 6B, a character movement pattern screen shown in FIG. 6C is displayed.

In the screen of FIG. 6 (C), for example, the character movement pattern is initially set to a straight line where the character changes constantly from the start point to the end point, and the character moves with the same movement (with no strong or weak accent). doing.
However, for example, if the finger is slid on the screen while part of the straight line of the pattern is touched with the finger in FIG. 6C, the pattern can be changed at that point. For example, if the finger is slid downward as shown by the arrow in FIG. 6D while touching a part of the straight line of the pattern with the finger, the straight part of the touched pattern is lowered along with the finger slide. To be curved. Then, the character's movement partially slows down at that point, and a movement pattern with strong and weak accents as shown in FIG. 6D is set.
If such an operation is applied, not only the speed and speed of movement, but also the position of the character can be shifted left and right.

Alternatively, the character movement pattern may be automatically changed based on the rhythm and tempo of the music data. For example, the character can be moved quickly or slowly according to the rhythm, tempo slowness, and strength of the music data.
This is easily realized if the time difference setting unit 114b has a function of determining a music piece, monitoring a change in sound intensity and a time change in frequency, and detecting a repetitive feature, a pattern shape, and the like. it can. Such repetition can be easily determined if the time of strength change appears many times and is approximately the same time. Since the rhythm may become faster or slower, it is better to determine the rhythm change rule and accurately determine the rhythm. Hereinafter, a representative example will be described.

  For example, touching “Return” in FIG. 6B returns to the screen in FIG. Further, when “Return” is touched in FIGS. 6C and 6D, the movement pattern is saved and the screen returns to the screen of FIG.

  FIGS. 7A to 7C show smartphone screens when “music selection” is selected on the screen of FIG. The “music” is divided into a plurality of phrases (one melody break, musical phrase) at automatic points, and the phrases 1, 2,. In “music specification”, music played in the position difference of each scene (position change by frame advance, position change of start and end images, movement distance in one scene), in other words, the character reaches from the start point to the end point. The music flowing in the section (section music) is set by selecting the phrase of the music for each scene.

When “music designation” is selected by touching in FIG. 6A, an initial screen of “music designation” shown in FIG. 7A is displayed on the smartphone.
In the initial screen of FIG. 7A, two music data “music 1” and “music 2” stored in the memory 20 and the external database 24 are displayed. If the downward triangle mark (▽) is touched, “Song 3” and the subsequent parts are displayed, and if the upward triangle mark (Δ) is touched, “Song” having a young song number is displayed.
For example, when “music 1” is touched, the waveform (volume) of phrase 1 of music 1 is displayed with the horizontal axis representing the music time and the vertical axis representing the music volume, as shown in FIG. 7B.

  In FIG. 7B, when the start point and the end point are determined by touching the screen on the waveform of phrase 1, the music (section music) to be played in the section from the start point to the end point is selected and specified. The Here, when the start point and end point are determined by touching the screen on the waveform of phrase 1, the time difference required from the start point to the end point (reproduction time difference, that is, the time required for one scene) is determined and set correspondingly. In this way, the reproduction time difference is set by “music specification”.

Touching “trial” after setting the segment music will select the phrase 1 and the segment music of the set part will flow. If the set section music is not liked, the section music selected by changing the positions of the start point and end point is changed. For example, if the start point and end point of the phrase 1 waveform are touched and slid left and right on the screen, the segment music adopted from the phrase 1 is changed.
If the positions of the start point and end point are changed, the reproduction time difference (time required for one scene) is also changed accordingly.

  If there is no favorite music (section music) in phrase 1 in FIG. 7B, touch the triangle pointing to the right in FIG. 7B to switch to the next phrase (phrase 2) screen (FIG. 7C )). The same operation as the designation of the section music of phrase 1 in FIG. 7B is repeated also in FIG. 7C to designate the section music from phrase 2.

If there is no favorite song in phrase 2, touch the triangle pointing to the right in Fig. 7 (B) to switch from phrase 2 to phrase 3, and touch the screen on the phrase 3 waveform to set the start and end points. Decide and select the segment music from phrase 3. When the left triangle mark in FIG. 7C is touched, the phrase 2 is returned to the phrase 1, and the selection and designation of the segment music from the phrase 1 are examined again. In this way, the section music played in the section from the start point to the end point is selected from the phrase.
Note that when “OK” is touched in FIGS. 7B and 7C, the screen settings are saved, and the screen returns to the initial screen in FIG. 7A where “Return” is touched.

  It is possible to set the section music from the phrase of a certain song for the time being, then repeat the setting of the section music from another phrase and the phrase of another music to set and replace (overwrite) the favorite section music. Good. That is, the section music on the screen where “OK” is finally touched may be determined and set as the section music.

FIGS. 8A and 8B show smartphone screens when “positional difference” is selected on the screen of FIG.
When “positional difference” is touched and selected in FIG. 4C, for example, images of the start point and end point in the interpolation guide are displayed on the smartphone 10b as shown in FIG. 8A. For example, a captured image (captured image) of the character at the start point of scene 1 in the interpolation guide is displayed in the upper half of the screen, and an image at the end point is displayed as a live view image in the lower half of the screen. .

  The user (photographer) moves the character and places the character at a preferred position from the relationship with the start point position to determine the end point position. In addition, the position of the character at the starting point is a predetermined position. If it is not preferable, the character is moved to change the starting point position, and the character is placed at the starting point position considered to be preferable.

  Various positions of the start point and end point are examined, and when the start point and end point are finally determined, a character is photographed at the start point and end point, and images at the start point and end point are determined. Then, for example, by superimposing two images on the basis of the background, the position difference (movement distance in one scene, Dm in FIG. 3C) is read (calculated) by the position difference determination unit 114a. 2/3 is judged (designated) on the screen.

If phrase 1 is selected from music 1, the screen of scene 1 in the interpolation guide is shown in FIG. 8B, and the section music specified in FIG. 7 flows and is confirmed.
Touching “RETURN” in FIGS. 8A and 8B saves the positional difference and switches to the screen of FIG. 4C.

Also in the prediction guide, the image of the start point and the image of the next position are photographed in the same manner as the interpolation guide, and the position difference is determined.
That is, in the prediction guide, the captured image of the character at the start point of scene 1 is displayed in the upper half of the screen of FIG. 8A, and the next image of the start point is displayed as a live view image in the lower half of the screen. Is displayed. The photographer moves the character and places the character at a preferred position based on the relationship with the starting point, and determines the next position. Also, the position of the starting point character is a tentatively determined position, and if it is not desirable, the character is moved to change the starting point position.
The starting point and the next position are examined in various ways. When the starting point and the next position are finally determined, the character is photographed at the starting point and the next position, and the images at the starting point and the next position are determined.

In the prediction guide, the position of the end point of the character is undecided, and the position difference (movement distance in one scene, Dm in FIG. 3D) is not fixed.
However, the time difference is set to 2 seconds with the setting of “time difference” in FIG. 6, and the section music time, that is, the playback time is set by specifying the section music with “music specification” in FIG. 7. For example, if the reproduction time is 6 seconds, two images (including the image at the end point) are predicted from the image at the next position to the image at the end point. Therefore, the position difference is three times the start point and the position change of the next image (ΔDm in FIG. 3D), and the position difference (Dm = ΔDmX3) is determined and determined (designated). Further, the position of the end point image is designated along with the determination of the position difference.

  Then, if “scene 1” is touched in FIG. 8A, the screen of scene 1 in which phrase 1 of music data 1 is incorporated is displayed in FIG. 8B, and the section music specified in FIG. Flows and is confirmed.

FIGS. 9A to 9D show smartphone screens when “Guide” is selected on the screen of FIG. 4C.
When “Guide” is selected by touching in FIG. 4C, the screen shown in FIG. 9A is displayed. When “Interpolation” is touched in FIG. 9A, an interpolation guide screen shown in FIG. 9B is displayed.

  In FIG. 9B, the screen set in FIG. 7, the phrases set in “Scene 1” and “Scene 1” are displayed. For example, if “phrase 1” is set as the section music of “scene 1”, that is, if the screen of FIG. 7B is set, “scene 1” and “phrase 1” are displayed. If “Phrase 2” is set as the section music of “Scene 1” instead of the screen of FIG. 7B, that is, if it is set as shown in the screen of FIG. “Scene 1” and “Phrase 2” are displayed.

  If “OK” is touched on the display screen of “Scene 1” and “Phrase 1” in FIG. 9B, the screen changes to FIG. 9C, and the section music set in FIG. The image of the positional difference determined in FIG. 8 is displayed in FIG. In addition, if “smoothness” is selected in FIG. 5 and “smoothness” is selected as “smoothness”, for example, “smoothness” is also displayed in FIG. 9C. In FIG. 9C, a positional difference, a time difference, a piece of music, smoothness, etc. are confirmed.

If “OK” is touched in FIG. 9C, the screen changes to FIG. 9D.
In FIG. 9D, the character of the interpolation guide from the start point to the end point passes through two guide images (interpolated images) obtained by dividing the start point and end point into three parts based on the position difference determined in FIG. The image is simulated and displayed as a scene 1 under the smoothness of “ParaPara” in the section 1 of the phrase 1 of the song 1.
In FIG. 9D, four circles on the phrase waveform indicate the start point, end point images, and two interpolated images between them.

  If the image of the simulated character is acceptable, “Adopt” is touched in FIG. 9D, the scene 1 is saved, and the process returns to FIG. 9B. If not, touch “Return” to return to FIG. Then, for example, returning to FIG. 4C, settings such as “positional difference” are performed again.

  If the simulated character image is acceptable, the character is placed at the position of the auxiliary image (interpolated image) between the start point and end point in the simulation before touching “Adopt” in FIG. , Press the release switch on the peripheral side of the smartphone to shoot the character. Alternatively, the start point and end point images (captured images) may be enlarged or reduced, and the interpolated image may be framed by overlapping the interpolated image.

In FIG. 9D, touch “recruit” to determine the scene 1, save it, and touch “return” to return from FIG. 9D to FIG. 9B.
When “Scene 1” is touched in FIG. 9B, “Scene 2” is displayed in FIG. 9B, and the phrase designated by “Scene 2”, for example, phrase 2 is displayed.

When “OK” is touched on the screen of FIG. 9B on which “Scene 2” and “Phrase 2” are displayed, and “OK” is touched on the screen of FIG. 9C that has been switched, FIG. 9D is displayed. The screen switches to. Then, as in the case of the scene 1, the scene 2 is simulated on the screen of FIG. If the simulation image is acceptable, after the interpolated image is photographed or the interpolated image is framed, “Adopt” is touched to determine and save the scene 2. If the simulation image is not satisfactory, touch “Return” and redo settings such as “Position Difference”.
Similarly, scene 3 and subsequent scenes are also determined, and images (moving images) of scenes 1.

  Since an auxiliary image for assisting shooting in frame advance is created, the image effect in frame advance can be easily confirmed. In addition, it is possible to easily create an image that is the basis of an animated movie, and it is also possible to easily create a scene with a song phrase.

  In addition, although the music data built in the smartphone 10b may be used, if music data is imported from outside using a network, a variety of music data can be used. Can be selected and set.

FIG. 10 shows a smartphone screen when “prediction” is selected on the screen of FIG.
When “prediction” is touched and selected on the screen of FIG. 9A, a “prediction” screen shown in FIG. 10A is displayed.

  If “scene 1” and the phrase set in “scene 1” in FIG. 7 are displayed in FIG. 10 (A), and if segment music is set from “phrase 1” in “scene 1”, FIG. “Scene 1” and “Phrase 1” are displayed in A). If “scene 1” or “phrase 1” is acceptable, touch “OK”. Then, the screen is switched to FIG. In the prediction guide, instead of the images at the start and end points in the interpolation guide, a screen displaying the start point and the next image on the left and right at the top of the screen is displayed as the scene 1 and phrase 1 screen in FIG. The

  Also, images of the section music set in FIG. 7B, the time difference of reproduction, and the position difference determined in FIG. 8 are displayed in FIG. Further, “smoothness” selected in FIG. 5, for example, “Parapara” is also displayed in FIG. 10B. In FIG. 10B, position difference, time difference, segment music, smoothness, etc. are confirmed.

Touching “OK” in FIG. 10B switches to FIG.
In FIG. 10C, based on the position difference determined in FIG. 8, the start point, the next image (from the start point), and the two images (from the next image from the start point) to the end point image (2 The image of the four characters (the predicted image is the end image) is simulated and displayed as the scene 1 with the smoothness of “Parapara” in the section music of the phrase 1 of the music data 1.

In FIG. 10C, the four circles on the phrase waveform indicate the start point, the next image, and two predicted images. When the circle at the right end corresponding to the predicted image at the end point position is touched and slid left or right on the screen, the segment music is extended or shortened. Here, if it slides to the left, a section music will be shortened and the number of prediction images will decrease, and if it slides to the right, a section music will be extended and the number of prediction images will increase.
In FIG. 10D in which the rightmost circle corresponding to the predicted image at the end point position is touched and slid to the right on the screen, the section music is extended and the number of predicted images is increased from 2 to 4, and 6 images are displayed. The number of circles on the waveform of the section music is also changed from 4 to 6.

If a simulation image is acceptable, a predicted image is taken, and then “adopt” is touched to determine scene 1 and save it. If the simulation image is not satisfactory, touch “Return” and redo settings such as “Position Difference”.
Similarly, scenes 2 and after are also determined, and images (moving images) of scenes 1.

FIG. 11 is a flowchart for creating a moving image for an animated moving image according to an embodiment of the present invention.
First, “external camera” is selected on the initial screen of the smartphone shown in FIG. 4A, “camera” is selected in FIG. 4B, the start point and end point images in the interpolation guide, and the start point and next in the prediction guide. Images are taken in advance.

Then, “animation mode” is selected in FIG. 4B, and the animation mode menu (FIG. 4C) is displayed in S1. In S2, it is determined whether all menus in the animation mode have been selected (set). If all the menus are not selected, the process branches from S2 to S21. Note that the number of menus selected by the menu selection in S2 is “positional difference”, “time difference”, and “smoothness”, and these are determined in this order.
As described above, the position difference is the subject position change between the start point and the end point (movement distance in one scene), the time difference is the frame feed time difference (time required for one frame) and the playback time difference (time required for one scene). Say.

  First, determination of “positional difference” is made in S21. When the “positional difference” is read and determined by the positional difference determination unit 114a (FIG. 8), the process proceeds to S23 and the setting of the “time difference” is determined. If the photographer forgets reading and determination by the position difference determination unit 114a in FIG. 8, the process branches to S22, where “position difference” is determined (FIG. 8), and the process returns to S1.

  When the “positional difference”, “time difference”, and “smoothness” are determined and set, the screen returns to the initial screen of the smartphone (FIG. 4A), and the menu screen of FIG. However, in order to avoid complication, the flow for returning to the initial screen of the smartphone is omitted.

When “time difference” is set in S23, the process branches to S24, where “time difference” is set (FIG. 6), and the process returns to S1.
When returning from S24 to S1, it is determined again whether the menu is selected in S2, and if there is a menu that has not been selected, the process branches to S21 to determine "positional difference" in S21. To be judged. When the “positional difference” is determined, the process proceeds from S21 to S23, and the setting of the “time difference” is determined. If “time difference” is set, the setting of “smoothness” is determined in S25.

  The setting of “smoothness” is determined in S25. If “smoothness” is set, “smoothness” is set in S26 (FIG. 5), and the process returns to S1. If not, the process returns from S25 to S1.

  Returning to S1 from S25 or S26, it is determined whether the menu is selected (determined or set) in S2, but if “positional difference”, “time difference”, or “smoothness” is selected, the process proceeds to S3. In step S3, “condition OK” is determined. That is, in S3, it is determined whether or not three menus (conditions) “positional difference”, “time difference”, and “smoothness” are determined and set. If there is something that has not been determined or set, the process branches to S4, a warning is given of no input (not determined, not set), and the process returns to S1. Then, S1 to S4 and S21 to S26 are repeated until “positional difference”, “time difference”, and “smoothness” are all input (determination, setting).

When all three menus of “positional difference”, “time difference”, and “smoothness” are determined and set, and “condition OK” is determined in S3, the process proceeds to S11 and an auxiliary image is created.
FIG. 12 shows a subroutine for creating an auxiliary image. FIGS. 13A to 13C show a character image at a start point and an end point in creation of an auxiliary image, and a live view image (background live view image) where no character exists.

In the auxiliary image creation subroutine, the difference between the two images is determined in S101. That is, it is determined which of the two captured images is the image of the character at the start point and the other is the other image (image of the character at the end point in the interpolation guide, image of the character next to the start point in the prediction guide). Is determined. For example, in a scene where the character moves in the direction of the photographer, of the two images, the relatively small character image is the start point image, the relatively large character image is the end point image, or the next to the start point. It is determined as an image.
Since FIG. 13 is an interpolation guide image, it is determined that the image of FIG. 13A is the start point image and the image of FIG. 13B is the end point image.

  When the difference between the two images is determined in S101, the process proceeds to S102 to determine the changed portion. In FIGS. 13A and 13B, the part surrounded by the broken line of the character is the part, and the change of the two parts is determined and recognized as the change part.

Then, the process proceeds to S103 where a moving object is determined. In the change (change part) of two parts, if there is nothing in a certain part and there is not in a certain part in that part, it is determined as a moving object. The character that exists in the region of FIG. 13A does not exist in the original region in the region of FIG. 13B, and the character that exists in the region of FIG. 13B does not exist in the region of FIG. It does not exist in the original site. Accordingly, the character at the site in FIGS. 13A and 13B is determined as a moving object.
Specifically, for example, a place where a plurality of images are overlapped and the degree of coincidence is high for each position is determined as the background. In addition, it is determined that a moving object (character) has a high degree of coincidence by changing the position and size of the image portion with low coincidence or by shifting the position.

  In step S104, a background image is created. The background image is created using a part where there is no moving object (character). In FIGS. 13A and 13B, a character is present in a part surrounded by a broken line, so that it is not used. For example, a part surrounded by a broken line in FIG. 13A is used as shown in FIG. A background image is created.

  When a background image is created in S104, it is determined in S105 whether an interpolation image or a predicted image is created as an auxiliary image. For example, if the image that is not the start point of the two images is the end point image, an interpolated image is created, and if the image is the next image after the start point, a predicted image is created.

When the creation of an interpolation image is selected in S105, the difference between the two images (start point and end point images) is internally divided (divided) in S106, and one of the two images of the moving object (character) is selected. By enlarging / reducing, an interpolated image is created and combined with the background image.
When the creation of the predicted image is selected in S105, the difference between the two images (start image and next image) is emphasized in S107, and the predicted image of the moving object (character) is synthesized with the background image at the predicted position. The
When the auxiliary image is combined with the background image in S106 and S107, the process returns to the main routine of FIG. 11 and proceeds to S12.

  Returning to the main routine of FIG. 11 and proceeding to S12, it is determined in S12 that the auxiliary image combined with the background image is adopted. If the auxiliary image is not adopted, the photographed image is adopted by photographing at S14. Shooting is performed when the subject (character) enters the position of the auxiliary image (when placed). It is determined in S15 whether or not shooting with the necessary frames has been completed, and the shooting is repeated by returning to S14 until all shooting with the required frames is completed. When all the necessary frames have been shot, the process proceeds from S15 to S16.

If it is determined in S12 that the auxiliary image is adopted, the auxiliary image is framed in S13 and the process proceeds to S16.
A framed auxiliary image is incorporated in S16 into two images (start point, end point image, start point image, and next image) taken in advance to create and record a moving image scene (continuous shot image). These are combined to produce an animation image.

The image processing apparatus (image processing unit) 114 according to the present invention only needs to create an auxiliary image as a material for animation moving image shooting. That is, although it is possible to create a moving image scene or combine them to produce an animation image, it is not always necessary.
If the steps of creating a moving image scene or producing an animated movie are processed by a device other than the image processing device, the function is specialized and the configuration is simplified, so that the image processing device can be obtained at a low cost.

  For example, a step of creating a moving image scene by incorporating an auxiliary image that is a material for moving image shooting created by the image processing device 114 or a step of creating an animation moving image is performed by a central control circuit 14 other than the image processing device. It may be made in the same manner as the processing. In addition, the auxiliary image created by the image processing device 114 is output from the communication control unit 114e to the outside, a moving image scene is created outside, an animation moving image is produced by combining them, and these are processed by the image processing device 114 or imaging. You may return to the apparatus 10. In other words, you may outsource the production of animated videos.

  However, the auxiliary image processing unit 114c may not only create an auxiliary image, but may also serve as a moving image creation unit that creates a moving image scene and combines them to produce an animation image. Moreover, even if the image processing apparatus 114 includes a moving image creation unit as an independent member.

The produced animation image may be recorded as a moving image file, or may be recorded as a group of still images including a photographed image and an auxiliary image. Here, since a group of still images including captured images and auxiliary images forms a series of continuous shot images as a whole, they may be recorded as one container file so as not to be separated. Further, information associated with each still image file may be embedded and recorded in the image file.
The auxiliary image creating unit 114c preferably includes a recording control unit that records the auxiliary image in association with each other when the auxiliary image creating unit 114c creates an auxiliary image that is a frame image in which a moving object moves on the background with actual shooting. The creation unit may also serve as the recording control unit, or the image processing device 114 may include a moving image creation unit as an independent member.

In the embodiment, a character is photographed at two positions of the start point and the end point in the interpolation guide, and at the two positions after the start point and the start point in the prediction guide, and an auxiliary image (interpolated image and predicted image) is created from the two images. Yes. However, the plurality of images prepared for creating the auxiliary image include at least two images including the start point image and the end point image in the interpolation guide, and two images including the start point image and the image after the start point in the prediction guide. That's enough.
For example, in the prediction guide, a character may be photographed at two positions following the start point and the start point, an approximate curve for character movement may be obtained from the three images, and a predicted image may be formed along the approximate curve.

  Further, in both the interpolation guide and the prediction guide, two auxiliary images are created as shown in FIGS. However, the number of auxiliary images to be created is not limited to two, and it is sufficient to create at least one auxiliary image.

  When the character is photographed in S14, a “warning” may be displayed when the character position is deliberately or accidentally shifted from the auxiliary image. In addition, if the position of the end point shifts and changes, the end point may be newly set and displayed.

  9 (C) and 9 (D), when “smoothness” is determined from a song, “smoothness” may be determined by determining a time interval between rhythms and beats from phrases of the selected song. . For example, in the phrase “Jajajajan”, the first half “Jajaja” corresponds to the beat and the rhythm corresponds to the volume, but the second half “Jaja” has the same pitch. Therefore, it is possible to determine the rhythm and beat with the clear “Jajaja” in the first half and use this as the time interval for division.

  The change in the image may be processed on the assumption that the pitch holding part of “Jan” in the latter half can be divided by the rhythm in the first half. In other words, “Jan” in the second half may change the image with the rhythm of the first half, and the interpolated image may not be output until the duration of “Jan” ends. Of course, “Jan” in the second half may be as smooth as “Jajaja” in the first half.

In other words, the rhythm of the music that is the background music can be detected, and the time difference setting unit can set the time difference in frame advance and the time difference of playback according to the rhythm, and the music phrase rhythm can be used as the music rhythm .
If the time difference in frame advance and the time difference in playback are set according to the rhythm of the music, the time difference in frame advance and the time difference in playback change according to the rhythm, and a natural video expression that matches the rhythm sense can be obtained .

In addition, as a music rhythm, it is preferable to set a time difference in frame advance and a reproduction time difference by a rhythm of a music phrase, for example, a cycle of a relatively large pulse sound having a specific frequency such as a bass or drum included therein. The period of a relatively large pulse sound of a specific frequency, such as a bass or drum, included in the rhythm of a musical phrase is clear and easy to understand for the human ear. Therefore, if the interval between pulse sounds is used when determining as a time difference, a natural moving picture expression corresponding to the phrase can be obtained, and a more optimal time difference in frame advance and reproduction time difference can be obtained.
The phrase data or file may have rhythm information as a time difference.

  The determination of the moving object in S103 will be described in detail. A plurality of images (two images in the embodiment) are formed having a moving object and a background, and the moving object and the background are determined and determined (determined). Therefore, the image processing apparatus 114 can include a determination unit that determines and determines the moving object and the background image of the auxiliary image.

  A character molded with clay or the like may change an expression or the like. Therefore, a moving object may be determined by ignoring the change in the expression using a face detection technique. Further, in addition to the degree of coincidence of images, it is preferable to determine a moving object when the distance becomes closer with reference to a change in distance.

If the image processing apparatus 114 includes a determining unit, the moving unit and the background are automatically determined and determined by the determining unit and are clearly distinguished, and the moving object can be easily determined. Therefore, the moving object can be clearly distinguished from the background and moved on the background, and a natural composite image can be created.
The position difference determination unit 114a also serves as the determination unit, and by automatically determining the moving object and the background, the user can easily determine the character. The position difference determination unit 114a may also serve as a determination unit, and the image processing apparatus 114 may include a determination unit as an independent member. However, in the configuration in which the position difference determination unit 114a also serves as the determination unit, an increase in the number of members can be avoided.

Furthermore, as shown in FIGS. 13A and 13B, if the moving distance of the moving object is increased, the overlapping of the moving objects is eliminated, and the moving object and the background can be easily distinguished. Therefore, the size and position of the moving object can be determined without error, and an auxiliary image can be created accurately.
The user (photographer) wants to save as much trouble as possible, and if an accurate auxiliary image can be created, the user uses the auxiliary image as it is, and the trouble of re-photographing is omitted (S13). In addition, in the past, a user who felt complicated because time-lapse photography for an animation video was necessary, the configuration of the embodiment requires only at least two shots. Time-lapse can be taken without feeling.

  In addition, since multiple moving images are placed on a common background, the image portion of the moving object is automatically and easily distinguished from the background image portion by image determination, so each image is processed appropriately. There is also an advantage of being a new image representation.

  In order to ensure the distinction and determination of the moving object and the background, it may be determined supplementarily that the background is a relatively peripheral part in the screen at the time of image determination. Furthermore, it is preferable that the contour shape and color arrangement of the moving object are substantially matched by enlarging / reducing when moving, and a warning may be issued if they do not substantially match.

The reproduction of the image (composite image) obtained by synthesizing the captured image and the auxiliary image on the liquid crystal display (display unit) 18 is not necessarily performed by moving image expression, and may be expressed as a still image or a combined photograph by arranging the combined images. Have a good taste and enjoy it.
Further, even if only the auxiliary image is reproduced, the auxiliary image is not actually taken, so that a new valuable image expression is created.

  Furthermore, when the auxiliary image is translucent and displayed on the liquid crystal display 18 as a photographing guide, the moving object of the semitransparent auxiliary image is arranged on the same background at the position of the moving object of the auxiliary image and is combined with the background. Is displayed. The composite image may be used not only as a photographing guide but also as a final image. Further, the selection of the composite image may be performed by a touch (tap) operation of the liquid crystal display 18 (smart phone screen).

  The auxiliary image can be converted into a moving image file or recorded in a single image file or container as a group of still images. By recording a single image file as a group of still images in a container, it is possible to easily process and view auxiliary images in different environments such as on the Internet.

  In other words, a position difference determination unit that determines a position difference of a moving object from at least two captured images having a common moving object and a background, a time difference in image frame advancement, and a time difference of image reproduction corresponding to the position difference are set. The moving object is displayed in a semi-transparent manner and arranged in the screen according to the position difference determined by the position difference determination section, the position difference determined by the position difference determination section, and the time difference in frame advance and playback time set by the time difference setting section. It is possible to provide a photographing guide device (image processing device) including an auxiliary image creation unit that creates at least one auxiliary image.

  When a moving object is placed at a position where an auxiliary image is displayed semi-transparently and re-photographed, the captured image can be changed to a distinguishable display different from translucent, for example, a transparent or black display. . If the display is changed in this way, the user can shoot with a favorite display. Further, if necessary, the display that does not have to be taken may be omitted, and the degree of freedom of photographing is increased. Furthermore, when re-photographing, the exposure may be set to a constant value or may be photographed while being changed little by little.

  As the display of the translucent moving object, only the outline of the moving object may be displayed. Here, the position of the translucent moving object may be clearly indicated by displaying an arrow indicating the position where the moving object is arranged in the outline.

  The guide display is clarified by displaying the shooting result superimposed on a live view image (through moving image) in which images are sequentially updated and displayed. Then, when the image of the translucent moving object is displayed on the live view image and the moving object is arranged in the vicinity of the display and photographed, the translucent image is displayed in another display other than the translucent, for example, The image may be expressed by a transparent or black display.

When at least two auxiliary images are created, if the exposure value or focus position changes, there is a possibility that the moving object and the background cannot be correctly distinguished and determined. In this case, for example, if the exposure value is matched with one of the images, the moving object and the background can be correctly distinguished and determined.
As for the focus, the focus is usually on the main moving object. However, when the moving object moves back and forth, the background blur changes, and when the moving object comes close to the user, the background blur becomes conspicuous. Therefore, you may focus on the background instead of the moving object. However, if the background blur is in an allowable range, it is preferable to focus on the moving object.
When the background is in focus and the blur of the moving object exceeds the allowable range, a configuration that encourages shooting with the moving object in focus (apart from shooting for judgment), for example, a configuration that issues a warning Good. In addition, the moving object may not be noticeable by reducing the aperture.

  Further, an auxiliary image whose exposure changes may be created. For example, in addition to the two shootings for determination, if the shooting with different exposure is performed separately and the change in exposure is reflected when the auxiliary image is created, an auxiliary image with changed exposure can be obtained. If the shooting guide is performed using an auxiliary image whose exposure changes, it is possible to eliminate the need to consider the problem of blur.

In addition, the present invention provides a function for determining a position difference of a moving object from at least two images having a common moving object and a background, a time difference in frame advancement of the image, and reproduction of the image corresponding to the position difference. A function for setting a time difference, and a function for creating at least one auxiliary image that is a frame image in which the moving object moves on the background, according to the determined position difference, the time difference in frame advance, and the time difference in reproduction. Can be said to be related to an image processing program for causing a computer to realize the above.
The moving object in the auxiliary image may be displayed in a translucent manner and arranged in the screen.

The image processing program is stored in a memory card such as a ROM card, an optical disk such as a CD-ROM or DVD, or a magnetic disk disk such as a hard disk. Then, each function is realized in the computer by the image processing program, and an auxiliary image is created. Therefore, the image effect in frame advance can be easily confirmed in advance.
The image processing program may be stored in the central control circuit 14 of the imaging apparatus, or may be read from an external memory such as a ROM card and used.

  As described above, in the present invention, an interpolated image is created and used as a material for moving image shooting. Therefore, an image in time-lapse shooting can be easily known in advance. Therefore, it is possible to accurately and easily produce animated images with smooth movement.

The above-described embodiments are for explaining the present invention, and are not intended to limit the present invention in any way, and all modifications and alterations made within the technical scope of the present invention are included in the present invention. Needless to say.
For example, the basic image does not need to be an actually captured image, and an auxiliary image of the present invention can be created from an image depicting a moving object (character) or background by handwriting or a personal computer operation. Further, an image depicting a moving object or background by handwriting or a personal computer operation may be used as a rough sketch to be an auxiliary image for photographing.

The image processing apparatus and the image processing method of the present invention can be widely applied to imaging apparatuses such as digital cameras, video cameras, and portable information terminal devices that perform image processing on captured images.
In addition, since it can be used for movement prediction, it can be applied to simulation equipment in various industrial fields. It can also be applied to the use of inspection equipment that analyzes how fast-moving things have changed. Furthermore, if you follow the idea of displaying an auxiliary image and shooting it while viewing it, you can apply it to reports and shooting guides for document creation, regardless of the video display.

DESCRIPTION OF SYMBOLS 10 Imaging device 10a Lens type digital camera 10b Smartphone 12 Imaging part 14 Central control circuit 16 Operation part 18 Display part 20 Recording part 22 Communication part 114 Image processing part (image processing apparatus)
114a Position difference determination unit 114b Time difference setting unit 114c Auxiliary image creation unit 114d Operation control unit 114e Communication control unit 114f Image capturing unit

Claims (13)

A position difference determination unit that determines a position difference of the moving object from at least two images having a common moving object and a background;
A time difference setting unit for setting a time difference in frame advance of the image and a time difference in reproduction of the image corresponding to the position difference;
According to the position difference determined by the position difference determination unit and the time difference in the frame advance and the time difference of reproduction set by the time difference setting unit, the moving object becomes at least one frame image that moves on the background. An auxiliary image creation unit for creating two auxiliary images;
An image processing apparatus comprising: A determination unit for determining and determining a moving object of the auxiliary image and a background image;
The image processing apparatus according to claim 1, further comprising:   The image processing apparatus according to claim 2, wherein the position difference determination unit also serves as the determination unit.   The image processing apparatus according to claim 1, wherein a rhythm of music serving as background music is detected, and the time difference setting unit sets a time difference in the frame advance and a reproduction time difference according to the rhythm.   The image processing apparatus according to claim 4, wherein the music rhythm is a music phrase rhythm.   The image processing apparatus according to claim 1, wherein the plurality of images include a start point image and an end point image, and an auxiliary image is created by interpolating between the start point and end point images.   The image processing according to any one of claims 1 to 5, wherein the plurality of images include an image of a start point and an image subsequent to the start point, and an auxiliary image is created by prediction from the image of the start point and images subsequent to the start point. apparatus.   The image processing apparatus according to claim 1, further comprising a moving image creating unit that creates a moving image by combining the plurality of images with the auxiliary image that is frame-advanced. A position difference determination unit that determines a position difference of the moving object from at least two images having a common moving object and a background;
A time difference setting unit for setting a time difference in frame advance of the image and a time difference in reproduction of the image corresponding to the position difference;
In accordance with the position difference determined by the position difference determination unit and the time difference in frame advance and playback time set by the time difference setting unit, the moving object is displayed in a translucent manner and arranged in the screen. An auxiliary image creation unit for creating at least one auxiliary image;
An imaging guide device comprising:   When the image of the translucent moving object in the auxiliary image is displayed on a live view image, and the moving object is arranged on the translucent display unit and photographed, the image of the translucent moving object is displayed. The photographing guide device according to claim 9, wherein the photographing guide device is changed to a distinguishable display different from translucent. A display part for displaying an image;
An image processing apparatus that processes the image and displays the processed image on the display unit;
Comprising
The image processing apparatus is
A position difference determination unit that determines a position difference of the moving object from at least two images having a common moving object and a background;
A time difference setting unit for setting a time difference in frame advance of the image and a time difference in reproduction of the image corresponding to the position difference;
According to the position difference determined by the position difference determination unit and the time difference in the frame advance and the time difference of reproduction set by the time difference setting unit, the moving object becomes at least one frame image that moves on the background. An auxiliary image creation unit for creating two auxiliary images;
An imaging apparatus comprising: Determining a position difference of the moving object from at least two images having a common moving object and a background;
Setting a time difference in the frame advance of the image and a time difference in reproducing the image corresponding to the position difference;
Creating at least one auxiliary image to be a frame image in which the moving object moves on the background according to the determined position difference and a time difference in the frame advance and a reproduction time difference;
An image processing method comprising: A function of determining a position difference of a moving object from at least two images having a common moving object and a background;
A function of setting a time difference in the frame advance of the image and a time difference in reproducing the image corresponding to the position difference;
A function of creating at least one auxiliary image that is a frame image in which the moving object moves on the background according to the determined position difference, a time difference in the frame advance, and a time difference in reproduction;
An image processing program for realizing a computer.

Images