JP2011139300A - Image processing apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To create a synthetic moving image with reality regarding an image processing apparatus and a program. <P>SOLUTION: A data processing part 13 synthesizes a background moving image which has a first frame rate with a foreground moving image which has a second frame rate to generate the synthetic moving image. In this case, a frame duplication part 63 compares the first frame rate with the second frame rate, and duplicates pieces of image data of the respective frames included in the moving image with lower frame rate according to reproduction speed of the moving image with higher frame rate of the foreground moving image and the background moving image. A frame synthesis part 64 sequentially synthesizes the image data of the moving image which is constituted of a plurality of frames including the frame duplicated by the frame duplication part 63 and the image data of the other moving image by unit of frame of the foreground moving image and the background moving image so that reproduction speeds in reproduction are mutually adapted. Thus, the image data of the synthetic moving image which is constituted of a plurality of frames which have regular field angles in which the background and the foreground are synthesized with each other are generated. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and a program, and more particularly to a technique that makes it possible to create a more realistic synthetic moving image.

  Conventionally, there is a technology to generate a composite movie as if it was captured in the original imaging area by dividing the imaging area with the same frame rate and shooting the movie separately, and then combining the divided areas (For example, refer to Patent Document 1).

JP 2005-244799 A

  However, since the technique described in Patent Document 1 simply divides the imaging region, there is a problem that if the contents (view angle and brightness) of the background images are different from each other, the reality is insufficient even if they are combined. It was.

  The present invention has been made in view of such a situation, and makes it possible to create a more realistic synthetic moving image.

According to a first aspect of the invention,
Image data of a first moving image having a normal angle of view and a first frame rate, and a main subject image area within the normal angle of view and an area excluding the main subject image area as a transmission information area In the image processing device that generates the image data of the synthesized moving image having the normal angle of view by combining the image data of the second moving image having the second frame rate,
The first frame rate and the second frame rate are compared, and a lower frame rate is selected in accordance with the playback speed of the higher frame rate video of the first video and the second video. A copying means for copying the image data of each frame included in the other video,
Image data of a moving image composed of a plurality of frames including the frame duplicated by the duplicating means of the first moving image and the second moving image so that the reproduction speeds at the time of reproduction match each other. Synthesizer that generates image data of the synthesized moving image by sequentially synthesizing the image data of the other moving image in units of frames;
Recording means for recording image data of the synthesized moving image generated by the synthesizing means as image data of a new moving image;
An image processing apparatus is provided.

According to a second aspect of the invention,
The synthesizing unit generates image data of the synthesized moving image including frames of the number of frames by synthesizing the number of frames corresponding to the reproduction time of the second moving image.
An image processing apparatus is provided.

According to a third aspect of the present invention,
Stores processing content information that is information in which the correspondence between the first frame rate and the second frame rate is associated with at least one of the processing content of the duplicating unit or the processing content of the combining unit. Further comprising processing content storage means;
At least one of the duplicating unit or the synthesizing unit is configured to correspond to the correspondence between the first frame rate of the first moving image to be combined and the second frame rate of the second moving image to be combined. The associated processing content is read from the processing content information stored in the processing content storage means, and the processing according to the read processing content is executed.
An image processing apparatus is provided.

According to a fourth aspect of the invention,
Computing means for computing the number of frames duplicated by the duplicating means from the first frame rate and the second frame rate,
An image processing apparatus further provided is provided.

According to a fifth aspect of the present invention,
Imaging means for imaging within the range of the normal angle of view;
The image capturing unit is continuously driven based on the first frame rate or the second frame rate to acquire image data of a plurality of images, and each of the acquired plurality of images is included as a frame. An image acquisition unit that generates image data of the first moving image or the second moving image, and adds to the image data, as additional information, information specifying a frame rate when the imaging unit is continuously driven;
An image processing apparatus is further provided.

According to a sixth aspect of the present invention,
Audio data is recorded in at least one of the first moving image or the second moving image,
The recording means further adds the audio data to the image data of the new moving image for recording;
An image processing apparatus is provided.

According to a seventh aspect of the present invention,
Image data of a first moving image having a normal angle of view and a first frame rate, and a main subject image area within the normal angle of view and an area excluding the main subject image area as a transmission information area A computer that controls an image processing apparatus that generates image data of a combined moving image having a normal angle of view by combining the image data of the second moving image having the second frame rate;
The first frame rate and the second frame rate are compared, and a lower frame rate is selected in accordance with the playback speed of the higher frame rate video of the first video and the second video. A replication function that replicates the image data of each frame included in the other video,
The image of the moving image composed of a plurality of frames including the frame duplicated by the realization of the duplication function of the first moving image and the second moving image so that the reproduction speeds at the time of reproduction match each other. A synthesis function for generating image data of the synthesized video by sequentially synthesizing data and image data of the other video in units of frames;
A recording control function for recording the image data of the combined moving image generated by the realization of the combining function as image data of a new moving image;
A program for realizing the above is provided.

  According to the present invention, a realistic synthesized moving image can be created.

It is a block diagram which shows the hardware constitutions of the digital camera which concerns on one Embodiment of the image processing apparatus of this invention. It is a functional block diagram which shows the functional structure of the data processing part of the digital camera of FIG. It is a flowchart which shows an example of the flow of the synthetic | combination moving image production | generation process which the data processing part of FIG. 2 performs. It is a flowchart which shows an example of the flow of the synthetic | combination moving image production | generation process which the data processing part of FIG. 2 performs. The example of the various display images displayed during the synthetic | combination moving image production | generation process of FIG.3 and FIG.4 is shown. The example of the various display images displayed during the synthetic | combination moving image production | generation process of FIG.3 and FIG.4 is shown. 3 shows an example of the data structure of processing content information related to frame duplication and synthesis by the data processing unit of FIG. A specific example of scheduling in the case of duplicating a frame according to the processing content information of FIG. 7 is shown. This is a specific example of scheduling in the case of duplicating a frame according to the processing content information of FIG. 7, and shows an example different from FIG. It is an external view of the mobile phone which concerns on the modification of the image processing apparatus of this invention, (a) is an open state front view, (b) is a closed state rear view. It is a block diagram which shows the structure of the hardware of the mobile telephone of FIG.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a hardware configuration of a digital camera 1 according to an embodiment of an image processing apparatus of the present invention.

  The digital camera 1 includes an imaging unit 11, a voice input unit 12, a data processing unit 13, and a user interface unit 14.

  The imaging unit 11 includes an optical lens unit 21 and an image sensor 22.

  The optical lens unit 21 includes a lens that collects light, such as a focus lens and a zoom lens, in order to photograph a subject included in a specified angle of view (hereinafter referred to as “regular angle of view”). The The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor 22. The zoom lens is a lens that freely changes the focal length within a certain range. The optical lens unit 21 is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

  The image sensor 22 includes a photoelectric conversion element, an AFE (Analog Front End), and the like. The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. A subject image is incident on the photoelectric conversion element from the optical lens unit 21 through a shutter unit (not shown). Therefore, the photoelectric conversion element photoelectrically converts (photographs) the subject image at regular intervals, accumulates image signals, and sequentially supplies the accumulated image signals to the AFE as analog signals. The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. By various signal processing, a digital signal is generated and output as an output signal of the image sensor 22. Hereinafter, the digital signal of the image signal is referred to as “image data”. In this way, image data is output from the image sensor 22 and supplied to the data processing unit 13.

  The voice input unit 12 includes a microphone, a voice data conversion unit, and the like. The microphone inputs voice when shooting by the imaging unit 11 and outputs it as an analog voice signal. The voice data converting unit performs various signal processing such as A / D conversion processing on the analog voice signal output from the microphone. Through various signal processing, a digital signal is generated and output as an output signal of the voice input unit 12. Hereinafter, the digital signal of the audio signal is referred to as “audio data”. In this way, voice data is output from the voice input unit 12 and supplied to the data processing unit 13.

  The data processing unit 13 includes a CPU (Central Processing Unit) 31, a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a memory 34, an output control unit 35, and an image processing unit 36. I have.

  The CPU 31 executes various processes according to programs recorded in the ROM 32. The RAM 33 appropriately stores data necessary for the CPU 31 to execute various processes.

  In the present embodiment, programs for realizing the functions of the image acquisition unit 61 to the file generation unit 66 shown in FIG. Therefore, each function of the image acquisition unit 61 to the file forming unit 66 can be realized by the CPU 31 executing processing according to this program.

  The memory 34 is configured by a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores image data output from the image sensor 22 and audio data output from the audio input unit 12. The memory 34 also stores various data necessary for various image processing and audio processing, such as image data, various flag values, threshold values, and the like. The memory 34 also includes a display memory area for saving and reading image data for image display (hereinafter referred to as “display image data”).

  The output control unit 35 reads the display image data stored in the display memory area of the memory 34, and displays an image expressed by the display image data (hereinafter referred to as “display image”) of the user interface unit 14. 42 is displayed. For example, the output control unit 35 generates an RGB signal based on the display image data and supplies the RGB signal to the display unit 42 to display the display image on the display unit 42. The RGB signal generated by the output control unit 35 can be output to an external device (not shown) by the communication unit 44 of the user interface unit 14. As a result, the display image can be displayed even by an external device such as a television receiver, a personal computer, or a projector.

  The output control unit 35 also reads out audio data stored in the memory 34, executes signal processing such as D / A conversion, generates an analog audio signal, and supplies the analog audio signal to the audio output unit 41. As a result, the sound expressed by the sound signal is output from the sound output unit 41.

  The image processing unit 36 is configured by a DSP (Digital Signal Processor) or the like, and executes various image processing such as white balance correction processing and γ correction processing on the image data stored in the memory 34. In the present embodiment, at least a part of various image processes executed by the image acquisition unit 61 to the filing unit 66 of FIG. 2 to be described later is executed by the image processing unit 36, and the remaining part is executed by the CPU 31. . That is, in the present embodiment, the image acquisition unit 61 to the file forming unit 66 shown in FIG. 2 to be described later are configured as a combination of hardware such as the CPU 31 and the image processing unit 36 and a program (software) stored in the ROM 32. Yes.

  The user interface unit 14 communicates between an audio output unit 41 configured with a speaker or the like, a display unit 42 configured with a liquid crystal display, an operation unit 43 that receives a user instruction operation, and an external device. A communication unit 44 for controlling and a drive 45 for reading and writing data with respect to the removable recording medium 51 are provided.

  Among the moving images to be processed by the digital camera 1 having such a configuration, the moving image having the normal angle of view and the first frame rate is hereinafter referred to as “background moving image”. In addition, the moving image having the second frame rate that includes the main subject image area within the normal angle of view and includes the area excluding the main subject image area as the transmission information area is hereinafter referred to as “foreground moving image”. . The digital camera 1 can generate image data of a combined moving image having a normal angle of view by combining the image data of the background moving image and the image data of the foreground moving image. Such processing of the digital camera 1 is hereinafter referred to as “composite moving image generation processing”.

  FIG. 2 is a functional block diagram showing a functional configuration of the data processing unit 13 for executing such a synthetic moving image generation process.

  The data processing unit 13 includes an image acquisition unit 61, a synthesis condition determination unit 62, a frame duplication unit 63, a frame synthesis unit 64, an audio addition unit 65, and a filing unit 66.

  Further, in the ROM 32, RAM 33, or memory 34 (FIG. 1) of the data processing unit 13, an index storage unit 71, a foreground moving image storage unit 72, a duplication processing content storage unit 73, a background moving image storage unit 74, And a combined moving image storage unit 75 is provided.

  The image acquisition unit 61 acquires image data of a moving image captured by the imaging unit 11. For example, the image acquisition unit 61 acquires image data of a plurality of images having normal angles of view sequentially supplied from the imaging unit 11 by continuously driving the imaging unit 11 based on the first frame rate. . That is, in the present embodiment, the image data of the plurality of images acquired by the image acquisition unit 61 in this way is acquired as the image data of the background moving image having the normal angle of view and the first frame rate. . Each of the plurality of images constituting the moving image is hereinafter referred to as a “frame”. That is, one moving image is composed of a plurality of frames.

  The image acquisition unit 61 converts the background moving image data acquired in this way into a file and stores it in the background moving image storage unit 74. That is, an image file including background moving image data is generated and stored in the background moving image storage unit 74. The image acquisition unit 61 further generates, for example, information for specifying the first frame rate or the like as meta information regarding the background moving image, and stores the information in the background moving image storage unit 74 by including the information in the image file. The information for specifying a predetermined frame rate is hereinafter referred to as “frame rate information”. That is, the moving image file stored in the background moving image storage unit 74 includes background moving image image data together with meta information such as frame rate information specifying the first frame rate. As a result, it is possible to easily create a synthesized movie as a background movie using a movie shot by the user, and the synthesized movie created in this way becomes a more realistic movie.

  In addition, the image acquisition unit 61 appropriately performs thumbnailing processing, reduction processing, thinning processing, and the like on the image data of the background moving image, so that an index image to be presented to the user at the time of determining a synthesis condition to be described later Data is generated and stored in the index storage unit 71 so as to be associated with each moving image file stored in the background moving image storage unit 74.

  In the present embodiment, a moving image having a second frame rate, that is, a foreground moving image including a main subject image region within a normal angle of view and including a region excluding the main subject image region as a transmission information region, It shall be given in advance. In other words, in this embodiment, it is assumed that the image data of the foreground moving image is included in a predetermined image file and stored in the foreground moving image storage unit 72 in advance. Further, it is assumed that the predetermined image file includes frame rate information for specifying the second frame rate, transmission information regarding the transmission information area, and the like as meta information. In addition, the index image data of the foreground moving image is stored in the index storage unit 71 so as to be associated with transmission information and the like and associated with each moving image file stored in the foreground moving image storage unit 72.

  The image acquisition unit 61 may acquire moving image data supplied from the imaging unit 11 as foreground moving image data by continuously driving the imaging unit 11 based on the second frame rate. . In this case, it is possible to easily create a composite video as a foreground video using a video shot by the user, and the composite video created in this way becomes a more realistic video.

  The synthesis condition determination unit 62 determines various synthesis conditions for generating a synthesized video. The synthesis condition includes, for example, a condition as to which of a plurality of background videos is to be synthesized, a condition as to which of a plurality of foreground videos is to be synthesized, and the like.

  In the present embodiment, many of the synthesis conditions can be set by the user, and a plurality of index image data is stored in the index storage unit 71 in order to support such setting by the user. Therefore, the synthesis condition determination unit 62 reads predetermined index image data from the index storage unit 71 and supplies it to the output control unit 35. Then, the output control unit 35 causes the display unit 42 to display an image expressed by the index image data. The user can perform an instruction operation for a predetermined composition condition by operating the operation unit 43 while viewing the image of the index image data. The synthesis condition determination unit 62 determines the predetermined synthesis condition according to the instruction operation. A specific example of index image data and a specific example of a synthesis condition determination method will be described later with reference to FIGS.

  The synthesis condition determination unit 62 generates index image data directly from the foreground video image data stored in the foreground video storage unit 72 and the background video image data stored in the background video storage unit 74. May be. In this case, the index storage unit 71 may be omitted.

  In this way, at least the background moving image and the foreground moving image to be combined are determined by the combining condition determination unit 62 as combining conditions. In this case, if the first frame rate of the background video to be synthesized matches the second frame rate of the foreground video to be synthesized, the background video and the foreground video can be synthesized as they are, Appropriate composite animation can be generated.

  However, if the first frame rate of the background moving image to be combined and the second frame rate of the foreground moving image to be combined do not match, if the background moving image and the foreground moving image are combined as they are, appropriate combining I can't create a video. For example, when playback is performed in accordance with the playback speed of the higher frame rate, a composite video is generated in which the subject image of the video with the lower frame rate is quickly moved and played back. Therefore, in order to generate an appropriate synthesized video, the number of frames of the lower frame rate video is increased in accordance with the playback speed of the higher frame rate video of the background video and the foreground video. It is necessary to let

  The method of increasing the number of frames in this case is not particularly limited, and for example, a method of inserting an interpolated frame or a newly created frame between existing frames can be employed. In the present embodiment, a method of inserting a duplicated frame from either one between existing frames is employed. For this reason, the data processing unit 13 is provided with a frame duplication unit 63.

  That is, the frame duplicating unit 63 compares the first frame rate and the second frame rate, and the lower frame rate is selected in accordance with the playback speed of the higher frame rate video of the foreground video and the background video. Duplicate the image data of each frame included in the other video.

  Here, the duplication processing content storage unit 73 stores the correspondence between the first frame rate and the second frame rate, and at least one of the processing content of the frame duplication unit 63 or the processing content of the frame composition unit 64 described later. Are associated with each other (hereinafter referred to as “processing content information”). Therefore, the frame duplicating unit 63 refers to the processing content information as appropriate, and executes the above-described processing for duplicating the image data of the frame. Thus, by preliminarily storing the processing content information, it becomes possible to reduce the time required to determine the processing content of the frame duplicating unit 63 or the processing content of the frame synthesizing unit 64 described later. It is possible to shorten the processing time of the synthetic moving image generation process. A specific example of the processing content information will be described later with reference to FIGS.

  In the following description, for convenience of explanation, it is assumed that the first frame rate of the background moving image is higher than the second frame rate of the foreground moving image. That is, in FIG. 2, it is assumed that data flows along a solid arrow. Therefore, the frame duplicating unit 63 duplicates the image data of each frame included in the foreground moving image. As a result, moving image data in which a duplicated frame is inserted between each existing frame is obtained as a synthesis target and supplied to the frame synthesis unit 64. On the other hand, the image data of the background moving image is supplied from the background moving image storage unit 74 to the frame composition unit 64.

  The frame synthesizing unit 64 combines the foreground moving image image data and the background moving image image data including a plurality of frames including the frames copied by the frame copying unit 63 so that the reproduction speeds at the time of reproduction match each other. Synthesize sequentially in frame units. In this case, the frame composition unit 64 refers to the processing content information stored in the duplication processing content storage unit 73 as appropriate, and executes a process of composing the frame image data. As a result, image data of a combined moving image that is a combined moving image in which the background moving image and the foreground moving image are combined and includes a plurality of frames having a normal angle of view is generated.

  Here, for convenience of explanation, it is assumed that the first frame rate of the background moving image is higher than the second frame rate of the foreground moving image, but of course, the first frame rate is the second frame rate. It may be lower than the frame rate. Even in such a case, the frame duplicating unit 63 and the frame synthesizing unit 64 can generate the image data of the synthesized moving image by executing the above-described processing in the same manner. In order to show that such processing is also possible, in FIG. 2, between the background moving image storage unit 74 and the frame replication unit 63 and between the foreground moving image storage unit 72 and the frame composition unit 64, respectively. On the other hand, a dotted arrow is drawn.

  By the way, audio may be recorded in at least one of the foreground moving image and the background moving image. In such a case, the audio adding unit 65 adds the audio data of the audio recorded in the foreground moving image or the background moving image to the image data of the combined moving image generated by the frame combining unit 64. As a result, since the corresponding audio is also output when the synthetic moving image is played back, it becomes possible to create a more realistic synthetic moving image.

  The filing unit 66 is a file (hereinafter referred to as “a” including image data of a synthesized moving image generated by the frame synthesizing unit 64, audio data of audio added by the audio adding unit 65 as necessary, and various metadata. (Referred to as “composite image file”) and stored in the composite moving image storage unit 75.

  At the time of playback of the composite video, among the video data of the composite video included in the composite image file stored in the composite video storage unit 75, the image data of the display target frame is stored as display image data in the display memory area of the memory 34. Is done. Then, the output control unit 35 reads the display image data, and causes the display unit 42 to display an image expressed by the display image data, that is, a display target frame. The output control unit 35 can reproduce the synthesized moving image by executing such processing for each frame constituting the synthesized moving image in accordance with the reproduction speed. Furthermore, when the synthesized image file includes audio data, the output control unit 35 causes the audio output unit 41 to output audio expressed by the audio data.

  Next, with reference to the flowcharts of FIGS. 3 and 4, a synthetic moving image generation process executed by the data processing unit 13 having such a functional configuration will be described. At that time, the display image shown in FIG. 5 or 6 is referred to as appropriate in order to facilitate understanding of the synthetic moving image generation processing.

  3 and 4 are flowcharts illustrating an example of the flow of the synthetic moving image generation process. 5 and 6 show specific examples of various display images displayed on the display unit 42 during the synthetic moving image generation process.

  In step S <b> 1 of FIG. 3, the composition condition determination unit 62 of the data processing unit 13 determines whether a selection operation in the background selection mode has been detected.

  If the background selection mode is not selected, it is determined as NO in step S1, and the combined moving image generation process ends as shown in FIG.

  On the other hand, for example, as shown in FIG. 5, in a state where the monitor image 101 or the like is displayed on the display unit 42, a “MENU button (not shown)” in the operation unit 43 is pressed, and PLAY When the menu image 102 is displayed and the item 120 “Create composite image” is selected by the “SET button (not shown)” of the operation unit 43, the composition condition determination unit 62 selects the background selection. Detects mode selection operations. Thereby, it determines with it being YES in step S1 of FIG. 3, and a process progresses to step S2.

  In step S <b> 2, the synthesis condition determination unit 62 refers to the index storage unit 71. In step S <b> 3, the composition condition determination unit 62 controls the output control unit 35 to read out index image data images corresponding to a plurality of image files (including background moving image files) from the index storage unit 71 and display the display unit 42. To display. In step S4, the synthesis condition determination unit 62 determines whether a selection instruction operation has been detected.

  For example, as shown in FIG. 5, the selection image 104 including the index image data image 121 indicating a predetermined background moving image is displayed on the display unit 42 in the process of step S <b> 3. The user presses a “left button (not shown)” or “right button (not shown)” in the operation unit 43 in a state where the selected image 104 is displayed. Can be switched from the image 121 to an image of index image data indicating another background moving image or background still image. In addition to the “left button (not shown)” and “right button (not shown)”, the user presses an “upper button (not shown)” and “lower button (not shown)” to be described later. When an operation instruction is viewed from the composition condition determination unit 62, it is called “external direction instruction”, and the above-described pressing operation is called “external direction instruction operation”. The user repeats a direction instruction operation from the outside several times to display an image of desired index image data in the selected image 104, and then presses a “SET button (not shown)” in the operation unit 43. By pressing the button, it is possible to select the background moving image or the background still image indicated by the image of the desired index image data.

  When the “SET button (not shown)” is not pressed, since the selection instruction operation is not detected by the synthesis condition determination unit 62, it is determined NO in step S4, and the process returns to step S3. The process is repeated. That is, unless the “SET button (not shown)” is pressed, the loop process of steps S3 and S4 is repeated. Thereafter, when a “SET button (not shown)” is pressed, the selection instruction operation is detected by the synthesis condition determination unit 62, so that it is determined as YES in Step S4, and the process proceeds to Step S5. .

  In step S5, the synthesis condition determination unit 62, as one of the synthesis conditions, the background moving image or the background still image indicated by the index image data image displayed in the selected image 104 when the selection instruction operation is detected. Are determined as background images to be synthesized. For example, when the “SET button (not shown)” is pressed when the selected image 104 is in the display state of FIG. 5, the background moving image indicated by the image 121 of the index image data is determined as the background moving image to be synthesized. .

  In step S <b> 6, the synthesis condition determination unit 62 refers to the index storage unit 71. In step S <b> 7, the synthesis condition determination unit 62 controls the output control unit 35 so that an image based on index image data corresponding to a plurality of image files (including foreground moving image files) including transparency information is stored in the index storage unit 71. The data is read out and displayed on the display unit 42. In step S8, the synthesis condition determination unit 62 determines whether a selection instruction operation has been detected.

  For example, as shown in FIG. 5, the selection image 105 including the index image data image 122 indicating a predetermined foreground moving image is displayed on the display unit 42 in the process of step S <b> 7. The user performs a direction instruction operation from the outside in a state in which the selected image 105 is displayed, thereby changing the display state in the selected image 105 from the image 122 to an index indicating another foreground moving image or foreground still image. It is possible to switch to an image of image data.

  In the present embodiment, foreground moving image or foreground still image data including a plurality of characters as main subject images is stored in the foreground moving image storage unit 72 in advance. Therefore, in order to prompt the user to select a foreground moving image or a foreground still image, a message “Please select a foreground to use” is displayed in the selected image 105 as shown in FIG. For example, the image 122 of the index image data shows a foreground moving image in which a witch riding on a kite moves so as to fly in the sky.

  The user repeats the direction instruction operation from the outside several times to display the image of the desired index image data in the selected image 105, and then presses the “SET button (not shown)” in the operation unit 43. By doing so, the foreground moving image or the foreground still image indicated by the desired image can be selected.

  Therefore, if the “SET button (not shown)” is not pressed, the selection instruction operation is not detected by the synthesis condition determination unit 62, so that it is determined NO in step S8, and the process returns to step S7. The subsequent processing is repeated. That is, as long as the “SET button (not shown)” is not pressed, the loop process of steps S7 and S8 is repeated. Thereafter, when a “SET button (not shown)” is pressed, the selection instruction operation is detected by the synthesis condition determination unit 62, so that it is determined as YES in Step S8, and the process proceeds to Step S9. .

  In step S <b> 9, the composition condition determination unit 62 controls the output control unit 35 to synthesize and display the index image data images, and differentiates the compositable period and displays it as a time bar. That is, the synthesis condition determination unit 62 includes, as one of the synthesis conditions, the foreground moving image or the foreground still image indicated by the index image data image displayed in the selected image 105 when the selection instruction operation is detected. Determined as the foreground image to be combined. The composition condition determination unit 62 generates a composite image obtained by combining the foreground image to be combined with the background moving image previously determined as a combination target, and image data of a selected image including a time bar. Then, the synthesis condition determination unit 62 controls the output control unit 35 to display the selected image on the display unit 42.

  For example, when the “SET button (not shown)” is pressed when the selected image 105 is in the display state of FIG. 5, as one of the synthesis conditions, the foreground video indicated by the image 122 is the foreground video to be synthesized. It is determined. For this reason, as shown in FIG. 6, a composite image obtained by combining an image 122 indicating a foreground moving image to be combined (moving image of a witch) and an image 121 indicating a background moving image to be combined, and a time bar 123 are included. The selected image 106 is displayed on the display unit 42.

  A time bar 123 displayed in the selected image 106 indicates a playback time zone of a foreground video and a background video that has a longer playback time, in this embodiment, a background video. A pointer 124 in the time bar 123 indicates a playback start timing of a foreground video and a background video having a shorter playback time, in this embodiment, the foreground video.

  The user can freely move the display position of the pointer 124 in the selected image 106 in the time bar 123 by performing a direction instruction operation from the outside while the selected image 106 is displayed. . The user repeats the direction instruction operation from the outside several times to display the pointer 124 at a desired position in the time bar 123, and then presses the “SET button (not shown)” in the operation unit 43. By doing so, it is possible to make a setting for starting the synthesis (reproduction) of the foreground moving image from the reproduction position of the background moving image indicated by the desired position. In other words, if it is understood that the foreground moving image and the background moving image are combined with the foreground moving image embedded in the background moving image, the user can set the foreground moving image embedding start timing by performing the above-described operation. In this case, the frame synthesis unit 64 synthesizes the number of frames corresponding to the playback time of the foreground video from the foreground video embedding start timing. As a result, the compositing process is executed for the minimum necessary current time zone, so that the processing time can be shortened. In some cases, the capacity of the generated composite video image data is reduced. It becomes possible. Although not shown, for the convenience of the user, a bar having a length corresponding to the playback time zone of the foreground video, that is, the so-called embedded time zone may be displayed in the time bar 123.

  In this way, after the selected image 106 is displayed in the process of step S9, in step S10, the synthesis condition determination unit 62 controls the output control unit 35 to display the direction indication detection result from the outside as a time bar. 123 is reflected in the display position. In other words, the direction instruction from the outside is an instruction to change the foreground video embedding start timing as described above, and the display position of the pointer 124 is displayed in the time bar 123 according to the detection result of the result of the change instruction. Change.

  In step S11, the composition condition determination unit 62 determines whether or not an instruction operation for determining the foreground video embedding start timing has been detected.

  If the “SET button (not shown)” is not pressed, the compositing condition determination unit 62 does not detect an instruction operation for determining the foreground video embedding start timing. Returning to S10, the subsequent processing is repeated. That is, unless the “SET button (not shown)” is pressed, the loop process of steps S10 and S11 is repeated. Thereafter, when the “SET button (not shown)” is pressed, the determination instruction operation is detected by the synthesis condition determination unit 62, so that it is determined as YES in Step S11.

  Thereby, the embedding start timing of the foreground moving image is determined as one of the synthesis conditions. Next, the synthesizing condition determination unit 62, as another synthesizing condition, combines the position of the main subject image of the foreground moving image (the witch character on the heel in the examples of FIGS. 5 and 6) in the background moving image. Need to be determined. Therefore, as shown in FIG. 6, the selected image 107 includes a composite image obtained by combining the image 122 indicating the foreground moving image and the image 121 indicating the background moving image to be combined, and a frame 125 surrounding the main subject image of the foreground moving image. Is displayed on the display unit 42.

  In a state where the selected image 107 is displayed, the user performs a direction instruction operation from the outside, so that the display position of the frame 125 in the selected image 107 is set within the background moving image, more precisely, the background moving image. It can be moved freely within the range of the image 121 shown. The user repeats the direction instruction operation from the outside several times to display the main subject image of the foreground moving image in the frame 125 at a desired position in the background image, and thereafter, the “SET button ( By pressing “not shown”, it is possible to determine the synthesis position of the main subject image of the foreground moving image within the background moving image. Further, until the user presses the “SET button (not shown)” in the operation unit 43, the user can press the “zoom button (not shown)” in the operation unit 43 to set the synthesis condition. Furthermore, the composite size of the main subject image of the foreground moving image, which is another one, can be changed.

  That is, after it is determined as YES in the process of step S11 and the selected image 107 is displayed, in step S12, the synthesis condition determination unit 62 controls the output control unit 35 to detect the direction indication from the outside. The result is reflected in the synthesis position of the main subject image of the foreground video. In other words, the direction instruction from the outside is an instruction to change the composition position of the main subject image of the foreground video as described above, and the frame 125 and the main subject of the foreground video are detected according to the detection result of the result of the change instruction. The image display position changes in the background video. If both the foreground video and the background video are preview-played during such processing, the main subject images will move. Therefore, for the user, the main subject image of the foreground video will be displayed while imagining the synthesized video after completion. Can be set.

  In step S13, the composition condition determination unit 62 determines whether or not an instruction operation for determining the composition position of the main subject image of the foreground moving image has been detected.

  If the “SET button (not shown)” is not pressed, the composition condition determination unit 62 does not detect the instruction operation for determining the composition position of the main subject image of the foreground video, so it is determined as NO in step S13. The process returns to step S12, and the subsequent processes are repeated. That is, unless the “SET button (not shown)” is pressed, the loop process of steps S12 and S13 is repeated. Thereafter, when the “SET button (not shown)” is pressed, the determination instruction operation is detected by the synthesis condition determination unit 62, so that it is determined as YES in Step S13.

  In this way, the synthesis position of the main subject image of the foreground video is determined as one of the synthesis conditions, and the synthesis size of the main subject image of the foreground video is also determined as another synthesis condition as necessary. The Thereby, in this embodiment, since all the synthetic | combination conditions were determined, a process progresses to step S14 of FIG.

  In step S <b> 14, the frame duplicating unit 63 reads frame rate information stored in the two image files to be combined from the foreground moving image storage unit 72 or the background moving image storage unit 74.

  In step S <b> 15, the frame duplication unit 63 refers to the processing content information stored in the duplication processing content storage unit 73 and executes a process of duplicating the image data of the frame. That is, in the present embodiment, the frame duplicating unit 63 recognizes the first frame rate and the second frame rate from the frame rate information, and matches the playback speed of the background video with a high frame rate according to the processing content information. The image data of each frame included in the foreground moving image with a low frame rate is duplicated.

  In step S <b> 16, the frame synthesis unit 64 refers to the processing content information stored in the replication processing content storage unit 73 and executes a synthesis process for synthesizing frames. That is, in the present embodiment, according to the processing content information, the foreground video image data composed of a plurality of frames including the frames copied by the frame replicating unit 63 and the background so that the playback speeds at the time of playback match each other. Sequentially synthesizes video image data in units of frames. As a result, image data of a combined moving image that is a combined moving image in which the background moving image and the foreground moving image are combined and includes a plurality of frames having a normal angle of view is generated.

  A specific example of steps S15 and S16 will be described later with reference to FIGS.

  In step S <b> 17, the sound adding unit 65 determines whether sound has been recorded in the foreground moving image or the background moving image to be synthesized.

  If no sound is recorded in either the foreground moving image or the background moving image to be synthesized, it is determined as NO in Step S17, and the process proceeds to Step S19 without performing the process in Step S18.

  On the other hand, when the sound is recorded in at least one of the foreground moving image and the background moving image to be synthesized, it is determined as YES in Step S17, and the process proceeds to Step S18. In step S18, the audio adding unit 65 cuts out the audio data corresponding to the synthesized portion and associates it with the image data of the synthesized moving image. In this embodiment, when audio is recorded in both the foreground video and the background video to be synthesized, the audio data of the audio of the background video is preferentially cut out and associated with the image data of the synthesized video. To do.

  In step S19, the filing unit 66 converts each data of the synthesized moving image into a file and controls the output control unit 35 to display a preview of the synthesized movie. That is, the filing unit 66 creates the image data of the synthesized moving image generated in the process of step S16, the audio data of the audio associated in the process when the process of step S18 is executed, the various metadata, Is generated and stored in the combined moving image storage unit 75. During this time, for example, an image 108 as shown in FIG. 6 is displayed on the display unit 42, and when the composite image file is stored in the composite video storage unit 75, the display image on the display unit 42 is the composite video 126. The screen is switched to an image 109 for preview display. As a result, the synthetic moving image generation process ends.

  Furthermore, with reference to FIG. 7 to FIG. 9, the processes of steps S15 and S16 executed with reference to the processing content information in the synthetic moving image generation process will be specifically described below.

  FIG. 7 shows an example of the data structure of the processing content information. Each of FIGS. 8 and 9 shows a specific example of scheduling when a frame is duplicated according to the processing content information of FIG.

  In the present embodiment, since the processing content information has a matrix structure, the collection of items in the horizontal direction in FIG. 7 is hereinafter referred to as “row”, and the collection of items in the vertical direction in FIG. ". One of the first frame rates of the background moving image is associated with the predetermined row. One of the second frame rates of the foreground moving image is associated with the predetermined column. That is, the item of i rows and j columns (i is a value of 1 or 2 and j is any value from 1 to 4) includes the first frame rate associated with the i row. , The processing contents of the frame duplicating unit 63 (frame duplicating technique) and the processing contents of the frame synthesizing part 64 (frame synthesizing technique) in the case of a combination with the second frame rate associated with the j-th row are stored. ing.

  For example, in FIG. 7, when the first frame rate of the background moving image is 24 (FPS) in the first row and the second frame rate of the foreground moving image is 5 (FPS) in the second column, 1 row and 2 columns. The contents of the processing stored in the item of “Foreground video are duplicated 5 times and one foreground video is synthesized for each background video (the frame number of the foreground video is the same and the background video is the same). The process contents of “the frame No. of the frame number sequentially incremented six times” is read from the duplication process contents storage unit 73 by each of the frame duplication unit 63 and the frame synthesis unit 64. Then, the frame duplication unit 63 duplicates each frame of the foreground video according to the scheduling shown in FIG. 8, and the frame synthesis unit 64 follows each frame of the background video and the foreground video according to the scheduling shown in FIG. Each frame (including duplicated frames) is synthesized.

  In FIG. 8, the background moving image frame number is a number assigned to each of the frames constituting the background moving image in the order of reproduction, and means “frame number of background moving image” in FIG. . The foreground moving image frame number is a number assigned to each of the frames constituting the foreground moving image in the order of reproduction, and refers to “frame number of foreground moving image” in FIG. That is, in FIG. 8, the presence of six identical foreground video frame numbers means that after the original frame of the foreground video frame number, a copy of the original frame five times is inserted sequentially. It means to be combined with the corresponding frame of the video. The items described in this paragraph also apply to FIG.

  For example, in FIG. 7, when the first frame rate of the background moving image is 30 (FPS) in the second row and the second frame rate of the foreground moving image is 20 (FPS) in the fourth column, 2 rows 4 The contents of processing stored in the column item, that is, “Foreground moving image frames are duplicated once every 5 frames and one foreground moving image is combined with one background moving image frame (the frame number of the foreground moving image is The processing contents of “same and generated by sequentially incrementing the frame number of the background video” are read from the duplication processing content storage unit 73 by each of the frame duplication unit 63 and the frame synthesis unit 64. Then, the frame duplicating unit 63 duplicates each frame of the foreground video according to the scheduling shown in FIG. 9, and the frame synthesis unit 64 follows each frame of the background video and the foreground video according to the scheduling shown in FIG. Each frame (including duplicated frames) is synthesized.

  As described above, the digital camera 1 according to the present embodiment uses a moving image having a normal angle of view and a first frame rate as a background moving image, includes the main subject image area within the normal angle of view, and The moving image having the second frame rate including the region excluding the main subject image region as the transmission information region is used as the foreground moving image, and the image data of the background moving image and the image data of the foreground moving image are combined to have a normal angle of view. Generate image data of a composite video. In this case, the frame duplicating unit 63 of the digital camera 1 compares the first frame rate and the second frame rate, and matches the playback speed of the higher frame rate video of the foreground video and the background video. The image data of each frame included in the moving image having the lower frame rate is duplicated. The frame synthesizing unit 64 of the digital camera 1 is composed of a plurality of frames including the frame duplicated by the frame duplicating unit 63 among the foreground moving image and the background moving image so that the reproducing speeds at the time of reproduction match each other. The image data of the moving image and the image data of the other moving image are sequentially combined in units of frames. As a result, image data of a combined moving image that is a combined moving image in which the background moving image and the foreground moving image are combined and includes a plurality of frames having a normal angle of view is generated. Therefore, the filing unit 66 of the digital camera 1 stores the synthesized moving image image data generated by the frame synthesizing unit 64 in the synthesized moving image storage unit 75 as a new moving image data included in the synthesized image file.

  In this way, the digital camera 1 according to the present embodiment can create a realistic synthesized moving image.

  It should be noted that the present invention is not limited to the above-described embodiment, and modifications, improvements and the like within a scope that can achieve the object of the present invention are included in the present invention.

  For example, in the above-described embodiment, the processing of the frame duplication unit 63 is executed according to the processing content information stored in the duplication processing content storage unit 73, but is not particularly limited thereto. For example, if the display time per frame of the foreground video is expressed as “Tc”, Tc = 0.2 (s) for 5FPC, Tc = 0.1 (s) for 10FPC, and 20FPC. Tc = 0.05 (s). On the other hand, when the display time per frame of the background moving image is expressed as “Tb”, Tc = 0.0416 (s) for 24 FPC, and Tc = 0.0333 (s) for 30 FPC. Accordingly, although not shown, a calculation unit is further provided for the functional block of FIG. 2, and the calculation unit calculates (Tc / Tb), thereby displaying a multiple of the length per frame of the foreground video. That is, it can be determined that the number of frames to be copied is a number corresponding to this multiple. Therefore, the frame duplicating unit 63 may duplicate the image data of the foreground moving image frame by an amount corresponding to this multiple for each frame (or alternately). As a result, it is not necessary to store processing content information or the like in advance, and appropriate frame duplication can be performed for any first frame rate or any second frame rate.

<Modification>
In the above embodiment, the case where the present invention is applied to a digital camera has been described in detail. However, the present invention is not limited to this, and wireless communication having an imaging function, a mail transmission function, and a network server upload function, such as a camera-equipped mobile phone. You may make it apply to an apparatus.

  Hereinafter, the case where it is applied to such a device will be described. In addition, about the structure which has the same operation | movement and a function as the said embodiment, the description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  FIGS. 10A and 10B are external views of the camera-equipped mobile phone 111 according to this modification. FIG. 10A is a front view in an open state, and FIG. 10B is a rear view in a closed state.

  The camera-equipped mobile phone 111 includes a first housing having the display unit 42 and a second housing having the operation unit 43. The first housing and the second housing are biaxial directions. By the hinge part that rotates in the vertical direction, the open state shown in FIG. 10 (a) and the closed state shown in FIG. 10 (b) can be taken. In the figure, reference numeral 46 denotes a telephone speaker.

  In this case, when acquiring by capturing a background image or a foreground image, it is performed in both forms of FIGS. 10 (a) and 10 (b). Further, the synthetic moving image generation process is performed only in the form of FIG.

  FIG. 11 is a diagram showing a hardware configuration of the camera-equipped mobile phone 111.

  Compared with the digital camera 1, the camera-equipped mobile phone 111 is roughly added with the call speaker 46 as an audio output unit, and a signal processing unit corresponding to a wireless communication system such as CDMA instead of the communication unit 44. In addition to this, the CPU 31 performs communication processing, e-mail creation processing, data upload, and data download processing in addition to image processing. As a matter of course, the ROM 32 further stores a program for executing these processes.

  Further, the image data and the moving image file to be stored in the index storage unit 71, the foreground moving image storage unit 72, and the background moving image storage unit 74 in FIG. 2 are acquired by the image acquisition unit 61 and a wireless communication network (not shown). The data received by the antenna 441 and the communication unit 440 can be stored.

  The composite video file created by the composite video generation process can be output to the outside of the camera-equipped mobile phone 111 by an attachment process to an e-mail or a direct upload process to a network server.

  In this way, according to the modification, not only can a realistic synthesized video be generated inside the apparatus, but also the foreground video and background video required for synthesis can be easily obtained via a wireless communication network. It is also possible to send a composite video to the outside.

  Thus, the present invention is not particularly limited to a digital camera, and can be applied to electronic devices in general. Specifically, for example, the present invention is applicable to a video camera, a portable navigation device, a portable game machine, and the like.

  The series of processes described above can be executed by hardware or can be executed by software.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  Although not shown, the recording medium including such a program is not only constituted by a removable medium distributed separately from the apparatus main body in order to provide a program to the user, but also in a state of being incorporated in the apparatus main body in advance. It consists of a recording medium provided to the user. The removable medium is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preinstalled in the apparatus main body is configured by, for example, the ROM 32 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 11 ... Imaging part, 12 ... Audio | voice input part, 13 ... Data processing part, 14 ... User interface part, 21 ... Optical lens part, 22 ... Image sensor, 31 ... CPU, 32 ... ROM, 33 ... RAM, 34 ... Memory, 35 ... Output control unit, 36 ... Image processing unit, 61 ... Image acquisition unit 62 ... Synthesis condition determining unit, 63 ... Frame duplicating unit, 64 ... Frame synthesizing unit, 65 ... Audio adding unit, 66 ... Filing unit, 71 ... Index storage unit, 72 ... foreground moving image storage unit, 73 ... replication processing content storage unit, 74 ... background moving image storage unit, 75 ... synthetic moving image storage unit, 111 ... mobile phone with camera

Claims (7)

Image data of a first moving image having a normal angle of view and a first frame rate, and a main subject image area within the normal angle of view and an area excluding the main subject image area as a transmission information area In the image processing device that generates the image data of the synthesized moving image having the normal angle of view by combining the image data of the second moving image having the second frame rate,
The first frame rate and the second frame rate are compared, and a lower frame rate is selected in accordance with the playback speed of the higher frame rate video of the first video and the second video. A copying means for copying the image data of each frame included in the other video,
Image data of a moving image composed of a plurality of frames including the frame duplicated by the duplicating means of the first moving image and the second moving image so that the reproduction speeds at the time of reproduction match each other. Synthesizer that generates image data of the synthesized moving image by sequentially synthesizing the image data of the other moving image in units of frames;
Recording means for recording image data of the synthesized moving image generated by the synthesizing means as image data of a new moving image;
An image processing apparatus comprising: The synthesizing unit generates image data of the synthesized moving image including frames of the number of frames by combining the number of frames corresponding to the reproduction time of the second moving image.
The image processing apparatus according to claim 1. Stores processing content information that is information in which the correspondence between the first frame rate and the second frame rate is associated with at least one of the processing content of the duplicating unit or the processing content of the combining unit. Further comprising processing content storage means;
At least one of the duplicating unit or the synthesizing unit is configured to correspond to the correspondence between the first frame rate of the first moving image to be combined and the second frame rate of the second moving image to be combined. The associated processing content is read from the processing content information stored in the processing content storage means, and the processing according to the read processing content is executed.
The image processing apparatus according to claim 1. Computing means for computing the number of frames duplicated by the duplicating means from the first frame rate and the second frame rate,
The image processing apparatus according to claim 1, further comprising: Imaging means for imaging within the range of the normal angle of view;
The image capturing unit is continuously driven based on the first frame rate or the second frame rate to acquire image data of a plurality of images, and each of the acquired plurality of images is included as a frame. An image acquisition unit that generates image data of the first moving image or the second moving image, and adds to the image data, as additional information, information specifying a frame rate when the imaging unit is continuously driven;
The image processing apparatus according to claim 1, further comprising: Audio data is recorded in at least one of the first moving image or the second moving image,
The recording means further adds the audio data to the image data of the new moving image for recording;
The image processing apparatus according to any one of claims 1 to 5. Image data of a first moving image having a normal angle of view and a first frame rate, and a main subject image area within the normal angle of view and an area excluding the main subject image area as a transmission information area A computer that controls an image processing apparatus that generates image data of a combined moving image having a normal angle of view by combining the image data of the second moving image having the second frame rate;
The first frame rate and the second frame rate are compared, and a lower frame rate is selected in accordance with the playback speed of the higher frame rate video of the first video and the second video. A replication function that replicates the image data of each frame included in the other video,
The image of the moving image composed of a plurality of frames including the frame duplicated by the realization of the duplication function of the first moving image and the second moving image so that the reproduction speeds at the time of reproduction match each other. A synthesis function for generating image data of the synthesized video by sequentially synthesizing data and image data of the other video in units of frames;
A recording control function for recording the image data of the combined moving image generated by the realization of the combining function as image data of a new moving image;
A program to realize

Images