JP4962219B2 - Composite image output apparatus and composite image output processing program - Google Patents

Description

  The present invention relates to a composite image output apparatus and a composite image output processing program for generating and outputting a composite image obtained by replacing a person image captured with a moving image with another character image.

  In recent years, with the advancement of special photography technology, it has become possible to create scenes that do not exist on the screen by miniature photography, composition of a person and a background image, composition of a computer graphic and a photographed image, and the like. Recently, an arbitrary character image corresponding to the motion of a model (target person) captured in advance as a moving image can be displayed as a moving image by the motion capture technology and the 3D image processing technology. In addition, image composition for adding comments to moving images has been generally performed using a net service.

  Conventionally, in the same applicant, a person image to be replaced is extracted from the photographed image according to distance information to each image portion included in the photographed image photographed by the digital camera, and the extracted target person An image output device is conceivable in which a mimic image composed of another character image corresponding to an image pose (posture) is replaced with a target person image and displayed (for example, refer to Patent Document 1).

  Further, the same applicant receives pose information indicating the posture of the person image to be replaced that is sent from the terminal device on the communication network, and the character image of the posture corresponding to the pose information is received from the database. An image distribution server that reads out and provides a transmission to the terminal device is also considered (for example, see Patent Document 2).

  According to such a conventional image output device or image distribution server, it is possible to replace an arbitrary target person image in a photographed image with another character image, and display or print it. You can enjoy using images by replacing them with other character images, or you can use the captured images more effectively in various situations by impersonating people who should not be seen by others and replacing them with other character images. .

Japanese Patent No. 3901015

Japanese Patent No. 3918632

  In the conventional technique for replacing a photographed person image with another character image, the target person image is replaced and synthesized as it is in either case of a still image or a moving image. As for special effects and animations, it is difficult to generate a composite image that forms a series of scenes by adding special effect images indicating the special techniques and attacks with the will of the target person.

  The present invention has been made in view of such problems, and for a human image taken in a moving image, a composite image that forms a series of scenes by adding a special effect image to a position corresponding to the target person according to the intention of the target person. It is an object of the present invention to provide a composite image output apparatus and a composite image output processing program that can generate the image.

The composite image output apparatus according to claim 1, a special effect data storage unit that stores preset gesture model data, special effect image data, and mark data indicating the special effect in association with each other; A captured image acquisition means for acquiring a series of captured image data, a person image extraction means for extracting a person image from the series of captured image data acquired by the captured image acquisition means, and the person image extraction means. Special effect gesture detection means for detecting a person image whose model data stored in the special effect data storage means matches the preset gesture from the extracted person image, and the special effect gesture detection means advance the photographic image data when gestures of the person image is detected which has been set, the synthesized image obtained by combining the image data of the special effect by And composite image generation means for generating data, and synthesized image output means for outputting a sequence of composite image data generated by the composite image generating means, wherein the composite image generating means, by the special effect gesture detecting means Photographed image data when a pre-set gesture human image is detected, a marked composite image generating unit that generates combined image data with a mark by combining mark data indicating the special effect, and the combined image with the mark Effect combined image generation means for generating composite image data obtained by combining the image data with the special effect on the composite image data with marks generated by the generation means, and the composite image output means includes the composite image with marks A marked composite image output means for outputting the marked composite image data generated by the generating means; It is characterized by comprising further comprising the effect combined image output means for outputting the synthesized image data obtained by combining the image data of the special effect which is generated, an the image generating means.

The composite image output apparatus according to claim 2, wherein the effect composite image generation unit synthesizes special effect image data at the detected position of the person image. It is characterized by that.

The composite image output device according to claim 3 is the composite image output device according to claim 1, wherein the effect composite image generation unit is configured to perform a special effect in one direction from the position of the detected person image. It is characterized by combining the image data .

The composite image output processing device according to claim 4 is the composite image output device according to any one of claims 1 to 3, wherein the special effect data storage means includes preset gesture model data and special data. The image data of the effect and the sound data are stored in association with each other, and the composite image generation unit is configured to capture the captured image data when the pre-set gesture human image is detected by the special effect gesture detection unit, and the special image It is characterized by generating combined image data obtained by combining the effect image data and generating combined image data in which the special effect audio data is associated .

According to the present invention, for the person image in the photographed image data, in accordance with gesture by target person, appropriately, as possible out to produce a composite synthetic image special effect image to the photographed image data. At that time, the combined image data with the combination of the photographed image data when the gesture human image is detected and the mark data indicating the special effect are generated and output, so that the special effect corresponding to the gesture Can be confirmed from the output data.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a conceptual diagram illustrating a moving image shooting state and a combined image generation state using a digital camera 10 according to an embodiment of a combined image output device of the present invention.
The digital camera 10 includes an LCD display unit 11, an input unit 12, and an audio output unit (speaker) 13 on the back surface thereof, and a moving image (captured image) G photographed according to the operation of the input unit 12 is real-time. It is displayed on the display unit 11.

  The digital camera 10 has, as main functions, a function of extracting a person / object image (P, K, A, B) included in the moving image G being shot, and a person image in which the extracted person images are registered in advance. A function of recognizing whether or not the image is (P) by recognizing the face image; if the extracted person image is a person image (P) registered in advance, it is associated with the registered person image (P) A function of adding a mark (posture transformation mark 24Mw) of a pre-set role image (24Gw), and a function of replacing the person image (P) to which the role transformation mark 24Mw is added with the pre-set role image (24Gw) , A function for detecting a pre-set gesture (motion) of the extracted person image, and a special effect image that is set in advance for the person image (K) from which the pre-set gesture has been detected. A function of adding a mark (23Gu1) (special effect mark 23Mu1), a function of adding a preset special effect image (23Gu1) in the vicinity of the person image (K) to which the special effect mark 23Mu1 is added, are extracted. If the object image is a destruction target image (A, B) registered in advance, a mark (destruction mark 15Ma) of the destruction image (15Ga3, 15Gb3) set in advance in association with the destruction target image (A, B) ', 15Mb'), a function for replacing the destruction target images (A, B) to which the destruction marks 15Ma ', 15Mb' are added with the preset destruction images (15Ga3, 15Gb3), and A motion capture process for capturing the motion of the extracted person image (P, K) and converting it into a 3D image of the role image (24Gw) Function, and a like 3D image processing function.

The face image recognition process, the motion capture process, and the 3D image process are well-known image processing techniques, and thus detailed description thereof is omitted.
When a digital camera 10 having such a main function is used by a family, for example, when a mother plays a series of scenes in which the father P serves as a monster and the child K serves as a man, In the digital camera 10, the face image of the father P is registered in advance in association with the monster image (the role image 24Gw), and the destruction target images of the beer bottle A and the paper airplane B correspond to the destruction images 15Ga and 15Gb of the tower and the airplane, respectively. And register in advance.

Then, as the father P defeats the beer bottle A or knocks down the flying paper airplane B, the child K makes a motion (special gesture) for special effects such as a special technique, and XX Man's Special Kill A mother photographs a series of scenes to try with a digital camera 10.
As shown in FIG. 1A, the moving image G being shot is displayed on the display unit 11 in real time and analyzed by the various functions of the digital camera 10. , A role transformation mark 24Mw indicating the corresponding monster image (a role image 24Gw) is additionally displayed, and when a preset gesture (motion) is detected in the child K person image, the corresponding XX man image is displayed. A special effect mark 23Mu1 indicating the special technique (special effect image 23Gu1) is additionally displayed. The beer bottle A is additionally displayed with a destruction mark 15Ma indicating the destruction image 15Ga of the corresponding tower, and the paper airplane B is additionally displayed with a destruction mark 15Ma indicating the destruction image 15Gb of the corresponding airplane.

As a result, the mother being photographed has the father P person image transformed according to his role, the special effect image of the special technique is added to the vicinity of the child K person image, and the object images of the beer bottle A and the paper airplane B are registered. It can be easily recognized that the replacement processing is performed as described above.
At this time, the voice data “Gao” and “Bibi Bibi” stored in advance in association with the monster image (role image 24Gw) and the XX man's special technique (special effect image 23Gu1) are read out and output as audio. By being output from the part 13, even if it is in the father P who is playing each role or the child K who is performing a special move, the transformation process and the special effect processing are carried out according to the role. Can be easily recognized.

A moving image in which the role transformation mark 24Mw, the special effect mark 23Mu1, and the destruction marks 15Ma and 15Mb are added to the photographed moving image (captured image) G is referred to as a marked photographed image Gm.
Thereafter, when the shooting of the moving image G of the series of scenes is finished and the captured image Gm with the mark is generated, the captured image Gm with the mark is analyzed by the various functions, and is shown in FIG. As shown, the monster role image 24Gw corresponding to the person image of the father P to which the role change mark 24Mw is added, and the XX man's deadly shot corresponding to the child K person image to which the special effect mark 23Mu1 is added. The special effect image 23Gu1 of the technique, and the destruction images 15Ga and 15Gb of the tower and the plane corresponding to the destruction target images of the beer bottle A and the paper airplane B to which the destruction marks 15Ma and 15Mb are added are set in advance. A synthesized moving image GG synthesized with the image BG is generated.

  Note that the destruction images 15Ga and 15Gb associated with the destruction target images of the beer bottle A and the paper airplane B are both the character change mark 24Mw and the transformation mark with respect to the destruction target images (A, B). It is detected that the father P or child K person image to which 14 Mu has been added has come into contact with the image and is replaced with a previously stored image after destruction.

This makes it easy for the user's family to create a special-effects-style synthesized moving image GG, and for the special move scene that XX Man has performed on the monster, it is close to the person image of the child K in the captured image. A series of synthesized moving images GG including images synthesized by adding special effect images indicating techniques are output and displayed on the display unit 11 or output and displayed on an external monitor. You can get some fun. Thus, it is possible to easily and effectively synthesize the image captured with the moving image with the destruction image of the object.
FIG. 2 is a block diagram showing a configuration of an electronic circuit of the digital camera 10 (communication terminal A).

The digital camera 10 includes a CPU 16 that is a computer.
The CPU 16 is a system program stored in the memory 17 in advance, a camera control program read from the external recording medium 18 such as a memory card into the memory 17 via a recording medium reading unit 19 such as a card slot, or a communication network ( N) The operation of each part of the circuit is controlled according to the camera control program read into the memory 17 from the server device 30 (in this case, the program server) via the power transmission control unit 20.

The CPU 16 is connected to the LCD display unit 11, the input unit 12, the audio output unit 13, the memory 17, the recording medium reading unit 19, and the power transmission control unit 20, as well as a solid-state imaging device (CCD), an imaging optical system, and the like. An image photographing unit 21 having a distance sensor and an illuminance sensor is connected.
The program memory 22 of the memory 17 includes a system program that controls the entire operation of the digital camera 10, a camera control program that controls the photographing operation, a server device 30 on the communication network N, and an external communication terminal B (PC: personal computer). ) A communication control program for controlling the communication operation with 40 is stored, and based on the captured image G captured by the image capturing unit 21, the above-described marked captured image Gm and composite image GG are generated and output. An image processing program that manages the various functions for storing the image is stored in advance.

The various programs stored in the program memory 22 are a key input signal from the input unit 12, a shooting input signal from the image shooting unit 21, and an external device (30, 40) via the transmission control unit 20. It is activated in response to the received input signal.
The memory 17 includes a transformation data memory 14, a special effect data memory 23, a destruction data memory 15, a transformation role data memory 24, a photographed image data memory 25, a marked photographed image data memory 26, a composite image data memory 27, And other working data memories.

FIG. 3 is a diagram showing the contents of the transformation data stored in advance in the transformation data memory 14 of the digital camera 10.
The transformation data memory 14 has storage areas for transformation motion model data 14P, transformation image data 14Gu, and transformation mark data 14Mu. The storage area for transformation image data 14Gu includes a plurality of types of transformation image data (transformation characters) and The audio data is stored as, for example, XX Man A's transformation image data 14Gu1 and its audio data “Swatch” 14Su1, the first XX Man's transformation image data 14Gu2 and its audio data “Ya Tah” 14Su2. Further, in the storage area of the transformation mark data 14Mu, transformation mark data 14Mu1 and 14Mu2 that respectively express the plurality of types of transformation image data 14Gu1 and 14Gu2 in a simplified manner are stored. In the storage area of the transformation motion model data 14P, transformation motion model data 14P1a to 14P1c and 14P2a to 14P2c including a series of motions (gestures) for transformation into the plural types of transformation image data 14Gu1 and 14Gu2 are stored. Is done.

  The transformed image data 14Gu1 and 14Gu2 are stored as image data itself representing various movements, or a combination of movement data of limb skeletons and facial feature points and 3D image generation data. The transformation motion model data 14P1a to 14P1c and 14P2a to 14P2c are stored as image data representing the transformation motion itself, or movement data of the limb skeleton and facial feature points.

FIG. 4 is a diagram showing the contents of special effect data stored in advance in the special effect data memory 23 of the digital camera 10.
The special effect data memory 23 has storage areas for special effect motion model data 23P, special effect image data 23Gu, and special effect mark data 23Mu. The storage area for the special effect image data 23Gu includes the plurality of types of transformed images. The special effect image data corresponding to each of the data 14Gu1 and 14Gu2 and the sound data thereof are, for example, the special effect image data 23Gu1 for XX Man A and the sound data “BibiBibi” 23Su1, the special effect image data 23Gu2 for the first XX man. And its voice data “Peeverly” 23Su2. In addition, special effect mark data 23Mu1, 23Mu2 for simply expressing the plurality of types of special effect image data 23Gu1, 23Gu2 is stored in the storage area of the special effect mark data 23Mu. A series of motions for generating and adding the plurality of types of special effect image data 23Gu1, 23Gu2 from predetermined positions of the corresponding transformation image data 14Gu1, 14Gu2 to the storage area of the special effect motion model data 23P. Special effect motion model data 23P1a to 23P1c and 23P2a to 23P2c including (gesture) are stored.

  The special effect image data 23Gu1 and 23Gu2 are stored as image data itself or 3D image generation data. The special effect motion model data 23P1a to 23P1c and 23P2a to 23P2c are stored as image data representing the special effect motion itself, or movement data of the limb skeleton and facial feature points.

FIG. 5 is a diagram showing the contents of the destruction data stored in advance in the destruction data memory 15 of the digital camera 10.
The destruction data memory 15 has storage areas for destruction target image data 15T, destruction image data 15G, destruction (front) mark data 15M, and destruction (rear) mark data 15M ′, and the storage area for the destruction image data 15G includes A series of a plurality of types of destruction image data and sound data in the middle of destruction include, for example, image data 15Ga1 to 15Ga3 before destruction of the tower, destruction of the tower, and sound data “Gusha” 15Sa during destruction, destruction of the airplane, The image data 15Gb1 to 15Gb3 before, during and after destruction, and the sound data “Dekan” 15Sb before destruction, the image data 15Gc1 to 15Gc3 before, during and after destruction of the building, and the sound data “Rattle” "Is stored as 15Sc. In addition, in each storage area of the destruction (previous) mark data 15M and the destruction (rear) mark data 15M ′, the plurality of types of image data 15Ga1, 15Ga3, 15Gb1, 15Gb3, 15Gc1, 15Gc3 before and after the destruction are respectively expressed in a simplified manner. Mark data 15Ma, 15Ma ′, 15Mb, 15Mb ′, 15Mc, 15Mc ′ before and after the destruction are stored. In the storage area of the destruction target image data 15T, an object image in a general household to be replaced with the plurality of types of destruction image data 15Ga1 to 15Ga3, 15Gb1 to 15Gb3, 15Gc1 to 15Gc3, for example, In the case of the destruction image data 15Ga1 to 15Ga3, it is stored as the beer bottle A1 or the plastic bottle A2, in the case of the destruction image data 15Gb1 to 15Gb3 of the airplane, the paper plane B, and in the case of the destruction image data 15Gc1 to 15Gc3 of the building, it is stored as the tissue box C. Is done.

  The destructive image data 15Ga1 to 15Ga3, 15Gb1 to 15Gb3, and 15Gc1 to 15Gc3 are stored as image data itself or 3D image generation data. The destruction target image data A1 (A2), B, and C are stored as image data obtained by photographing the object, or shape data indicating the characteristics of the object.

FIG. 6 is a view showing the contents of the transformation character data stored in the transformation character data memory 24 of the digital camera 10.
This transformation role data memory 24 includes transformation target image data 24T, transformation (1) image data 24Gw and transformation (1) mark data 24Mw, transformation (2) image data 24Gw 'and transformation (2) mark data 24Mw. ′, A storage area for transformation condition data 24H, and transformation (1) In the storage area for image data 24Gw, a plurality of types of first stage transformation image data (transformation characters) and their voice data are stored in, for example, monster A. Transformation (1) Image data 24Gw1 and its voice data “Gao” 24Sw1, transformation of monster B (1) Image data 24Gw2 and its voice data “Gaga” 24Sw2 are stored. Further, in the storage area of the transformation (1) mark data 24Mw, transformation (1) mark data 24Mw1 and 24Mw2 that simply express the plural types of transformation (1) image data 24Gw1 and 24Gw2 are stored. Further, in the storage area for the transformation (2) image data 24Gw ′, the second-stage transformation image data 24Gw1 ′ and 24Gw2 ′ corresponding to the plurality of types of first-stage transformation image data 24Gw1 and 24Gw2, respectively, and the audio data thereof. 24Sw1 'and 24Sw2' are stored. Further, in the storage area for the transformation (2) mark data 24Mw ′, transformation (2) mark data 24Mw1 ′ and 24Mw2 ′ for simply expressing the transformation (2) image data 24Gw1 ′ and 24Gw2 ′ are stored. Further, in the storage area of the transformation condition data 24H, the first stage transformation (1) image data 24Gw1, 24Gw2 to the second stage transformation (2) condition data 24H1 for transformation into image data 24Gw1 ′, 24Gw2 ′. , 24H2 is stored.

In this embodiment, since the second stage transformation (2) image data 24Gw1 ′ and 24Gw2 ′ is not set, the transformation (2) mark data 24Mw1 ′ and 24Mw2 ′ and transformation condition data 24H1 and 24H2 are not set. .
In the storage area of the transformation target image data 24T, the facial images of persons to be transformed and replaced with the plurality of types of transformation image data 24Gw1 and 24Gw2, respectively, are, for example, the father's facial image P, the mother's facial The user is registered and stored as an image M.

The transformation (1) and (2) image data 24Gw and 24Gw ′ are image data representing various movements, or a combination of movement data of limb skeletons and facial feature points and 3D image generation data. Remembered.
The photographed image data memory 25 stores photographed image data G input by taking a still image or a moving image by the image photographing unit 21.

  In the marked-captured image data memory 26, the marked-captured image data Gm (generated in real time for shooting according to the image processing program based on the captured image data G captured and stored in the captured image data memory 25). 1A) are sequentially stored.

In the composite image data memory 27, composite image data GG generated according to the image processing program based on the marked captured image data Gm generated and stored in the marked captured image data memory 26 (see FIG. 1B). Is memorized.
The digital camera 10 having the above configuration generates and displays the marked photographic image data Gm (see FIG. 1A) in real time based on the photographic image data G captured by the digital camera 10 itself. Thereafter, the composite image data GG (see FIG. 1B) is generated from the photographed image data Gm with the mark and displayed or output to the outside. On the other hand, photographic image data Gm with a mark (see FIG. 1 (A)) is generated from the photographic image data G taken with the moving image other than the photographic function of the digital camera 10, and the photographic image data Gm with the mark is generated. Various functions for generating the composite image data GG (see FIG. 1B) are provided in the server device 30 on the communication network N, and a digital camera (communication terminal A) 10 ′ or PC (with communication function) is installed. Based on the moving image shooting data G transferred from the communication terminal B) 40, the server device 30 generates the marked shooting image data Gm and the composite image data GG, and the communication terminals (A) 10 'and (B). It is good also as a structure which responds to 40 and provides a service.

FIG. 7 is a block diagram showing the configuration of the electronic circuit of the server apparatus 30 according to the embodiment of the composite image output apparatus of the present invention.
The server device 30 includes a CPU 31 that is a computer.
The CPU 31 operates in accordance with a system program stored in advance in the memory 32 or a server control program read into the memory 32 from an external recording medium 33 such as a CD-ROM via a recording medium reading unit 34 such as an optical disk drive. Control the behavior.

  In addition to the memory 32 and the recording medium reading unit 34 being connected to the CPU 31, data transmission control between the communication terminal A (digital camera) 10 ′ and the communication terminal B (PC) 40 on the communication network N is performed. Are connected to an electric transmission control unit 35, an input unit 36 such as a keyboard / mouse, and an LCD display unit 37.

  The program memory 32A of the memory 32 includes a system program that controls the overall operation of the server device 30, and communication operations with each communication terminal A (digital camera) 10 'and communication terminal B (PC) 40 on the communication network N. The communication control program for managing the image is stored, and based on the moving image photographic image data G transferred from each of the external communication terminals A10 ′ and B40, the marked photographic image Gm and the composite image are obtained as in the digital camera 10 described above. An image processing program for managing various functions described with reference to FIG. 1 for generating or outputting (distributing) GG is stored in advance.

Various programs stored in the program memory 32A are activated in response to an input signal from the input unit 36 and a received input signal from each of the external communication terminals A10 'and B40 via the transmission control unit 35. .
The memory 32 includes a transformation data memory 32B, a special effect data memory 32C, a destruction data memory 32D, a transformation data, which are substantially the same data contents as those in the digital camera 10 described with reference to FIGS. A role data memory 32E, a photographed image data memory 32F, a marked photographed image data memory 32G, a composite image data memory 32H, and other work data memories are provided.

  Thereby, in the server apparatus 30, based on the moving image photographic image data G transferred from the communication terminal A (digital camera) 10 'or the communication terminal B (PC) 40 connected to the communication network N, the same marked shooting as described above is performed. Image data Gm (see FIG. 1 (A)) and composite image data GG (see FIG. 1 (B)) are generated and distributed to the communication terminals A10 ′ and B40 that are the transfer source of the photographed image data G. Service.

Next, the operation of the composite image output apparatus by the digital camera 10 or the server apparatus 30 having the above configuration will be described.
FIG. 8 is a flowchart showing the overall process of the composite image output process by the composite image output apparatus.
FIG. 9 is a flowchart showing target image extraction processing for extracting target images of persons (P, M, K) and objects (A, B, C) existing in the captured image (G) in accordance with the composite image output processing. It is.

FIG. 10 is a flowchart showing a role transformation mark process for adding a transformation mark (24 Mw) to a person image (P, M) whose role has been registered along with the composite image output process.
FIG. 11 is a flowchart showing destruction mark processing for adding a destruction mark (15M, 15M ′) to the object image (A, B, C) extracted with the composite image output processing and outputting the corresponding sound (15S). It is.

FIG. 12 is a flowchart showing a motion transformation mark process for adding a transformation mark (14Mu) to a person image (K) in which a transformation motion (14P) is detected in accordance with the synthetic image output process and outputting a corresponding voice (14Su). It is.
FIG. 13 shows a special effect mark process for adding a special effect mark (23Gu) to a person image (K) in which a special effect motion (23P) is detected in accordance with the composite image output process and outputting a corresponding sound (23Su). It is a flowchart to show.

  In FIG. 14, the person image (P) (K) to which the transformation mark (24Mw) (14Mu) is added in accordance with the synthetic image output processing is replaced with the transformation image (24Gw) ((14Gu) and synthesized, and the corresponding voice (24Sw ) (14Su) is a flowchart showing a transformation mark composition process for setting the output of (14Su).

In FIG. 15, the object images (A, B) to which the destruction marks (15M, 15M ′) are added in accordance with the synthetic image output processing are replaced with the destruction images (15G), and the corresponding sound (15S) output is set. It is a flowchart which shows the destruction mark synthetic | combination process to perform.
FIG. 16 shows a special effect in which the special effect image 23Gu is additionally synthesized at a predetermined position of the person image (K) to which the special effect mark (23Gu) is added in accordance with the synthetic image output process and the output of the corresponding sound (23Su) is set. It is a flowchart which shows an effect mark synthetic | combination process.

  FIGS. 17 to 25 are diagrams showing image processing states based on the moving image captured image (G) accompanying the composite image output processing in order of typical frame advance (part 1) to (part 9). FIG. 3A is a diagram showing a captured image G, FIG. 4B is a diagram showing a marked captured image Gm, and FIG. 4C is a diagram showing a composite image GG.

  In the combined image output processing in steps S1 to S3 in FIG. 8, as shown in FIGS. 17A to 25A, a captured image (G) of each frame to be captured is acquired. As a result, the captured image (Gm) with a mark as shown in FIGS. 17B to 25B is generated and output.

  Then, the process of steps S4 to S7 of the composite image output process in FIG. 8 is executed based on a series of marked photographed images (Gm) generated as shown in FIGS. 17 (B) to 25 (B). Then, a composite image (GG) as shown in FIGS. 17C to 25C is generated and output.

  Here, the case where the composite image output process in FIG. 8 is executed in the digital camera 10 (see FIG. 2) will be mainly described, and the composite image output process is executed in the server device 30 (see FIG. 7). Since the processing contents are substantially the same, the description thereof is omitted.

  When a series of scenes played by a parent and child are photographed by the image photographing unit 13 of the digital camera 10, for example, as shown in FIGS. 17A to 25A, the photographed images G1 to G9 are sequentially captured image data. It is stored and stored in the memory 25 (step S1), and the process proceeds to the target image extraction process in FIG. 9 in real time (step SA).

  This target image extraction processing has two methods. In the target image extraction processing (method 1) shown in FIG. 9A, the captured image data Gn sequentially stored in the captured image data memory 25 is stored in the work memory. Are temporarily transferred and stored (step A11), and the changed portion of the image between the previous captured image data Gn-1 and the current captured image data Gn is blocked and extracted (step A12).

Then, the position information blocked on the photographed image data Gn becomes the position information of the target image that is a person or an object, and is stored in the photographed image data memory 25 as additional information of the photographed image data Gn (step). A13).
On the other hand, in the target image extraction process (method 2) shown in FIG. 9B, it is acquired by the distance sensor of the image photographing unit 21, which is additional information of the photographed image data Gn sequentially stored in the photographed image data memory 25. The distance information to the subject is read (step A21), and the image portion existing at a short distance is extracted as a block (step A22).

Then, the position information blocked on the photographed image data Gn becomes the position information of the target image that is a person or an object, and is stored in the photographed image data memory 25 as additional information of the photographed image data Gn (step). A23).
That is, the target image extraction process (method 2) shown in FIG. 9B requires distance information to the subject as additional information of the captured image data Gn, and is therefore suitable for processing by the digital camera 10, The target image extraction processing (method 1) shown in FIG. 9A is suitable for processing in the server device 30 because it does not require distance information to the subject in the captured image data Gn.

As described above, when the person image (P, K) and the object image (A, B) existing in the captured image data Gn are extracted by the target image extraction process (step SA), the role transformation in FIG. The process proceeds to mark processing (step SB).
In this role transformation mark processing, the transformation role data memory 24 (see FIG. 6) is used for the person images (P, K) and object images (A, B) in the photographed image data Gn extracted by the target image extraction processing. ) Is included using the face image recognition process (step B1). The person image (P or M) as the transformation target (face) image data 24T registered by the user is included.

  Here, as shown in each of the captured image data G1 to G9 in FIGS. 17A to 25A, it is identified that the father's person image P as the transformation target (face) image data 24T is included. When it is done (step B1 (Yes)), the transformation mark data 24Mw1 of the monster A stored in association with the transformation role data memory 24 (see FIG. 6) is read and the identified father's person The image P is added corresponding to the recognized position of the face image, and is temporarily stored in the work data memory as marked image data Gm1 to Gm9 as shown in FIGS. 17 (B) to 25 (B). It is displayed on the display unit 11 (step B2).

The transformation mark data 24Mw1 added corresponding to the position of the face image of the registered person image P is either added to the face image or added in the vicinity of the face image. There may be.
If it is determined that the digital camera 10 is currently photographing (step B3 (Yes)), the voice data “Gao” stored in association with the transformation character data memory 24 (see FIG. 6) is stored. 24Sw1 is read out and output from the audio output unit 13 as shown in FIGS. 17B to 25B (step B4).

Then, the process proceeds to the destruction mark process in FIG. 11 (step SC).
In the destruction mark process, first, the destruction data memory 15 (see FIG. 5) is used for the person image (P, K) and the object image (A, B) in the photographed image data Gn extracted by the target image extraction process. ) Includes an object image (A, B, or C) as the destruction target image data 15T stored in advance (step C1).

  Here, the object images of the beer bottle A1 and the paper airplane B as the destruction target image data 15T are included as shown in the captured image data G1 to G9 of FIGS. If the object image is identified (step C1 (Yes)), whether the object image of the identified beer bottle A1 or paper airplane B is in a state after destruction, whether each object image (A1, B) Judgment is made by image recognition as to whether or not it is in an inverted state (step C2).

  Here, for example, as shown by the beer bottle A1 in FIG. 17A or the paper airplane B in FIG. 18A, each of the object images A1 and B is image-recognized unless they are in the overturned state or the inverted state. If it is determined that the state is not in the state (step C2 (No)), the tower and airplane destruction (previous) mark data 15Ma and 15Mb stored in association with the destruction data memory 15 (see FIG. 5) are stored. Is added corresponding to the position of each of the object images A1 and B, and is recorded in the work data memory as marked image data Gm1 and Gm2 as shown in FIG. 17B or 18B. It is temporarily stored and displayed on the display unit 11 (step C3).

The destruction (previous) mark data 15Ma and 15Mb added corresponding to the positions of the object images A1 and B to be destroyed are added to the object images A1 and B, or are added to the object images A1 and B. It may be added in the vicinity of A1 and B.
If it is determined that the digital camera 10 is currently photographing (step C4 (Yes)), the destruction of the airplane stored in association with the destruction data memory 15 (see FIG. 5) (previous) ) The audio data “Bune” 15Sb1 of the image data 15Gb1 is read out and output from the audio output unit 13 as shown in FIG. 18B (step C5).

  Then, the position of the destruction target image data (object image) of the beer bottle A1 or the paper airplane B, which is a portion corresponding to the tower or airplane destruction (previous) mark data 15Ma, 15Mb additionally displayed in step C3, is the target. It is identified from the position information of the block obtained as additional information of the captured image data Gn by the image extraction process (step SA) (step C6), and it is determined whether or not the image portion of another target image extraction block has been touched. (Step C7).

  Here, for example, the contact state between the object image of the beer bottle A1 and the father's person image P in the photographed image data G2 in FIG. 18A, or the object image of the paper airplane B in the photographed image data G3 in FIG. As shown in the state of contact with the father's person image P, when it is determined that the position of the specified destruction target image data (A1 or B) is in contact with another extracted image portion (step C7). (Yes)), it is determined whether or not the other extracted image portion determined to be in contact is the portion of the person image to which the transformation mark data 24Mw is added in the role transformation mark processing (step SB) (step C8). ).

  In this case, since the other extracted image part determined to be in contact is determined to be the person image part of the father P to which the transformation mark data 24Mw1 of the monster A is added (step C8 (Yes)), FIG. As shown in B), the tower destruction (after) mark data 15Ma ′ stored in association with the destruction data memory 15 is read out at the position corresponding to the destruction target image data of the beer bottle A1. The image data Gm2 with a mark is added and temporarily stored in the work data memory and displayed on the display unit 11 (step C9). Similarly, as shown in FIG. 19B, the destruction (rear) mark of the airplane stored in association with the destruction data memory 15 at the position corresponding to the destruction target image data of the paper airplane B. Data 15Mb ′ is read and added, and is temporarily stored in the working data memory as marked image data Gm3 and displayed on the display unit 11 (step C9).

  If it is determined that the digital camera 10 is currently photographing (step C10 (Yes)), the tower is stored in the destruction data memory 15 (see FIG. 5) in association with the destruction (medium). ) The audio data “Gusha” 15Sa of the image data 15Ga2 is read and output from the audio output unit 13 as shown in FIG. 18B (step C11). Similarly, the audio data “docan” 15Sb2 of the aircraft destruction (medium) image data 15Gb2 stored in association with the destruction data memory 15 (see FIG. 5) is read out, and FIG. Is output from the audio output unit 13 (step C11). In this way, the performer (such as father P) can easily and audibly recognize that the composite image processing of the destruction is appropriately performed by the voice data “Gusha”, “Dokan”, etc. of the data being destroyed. it can.

  Thereafter, in the step C2 in the break mark processing, for example, as shown by the beer bottle A1 in FIGS. 19 (A) to 25 (A) or the paper airplane B in FIGS. 20 (A) to 25 (A), When each of the object images A1 and B is recognized as being in an overturned state or an inverted state and is determined to be in a state after destruction (step C2 (Yes)), the destruction data memory 15 (see FIG. 5). ) And the destruction (after) mark data 15Ma ′ and 15Mb ′ of the tower and the plane stored in association with each other) are read out and added corresponding to the positions of the object images A1 and B, respectively. ) To 25B or 20B to 25B, the marked image data Gm3 to Gm9 or Gm4 to Gm9 is temporarily stored in the work data memory. Is displayed on the display unit 11 is (step C12).

Then, the process proceeds to the motion transformation mark process in FIG. 12 (step SD).
In this motion transformation mark processing, the transformation data memory 14 (FIG. 3) is used for the person images (P, K) and object images (A, B) in the series of photographed image data Gn extracted by the target image extraction processing. It is determined whether or not an image portion corresponding to the series of transformation motion model data 14P1a to 14P1c of XX Man A and the series of transformation motion model data 14P2a to 14P2c of the first XX man is stored in advance. (Step D1).

  Here, as shown in the photographed image data G4 to G6 in FIGS. 20A to 22A in sequence, the transformation data memory 14 for a series of child person images K extracted by the target image extraction process. If it is detected that the image portion is identical in gesture with a series of transformation motion model data 14P1a to 14P1c of XX man A stored in advance (see FIG. 3) (step D1 (Yes)), the transformation data memory 14 (see FIG. 3) and stored in correspondence with the position of the face image of the child image K of the child whose motion has been detected, and the transformation mark data 14Mu1 of OO Man A stored in correspondence with the data 14 is read out. Added, and temporarily stored in the work data memory as marked image data Gm6 to Gm9 as shown in FIGS. Is displayed on the 1 (step D2).

It should be noted that the transformation mark data 14Mu1 added corresponding to the position of the face image of the person image K detected in motion is either added over the face image or added in the vicinity of the face image. It may be.
If it is determined that the digital camera 10 is currently photographing (step D3 (Yes)), the XXX man A stored in association with the transformation data memory 14 (see FIG. 3) is stored. The audio data “Schwatch” 14Su1 is read out and output from the audio output unit 13 as shown in FIGS. 22B to 25B (step D4).

Then, the process proceeds to the special effect mark process in FIG. 13 (step SE).
In the special effect mark processing, the special effect data memory 23 (FIG. 5) is used for the human images (P, K) and object images (A, B) in the series of photographed image data Gn extracted by the target image extraction processing. 4), it is determined whether or not an image portion matching the series of transformation motion model data 23P1a to 23P1c of special effect A and the series of transformation motion model data 23P2a to 23P2c of special effect B stored in advance is detected (see FIG. 4). Step E1).

  Here, as shown in the photographed image data G7 to G9 in FIG. 23A to FIG. 25A in sequence, the special effect data memory for the series of child person images K extracted by the target image extraction process. 23 (see FIG. 4), if it is detected that the image portion has the same gesture as the series of transformation motion model data 23P1a to 23P1c of special effect A stored in advance (step E1 (Yes)), the special effect data The special effect mark data 23Mu1 of the special effect A stored in association with the memory 23 (see FIG. 4) is read out, and a predetermined position (for example, 3 minutes from the top of the image) of the child image K of the motion detected. Corresponding to the person image (father) P to which the transformation mark data 24Mw1 of the monster A is added. Figure 25 is temporarily stored in a working data memory as marked photographic image data Gm9 as shown in (B) is displayed on the display unit 11 (step E2).

Whether the special effect mark data 24Mu1 added corresponding to a predetermined position of the person image K detected in motion is added to the person image K or in the vicinity of the person image K. Either may be sufficient. As a result, the mother who is photographing can easily recognize that the special effect processing is appropriately executed by the computer and the composite image to which the special effect image is added is created as a result of the child K performing the special effect motion. be able to.
If it is determined that the digital camera 10 is currently photographing (step E3 (Yes)), the special effect A stored in association with the special effect data memory 23 (see FIG. 4) is stored. The audio data “BibiBibi” 23Su1 is read and output from the audio output unit 13 as shown in FIG. 25B (step E4). As described above, the special effect processing is performed properly on the computer by outputting the sound data “Bibi Bibi” 23Su1 of the special effect A from the digital camera, but the performer (child, father) can recognize the sound by sound. Also, the performer can perform with a realistic feeling by using the sound data of the special effect.

  Then, as shown in FIG. 17B to FIG. 25B, the continuous photographed image data Gm1 to Gm9 with marks generated on the work data memory in accordance with the processing of the steps SA to SE are marked. It is stored and saved in the attached photographed image data memory 26 (step S2).

  When it is determined that the shooting input of the series of shot image data G1 to G9 from the image shooting unit 21 has been completed (step S3 (Yes)), the series of shot images generated and stored in the marked shot image data memory 26. It is determined whether or not to proceed to the composite image generation process based on the marked image data Gm1 to Gm9 (step S4).

  Here, when it is determined that the process proceeds to a composite image generation process in response to a user operation for generating a composite image on the input unit 12 or automatically upon completion of acquisition of the series of captured image data (step S4). (Yes)), the series of marked photographed image data Gm1 to Gm9 is read from the marked photographed image data memory 26 to the working memory, and the process proceeds to the transformation mark composition process in FIG. 14 (step SF).

  In this transformation mark composition process, first, the marked photographed image data Gm1 to Gm9 read from the marked photographed image data memory 26 to the working memory is stored in advance in the transformation character data memory 24 (see FIG. 6). It is determined whether or not an image to which the transformation mark data 14mu stored in advance is included in the transformation mark data 24Mw being transformed or the transformation data memory 14 (see FIG. 3) (step F1).

  Here, as shown in FIGS. 17B to 25B, an image to which the transformation mark data 24Mw1 of the monster A is added, and as shown in FIGS. 22B to 25B, If it is determined that there is an image to which the transformation mark data 14Mu1 of Man A is added (step F1 (Yes)), the person image P in the vicinity of the mark to which the transformation mark data 24Mw1 and 14Mu1 are added correspondingly. , K are specified from the block position information obtained as additional information of the captured image data Gn by the target image extraction process (step SA) (step F2).

  Then, the identified person images P and K are subjected to motion capture processing, and the transformation (1) image corresponding to the transformation mark data 24Mw1 of the monster A stored in the transformation role data memory 24 (see FIG. 6). Based on the transformation image data 14Gu1 corresponding to the transformation mark data 14Mu1 of the XX man A stored in the data 24Gw1 and the transformation data memory 14 (see FIG. 3), FIG. 17 (C) to FIG. 25 (C). The transformation image data 24Gw1 of the monster A in the same pose (posture) as the person image (father) P is the same as the person image (child) K as shown in FIGS. 22 (C) to 25 (C). The transformed image data 14Gu1 of the pose (posture) XX man A is replaced with the preset background image BG and synthesized, and synthesized image data GG ~GG9, GG6~GG9 is generated (step F3).

At this time, as shown by the arrow x in FIG. 22C, the transformation image data 14Gu1 of the XX man A having the same pose (posture) as the person image (child) K is enlarged and replaced and synthesized.
Then, the voice data “Gao” 24Sw1 of the monster A stored in association with the transformation character data memory 24 (see FIG. 6) is read out and shown in FIGS. 17 (C) to 25 (C). Is stored in association with the synthesized images GG1 to GG9 generated as described above and stored in association with the transformation data memory 14 (see FIG. 3). Are stored in association with the synthesized images GG6 to GG9 generated as shown in FIGS. 22C to 25C (step F4).

Then, the process proceeds to the destruction mark composition process in FIG. 15 (step SG).
In this destruction mark composition process, first, the marked photographed image data Gm1 to Gm9 read from the marked photographed image data memory 26 to the working memory are stored in advance in the destruction data memory 15 (see FIG. 5). It is determined whether or not an image to which the destruction (front) mark data 15M and the destruction (rear) mark data 15M ′ are added is included (step G1).

  Here, as shown in FIGS. 17B to 25B, an image to which the tower destruction (front / rear) mark data 15Ma / 15Ma ′ is added, and FIGS. 18B to 25B. ), When it is determined that there is an image to which the airplane destruction (front / rear) mark data 15Mb / 15Mb ′ is added (step G1 (Yes)), the respective destruction (front / rear) marks The position of destruction target image data (beer bottle) A1 (paper airplane) B in the vicinity of the mark to which data 15Ma / 15Ma 'and 15Mb / 15Mb' are added is taken by the target image extraction process (step SA). It is specified from the position information of the block obtained as additional information of the data Gn (step G2).

And the position of the specified destruction target image data (beer bottle) A1, (paper airplane) B is in contact with the image portion of another target image extraction block extracted by the target image extraction process (step SA). Is determined (step G3).
Here, for example, in the destruction target image data (beer bottle) A1 in the marked photographed image data Gm1 in FIG. 17B, or in the destruction target image data (paper airplane) B in the marked photographed image data Gm2 in FIG. 18B. As shown, when it is determined that the position of the specified destruction target image data (A1 or B) is not in contact with another extracted image portion (step G3 (No)), the specified Whether or not the image data to be destroyed of the beer bottle A1 or the paper airplane B is in a state after destruction is determined by image recognition of whether or not each object image (A1, B) is in a fallen state or a reversed state. (Step G4).

  Here, for example, as shown by the beer bottle A1 in FIG. 17B or the paper airplane B in FIG. 18B, each of the object images A1 and B is image-recognized unless it is in the overturned state or the inverted state. If it is determined that it is not in the state (step G4 (No)), the image data to be destroyed of the specified beer bottle A1 or paper plane B is shown in FIG. 17C as the destruction data memory 15 (FIG. 5), the destruction (previous) image data 15Ga1 corresponding to the tower destruction (previous) mark data 15Ma stored in the tower, or the destruction data memory 15 (see FIG. 5) as shown in FIG. On the background image BG previously set in the destruction (previous) image data 15Gb1 corresponding to the destruction (previous) mark data 15Mb of the airplane stored in FIG. Synthesized been replaced come, the composite image data GG1, GG2 is generated (step G5).

Then, the pre-destruction voice data “Bune” 15Sb1 of the airplane stored in association with the destruction data memory 15 (see FIG. 5) is read out and generated as shown in FIG. 18C. The image is stored in association with the image GG2 (step G6).
On the other hand, for example, the destruction target image data (beer bottle) A1 and the person image (father) P in the marked image data Gm2 in FIG. 18B, or the destruction in the marked photographed image data Gm3 in FIG. 19B. As shown in the contact state between the target image data (paper airplane) B and the person image (father) P, the position of the specified destruction target image data (A1 and B) is the other extracted image part in operation. If it is determined that contact has been made (step G3 (Yes)), it is further determined whether or not the image portion with the transformation mark has been contacted. (Step G7). When contact with the image portion with the transformation mark is detected, the destruction target image data of the beer bottle A1 or the paper airplane B specified as the destruction target image is the destruction data memory as shown in FIG. 15 (see FIG. 5), the destruction (medium) image data 15Ga2 corresponding to the destruction (after) mark data 15Ma ′ stored in the tower 15 (see FIG. 5), or the destruction data memory 15 (see FIG. 19C). (See FIG. 5), the destruction (medium) image data 15Gb2 corresponding to the destruction (rear) mark data 15Mb 'of the airplane stored in FIG. 5 is respectively replaced with the preset background image BG and synthesized. Image data GG2 and GG3 are generated (step G8).
In step G7, it is determined whether or not the image portion with the transformation mark is touched. The transformation mark is stored in the transformation data storage means (14 in FIG. 3) and the gesture model data. When a person image with a matching face is detected and a transformation mark is attached to the person image (SD in FIG. 8), a person image with a matching face image stored in the face transformation data storage means is detected and the person image is detected. There is a case where a transformation mark is put on (SB in FIG. 8). In this way, the destruction mark is attached only in accordance with the contact with the extracted image part with the transformation mark, thereby eliminating the case where the moving image part and the destruction target image happen to overlap on the screen. The destruction image can be appropriately generated only for the contact with the person with the transformation mark.

  Then, the destruction sound data “Gusha” 15Sa of the tower stored in association with the destruction data memory 15 (see FIG. 5) is read, and the composite image generated as shown in FIG. It is stored in association with GG2, and the destruction sound data “Docan” 15Sb2 of the airplane is read out and stored in association with the synthesized image GG3 generated as shown in 19 (C) (step G9). ).

  On the other hand, for example, the destruction target image data (beer bottle) A1 in the marked photographed image data Gm3 to Gm9 in FIGS. 19B to 25B, or the marked photographed images in FIGS. 20B to 25B. As shown in the destruction target image data (paper airplane) B in the data Gm4 to Gm9, it is determined that the position of the specified destruction target image data (A1 or B) is not in contact with other extracted image portions. Even if it is a case (step G3 (No)), when each of the destruction target image data A1 and B is image-recognized as being in the overturned state or the inverted state and is judged to be in the state after the destruction (step G4 (Yes)), the destruction target image data of the specified beer bottle A1 or paper plane B is the destruction data memory 15 (FIG. 5) as shown in FIGS. 19 (B) to 25 (B). The destruction (rear) image data 15Ga3 corresponding to the destruction (rear) mark data 15Ma 'of the tower stored in (b)) or the destruction data as shown in FIGS. 20 (B) to 25 (B). The destruction (rear) image data 15Gb3 corresponding to the destruction (rear) mark data 15Mb 'of the airplane stored in the memory 15 (see FIG. 5) is replaced with the preset background image BG and synthesized. Then, composite image data GG3 to GG9 or GG4 to GG9 are generated (step G10).

Then, the process proceeds to the special effect mark composition process in FIG. 16 (step SH).
In this special effect mark composition processing, first, the marked image data Gm1 to Gm9 read from the marked image data memory 26 to the working memory is stored in advance in the special effect data memory 23 (see FIG. 4). It is determined whether or not an image to which the stored special effect mark data 23Mu is added is included (step H1).

  If it is determined that there is no image Gm9 to which the special effect mark data 23Mu1 for the special effect A is added as shown in FIG. 25 (B) (step H1 (No)), the process is terminated and exists. If it is determined (step H1 (Yes)), the position of the person image (child) K in the vicinity of the mark to which the special effect mark data 23Mu1 is associated is added to the target image extraction process (step SA). Thus, the position information of the block obtained as additional information of the photographed image data Gn is specified (step H2).

  Then, the special effect image data 23Gu1 stored in association with the special effect mark data 23Mu1 in the special effect data memory 23 (see FIG. 4) is read out, and the specified person image (child) K is predetermined. Corresponding to the position (for example, one third of the position from the top of the image), the image is added and synthesized in the direction of the person image (father) P to which the transformation mark data 24Mw1 of the monster A is added, and the combined image data GG9 is obtained. Is generated (step H3).

Furthermore, the special effect image data 23Gu1 additionally synthesized at a predetermined position of the person image (child) K is transformed into a monster A whose explosion effect image part 23Gu1 ′ at the tip thereof corresponds to the person image (father) P ( 1) The image data 24Gw1 is extended to a position that reaches the image data 24Gw1, and is additionally combined to generate combined image data GG9 (step H4).
Then, the audio data “Bibi Bibi” 23Su1 of the special effect A stored in association with the special effect data memory 23 (see FIG. 4) is read out and generated as shown in FIG. 25 (C). The image is stored in association with the composite image GG9 (step H5).

Then, as shown in FIGS. 17C to 25C, a series of synthesized image data GG1 to GG9 generated on the work data memory in accordance with the processing of the steps SF to SH is obtained as a synthesized image data memory. 27 and stored (step S6).
A series of synthesized image data (including associated audio data) GG1 to GG9 stored in the synthesized image data memory 27 is displayed as shown in FIGS. 17 (C) to 25 (C). Reproduced and displayed by the unit 11 and reproduced and output by the audio output unit 13 (step S7). When the composite image output device is configured as the server device 30 (see FIG. 7), it is on the communication network N that is the transmission source of the series of captured image data G1 to G9 received in step S1. For the communication terminal A (digital camera) 10 ′ and the communication terminal B (PC) 40, a series of synthesized image data (including associated audio data) GG1 to GG9 stored in the synthesized image data memory 27. The read response is transmitted (step S7).

  Therefore, according to the composite image output device using the digital camera 10 or the server device 30 having the above configuration, when a series of captured image data Gn captured by the image capturing unit 21 is input, the captured image data Gn includes the captured image data Gn. A person image (P, K) is extracted, and the presence / absence of a person image (K) having a preset motion (gesture) is detected. When a person image (K) having a preset motion (gesture) included in the captured image data Gn is detected, it is associated with the model data 14P of the set motion and stored in the special effect data memory 23. Based on the stored special effect data 23Gu1 of XX man A's special move, for example, it is generated and output as composite image data GGn with special effect data 14Gu1 added in the vicinity of the person image (K) of the photographed image data Gn. Therefore, the person who becomes the subject at the time of the video shooting simply performs a special effect motion (gesture) for an anime hero or the like to perform a special technique or attack, and in the case of the special effect motion, in the vicinity of the person image of the subject In addition, a composite image to which a special effect image is added can be easily generated.

Therefore, for a human image taken with a moving image, a composite image that is a series of scenes can be easily created by adding a special effect image to a position corresponding to the target person according to the will of the target person.
As a result, for example, if video shooting is performed with a child acting as a hero in a user family, a special effect-like synthesized moving image GGn can be easily created, and the generated synthesized moving image GGn is output and displayed on the display unit 11. You can get a new level of fun by watching it or by displaying it on an external monitor.

  In addition, according to the composite image output device using the digital camera 10 or the server device 30 configured as described above, a preset motion (gesture) of the person image (K) that is the subject is detected and corresponding to this, for example, When a synthesized image GG is generated by adding the special effect image data 23Gu1 of Man A to the vicinity of the person image (K), the audio data “Guide image data 23Gu1 stored in the transformation data memory 23 is stored. Since “BibiBibi” 23Su1 is stored and output in association with the composite image GG, it is possible to easily and more realistically create a special effect-like composite moving image GGn.

  Further, according to the composite image output device using the digital camera 10 having the above-described configuration, the preset motion (gesture) human image (gesture) of the captured image data Gn displayed in real time on the display unit 11 during the moving image capturing ( When K) is detected, the marked image obtained by adding transformation mark data 23Mu1 indicating that the person image (K) has been transformed into, for example, the special effect image data 23Gu1 of XX Man A is added. The image data Gmn is displayed on the display unit 11 and the audio data “Bibi Bibi” 23Su1 of the transformation image data 23Gu1 is also output from the audio output unit 13, so that, for example, the family during the shooting operation is a person image (child) of the subject. K can easily recognize that special effects such as Special Moves have been added as intended. Flip even in in the middle are the subject itself (child K), it can be easily recognized by the speech that it is special effects Special Move etc. itself intended.

  In the embodiment, the special effect motion model data 23P for detecting a preset motion (gesture) of the person image (K) extracted as the target image from the photographed image data Gn, and the transformation mark data corresponding thereto. 23Mu and transformation image data 23Gu are stored in advance in the transformation data memory 23, but the special effect image data (a special technique, a special technique image from the server device 30 on the communication network N (in this case, the data providing server of the service site)). It is also possible to select a desired type of attack, etc.), download it, and newly store it as the special effect data memory 23 so as to function in the same manner as described above.

  Note that each processing method by the composite image output apparatus described in the embodiment, that is, the composite image output process shown in the flowchart of FIG. 8, the target image extraction process accompanying the composite image output process shown in the flowchart of FIG. The process change mark process accompanying the composite image output process shown in the flowchart of FIG. 10, the destruction mark process accompanying the composite image output process shown in the flowchart of FIG. 11, and the motion change mark accompanying the composite image output process shown in the flowchart of FIG. Processing, special effect mark processing accompanying the composite image output processing shown in the flowchart of FIG. 13, transformation mark composition processing accompanying the composite image output processing shown in the flowchart of FIG. 14, and the composite image output processing shown in the flowchart of FIG. Destruction mark compositing process with accompanying, shown in flowchart of FIG. Each method such as the special effect mark synthesis process associated with the synthesized image output process is a program that can be executed by a computer, such as a memory card (ROM card, RAM card, etc.), magnetic disk (floppy disk, hard disk, etc.). It can be stored and distributed in an external storage medium 18 (33) such as an optical disk (CD-ROM, DVD, etc.), semiconductor memory or the like. Various computer terminals storing the transformation data 14 (32B), the special effect data 23 (32C), the destruction data 15 (32D), and the transformation character data 24 (32E) in the memory 17 (32) are stored in the external storage medium. The program stored in 18 (33) is read into the program memory 22 (32A), and the operation is controlled by the read program, so that the transformed image from the moving image captured image data Gn described in the above embodiment is used. The function of creating and outputting the composite image GGn can be realized, and the same processing can be executed by the method described above.

  Further, program data for realizing each of the above methods can be transmitted on a communication network (Internet) N as a program code form, and a computer terminal (program server) connected to the communication network (Internet) N ) From the above-mentioned program data, and a function of creating and outputting a composite image GGn using a transformed image from the above-described moving image captured image data Gn can be realized.

  Note that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention at the stage of implementation. Further, each of the embodiments includes inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in each embodiment or some constituent elements are combined in different forms, the problems described in the column of the problem to be solved by the invention If the effects described in the column “Effects of the Invention” can be obtained, a configuration in which these constituent requirements are deleted or combined can be extracted as an invention.

The conceptual diagram which shows the imaging | photography state of a moving image using the digital camera 10 which concerns on embodiment of the synthesized image output apparatus of this invention, and the production | generation state of a synthesized image. The block diagram which shows the structure of the electronic circuit of the said digital camera 10 (communication terminal A). The figure which shows the content of the transformation data previously stored in the transformation data memory 14 of the said digital camera 10. FIG. The figure which shows the content of the special effect data previously stored in the special effect data memory 23 of the said digital camera 10. FIG. The figure which shows the content of the destruction data previously stored in the destruction data memory 15 of the said digital camera 10. FIG. The figure which shows the content of the transformation character data memorize | stored in the transformation character data memory 24 of the said digital camera. The block diagram which shows the structure of the electronic circuit of the server apparatus 30 which concerns on embodiment of the synthesized image output apparatus of this invention. 6 is a flowchart showing an overall process of a composite image output process by the composite image output apparatus. The flowchart which shows the target image extraction process which extracts the target image of the person (P, M, K) and the object (A, B, C) which exist in the picked-up image (G) with the said composite image output process. The flowchart which shows the role transformation mark process which adds a transformation mark (24Mw) to the person image (P, M) by which the role registration was carried out with the said composite image output process. The flowchart which shows the destruction mark process which adds a destruction mark (15M, 15M ') to the object image (A, B, C) extracted with the said synthetic image output process, and outputs corresponding sound (15S). The flowchart which shows the motion transformation mark process which adds a transformation mark (14Mu) to the person image (K) by which transformation motion (14P) was detected with the said composite image output process, and outputs corresponding sound (14Su). The flowchart which shows the special effect mark process which adds a special effect mark (23Gu) to the person image (K) from which the special effect motion (23P) was detected with the said composite image output process, and outputs corresponding sound (23Su). The person image (P) (K) to which the transformation mark (24Mw) (14Mu) is added in accordance with the synthetic image output processing is replaced with the transformation image (24Gw) ((14Gu) and synthesized, and the corresponding voice (24Sw) (14Su). FIG. 6 is a flowchart showing a transformation mark composition process for setting the output. Destructive mark composition for replacing the object image (A, B) to which the destructive mark (15M, 15M ′) is added with the destructive image (15G) and setting the output of the corresponding voice (15S) along with the synthetic image output processing The flowchart which shows a process. A special effect mark synthesis process for additionally synthesizing the special effect image 23Gu at a predetermined position of the person image (K) to which the special effect mark (23Gu) is added in accordance with the synthetic image output process and setting the output of the corresponding sound (23Su). The flowchart which shows. FIG. 2 is a diagram (part 1) illustrating image processing states based on a moving image captured image (G) associated with the composite image output process in order of typical frame advance, in which FIG. (A) illustrates a captured image G1; FIG. 4B is a diagram showing a marked-captured photographed image Gm1, and FIG. 4C is a diagram showing a composite image GG1. FIG. 2 is a diagram (part 2) illustrating image processing states based on the moving image captured image (G) associated with the composite image output processing in order of typical frame advance, in which (A) is a diagram illustrating the captured image G2. FIG. 5B is a diagram showing a marked photographed image Gm2, and FIG. 5C is a diagram showing a composite image GG2. FIG. 3 is a diagram (part 3) illustrating image processing states based on the moving image captured image (G) associated with the composite image output process in order of typical frame advance, in which FIG. (A) is a diagram illustrating the captured image G3; FIG. 4B is a view showing a marked-captured photographed image Gm3, and FIG. 4C is a view showing a composite image GG3. FIG. 4 is a diagram (part 4) illustrating image processing states based on the moving image captured image (G) associated with the composite image output process sequentially as representative frame advance, and FIG. (A) is a diagram illustrating the captured image G4; FIG. 4B is a diagram showing a marked-captured photographed image Gm4, and FIG. 4C is a diagram showing a composite image GG4. FIG. 5 is a diagram (part 5) illustrating image processing states based on the moving image captured image (G) associated with the composite image output processing in order of typical frame advance, in which FIG. (A) illustrates the captured image G5; FIG. 5B is a view showing a marked photographic image Gm5, and FIG. 4C is a view showing a composite image GG5. FIG. 6 is a diagram (part 6) illustrating image processing states based on the moving image captured image (G) associated with the composite image output process sequentially as a representative frame advance, in which FIG. (A) illustrates the captured image G6; FIG. 5B is a view showing a marked photographed image Gm6, and FIG. 4C is a view showing a composite image GG6. FIG. 7 is a diagram (part 7) illustrating image processing states based on the moving image captured image (G) associated with the composite image output process sequentially as a representative frame advance, in which FIG. (A) illustrates the captured image G7; FIG. 4B is a diagram showing a marked photographic image Gm7, and FIG. 4C is a diagram showing a composite image GG7. FIG. 8 is a diagram (No. 8) illustrating image processing states based on the moving image captured image (G) accompanying the composite image output processing in order of typical frame advance, in which FIG. (A) is a diagram illustrating the captured image G8; FIG. 4B is a diagram showing a marked photographic image Gm8, and FIG. 4C is a diagram showing a composite image GG8. FIG. 9 is a diagram (No. 9) illustrating image processing states based on the moving image captured image (G) associated with the composite image output processing sequentially as representative frame advance, in which FIG. (A) is a diagram illustrating the captured image G9; FIG. 4B is a diagram showing a marked photographic image Gm9, and FIG. 4C is a diagram showing a composite image GG9.

Explanation of symbols

10 ... Digital camera (communication terminal A)
11 (37) ... Display unit 12 (36) ... Input unit 13 ... Audio output unit 14 (32B) ... Transformation data memory 14P ... Transformation motion model data 14Gu ... Transformation image data 14Su ... Transformation voice data 14Mu ... Transformation mark data 15 ( 32D) ... Destruction data memory 15T ... Destruction target image data 15G ... Destruction image data 15S ... Destruction audio data 15M ... Destruction (front) mark data 15M '... Destruction (rear) mark data 16 (31) ... CPU
17 (32) ... Memory 18 (33) ... External recording medium 19 (34) ... Recording medium reading section 20 (35) ... Transmission control section 21 ... Image photographing section 22 (32A) ... Program memory 23 (32C) ... Special effects Data memory 23P ... Special effect motion model data 23Gu ... Special effect image data 23Su ... Special effect sound data 23Mu ... Special effect mark data 24 (32E) ... Transformation role data memory 24T ... Transformation target (face) image data 24Gw ... Transform image data 24 Sw ... Transformation voice data 24 Mw ... Transformation mark data 24H ... Transformation condition data 25 (32F) ... Captured image data memory 26 (32G) ... Marked captured image data memory 27 (32H) ... Composite image data memory 30 ... Server device 40 ... Communication terminal B (PC)
N ... Communication network (Internet)
BG ... Background image data Gn ... Shooting image data Gmn ... Marked shooting image data GGn ... Composite image data P ... Person image (father)
K ... person image (child)
M ... person image (mother)
A1 ... Object image (beer bottle)
A2 ... Object image (plastic bottle)
B ... Object image (paper airplane)
C ... Object image (tissue box)

Claims (5)

Special effect data storage means for storing pre-set gesture model data, special effect image data, and mark data indicating the special effect in association with each other;
Captured image acquisition means for acquiring a series of captured image data taken by a movie,
A person image extracting means for extracting a person image from a series of photographed image data obtained by the photographed image obtaining means;
Special effect gesture detection means for detecting a person image whose model gesture stored in the special effect data storage means matches the preset gesture from the person images extracted by the person image extraction means;
The special and effects gesture detecting means captured image data when the human image of preset gesture is detected by a composite image generating means for generating a composite image data obtained by combining the image data of the special effects,
A composite image output means for outputting a series of composite image data generated by the composite image generation means;
Equipped with a,
The composite image generation means includes
Marked composite image generation means for generating photographed image data when a pre-set gesture human image is detected by the special effect gesture detection means and combined image data with a mark obtained by combining the mark data indicating the special effect When,
Effect synthesized image generating means for generating synthesized image data obtained by synthesizing the image data of the special effect on the marked synthesized image data generated by the marked synthesized image generating means;
With
The composite image output means includes
A marked composite image output means for outputting the marked composite image data generated by the marked composite image generation means;
Effect composite image output means for outputting composite image data obtained by combining the special effect image data generated by the effect composite image generation means;
Composite image output apparatus characterized by comprising a. The composite image output apparatus according to claim 1, wherein the effect composite image generation unit synthesizes special effect image data at the position of the detected person image. The composite image output apparatus according to claim 1, wherein the effect composite image generation unit synthesizes special effect image data in one direction from the position of the detected person image. The special effect data storage means stores preset gesture model data and special effect image data and audio data in association with each other,
The composite image generating means, wherein the preset shooting when the gestures of the person image is detected the image data by the special effect gesture detecting means, to generate a synthetic image data obtained by combining the image data of the special effects, Generating composite image data in which the sound data of the special effect is associated;
The composite image output device according to any one of claims 1 to 3 . Computer
Special effect data storage means for storing pre-set gesture model data, special effect image data, and mark data indicating the special effect in association with each other;
Captured image acquisition means for acquiring a series of captured image data taken with a movie,
A person image extracting means for extracting a person image from a series of photographed image data acquired by the photographed image acquiring means;
Special effect gesture detection means for detecting a person image whose model gesture stored in the special effect data storage means matches the preset gesture from the person images extracted by the person image extraction means,
The special and effects gesture captured image data when the human image of preset gesture is detected by the detecting means, the composite image generation means for generating a composite image data obtained by combining the image data of the special effects,
A composite image output means for outputting a series of composite image data generated by the composite image generation means;
To function as,
The composite image generation means includes
Marked composite image generation means for generating photographed image data when a pre-set gesture human image is detected by the special effect gesture detection means and combined image data with a mark obtained by combining the mark data indicating the special effect When,
Effect synthesized image generating means for generating synthesized image data obtained by synthesizing the image data of the special effect on the marked synthesized image data generated by the marked synthesized image generating means;
Function to have
The composite image output means includes
A marked composite image output means for outputting the marked composite image data generated by the marked composite image generation means;
Effect composite image output means for outputting composite image data obtained by combining the special effect image data generated by the effect composite image generation means;
To work with
A computer-readable composite image output processing program characterized by the above.

Images