JP2011228913A - Image processing apparatus, reply image generation system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which processes an interesting image with a specified image, a reply image generation system and a program.SOLUTION: The image processing apparatus comprises: a first image acquisition part for acquiring a first image; and an image composition part for combining a second image acquired with the first image acquired by the first image acquisition part while acquiring the second image. The image composition part detects a first region which satisfies a predetermined reference on the first image acquired by the first image acquisition part, superimposes the second image on a region in the vicinity of the first region detected, and generates a third image by combining the first image with the second image.

Description

  The present invention relates to an image processing device, a reply image generation system, and a program.

  For example, Patent Literature 1 discloses a display control method for displaying image data transmitted from a communication terminal on a display device, and the current state of the user of the communication terminal estimated from position information of the communication terminal that transmitted the image data. A display control method for outputting the image data together with image data to a display device is disclosed.

  Patent Document 2 discloses a display control method for displaying image data transmitted from a mobile phone or a personal computer on a DPF, which is received without requiring a DPF user to perform a display method setting operation. There has been disclosed a display control method for setting a display method based on setting information and automatically replenishing digital photo data that is deficient when the setting is changed.

JP 2009-181210 A JP 2009-151369 A

  The techniques described in Patent Document 1 and Patent Document 2 have not been able to process a predetermined image into an interesting image.

  The present invention has been made in view of the above points, and an object thereof is to provide an image processing apparatus, a reply image generation system, and a program that process a predetermined image into an interesting image.

An image processing apparatus according to a first aspect of the present invention includes:
First image acquisition means for acquiring a first image;
An image combining unit that acquires a second image and combines the acquired second image with the first image acquired by the first image acquiring unit;
The image synthesizing unit detects a first region satisfying a predetermined criterion in the first image acquired by the first image acquiring unit, and superimposes the second image on a region near the detected first region. Then, a third image obtained by synthesizing the first image and the second image is generated.

The reply image generation system according to the second aspect of the present invention is:
A reply image generation system for generating a reply image for the first image based on a first image transmitted from a first terminal to a second terminal,
A photographed image obtained by photographing the front of the display surface of the second terminal displaying the first image is obtained, a predetermined image is extracted based on the obtained photographed image, and a first image representing the extracted prescribed image is obtained. Image generating means for generating two images;
Image synthesizing means for synthesizing the first image and the second image generated by the image generating means,
The image composition means detects a first region that satisfies a predetermined criterion in the first image, superimposes the second image on a region in the vicinity of the detected first region, and the first image and the second image A third image synthesized with the image is generated.

A program according to the third aspect of the present invention is:
Computer
First image acquisition means for acquiring a first image;
Acquiring a second image, and causing the acquired second image to function as an image combining unit that combines the first image acquired by the first image acquiring unit;
The image synthesizing unit detects a first region satisfying a predetermined criterion in the first image acquired by the first image acquiring unit, and superimposes the second image on a region near the detected first region. Then, a third image obtained by synthesizing the first image and the second image is generated.

  According to the image processing device, the reply image generation system, and the program according to the present invention, a predetermined image can be processed into an interesting image.

It is a block diagram which shows the structural example of the reply image generation system which concerns on one Embodiment of this invention. It is a figure which shows the structure of the 1st terminal which concerns on one Embodiment of this invention. It is a figure which shows the structure at the time of using CPU etc. of the 1st terminal which concerns on one Embodiment of this invention. It is a figure which shows the structure of the 2nd terminal which concerns on 1 embodiment of this invention. It is a figure which shows the structure at the time of using CPU etc. of the 2nd terminal which concerns on 1 embodiment of this invention. It is a figure which shows the structure of the server which concerns on one Embodiment of this invention. It is a figure which shows the structure at the time of using CPU etc. of the server which concerns on one Embodiment of this invention. It is a figure which shows the flowchart of the 1st process of the image transmission / reception process which the reply image generation system which concerns on one Embodiment of this invention performs. It is a figure explaining the data structure of the data table for image transmission / reception in one Embodiment of this invention. It is a figure which shows the flowchart of the 2nd process of the image transmission / reception process which the reply image generation system which concerns on one Embodiment of this invention performs. It is a figure which shows an example of a 2nd image. It is a figure which shows the flowchart of the 3rd image generation process of the image transmission / reception process C which the server which concerns on 1 embodiment of this invention performs. (A) is a figure for demonstrating the reference | standard face in a 1st image. (B) is a figure for demonstrating the face area which should not superimpose the reference | standard face and 2nd image in a 1st image. It is a figure which shows the flowchart of the position / size determination process of a 2nd image in the 3rd image generation process which the server which concerns on one Embodiment of this invention performs. It is a figure which shows the relationship between a collection area | region and a reference | standard face in a 1st image in the case of the gesture of "stroking", arrangement | positioning of a collection | recovery area | region, etc., (a) is when a reference | standard face is large, (b) is It is a figure which shows the case where a reference | standard face is small. FIG. 10 is a diagram showing the relationship between the collection area and the reference face in the first image, the arrangement of the collection area, and the like in the case of the gesture “pointing finger”, where (a) is (b) when the reference face is large ) Is a diagram showing a case where the reference face is small. FIG. 10 is a diagram illustrating a relationship between a collection area and a reference face in the first image, an arrangement of the collection area, and the like in the case of a gesture of “applause” or “waving hands” (waving hands with both hands); FIG. 4A is a diagram illustrating a case where the reference face is large, and FIG. It is a figure which shows the relationship between a collection area | region and a reference | standard face in a 1st image in the case of the gesture of "enveloping", arrangement | positioning of a collection | recovery area | region, etc., (a) when a reference | standard face is large, (b) is It is a figure which shows the case where a reference | standard face is small. It is a figure for demonstrating the production | generation of the 3rd image in the reply image production | generation system which concerns on one Embodiment of this invention.

  An embodiment according to the present invention will be described with reference to the drawings. In addition, this invention is not limited by description of the following description and drawing. It goes without saying that changes (including deletion of components) can be added to the contents of the following description and drawings. Further, in the following description, in order to facilitate understanding of the present invention, descriptions of known technical matters are omitted as appropriate.

  First, the configuration of a reply image generation system 1 according to this embodiment will be described with reference to FIG. The reply image generation system 1 includes a first terminal 100, a second terminal 200, and a server 300. The reply image generation system 1 is also a system that transmits and receives image data between the first terminal 100 and the second terminal 200.

  The first terminal 100, the second terminal 200, and the server 300 are connected to the network 900 and can communicate with each other. In the present embodiment, the reply image generation system 1 includes the first terminal 100, the second terminal 200, and the server 300, but the reply image generation system 1 includes the first terminal 100. And at least one of the second terminal 200 and the server 300 may be provided. The network 900 is the Internet, for example.

  In the present embodiment, the first terminal 100 and the second terminal 200 are digital photo frames. The first terminal 100 and the second terminal 200 may be other devices such as a personal computer and a mobile phone. The first terminal 100 and the second terminal 200 transmit and receive data such as image data via the server 300.

  The server 300 relays data such as image data transmitted and received by the first terminal 100 or the second terminal 200. The server 300 is configured by a predetermined computer.

  The first terminal 100, the second terminal 200, and the server 300 process image data, but the data format of the image data may be changed as appropriate during processing.

  Next, the configuration of the first terminal 100 will be described with reference to FIGS. 2 and 3.

  The first terminal 100 includes a control unit 110, an internal storage unit 120, an operation unit 130, a display unit 140, a communication unit 150, and an I / F unit 160.

  The operation unit 130 is configured by an input device such as a touch sensor, for example. The operation unit 130 outputs an operation signal corresponding to an operation input from the user under the control of the control unit 110 and supplies the operation signal to the control unit 110. The control unit 110 performs processing according to the supplied operation signal. As a result, the control unit 110 performs processing according to the user's operation content on the operation unit 130.

  The display unit 140 includes, for example, a display panel and a display controller. Data such as image data is supplied from the control unit 110 to the display controller. The display controller drives the display panel based on the supplied image data, and displays an image on the display panel. In this way, the control unit 101 displays a desired image on the display unit 140.

  Note that the operation unit 130 and the display unit 140 may be configured by a touch panel or the like. In this case, the input device built in the touch panel constitutes the operation unit 130.

  The communication unit 150 is configured by an appropriate communication device such as a modem, for example. The communication unit 150 is connected to the network 900 by wire or wireless. The control unit 110 transmits and receives data via the communication unit 150 and the network 900.

  The I / F unit 160 is an interface (I / F) with the outside. The I / F unit 160 is configured by a reading / writing device including a media controller or the like, for example. The control unit 110 reads / writes data from / to the storage medium 190 via the I / F unit 160.

  The storage medium 190 is configured by, for example, a memory card including a flash memory or the like, and is attached to the first terminal 100. The storage medium 190 stores image data and the like.

  The control unit 110 includes an image transmission unit 110a, a display control unit 110b, and an image reception unit 110c. The control unit 110 (the image transmission unit 110a, the display control unit 110b, and the image reception unit 110c) includes, for example, a CPU (Central Processing Unit) 111 and a RAM (Random Access Memory) 112. Process A is performed. The internal storage unit 120 includes, for example, a flash memory 121, and appropriately stores data used by the control unit 110.

  The CPU 111 controls the entire first terminal 100 according to the program recorded in the flash memory 121 and further uses data and the like recorded in the flash memory 121, and the image transmission / reception processing A described later performed by the control unit 110. I do. Programs, data, and the like recorded in the flash memory 121 may be read once into a RAM (Random Access Memory) 121.

  At least a part of the processing performed by the CPU 111 may be executed by various dedicated circuits. That is, at least a part of the control unit 110 may be configured by various dedicated circuits.

  The RAM 112 stores data used by the CPU 111, data generated by the CPU 111, data supplied to the CPU 111, and the like.

  The flash memory 121 stores a program, various data used by the CPU 111, and the like. The internal storage unit 120 may include other storage devices such as a ROM (Read Only Memory) and a hard disk, and the other storage devices may store programs, various data used by the CPU 111, and the like.

  The internal storage unit 120 and the storage medium 190 constitute a storage unit that stores data used by the control unit 110, and the control unit 110 acquires data from the storage unit and performs predetermined processing. Note that the storage unit only needs to be capable of reading and writing data by the first terminal 100, and may be outside the first terminal 100.

  Next, the configuration of the second terminal 200 will be described with reference to FIGS. 4 and 5.

  The second terminal 200 has a configuration in which a photographing unit 270 is further added to the configuration of the first terminal 100. Although the storage medium 290 is attached to the second terminal 200, the storage medium 290 can be the same as the storage medium 190.

  The second terminal 200 includes a control unit 210, an internal storage unit 220, an operation unit 230, a display unit 240, a communication unit 250, an I / F unit 260, and an imaging unit 270.

  The control unit 210 corresponds to the control unit 110. The control unit 210 includes an image transmission unit 210a, an image generation unit 210b, an image reception unit 210c, a display control unit 210d, and an imaging control unit 210e, and performs image transmission / reception processing B described later. The control unit 210 includes, for example, a CPU 211 and a RAM 212, and the CPU 211 performs an image transmission / reception process B described later according to a program recorded in the flash memory 221. Since the other description about the control part 210 is the same as the description about the control part 110, description is abbreviate | omitted.

  The imaging unit 270 includes an imaging device, an imaging lens group, an imaging drive unit, an AFE (Analog Front End), and a DSP (Digital Signal Processor). These operate under the control of the control unit 110 (imaging control unit 210e). The imaging unit 270 images the front of the second terminal 200 (the normal direction of the display screen of the display unit 240 and the direction on the user side).

  The imaging device is configured by an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Under the control of the control unit 210, the image sensor captures an image of light incident through the imaging lens group, performs photoelectric conversion for each pixel, and uses a signal based on the charge for each pixel generated by the photoelectric conversion as an imaging signal. Supply to AFE.

  The imaging lens group includes a lens group such as a focus lens and a zoom lens. The imaging drive unit is a mechanism that controls the driving of the imaging lens group, the control of the amount of light incident on the imaging device, the control of the time during which light is applied to the imaging device (the control of the time that the light receiving element of the imaging device receives light) . The imaging lens group and the imaging drive unit operate under the control of the control unit 210, and the imaging element performs imaging.

  Under the control of the control unit 210, the AFE performs predetermined processing (hereinafter referred to as processing A) on the imaging signal supplied from the imaging device, and represents digital data (original image data) representing a captured image captured by the imaging device. ) Is generated and output. The process A is a process including an OB (Optical Black) clamp process, a CDS (Correlated Double Sampling) process, an AGC (Automatic Gain Control) process, an A / D (Analog / digital) process, and the like.

  The DSP performs various processes (hereinafter referred to as process B) on the original image data supplied from the AFE, and generates image data in a YUV (luminance / color difference) format or the like. The process B includes a process for improving the image quality, such as contour enhancement, auto white balance, and auto iris. Thus, image data representing a captured image captured by the image sensor, that is, captured image data captured by the capturing unit 270 is generated.

  With the above configuration, the imaging unit 270 captures captured images such as a person and a background in the imaging direction under the control of the control unit 210 (imaging control unit 210e), and represents the captured image. A signal is generated, original image data representing the captured image is generated based on the generated imaging signal, captured image data is generated based on the generated original image data, and is supplied to the control unit 210. Note that the imaging unit 270 includes a lamp or the like provided in the housing of the second terminal 200, and this lamp is lit under the control of the control unit 210.

  Note that the processing performed by the DSP may be performed by the control unit 210 (image generation unit 210b). In this case, the control unit 210 receives the image original data from the imaging unit 270 (AFE), and generates image data (YUV format or the like) by performing the same process as the process B on the received image original data. To do. Thereby, the control unit 210 acquires image data representing the captured image captured by the imaging unit 270.

  The internal storage unit 220, the operation unit 230, the display unit 240, the communication unit 250, and the I / F unit 260 are the internal storage unit 120, the operation unit 130, and the first terminal 100, respectively. Since it corresponds to the display unit 140, the communication unit 150, and the I / F unit 160 and is substantially the same, detailed description thereof is omitted.

  The server 300 includes a control unit 310, an internal storage unit 320, an operation unit 330, a display unit 340, and a communication unit 350. These are substantially the same as each part of the first terminal 100. The internal storage unit 320, the operation unit 330, the display unit 340, and the communication unit 350 are the internal storage unit 120, the operation unit 130, the display unit 140, and the communication unit of the first terminal 100, respectively. 150. The operation unit 330 and the display unit 340 are used for managing the server 300. The server 300 may not include the operation unit 330 and the display unit 340.

  The operation unit 330 may be configured by an input device such as a keyboard, for example. The control unit 310 includes an image transmission unit 310a, a display control unit 310b, and an image reception unit 310c, and performs image transmission / reception processing C described later. Note that the control unit 310 includes, for example, a CPU 311 and a RAM 312, and the CPU 311 performs processing performed by the image transmission unit 310 a, the display control unit 310 b, and the image reception unit 310 c, which will be described later, according to a program recorded on the hard disk 321. A transmission / reception process C is performed. The internal storage unit 320 is constituted by, for example, a hard disk 321 and appropriately stores data used by the control unit 310. The other description is the same as the description in the first terminal 100, and the detailed description is omitted.

  Next, operations of the reply image generation system 1 (operations of the first terminal 100, the second terminal 200, and the server 300) will be described. The reply image generation system 1 performs image transmission / reception processing.

  First, a first process in the image transmission / reception process (a process of transmitting a first image described later from the first terminal 100 to the second terminal 200) will be described with reference to FIG. In this process, the first terminal 100, the second terminal 200, and the server 300 perform the first process in the image transmission / reception processes A to C, respectively. The first terminal 100 starts the first process of the image transmission / reception process A when, for example, the first user operates the operation unit 130 and makes a request to start the first process. In addition, it is assumed that the second terminal 200 and the server 300 are on standby until image data is received.

  When the first process of the image transmission / reception process A is started, the display control unit 110b displays on the display unit 140 images of a list of images (for example, a list of thumbnails of images) represented by the image data recorded in the storage medium 190. (Step S101). Accordingly, the display control unit 110b prompts the first user to specify an image to be transmitted. Here, the first user is a user of the first terminal 100.

  Next, the display control unit 110b determines whether an image is selected (step S102). When the first user operates the operation unit 130 while viewing the list image on the display unit 140 and selects an image to be transmitted from the list, the display control unit 110b receives an operation signal corresponding to the selection from the operation unit 130. Supplied. The display control unit 110b determines that an image has been selected by supplying the operation signal (step S102; YES). At this time, the display control unit 110b specifies image data representing the image selected by the first user, which is specified by the operation signal, as image data to be transmitted (step S103).

  On the other hand, when the operation signal is not supplied from the operation unit 130 to the display control unit 110b, the display control unit 110b determines that no image is selected (step S102; NO), and performs the process of step S102 again. In this way, the display control unit 110b waits until an image is selected.

  It is assumed that the first user selects an image (here, a digital photograph) showing a human face as an image to be transmitted. Hereinafter, this image and image data representing this image are referred to as a first image and a first image data, respectively.

  Next, the display control unit 110b displays an image of a list of transmission destinations specified by the address data recorded in the storage medium 190 on the display unit 140 (step S104). Accordingly, the display control unit 110b prompts the first user to specify the transmission destination of the first image.

  Next, the display control unit 110b determines whether a transmission destination has been selected (step S105). When the first user operates the operation unit 130 while viewing the list image on the display unit 140 and selects a transmission destination from the list, an operation signal corresponding to the selection is supplied from the operation unit 130 to the display control unit 110b. Is done. The display control unit 110b determines that the transmission destination has been selected by supplying the operation signal (step S105; YES). At this time, the display control unit 110b specifies the address data specifying the transmission destination selected by the first user, which is specified by the operation signal, as the transmission destination address data to be transmitted (step S106). Note that the transmission destination may be input directly by the user by operating the operation unit 130. Here, it is assumed that the second terminal 200 is designated as the transmission destination.

  On the other hand, when the operation signal is not supplied from the operation unit 130 to the display control unit 110b, the display control unit 110b determines that the transmission destination is not selected (step S105; NO), and performs the process of step S105 again. In this way, the display control unit 110b waits until a transmission destination is selected.

  When the display control unit 110b specifies the address data of the transmission destination and the image data to be transmitted (this specification is performed by holding the image data in the RAM 112, for example), an image for selecting whether to perform a transmission instruction. (For example, an image including a character string “Do you want to send?”, A “YES” button, and a “NO” button) is displayed on the display unit 140 (step S107), and the first user To select whether to transmit the first image. Then, the display control unit 110b determines whether a transmission instruction has been given (step S108).

  The first user looks at the image displayed on the display unit 140 to select whether to give a transmission instruction, and operates the operation unit 130 to select whether to give a transmission instruction. An operation signal corresponding to this selection is supplied from the operation unit 130 to the display control unit 110b.

  When the first user inputs that the transmission instruction is not performed via the operation unit 140, the display control unit 110b displays an operation signal corresponding to the operation (for example, an operation indicating that the “NO” button is selected). Signal). When this operation signal is supplied, the display control unit 110b determines that there is no transmission instruction (step S108; NO), and performs the process of step S101 again. As a result, the image selection and the like are performed again by the first user.

  When the first user inputs a transmission instruction via the operation unit 140, the display control unit 110b receives an operation signal corresponding to this operation (for example, an operation signal indicating that the “YES” button has been selected). ) Is supplied. When this operation signal is supplied, the display control unit 110b determines that a transmission instruction has been given (step S108; YES).

  When it is determined that the display control unit 110b has been instructed to operate (step S108; YES), the image transmission unit 110a uses the image data specified by the display control unit 110b and the address data specified by the display control unit 110b ( Hereinafter, it is referred to as transmission destination address data) and address data indicating its own address (address data indicating the address of the first terminal 100, which is stored in the internal storage unit 120 in advance). The address data is transmitted to the server 300 via the communication unit 150 and the network 900 together with the transmission source address data below (step S109).

  The image receiving unit 310c of the server 300 receives the image data, the transmission source address data, and the transmission destination address data transmitted from the first terminal 100 via the communication unit 350 (Step S110). The image receiving unit 310c records the received image data in the internal storage unit 320 (step S111). At this time, the image receiving unit 310c attaches an image ID (information for identifying the image data, such as a serial number) to the image data, and associates the received image with the data indicating the image ID. Data, destination address data, and source address data are recorded in the internal storage unit 320. The image data is recorded in association with the image ID via data that specifies the recording location of the image data (recording location in the internal storage unit 320). That is, data indicating the recording location of the image data is recorded in association with the image ID. The image data may be recorded in association with the direct image ID.

  FIG. 9 shows an example of the data structure of the image ID data, the data indicating the recording location of the image data, the transmission destination address data, and the transmission source address data recorded in the internal storage unit 320. The internal storage unit 320 stores a data table including the above data (hereinafter referred to as an image transmission / reception data table). FIG. 9 shows the data structure of this data table. This data table identifies an “image ID” for identifying image data, an “image data recording location” indicating the image data recording location, and a terminal (in this case, the second terminal 200, etc.) that transmits the image. Data (corresponding to each other) indicating the “transmission destination address” and the “transmission source address” specifying the terminal (here, the first terminal 100 or the like) that transmitted the image. Data structure.

  When the image receiving unit 310c records the first image data or the like in the internal storage unit 320, the image transmitting unit 310a transmits the destination address data (the first address data) corresponding to the first image data via the communication unit 350 and the network 900. The first image data and the image receiver 310c together with the first image data are sent to the terminal (here, the second terminal 200) indicated by the destination address data received by the image receiver 310a together with the one image data. The received transmission source address data and data indicating the image ID are transmitted (step S112).

  The image receiving unit 210c of the second terminal 200 receives the first image data, the transmission source address data, and the data indicating the image ID transmitted from the server 200 via the communication unit 350 (step S113). When the image receiving unit 210c receives the data, the display control unit 210d displays, for example, an image including a mail mark indicating that the first image data has been received on the display unit 240, and receives the first image data. The second user is notified of this (step S114). The image receiving unit 210c records the source address data, the first image data, and the data indicating the image ID in the internal storage unit 220 in association with each other. The second user is a user of the second terminal 200.

  Next, a second process in the image transmission / reception process (a process of returning a third image to be described later to the first terminal 100) will be described with reference to FIG. In this process, the first terminal 100, the second terminal 200, and the server 300 perform the second process in the image transmission / reception processes A to C, respectively. The second terminal 200 displays a request (for example, one of a plurality of images represented by a plurality of received image data) by the first user operating the operation unit 230 to start the second process. The second process of the image transmission / reception process B is started. Further, it is assumed that the first terminal 100 and the server 300 have started the second processing of the image transmission / reception processes A and C and are waiting until the image data is received.

  First, the display control unit 210d acquires first image data (for example, first image data designated by the user) from, for example, the internal storage unit 220, and supplies the acquired first image data to the display unit 240. Then, the first image represented by the first image data is displayed on the display unit 240 (step S201).

  When the display control unit 210d displays the first image on the display unit 240, the image generation unit 210b controls the shooting unit 270 to take a picture in front of the display unit 240 (step S202). Note that when shooting is started, the shooting control unit 210e turns on the lamp and the like included in the imaging unit 270, for example, and notifies the second user that shooting is started.

  Note that photographing is performed for a predetermined period (for example, 30 seconds). Further, the shooting control unit 210e performs shooting by shooting still images at predetermined time intervals (for example, every second). The imaging control unit 210e sequentially records captured image data (image data representing the still image) sequentially supplied by the imaging in the internal storage unit 220.

  The imaging control unit 210e supplies the captured image data sequentially supplied to the display unit 240, and sequentially displays the still images represented by the captured image data toward a predetermined display area (for example, toward the display screen) of the display screen of the display unit 240. It may be displayed in the upper right corner). As a result, the second user can check his / her own operation on the screen. In addition, the image generation unit 210b described later performs detection of a hand region described later in real time, and when the hand region is detected, the imaging control unit 210e turns on a predetermined lamp to notify the second user. It may be fed back that the hand region has been detected.

  During the above photographing, the second user in front of the display unit 240 looks at the first image displayed on the display unit 240 and performs a predetermined gesture with his hand in front of the display unit 240. This gesture is a gesture or hand imitation performed by the user, and includes operations such as “applause”, “stroking”, “pointing fingers”, “waving hands”, “wrapping”, and the like. In addition, the second user performs the gesture on a desired face captured in the first image. In addition, it is assumed that the second user performs only one of the operations. The photographing control unit 210e uses the photographing unit 270 to photograph the hand gesture.

  When the shooting control unit 210e performs shooting for the predetermined period, that is, when shooting is finished, the image generation unit 210b generates a second image representing the hand region (step S203). The image generation unit 210b generates a second image based on the captured image (captured image data recorded in the internal storage unit 220) captured by the capture control unit 210e.

  For example, the image generation unit 210b performs image analysis on the captured image, and detects the hand (second user's hand) region by dividing the captured image into a second user's hand region and other regions. This hand region detection can use image recognition technology for operating home appliances with hand gestures. The detection of the hand region is performed, for example, by template matching using a hand template image. It is assumed that the template image data is stored in advance in the internal storage unit 220. In addition, this template image is an image of a hand when stroking the head (corresponding to the gesture of “stroking” above), an image of a hand when pointing a finger (corresponding to the gesture of “pointing a finger” above), and applauding Images of hands (corresponding to the gesture of “applause” above), images of hands when waving (corresponding to the gesture of “waving hands” above), images of hands when wrapping the face (above “envelop”) The image corresponding to the gesture of the hand that the second user will perform. If the hand region cannot be detected in one template image, other template images are used in sequence. If the hand area cannot be detected for all template images, the hand area is not detected in the captured image.

  When a plurality of hand regions are detected in the captured image, the image generation unit 210b detects, for example, the largest hand region. This is because the hand is considered to be in a position close to the photographing unit 270 and is likely to be the hand of the second user.

  When detecting the hand region, the image generation unit 210b divides the hand region into other hand regions and generates an image in which only the hand region is extracted.

  When the image generation unit 210b generates the image, the image generation unit 210b inverts the image (inverted horizontally). Since the photographing unit 270 is photographing the front of the display unit 240, that is, since the hand is photographed from the display unit 240, the hand viewed by the second user is reverse to the actually photographed hand. Because it becomes. Next, the inverted image is painted black and converted into a shadow-like image (hand region image).

  Note that the image generation unit 210b may set an image representing a finger or the like as a hand region image by putting a predetermined line in the shadow picture. In addition, the hand region image does not necessarily have to be painted black.

  An example of an image generated by the above method will be described with reference to FIG. As shown in FIG. 11, the hand region image 30 is an image like a shadow picture represented by a bitmap, for example, in which the hand region is painted black. In addition, the rectangle in FIG. 11 shows the outer edge of the original captured image. The hand area image data representing the hand area image appropriately includes a position indicating the position of the hand area (position of the center of the hand area), direction (direction of the long direction of the hand area), and the like in the original captured image. Direction data shall be included. That is, according to the hand area image data, the position of the hand area image in the original captured image can be known.

  The series of processing described above is performed in time series for each captured image. In the template matching described above, the template image used when the hand region can be detected for the first time is used for template matching in the subsequent captured image. A second image is constituted by a set of hand region images. That is, by the series of processes described above, a second image that is an image including an image of a human hand and includes a plurality of hand region images is generated. The image data representing the second image (hereinafter referred to as second image data) is, for example, a set of a plurality of hand region image data representing each hand region image (arranged in time series) constituting the second image.

  When the image generation unit 210b generates the second image, that is, when the second image data is generated, what kind of gesture is the hand of each still image constituting the second image (the type of gesture) ) To specify data (hand-type data). This can be specified by, for example, the type of template image used when the hand region can be detected in the template matching described above.

  The image transmission unit 210a uses the second image data generated by the image generation unit 210b as data specifying the image ID of the first image currently displayed on the display unit 240 (for example, recorded in the internal storage unit 220). The image transmission unit 210a obtains this and the hand type data are transmitted to the server 300 via the network 900 and the communication unit 250 (step S204). At this time, the image transmission unit 210a stores the source image data and the second terminal 200 itself together with the second image data recorded in the internal storage unit 220 in association with the data specifying the image ID. The set address data (the transmission destination address data) is transmitted.

  The image receiving unit 310c of the server 300 transmits the second image data, the hand type data, the data specifying the image ID, the transmission source address data, and the transmission transmitted by the image transmission unit 210a via the communication unit 350. The destination address data is received (step S205). When the image receiving unit 310c receives the data, the image composition 310b refers to the image transmission / reception data table stored in the internal storage unit 320, and based on the data specifying the image ID received by the image receiving unit 310c. First image data corresponding to the data is specified, the first image data is acquired from the internal storage unit 320, and the first image represented by the acquired first image data and the second image data received by the image receiving unit 310c are The second image to be represented is synthesized to generate a third image (step S206).

  Details of the process in step S206 will be described with reference to FIG.

  The image composition unit 310b performs face recognition based on the acquired first image data (step S301). Face recognition is performed by, for example, template matching using a template image of a face image.

  The image composition unit 310b identifies the recognized face as a reference face (step S302). In particular, when there is one recognized face, the image composition unit 310b identifies that face as a reference face (a face to be used in image composition described later). In addition, when there are a plurality of recognized faces, the image composition unit 310b specifies the face having the largest size as the reference image. This is because it is considered that the largest face is the main subject in the first image and the face that the second user or the like sees. For example, as shown in FIG. 13A, when there are a plurality of recognized faces, the largest face is specified as the reference face. In FIG. 13A, a face surrounded by a dotted line is a reference face. Such a face is considered to be highly likely to be an object on which the second user performs a gesture (for example, a face of an object stroking the head). For this reason, this method improves the accuracy of the reference face. Further, according to such a method, the reference face can be easily specified.

  When there are a plurality of recognized faces, for example, the image composition unit 310b uses a pre-registered face (for example, a face of a second user or the first user, a face of a friend, family, acquaintance, etc.) as a reference. It may be specified as a face. In this case, image data of a template image of a predetermined person's face is recorded for each image ID data, and the image composition unit 310b is a template image represented by corresponding image data for each image ID. Perform template matching. It is considered that such a face is likely to be a target for the second user to perform a gesture. For this reason, this method improves the accuracy of specifying the reference face.

  In addition, when there are a plurality of recognized faces, for example, the image composition unit 310b adds, to the first image, the first image based on data specifying the position and direction in the captured image included in the hand region image data. Overlay the hand area image in the position and direction (assuming that the image size of the captured image is the same as the image size of the first image), and specify the face closest to this hand area image as the reference face For example, if the hand area image is a hand when pointing with a finger, the face closest to this part on the extension line of the tapered part in the hand area is used as the reference face. You may specify. Further, for example, if the hand region image is an enveloping hand, the face between the two hand regions may be specified as the reference face. The face specified by the method of specifying the reference face according to the shape of the hand area (type of gesture) of the hand area image is an object that the second user performs a gesture (for example, an object that strokes the head). The face of the subject pointed to by the finger, the face of the subject to be wrapped with the hand). For this reason, in these methods, the accuracy of specifying the reference face is improved. The shape of the hand region (the type of gesture) is specified by the hand type data.

  In addition, when there are a plurality of recognized faces, for example, the image composition unit 310b sets the face facing the front as the reference face, or sets the face in the center area of the first image as the reference face. Also good. In any case, it is considered that the face identified by these methods is highly likely to be a target for the second user to perform a gesture. For this reason, in these methods, the accuracy of specifying the reference face is improved.

  Next, the image compositing unit 310b should superimpose the second image on a face area (area where the difference in area between the two is equal to or less than a threshold) in the first image that is the same size as the face specified as the reference face. Is identified as a non-face region (step S303). Like the reference face, such a face is also an important person's face for the second user or the like. If the second image is overlapped with a predetermined area or more as described later, the important person's face is hidden. Because it ends up. For example, as shown in FIG. 13B, a face 12 or 14 having the same size as the reference face 11 or 13 (face within a dotted line) is specified as a face region (face within a dotted line).

  Note that the method used for specifying the reference face (in particular, a method other than the method for specifying the reference face in accordance with the type of gesture) may be used for specifying the face area here.

  Further, this step S303 is a process performed when a plurality of faces are recognized, and when only one face is recognized, the image composition unit 310b skips this process. Further, the image composition unit 310b proceeds to the next process even when there is no face other than the reference face having the same size as the face specified as the reference face.

  Next, the image composition unit 310b performs processing for superimposing (combining) the second image on the first image, but before that, the position in the first image and the size of the second image on which the second image is superimposed are determined. Processing to determine is performed (step S304).

  This process will be described in detail with reference to FIG.

  First, the image composition unit 310b calculates the length of the reference face image in the long direction (for example, the length in the long direction of the rectangle surrounding the reference face image) in the first image (step S401). Further, the image composition unit 310b calculates the average length in the long direction of the hand region image included in the second image (for example, the average length in the long direction of the rectangle surrounding the hand region image) (step S402). ). Even if the hand of the hand region image is held. The length of this image in the longitudinal direction is calculated.

  Next, the image composition unit 310b obtains a size coefficient (step S403). From the two values calculated above, the image composition unit 310b calculates the size coefficient = A × (length of the reference face image in the longitudinal direction) / (average length in the longitudinal direction of the rectangle surrounding the hand region image). ) A is a predetermined constant, and is set in advance according to how many times the length of the reference face image in the longitudinal direction is the average of the length in the longitudinal direction of the rectangle surrounding the hand region image.

  The image composition unit 310b resizes each hand area image included in the second image by multiplying the size (for example, vertical and horizontal dimensions) of the hand area image by a size coefficient (step S404). As a result, the average length of the hand region image is A times that of the reference face.

  Next, the image compositing unit 310b superimposes the resized hand region images with the same center, and collects all the hand region images (for example, a sum set obtained by superimposing the hand region images). Region) is obtained, and the obtained aggregate region is superimposed on the first image (step S405). At this time, the position on the first image (the center position of the gathering area) is specified according to the type of gesture specified by the hand type data, for example.

  For example, in the case of the “stroking” gesture, since the hand is usually positioned on the reference face, the position above the center position of the reference face and at the first distance is set as the center position of the collection area.

  For example, in the case of the “pointing finger” gesture, since the finger usually points to the face, the second position is shifted from the center position of the reference face in a direction opposite to the tapering direction in the gathering region. This position is set as the center position of the collection area.

  For example, in the case of the above “applause” gesture, both hands do not have to overlap the reference face. Therefore, a position shifted by a third distance from the center position of the reference face in an arbitrary direction is set as the center position of the collection area.

  For example, in the case of the “waving hand” gesture, both hands do not have to overlap the reference face. Therefore, a position shifted by a fourth distance from the center position of the reference face in an arbitrary direction is set as the center position of the collection area. .

  For example, in the case of the “envelop” gesture, both hands wrap the reference face, so the positions shifted by the fifth distance in the left-right direction from the center position of the reference face are set as the center positions of both hands of the collective region.

  Next, the image composition unit 310b determines whether or not the overlapped aggregate region overlaps the specified face and the face region that should not be superimposed with the second image (step S406). If there is no overlap (step S406; NO), the process proceeds to step S408. On the other hand, when the overlapped aggregate region overlaps with a predetermined area or more (step S406; YES), it is moved to a position where it does not overlap with the predetermined area at the minimum distance (step S407). In the process of step S407, for example, the image generation unit 310b shifts the collection area from the original position vertically, horizontally, 1 pixel, 2 pixels, 3 pixels,... In the first image. And, for example, a position where the set area does not overlap with the reference area and the face area where the second image should not be superimposed on the face area where the second image should not overlap with the minimum area at the minimum distance. And move the collection area. Then, the center position of the collective region in the first image is specified in advance. If the gathering area is composed of two separate areas for both hands, the two are moved independently as appropriate.

  Next, the image composition unit 310b determines whether or not the collective region superimposed on the first image protrudes outside the first image (step S408), and determines that it does not protrude (step S408; NO). ), The process of step S305 is performed. When it is determined that the image composition unit 310b has protruded (step S408; YES), the reduction ratio (the left and right reduction ratios are the same) is reduced so that the aggregate area is reduced and fits in the first image with the maximum size. .) The image composition unit 310b gradually increases the reduction ratio to sequentially reduce the collection area, and first specifies the reduction ratio at which the reduced collection area does not protrude the first image as the reduction ratio. . Note that the image generation unit 310b performs reduction around a predetermined point close to the reference face side when reducing the collective region. For this reason, the center position of the aggregated area after the reduction changes compared with that before the reduction. The image generation unit 310b also specifies the center position. As described above, the image generation unit 310b specifies the reduction ratio at which the reduced aggregate area does not protrude from the first image as the reduction ratio, and also specifies the position of the center of the aggregate area in the first image. (Step S409). Thereafter, the image generation unit 310b performs the process of step S305.

  An example of the relationship between the collective area and the reference face will be described with reference to FIGS. 15 to 18. Each set area is schematically represented by a simple figure such as a rectangle for convenience. Also, when there is one face (one reference face)

  FIG. 15 is a diagram in the case of the “stroking” gesture. As in (a), when the collection area (black rectangular portion) overlaps the reference face, the image composition unit 310b moves the collection area (see the dotted rectangle in the lower diagram), but after the movement The gathering area is reduced because it protrudes from the first image. As shown in (b), when the collective region (black rectangular portion) overlaps the reference face, the image composition unit 310b moves the collective region (see the lower diagram). In (b), the reference face is small and the gathering area is also small accordingly. For this reason, the collective area after the movement does not protrude from the first image.

  FIG. 16 is a diagram in the case of the gesture “pointing finger”. As in (a), when the set area (black polygonal portion) overlaps the reference face, the image composition unit 310b moves the set area (see the polygon in the lower diagram), but after the move The gathering area is reduced because it protrudes from the first image. As shown in (b), when the set area (black polygonal portion) overlaps the reference face, the image composition unit 310b moves the set area (see the lower diagram). In (b), the reference face is small and the gathering area is also small accordingly. For this reason, the collective area after the movement does not protrude from the first image.

  FIG. 17 is a diagram in the case of a gesture of “applause” or “waving hands” (waving hands with both hands). As in (a), when the collection area (black rectangular portion) overlaps the reference face, the image composition unit 310b moves the collection area (see the dotted rectangle in the lower diagram), but after the movement The gathering area is reduced because it protrudes from the first image. As shown in (b), when the collective region (black rectangular portion) overlaps the reference face, the image composition unit 310b moves the collective region (see the lower diagram). In (b), the reference face is small and the gathering area is also small accordingly. For this reason, the collective area after the movement does not protrude from the first image. Note that at this time, a single gathering area including both hands is used.

  FIG. 18 is a diagram in the case of the gesture of “wrapping”. As in (a), when the collective region (black rectangular portion) overlaps the reference face, the image composition unit 310b moves the collective region (see the lower diagram). As shown in (b), when the collective region (black rectangular portion) overlaps the reference face, the image composition unit 310b moves the collective region (see the lower diagram). In both (a) and (b), the gathered area after the movement does not protrude the first image. In the case of such a gesture, the collective region includes two regions that are separated from each other, and the image composition unit 310b performs movement, reduction, and the like for each region.

  Note that the above-described movement, reduction, and the like of the image and area are performed by, for example, coordinate conversion for each pixel.

  Next, the image composition unit 310b composes the first image with the second image to generate a third image (step S305). The image composition unit 310b generates a third image based on the first image data and the second image data. For example, the image composition unit 310b superimposes each hand region image represented by the second image data on the first image to generate a plurality of third images. That is, the number of third images generated here is the same as the number of hand region images. The image compositing unit 310b superimposes each hand region image generated in step S404 and multiplied by the size coefficient (reduced or enlarged) on the first image. Further, each hand region image is superimposed on the first image so that the center position of each hand region image becomes the center position of the collective region specified in Steps S405, S407, and S409. If the reduction ratio is specified, each hand area image is reduced at the specified reduction ratio, and each reduced hand area image is superimposed on the first image. As described above, the collective area does not overlap with the face area that should not be superimposed with the reference face and the second image more than a predetermined area, so that each hand area image that constitutes the collective area does not overlap with the face area more than a predetermined area. Become. In this way, the image composition unit 310b generates a plurality of third images obtained by compositing the second image (each hand region image constituting the first image) with the first image.

  Next, when the image composition unit 310b generates the third image, the image transmission unit 310a transmits the generated third image data to the transmission source address data recorded in the internal storage unit 320 (transmitted from the second terminal 200). The data is recorded to the transmission source (first terminal 100) indicated by the data (corresponding to the data specifying the image ID) (step S207). At this time, the image transmission unit 310a may transmit destination address data together with the third image data as appropriate. Thereby, in the first terminal 100, the first user can specify the reply source. The image composition unit 310b transmits the data via the communication unit 350 and the network 900.

  Next, when the image receiving unit 110c receives the third image data via the communication unit 150 (step S208), the display control unit 110b supplies the third image data to the display unit 140, and the third image data Is displayed on the display unit 140 (step S209). Here, since there are a plurality of third images, they are sequentially displayed on the display unit 140 in time series. Therefore, the display unit 140 displays the third image with a configuration such as a slide show or a moving image. For this reason, the movement of the hand of the second user is reproduced, and the image displayed on the display unit 140 is an interesting image.

  In this way, the third image obtained by superimposing the second user's gesture image (second image) on the first image is supplied to the first terminal 100 as a reply image of the first image and displayed. Will be.

  FIG. 19 shows an outline of the second process of the image transmission / reception process.

  When the first image (the image transmitted from the first terminal 100) is displayed on the display unit 140 of the second terminal 200 as illustrated in FIG. 19, the second user is, for example, a person included in the first image. A gesture such as stroking his face is performed in front of the display unit 240. The imaging unit 270 images the state. The second terminal generates a second image based on the captured image and transmits it to the server 300. The server 300 combines the second image and the first image, generates a combined third image, and transmits it to the first terminal 100. The first terminal 100 displays the received third image on the display unit 140. At this time, the position and size of the image of the second user's hand are changed according to the size of the subject (face) of the first image. For this reason, the hand 50 (shadow) is reduced in (a). Moreover, in (b), the hand 50 is not reduced.

  With the configuration as described above, when returning to the photograph (first image) sent via the network 900, the gesture of the hand of the responder (second user) is photographed by the imaging unit 270 of the second terminal 200. Then, a third image superimposed on the first image is generated as a moving picture (second image). This makes it possible to easily generate rich contents with movement without using an input device such as a keyboard and a mouse, and to reply easily and happily. This is particularly true when the first terminal 100 or the like is a digital photo frame. In addition, hand and face image recognition is used when generating the third image. By automatically adjusting and superimposing the extracted hand movement in consideration of the position and size of the face, a silhouette with an appropriate arrangement can be generated without the user specifying a fine position and size.

  An image display device such as a digital photo frame is not equipped with a rich input device such as a keyboard due to a request for design even if it becomes highly functional, and even if a photograph is received via the network 900, it is a message or moving content. It was difficult to reply. In this embodiment, a hand is photographed using the imaging 270, and a shadow picture used for reply is generated. Since the position and size for superimposing the shadow picture are automated by using the face recognition information, it is possible to generate a rich reply content with movement without a special input device other than the imaging unit 270. Further, since the movement of the hand is free, there are many variations as compared with a selective type image reply method such as a standard message or a stamp. In addition, the image capturing unit 270 only needs to be able to capture an image with a quality that allows the outline of the hand to be recognized. Therefore, the image capturing unit 270 can be used in common with the image capturing unit 270 that uses an image sensor with a low pixel number for QR code recognition.

  In the above description, since the second image representing the hand region, etc., is sent to the server instead of the captured image, the amount of data sent from the second terminal 200 to the server 300 is reduced. For this reason, even when the second terminal 200 is a low-speed and inexpensive network line and the uplink is thin, data can be transmitted at a sufficiently high speed. In addition, when there is a first image in the first terminal 100, if only the second image is sent and the third image is generated by the first terminal 100 at the time of reproduction, the first terminal 100 is transmitted from the server 300. The network load in the downstream direction can also be reduced.

  In the present embodiment, the server 300 (image processing apparatus) acquires the first image acquisition unit (here, the image reception unit 310a) that acquires (receives) the first image, the second image, and the second image by the above-described configuration. An image combining unit 310b that combines the acquired second image and the first image acquired by the image receiving unit 310a; Further, the image composition unit 310b detects a first region (here, a region of a face and including a reference face region) that satisfies a predetermined criterion in the first image, and is in the vicinity of the detected first region A second image is superimposed on the area of the first image, and a third image is generated by combining the first image and the second image.

  In the present embodiment, the reply image generation system 1 generates a reply image for the first image based on the first image that the first terminal 100 transmits to the second terminal 200 with the above configuration. In the system 1, a captured image obtained by capturing the front of the display surface of the second terminal 200 displaying the first image is acquired, a predetermined image is extracted based on the acquired captured image, and the extracted predetermined image An image generation unit 210b that generates a second image representing the image, and an image combination unit 310b that combines the first image and the second image generated by the image generation unit 210b. In addition, the image composition unit 310b detects a first region that satisfies a predetermined criterion in the first image, superimposes the second image on a region near the detected first region, and combines the first image and the second image. A synthesized third image is generated.

  With the configuration as described above, in the present embodiment, the second image is superimposed on a region in the vicinity of the first region that satisfies the predetermined criterion, so that the predetermined image can be processed into an interesting image.

  Further, with the above configuration, the image composition unit 310b adjusts the size of the second image according to the size of the first region, and superimposes the second image on a region near the first region. As a result, even if the second image is too large or too small with respect to the first region, the second image can be superimposed with an appropriate size, so that an interesting and accurate image can be generated. can do.

  In the above, the first image is an image showing a person's face, but may be another image. In the above, the second image is an image representing an image of a human hand. It may be an image. Furthermore, the first area is an area including a human face, but may be another area.

  In the above, the second image includes a plurality of still images, and the image composition unit 310b generates a plurality of third images in which the plurality of still images included in the second image are superimposed on the first image. . As a result, a moving third image is generated, and a more interesting image is generated.

  The image composition unit 310b superimposes the second image on the first image so as not to overlap the first region by more than a predetermined area (including the case where the second image does not overlap at all). The relationship with the image becomes clear, and a more interesting image is generated.

  Further, the first image is a transmission image transmitted from the first terminal 100 to the second terminal 200 via the network 300, and the third image is a reply to the transmission image displayed by the first terminal 100. It is an image. Thus, in the above, an interesting image can be generated for the reply image.

  In the above description, the third image is composed of a plurality of still images. However, the third image may be a so-called dynamic photo in which each hand region image is sequentially superimposed on one first image. The third image may be in the form of a slide show playlist. Thus, the format of the third image may be one image or any other format.

  Further, the second image may be a single still image. In this case, the above process is performed for one still image. The third image is also a still image.

  In addition, each device included in the first terminal 100, the second terminal 200, and the server 300 may be included in another device. For example, the server 300 may include the image generation unit 210b. In this case, the second terminal 200 transmits the captured image as it is to the server 300, and the image generation unit of the server 300 generates a third image. Similarly, the second terminal 200 may include the image composition unit 310b of the server 300. In this case, the second image is generated by the second terminal 200 and the third image is returned to the first terminal 100 via the server 300. Similarly, the first terminal 100 may include an image generation unit 210b and an image composition unit 310b. In this case, a captured image is supplied to the first terminal 100, and the first terminal 100 Two images and a third image are generated. In this manner, any device may be included in each unit. For this reason, depending on the configuration, the image processing apparatus may be the first terminal 100 or the second terminal 200. Moreover, said acquisition includes acquiring by producing | generating an image etc. other than acquiring an image etc. from a memory | storage part, acquiring an image etc. via a communication part.

  The program may cause the CPU 111 (CPU 211 or CPU 311) to perform image transmission / reception processing A (B or C) described later in cooperation with an OS (Operation System). In this case, the OS is also recorded in the flash memory 121 (flash memory 221 or hard disk 321). The program is recorded on a portable storage medium (for example, CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory)), and the first terminal 100 (second terminal). 200 or server 300) and may be recorded in the flash memory 121 (flash memory 221 or hard disk 321). The program may be supplied to the first terminal 100 (the second terminal 200 or the server 300) via the network 900 and recorded in the flash memory 121 (the flash memory 221 or the hard disk 321).

  A storage device such as the flash memory 121 (the flash memory 221 or the hard disk 321) or a portable storage medium in which the program is recorded becomes a computer-readable storage medium storing the program. The program may operate a device (computer) in which at least a part of the functions is realized by a dedicated circuit. That is, the first terminal 100 (the second terminal 200 or the server 300) may be anything as long as it performs an image transmission / reception process A (B or C), which will be described later, and the program may be such a first Any device that operates the terminal 100 (the second terminal 200 or the server 300) may be used.

  In the present embodiment, the second image is superimposed on the first image. However, if the second image is simply superimposed as it is, an appropriate third image cannot be obtained. For example, the hand, the terminal, and the size / position relationship between the imaging unit captures the hand too much, and if the second image is superimposed as it is, most of the first image is hidden. If a camera such as a large-screen TV is mounted with a camera that can cover the entire range in front of the screen, the hand becomes too small.

  Therefore, as a rule for arrangement of the second image, a predetermined face in the first image is not hidden when the second image is arranged. When there are a plurality of faces, the largest face is used as a reference face, and all faces having the same size are arranged so as not to hide them. If the face is large, the shadow hand is also enlarged.

  There are various second images to be added to the first image. However, if the situation is limited to the “reply to received image” situation as in this embodiment, the types can be limited to some extent. Until now, many of the images presented in albums (including analog photos) have been photographed, such as travel photos taken together and event photos of grandchildren living apart. For this reason, it is estimated that the image used for the reply is overwhelmingly often used when a photograph in which an important character appears larger than the background character is used. In addition, in such cases, the hand does not move to hide important characters, and often moves to target people, such as stroking, pointing, waving, wrapping. It is estimated (although it is difficult to hide people intentionally in order to make interesting news photos, it is difficult to hide the person who responded by thanks to photos of family, friends, and acquaintances). Therefore, by superimposing the second image on the first image according to the above rules, the position of the person who is the target of the hand movement in the position avoiding the main subject person in the photograph (a group with a large face) A hand of a size proportional to the face can be superimposed.

  Note that, as shown in FIG. 17A, the position may be greatly shifted as in the case of a large face of “applause and shake hands”. However, as described above in the above rules, since a gesture that hides the face of an important character is not preferred in a reply situation, such a case is unlikely to occur. In addition, it is desirable to shift in this way even when the user mistakenly makes a gesture that largely hides the face.

DESCRIPTION OF SYMBOLS 1 ... Reply image generation system, 100 ... 1st terminal, 110 ... Control part, 110a ... Image transmission part, 110b ... Display control part, 110c ... Image reception part, 111 ... CPU, 112 ... RAM, 120 ... internal storage unit, 121 ... flash memory, 130 ... operation unit, 140 ... display unit, 150 ... communication unit, 160 ... -I / F unit, storage medium ... 190, 200 ... second terminal, 210 ... control unit, 210a ... image transmission unit, 110b ... image generation unit, 110c ... image Receiving unit, 110d ... display control unit, 110e ... photographing control unit, 211 ... CPU, 212 ... RAM, 220 ... internal storage unit, 221 ... flash memory, 230 ... Operation unit, 240... Display unit, 250. Communication unit 260 ... I / F unit, 270 ... shooting unit, storage medium ... 290, 300 ... server, 310 ... control unit, 310a ... image transmission unit, 310b ..Image composition unit, 310c ... Image receiving unit, 311 ... CPU, 312 ... RAM, 320 ... Internal storage unit, 321 ... Hard disk, 330 ... Operation unit, 340 ... -Display unit, 350 ... Communication unit, 900 ... Server

Claims (8)

First image acquisition means for acquiring a first image;
An image combining unit that acquires a second image and combines the acquired second image with the first image acquired by the first image acquiring unit;
The image synthesizing unit detects a first region satisfying a predetermined criterion in the first image acquired by the first image acquiring unit, and superimposes the second image on a region near the detected first region. Generating a third image obtained by synthesizing the first image and the second image;
An image processing apparatus. The image synthesizing unit adjusts the size of the second image according to the size of the first region, and superimposes the second image on a region near the first region;
The image processing apparatus according to claim 1. The first image is an image showing a person's face,
The second image is an image representing an image of a human hand,
The first area is an area including the face of the person.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The second image includes a plurality of still images,
The image synthesizing unit generates one or more third images obtained by superimposing the plurality of still images included in the second image on the first image;
The image processing apparatus according to claim 1. The image composition means superimposes the second image on the first image so as not to overlap the first region by a predetermined area;
The image processing apparatus according to claim 1. The first image is a transmission image transmitted from the first terminal to the second terminal via the network,
The third image is a reply image to the transmission image displayed by the first terminal.
The image processing apparatus according to claim 1. A reply image generation system for generating a reply image for the first image based on a first image transmitted from a first terminal to a second terminal,
A photographed image obtained by photographing the front of the display surface of the second terminal displaying the first image is obtained, a predetermined image is extracted based on the obtained photographed image, and a first image representing the extracted prescribed image is obtained. Image generating means for generating two images;
Image synthesizing means for synthesizing the first image and the second image generated by the image generating means,
The image composition means detects a first region that satisfies a predetermined criterion in the first image, superimposes the second image on a region in the vicinity of the detected first region, and the first image and the second image Generating a third image synthesized with the image;
A reply image generation system characterized by that. Computer
First image acquisition means for acquiring a first image;
Acquiring a second image, and causing the acquired second image to function as an image combining unit that combines the first image acquired by the first image acquiring unit;
The image synthesizing unit detects a first region satisfying a predetermined criterion in the first image acquired by the first image acquiring unit, and superimposes the second image on a region near the detected first region. Generating a third image obtained by synthesizing the first image and the second image;
A program characterized by that.

Images