JP2003244425A - Method and apparatus for registering on fancy pattern of transmission image and method and apparatus for reproducing the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and an apparatus for registering the fancy pattern of a transmission image and a method and an apparatus for reproducing the same in which a number of patterns for improving the amusement property of a display image on an image display part during communications mutually between communication devices such as portable telephones can be easily registered to be freely used by persons in communication and the modification pattern dedicated to the person concerned can be abundantly prepared. <P>SOLUTION: While the first communication device provided with an image display part for the person in communication and the second communication device provided with an image display part for a communicating party mutually communicate via a repeater station, contents attached pith the modification pattern for modifying the transmission image to be displayed on the image display part in the communication device of the communicating party are read, the modification pattern is copied from the read contents and the copied modification pattern is registered availably for the person in communication or for the first communication device. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a transmission image displayed on an image display unit, for example, a communication partner's image taken during communication between communication devices having an image display unit such as a mobile phone in order to improve entertainment. A modification pattern for modifying an image, a person image, or the like, for example, image information such as a modified image displayed by being combined with a transmission image, audio information, and the like, and reproducing these.
For example, a modification pattern including a program for synthesizing a modification image or the like, or a program for processing a transmission image,
BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for registering a modification pattern of a transmission image, which is registered in a large number, for example, in a database of a relay station of a communication network connecting communication devices or in a memory of the communication device so that a correspondent can freely use it. Further, the present invention reproduces the modification pattern registered in this way on the image display unit of the communication device or the image display device of the relay station, for example, by modifying the modification pattern of the transmission image to be combined with the modification of the transmission image for reproduction. The present invention relates to a reproducing method and device.

[0002]

2. Description of the Related Art Conventionally, as image supply means for providing a viewing image, a photograph (print) for reproducing a still image, a movie (projector and screen) for reproducing a moving image, a television (TV: cathode ray tube (CRT)), etc. But in recent years
Various image display means such as a liquid crystal display (LCD), a plasma display, and still image display means such as electronic paper have been developed as image display means for moving images and still images such as television and computer monitors. Also,
At present, the various image display means described above can display not only images such as moving images and still images but also character information. In recent years, these image display means, especially LCDs, are not only used as a monitor of a television or a computer but also can be of various sizes, such as cameras, telephones, various communication devices, It has also come to be used as an image display unit for home appliances, other household products, and industrial products.

In addition, image capturing devices such as video cameras, digital cameras and digital video movies, and communication devices such as fixed-line phones, portable phones, videophones, mobile phones, mobile phones, and mobile videophones use image and character information. At the same time, voice can be captured, and the captured image and character information can be displayed on the image display unit described above. In particular, some mobile phones having an image display unit have a photographing function, and can display the person image of the correspondent on the image display unit of the communication partner's mobile phone during a call or character communication between the mobile phones. Is starting to appear.

However, the above-mentioned conventional image capturing device and communication device are capable of capturing and communicating images, communicating character information, and also capable of communicating and recording voice. However, the captured audio data is simply played as sound as it is, and the main focus is to reproduce the image as it is captured or as it receives the image and character information as faithfully as possible. However, there was a problem that the point of entertainment was not taken into consideration at all.

[0005]

SUMMARY OF THE INVENTION A first object of the present invention is to solve the above-mentioned problems of the prior art and to improve amusement during communication between communication devices having an image display unit such as a mobile phone. In addition, a transmission image displayed on the image display unit, for example, a modification pattern for modifying a captured image or a person image of a communication partner, for example, a modification image or voice displayed by being combined with the transmission image, for example, Modification including a modification image and voice additionally displayed on a person image to represent information according to emotions and communication contents, and a processing program for synthesizing the modification pattern and a program for processing a transmission image A large number of patterns can be registered so that the correspondents can use them freely, for example, they can be registered in the database of the relay station of the communication network connecting the communication devices or in the memory of the communication device, And to provide a registration method and apparatus for modifying the pattern of the transmitted image can be aligned with decorative patterns rich.

A second object of the present invention is to freely reproduce the modification pattern thus registered on the image display unit of the communication device or the image display device of the relay station, for example, to synthesize it with a transmission image for modification. It is therefore an object of the present invention to provide a method and apparatus for reproducing a modification pattern of a transmission image that can be reproduced freely. A third object of the present invention is to modify the image by visualizing the content to be emphasized in the captured image of the communication partner, particularly in the transmission image such as the person image, according to the emotional type of the person. Modification patterns that improve the enjoyment of creating a pattern and enhance the entertainment of the image displayed on the image display unit during communication between communication devices can be easily registered according to preference, and the registered modification can be performed. It is an object of the present invention to provide a method and an apparatus for registering a modified pattern of a transmission image that can freely reproduce the pattern, and a reproducing method and an apparatus.

[0007]

In order to solve the above first and third problems, a first aspect of the present invention is directed to a first communication device having an image display section of a correspondent and an image of a communication partner. When registering a modification pattern for modifying the transmission image displayed on the image display unit of the communication device of the communication partner while the second communication device including the display unit is communicating with each other through the relay station, the modification is performed. Reading the content with the pattern attached, copying the modified pattern from the read content, and registering the copied modified pattern in a state where it can be used by the correspondent or the first communication device. It is intended to provide a method of registering a characteristic modification pattern of a transmission image.

Here, it is preferable that the modification pattern includes at least one of image information and audio information and a program for reproducing the information, or a program for processing the transmission image. Further, the modification pattern is
It is preferable to modify the relevant image associated with the correspondent in the transmitted image. Further, it is preferable that the copied modification pattern is registered in a database of the relay station or a memory of the first communication device in association with related information related to the correspondent or the first communication device thereof. . The content is a transmission image from the second communication device displayed on the image display unit of the first communication device in communication with the communication partner, and the copied modification pattern is: The first communicating with the communication partner
It is preferable that the modification pattern is a modification pattern that modifies the transmission image from the second communication device reproduced and displayed on the image display unit of the communication device.

Further, the content is a transmission image from the second communication device displayed on the image display unit of the first communication device in communication with the communication partner, and the copied modification. The pattern is a modification pattern that modifies a transmission image from the second communication device that is being reproduced and displayed on the image display unit of the first communication device that is communicating with the communication partner. Registered in the database of the relay station or the memory of the first communication device in association with related information related to a person or its first communication device, or transmitted from the relay station to the first communication device It is preferable that it is stored in the memory.

Further, the modification pattern is registered as modification pattern data in advance in the database of the relay station in association with the related information, and the transmission displayed on the image display unit of the communication device. The identification number of the reproduced decoration pattern is added to the image, and the first communication device cuts out the identification number of the decoration pattern from the transmission image displayed on the image display unit, and cuts out the decoration. The relay station that transmits the copy request data including the pattern identification number to the relay station and receives the copy request data is associated with the related information related to the communication partner or the second communication device thereof, and the database is previously stored. From the identification number of the modification pattern, the modification pattern registered in the Preferably additionally registered in the database in association with the relevant information related to the first communication device. Further, it is preferable that the copied modification pattern is counted every time it is copied or used, and the owner of the second communication device is provided with a service according to the count number.

Further, the modification pattern is registered in advance in the memory of the second communication device as modification pattern data, and is reproduced in the transmission image displayed on the image display unit. The identification number of the decoration pattern is added, and the first communication device cuts out the identification number of the decoration pattern from the transmission image displayed on the image display unit and makes a copy including the identification number of the cut-out decoration pattern. The second communication device, which transmits the request data to the second communication device and receives the copy request data, reads and reads the modification pattern registered in the memory from the identification number of the modification pattern. The issued modified pattern data is returned to the first communication device,
It is preferable that the first communication device additionally registers the received decoration pattern data in the memory.

Further, the decoration pattern is used to combine the related image related to the correspondent in the transmission image with the related image and display the image as the transmission image. The modification pattern is preferably a modification pattern that is combined in a transmission image from the second communication device displayed on the image display unit of the first communication device that is communicating with the communication partner. .

The content is a video scene being viewed by the correspondent and being broadcast on an image display device, and the copied modification pattern is a video scene being viewed by the correspondent. It is preferably predetermined video information and / or audio information, and a reproduction program thereof. The content is a video scene that is being viewed by the correspondent and is being broadcast on an image display device, and the copied modification pattern is a special effect scene in the broadcast image that is being viewed by the correspondent. Video information and / or audio information, and a reproduction program thereof, wherein the copied modification pattern is associated with the related information related to the correspondent or the first communication device thereof, or the database of the relay station or the It is preferably registered in the memory of the first telecommunication device.

Further, the modification pattern is associated with the related information in the database of the relay station in advance,
It is registered as modification pattern data,
Alternatively, it is registered as modified pattern data in the database of the source of the broadcast image, and a scene frame identification number is added to the broadcast image in the image display device, and the scene frame identification number is added to the broadcast image in the image display device. Copy request data including the identification number of the scene frame is transmitted to a supply source that supplies a broadcast image to the image display device, and the supply source that receives the copy request data is the broadcast source from the identification number of the scene frame. The special effect scene image is searched, and the modification pattern in the searched special effect scene image is additionally registered as a modification pattern in the database of the relay station in association with related information related to the correspondent or the communication device. preferable.

The modification pattern is previously registered as modification pattern data in the database of the relay station in association with the related information, or as modification pattern data in the database of the broadcast image supply source. It is registered, and a scene frame identification number is added to the broadcast image on the image display device. From the first communication device to the relay station,
When the copy request data including the identification number of the scene frame is transmitted, the relay station that receives the copy request data transmits the copy request data to the supply source, and the supply source that receives the copy request data is the The special effect scene image broadcast from the identification number of the scene frame is searched, and the relay is performed by associating the modification pattern in the searched special effect scene image with the relevant information related to the correspondent or the communication device as the modification pattern. Additional registration in the station's database is preferred.

Further, the decoration pattern is used to combine the related image in the transmission image with the related image to display the transmission image, in order to modify the related image associated with the correspondent in the transmission image. The modification pattern is preferably a modification pattern in the special effect scene image of the image being displayed on the image display device.

Further, the content is a video recording medium viewed by the correspondent, and the copied modification pattern is reproduced by the correspondent on the video recording medium.
It is preferably video information and / or audio information in the reproduced video that is being viewed, and a reproduction program thereof.
Further, the video recording medium is a video data recording medium in which at least the video information is recorded as video data,
The modification pattern is recorded as modification pattern data on the video data recording medium and copied from the video data recording medium, or by using identification information of an access destination recorded on the video data recording medium, It is preferably copied from the access destination.

The video recording medium is at least
A hard copy medium that displays the video information as a hard copy image, and includes, as the video information, image information of the modification pattern and its identification information, and the modification pattern is displayed on the hard copy medium as modification pattern data. It is preferable that the identification information of the modified pattern is used to copy from a predetermined access destination. Further, the modification pattern data is transmitted from the predetermined access destination based on the modification pattern identification information transmitted from the first communication device to the relay station, the modification pattern data being transmitted from the first communication device. Preferably, it is registered in the database of the relay station in association with the correspondent.

Further, in each of the above aspects, the related image is a photographed image of the correspondent, and the modification pattern is an expression or gesture of the correspondent extracted from the photographed image or is attached to the photographed image. It is preferable to represent the emotion type determined based on any one or more of the audio data to be generated. Further, it is preferable that the communication device is a communication device having a photographing function.

Further, in order to solve the above first and third problems, a second aspect of the present invention comprises a first communication device having an image display section of a correspondent and an image display section of a communication partner. A reading unit for reading the content accompanied by a modification pattern for modifying the transmission image displayed on the image display unit of the communication device of the communication partner while the second communication device is communicating with each other via the relay station; A copy means for copying the decoration pattern from the content read by the reading means; and the decoration pattern copied by the copying means in a state where the correspondent or the first communication device can use it. The present invention also provides a registration device for a modified pattern of a transmission image, the device including a storage unit that stores the modified pattern.

Here, the storage means stores the copied modification pattern in association with the relevant information related to the correspondent or the first communication device thereof, the database of the relay station or the first communication. It is preferably the memory of the device. Further, it is preferable that the registration device is a server of the relay station or the first communication device.
Further, it is preferable that the communication device is a mobile phone having a photographing function.

In order to solve the above second and third problems, the third aspect of the present invention is registered by the method for registering the modification pattern of each transmission image according to the first aspect of the present invention. The modification pattern is read from the database of the relay station or the memory of the first communication device by the first communication device, and the image display unit of the first communication device and the second communication device A method of reproducing a modified pattern of a transmission image, which is reproduced and displayed on an image display unit or an image display device of the relay station. Here, the transmission image transmitted from the first communication device and displayed on the image display unit of the second communication device is preferably displayed after being modified by the modification pattern.

Further, in order to solve the above-mentioned second and third problems, a fourth aspect of the present invention comprises a first communication device having an image display section of a correspondent and an image display section of a communication partner. While the second communication device is communicating with each other via the relay station, the first content is read out, which is accompanied by a modification pattern for modifying the transmission image displayed on the image display unit of the communication device of the communication partner. A reading means, a copying means for copying the decoration pattern from the content read by the reading means, and a state in which the decoration pattern copied by the copying means can be used by the correspondent or the first communication device. Storage means to register and store with,
Second reading means provided by the first communication device for reading the modification pattern registered from the storage means; the image display unit of the first communication device;
The present invention provides a reproducing device for a modified pattern of a transmission image, which is reproduced and displayed on the image display unit of the communication device or the image display device of the relay station. Here, it is preferable that the second communication device displays on the image display unit the transmission image transmitted from the first communication device and modified by the modification pattern.

[0024]

BEST MODE FOR CARRYING OUT THE INVENTION A method and apparatus for registering a modification pattern of a transmission image and a reproducing method and apparatus according to the first to fourth aspects of the present invention will be described in detail below with reference to the preferred embodiments shown in the accompanying drawings. explain. First, the present invention basically relates to a communication device such as a telephone including an image display unit,
During communication or during a call, when displaying a related image such as a photographed image or a person image of the correspondent as a transmission image on the image display unit of the communication device of the communication partner A modification pattern that modifies a related image such as a captured image of a communicator or a person image in the image, for example, a modified image that represents communication content, call content, or emotion of the communicator, and this is combined with the transmission image or the related image. In order to display a program, a modification pattern (image processing pattern) including a program for processing a transmission image, etc., can be freely used by the correspondent, for example, a database of a relay station of the communication device or a memory of the communication device itself. Is easy to register,
It can be freely and easily reproduced and displayed on the image display unit of the communication device or the image display device of the relay station.

According to the present invention, when a modification pattern is registered, contents with various modification patterns, such as a transmission image modified by a modification pattern from a communication partner, and a modification pattern such as a special effect scene can be acquired by television broadcasting. A video data recording medium such as a DVD on which a modification pattern is recorded, a modification image of the modification pattern, and a hard copy medium such as a magazine in which the image processing pattern can be obtained. From the above, the modification pattern is acquired and copied to be registered in a form freely usable by the correspondent.

A method of determining or selecting a modification pattern (image processing pattern) to be combined with a related image such as a photographed image of a correspondent or a person image, which is a premise of the present invention, in particular, the correspondent from the contents of communication and the contents of a call Method of selecting a decoration pattern that expresses emotions of a person, a method of synthesizing a decoration image of the determined or selected decoration pattern with a related image such as a photographed image of a correspondent or a person image, and a related image with the selected decoration pattern The present inventor has proposed a method for performing image processing on the image and a method of creating and registering a modification pattern itself representing the emotion of the correspondent in advance so that the modification pattern can be selected. No. 111361 specification.

According to each of the methods proposed here, the communicator uses the modification pattern of the transmission image of the communication partner displayed on the image display unit of his communication device, that is, the modification pattern during the call or communication. The displayed images (modified images) can be enjoyed together. However, the modification pattern of the transmission image that the user transmits to the communication partner is selected from the modification patterns prepared in advance. However, the modification pattern of the transmission image includes the modification image information (image data) and the modification sound information (sound data) that modify the transmission image, as well as the program for synthesizing the modification image information and the modification information (data). Since the program includes a program for performing processing such as image processing and audio processing on a transmission image by using, it is necessary to create the modification pattern by yourself to prepare the modification pattern in advance, which is troublesome. Furthermore, even if a modification pattern is prepared in advance, if there is no modification pattern suitable for the transmission image, a new modification pattern is created, or a similar modification pattern or a related modification pattern is used instead. ,
Since it is necessary to stop synthesizing the modified image of the modified pattern or to select either, it takes time and it is not in time for real-time communication, entertainment is deteriorated, or entertainment is lost. It will be. Therefore,
In order to further improve the entertainment property, it is necessary to make rich the modification patterns of the transmission image of oneself in advance, but for that purpose, it is necessary to create the modification patterns by oneself as described above, which is troublesome. It takes.

Therefore, the first and second aspects of the present invention provide a method and an apparatus for registering a modification pattern of a transmission image that can easily enrich the modification pattern of the transmission image of the user. . In addition, the third and fourth aspects of the present invention are suitable for a transmission image freely or according to preference from among abundant modification patterns of the transmission image registered by such a registration method and device. (EN) A method and apparatus for reproducing a modified pattern of a transmission image, which allows a pattern to be easily selected and reproduced.
Regarding the method for deciding or selecting a modification pattern, the method for synthesizing the decided or selected modification pattern, and the method and apparatus for registering the modification pattern for selection, which are the premise of the present invention, the above-mentioned Japanese Patent Application No. 2001-111361. However, these methods and apparatus will first be described below.

Here, these premise methods of the present invention are
Basically, from the audio data attached to the captured image scene, the type of emotion of the person in the image scene is determined, and the person is identified by a modification pattern that emphasizes the emotion corresponding to the type of emotion. By modifying the image, for example, by attaching (synthesizing) a modified image such as a mark to the person image, the entertainment of image representation such as real-time image display of a videophone or a mobile videophone is improved. Is. In the following description, a mobile phone system capable of image communication capable of implementing the method of the present invention (hereinafter referred to as a mobile video phone system) is taken as a representative example, and image processing for synthesizing a modified image such as a mark on a display image is performed. Although the pattern is described as a typical example of the modified pattern, the present invention is not limited to this.

FIG. 1 is a schematic block diagram of an embodiment of a portable videophone system for implementing the method of the present invention. The mobile videophone system shown in FIG. 1 has mobile videophone terminals with image display units 34 and 44 (hereinafter referred to as terminals).
It is composed of 30 and 40, and a relay station (communication control device of a relay company) 50 of a portable videophone that connects them. Here, it goes without saying that the relay station 50 of the present embodiment includes a communication control device (a communication control device of a relay company) that relays and controls the communication between the terminals 30 and 40. It also includes an image processing device that performs image synthesis processing.

In this portable videophone system, the terminals 30 and 40 are respectively image sensors 32 and 42.
Image display units 34 and 44, confirmation display frames 36 and 46, processing units 37 and 47, and memories 38 and 4.
8 and. The relay station 50 also includes a processing unit 52 and a database 54. In this embodiment, the image synthesizing process, which is each modification pattern of each display image on the image display units 34 and 44, is performed on the relay station 50 side of the portable videophone. The image composition processing will be described later.

In the present embodiment, the database (server) of the relay station 50 of the telecommunications carrier is previously set through the terminals 30 and 40.
Reference numeral 54 denotes face image data (a facial expression for each mode for determining the mode corresponding to the emotion type), voice data (keyword for extracting each mode), and modification in each mode for each user (communicator). Register the pattern. Note that selection and registration of face image data, voice data, and modification patterns in each mode will be described later.

At the time of communication, as shown in FIG.
From the terminal 30 of the communication partner B, the captured image of the communication party A and the voice of the communication party A, and from the terminal 40 of the communication partner B, the image sensor 42.
The captured image of the communication partner B and the voice of the communication partner B captured by are respectively transmitted to the relay station 50. In the processing unit (server) 52 of the relay station 50, faces are always extracted from the photographed images of the correspondent A and the correspondent B, and matching with the registered facial expression is performed, and the keywords registered in the call voice are A check is made to see if it appears.

Then, for example, when the impatient mode is detected from the conversation of the correspondent A, a process (to be described later) of synthesizing a sweat mark as a modification pattern is performed on the photographed image of the correspondent A, for example. The image is transmitted to the terminal 40 of the communication partner B as a transmission image. At this time, the correspondent A
The processed image may be sent to the terminal 30 of the correspondent A as a confirmation for confirmation. Terminal 30 of correspondent A
Although the face image of the communication partner B is displayed on the display screen of the image display unit 34, a display frame 36 is provided in the corner for confirming the processed image of the communication party A, and the communication frame is displayed in this display frame 36. The processed image of person A is displayed. At this time, similarly, the processed image of the correspondent A is displayed on the display screen 44 of the terminal 40 of the communication partner B, and the display frame 4 for confirming the corner is displayed.
At 6, a reduced version of the processed image of the communication partner B sent from the relay station 50 is displayed.

Note that face image data, voice data, and modification patterns for each user, in this embodiment, image composition patterns and the like are registered in the user's terminal (see FIG. 4), and the terminal side of the portable videophone is connected. Then, the image composition processing may be performed from the mode detection. Further, the above data may be registered in both the relay station and the terminal so that either of them can perform the processing. Further, as described later, the image synthesizing process is not limited to the case of transmitting, and the synthesizing process may be applied to the received voice / image data. For example,
In FIG. 1, the compositing process for the image received from the communication partner B may be performed at the terminal of the correspondent A. In this case, amusement is improved by attaching a modification pattern that is preferred by Mr. A, who is the receiver.

Next, a method of selecting a modification pattern for a photographed image in the relay station 50 and performing a combining process using the selected modification pattern will be described. First, in this embodiment, for example, a still image, which is a captured image captured by the terminals 30 and 40 having a capturing function such as the image sensors 32 and 42 and attached as additional information to voice data, The modification pattern corresponding to the data is selected or created, and the image composition processing is performed on the photographed image according to the selected modification pattern. FIG. 2 is a block diagram showing an outline of a modification pattern selection / synthesis processing device including an image processing device for implementing the image synthesis processing method according to the present embodiment. The modification pattern selection / synthesis processing device (hereinafter, simply referred to as selection / synthesis device) 10 shown in FIG. 2 is provided or connected to the relay station 50, and mainly includes the information input / output means 12. It has an image processing device 14 and a database 16. Here, the image processing device 14 and the database 16 can be used as the processing unit 52 and the database 54 shown in FIG. 1, respectively.

The image processing device 14 executes the image combining processing method applied to the present invention and various other image processing. In addition, the image processing device 14 is configured to input and set conditions relating to various image processing, select and instruct processing,
An operation system 18 having a keyboard 18a and a mouse 18b for inputting an instruction such as color / density correction, an image input from the information input / output means 12, various operation setting / registration screens, etc. Monitor 2 displaying
0 is connected. Note that the monitor 20 also displays a combined processed image of the photographed image and the modified image of the modified pattern (hereinafter referred to as the modified pattern image), which is combined by the image processing device 14 for confirmation. Furthermore, the image processing device 14
, The photosensitive material (photographic paper) is image-exposed with a light beam modulated according to the image data output from the image processing device,
An image recording device 22 may be connected that outputs a developed (finished) image as a print.

The information input / output means 12 obtains the image data and the audio data which are taken by the terminals 30 and 40 and transmitted to the relay station, and sends them to the image processing device 14 where they are combined and processed. It is a bidirectional transmitting / receiving means for receiving the composite image of the photographed image and the modified pattern image and transmitting it to the terminals 30 and 40. The database 16 is
Here, a plurality of modification patterns for modifying the photographed image by the image processing device 14 are attached with predetermined modification pattern identification information (ID code or ID number) to the predetermined customer area (customer identification information, (Specified by customer ID), for example, as shown in FIG.
It can be stored as a table as shown.

The information input / output unit 12 is not limited to transmitting and receiving voice data and image data (captured image, synthetic image) data during communication between the terminals 30 and 40, and new creation of a modification pattern, When registering or adding new modification patterns to the database, setting or checking the conditions of these new modification pattern images to the captured image, or registering or adding to the database, the voice recording function is required. It is preferable that the audio data and the captured image data from the image capturing device with a recording function such as a digital camera that it has can be read from a recording medium or the like.

The image synthesizing process of the modified image by the modified pattern will be described below with reference to the flowchart of FIG. First, as a premise, in step 100, modification patterns corresponding to various emotion types, that is, modification images and image combination patterns are registered in the database 16 in advance. However, human emotions are so diverse that it is impossible to deal with all emotions, and here we deal only with the typical ones that appear relatively clearly. FIG. 4 shows an example of setting of the modification pattern (modification image and image combination pattern) and registration in the database 16. As you can see in the column labeled Mode in Figure 4, the types of emotions handled here are:
“Irritation”, “surprise”, “anger”, “sorrow”, “worry (question)”, “love”, “joy” ... etc.

In the present embodiment, the kind of emotion is determined based on the audio data (audio information) attached to the captured image scene. As a modification pattern, that is, an image processing pattern, a modification image (mark) that emphasizes each emotion is combined with each emotion type. Therefore, setting a modification pattern (image synthesis pattern) corresponding to the type of emotion means that
This is to set a decoration pattern (image synthesis pattern) corresponding to specific voice information. That is, as the items to be set in advance, there are a mode indicating the kind of emotion, a language for extracting each mode (keyword, omitted in the table of FIG. 4), and an image synthesis pattern corresponding to each mode. As a modification pattern (image composition pattern), a modification image (mark) to be composed, a position to be composed (relative position to the face position in the image), a size (size) of the composition image, a direction of the composition image with respect to the face, etc. There is. Here, the modification pattern image column of FIG. 4 will be described. The illustrated image is an image diagram showing the processing result of the modification pattern. For example,
In the “impulsiveness” mode, the face part shows a photographed image of the face of the caller, and only the sweat part is the modified image (mark) of the modified pattern. Note that the same applies to other modes, and a part other than the face part shows a modified image (mark). The method of synthesizing the modified image (mark) that emphasizes the emotion determined by the audio data attached to the captured image scene will be specifically described below.

For example, when a keyword such as “uuuu”, “yaibai”, or “yabai” is present in the voice information, the “impairment” mode corresponds to the result of processing the modification pattern on the person image. The "sweat mark" is synthesized. The composite image is not limited to one, and there may be a plurality of composite images. Further, the composite position is specified by the relative position with respect to the position of the face in the image, and is normalized by the face width. The size of the composite image is also specified on the basis of the face. The direction in which the composite image is combined is also set with reference to the face of the person in the image. For example, in the case of FIG. 4, when the coordinates of the center of the two sweat marks (one set of two) are at the position of the side of the eye (starting from the eye) (x1, y1) and the size is the face width of 1 , 0.1 and is synthesized on the left side of the face.

When a keyword such as “Uhyo”, “Hee”, “Bikuri” is included in the voice information,
The "surprise" mode corresponds, and the "surprised and popped-out eye mark" is synthesized at the coordinates (x2, y2) and (x3, y3) of the coordinates of the center of each eye mark. Further, the “orientation determination result” in FIG. 4 means whether the orientation of the human face with respect to the camera is frontal, diagonally right, or diagonally left by a pattern matching method or the like. It is shown that the patterns in which the pop-out direction is changed are combined.

When a keyword such as "cold" or "sabu" is present in the voice information, "sorrow" mode corresponds, a "shadow (diagonal line) mark" is put on the face, and "dead leaves" around. The mark ”is synthesized. At this time, the face color may be made white (white) instead of putting the diagonal lines on the face. Further, the background may be changed to a monotone so as to give a more sorrowful feeling. In addition, if there are other words such as "umm" and "umm", the "hatena mark" should be combined as the "worry (question)" mode, and the words such as "ummum" and "puki" should be used. For example, "heart" may be combined as the "affection" mode, and if there is a word such as "Wai" or "I did it", the "joy" mode may be combined and "launch fireworks" may be combined. . These image processing patterns are prepared to some extent on the system side,
The customer may create it and add data to it. At this time, a word that a customer often uses as a habit may be set as a word for extracting a mode, and a composite image or various emotion mode keywords may be registered as a customer's hobby. The above-described image processing pattern (modification pattern) is set once before each image synthesis process, and the set modification pattern, here the image synthesis pattern, is added with identification information and stored in the database. It may be registered in 16 predetermined customer areas (predetermined areas specified by customer identification information).

Next, in step 110, the information including the voice data and the photographing data attached as the attached information is input to the information input / output means 12. The voice data and the image data of the input information are sent to the image processing device 14, respectively. Next, in step 120, the image processing apparatus 14 specifies the emotion type from the audio data. First, the voice data is recognized, and matching is performed if a keyword that guides a mode indicating the registered emotion type is included. When a specific keyword is detected from voice data, a modification pattern that is a specific image processing pattern is specified by the mode corresponding to the keyword.

In the next step 130, the photographed image is combined using the modification pattern (image processing pattern) specified in the above step, that is, the modified image is combined with the photographed image. At this time, the face of the person is extracted from the captured image by a known method. As a face extraction method, for example, there is a method disclosed in Japanese Patent Application Laid-Open No. 8-122944 by the present applicant. As a method of extracting a specific area such as a face area of a person, in addition to the above, JP-A-4-346333, JP-A-5-158164, and 5-1.
No. 65120, No. 6-160993, No. 8-1849.
No. 25, No. 9-101579, No. 9-138470.
The methods disclosed in Japanese Patent Nos. 9-138471, 9-146194, 9-197575 and the like can also be suitably used.

When the face of the person is extracted from the photographed image, the eyes are extracted from the face, and the width of the face and the position of the eyes are calculated. Then, based on these data, the modification pattern image is synthesized in accordance with the position, size, and orientation designated by the modification pattern (image processing pattern). In addition, if there is a designation such as changing the background or the color of the entire screen, those processes are performed. After that, normal image processing is also performed to create an output image.

It is not always necessary to extract the person from the photographed image when determining the emotion type from only the voice data as described above. On the other hand, when the type of emotion is determined from the facial expression or gesture of the person, or the voice data and the facial expression or gesture of the person, it is necessary to extract the person from the image. In addition, even if it is not necessary to extract a person from the image for emotion determination,
When synthesizing a modified image with a captured image according to the modified pattern, extraction of a person is generally required. However, if the background is changed or the color of the entire screen is changed as a modification pattern, or if, for example, dead leaves are fluttered without regard to the position, it is not necessary to extract the person. . In this way, the extraction of the person from the image may be performed as needed. Therefore, for example, it is preferable to be able to select and set the method of determining the kind of emotion in advance.

In this way, the image processing apparatus 14 can obtain a composite image of the photographed image subjected to the image composition processing and the modified image of the modification pattern. For confirmation, the composite image is displayed on the monitor 20. Alternatively, the composite image (print) may be output from the image recording device 22 at the time of confirmation or confirmation of registration in the database 16. The composite image print may be delivered to the customer. Finally,
In step 140, the composite image subjected to the image composition processing by the image processing device 14 according to the modification pattern is output to the information input / output unit 12 and transmitted from the information input / output unit 12 to the communication partner terminal 40 or 30 together with the voice data. .

In the example described above, the selecting / synthesizing device 1 installed or connected to the relay station 50 on the relay station 50 side.
Although image processing is performed at 0 to create a composite image in which marks (modification pattern images) emphasizing emotions are combined, the image processing may be performed at the terminal 30 or 40 side. When processing is performed on the terminal 30 or 40 side, the image processing pattern or the like is set in advance on the terminal 30 or 40 side. Further, as described above, the modification pattern processing may be automatically performed based on the voice data recognition and the face extraction from the photographed image, or the operator may input the operation key or the like according to the customer's instruction. It may be performed. Even when the processing with the modified pattern is performed on the terminal 30 or 40 side, the software for executing the image processing may be incorporated in the terminal 30 or 40 to automatically perform the image combining processing. Keys / buttons etc. for terminals 30 and 4
The combination processing may be performed by giving an instruction to 0.

The modification pattern used in the present invention includes modification data such as modification pattern image data such as a mark for emphasizing emotions, modification sound data and the like, and a combination processing pattern thereof. Is a processing program for synthesizing a transmission image, in particular, a related image such as a captured image, and executing a series of processes until a synthesized image is output. Such a processing program includes a plurality of processing modules, particularly an image processing module.

Examples of the image processing module include face extraction, face element extraction such as eye / nose / mouth / ear extraction, hair extraction, various extraction processing modules such as body extraction, shape pattern matching calculation, density Examples include processing modules such as color conversion, enlargement / reduction, rotation, movement, size normalization, and image composition. In addition, the processing module may include an audio processing module. Of course, the original program created by the correspondent or the relay company of the relay station 50 may be used. It is preferable that the image processing modules having high commonality are stored as a common module group in the server of the relay station 50, for example, the database 54 and are held by the relay station.

As the modification data, for example, FIG.
Image data (including a time-series movement amount in the case of a moving image pattern) constituting elemental marks as shown in FIG. 4, keyword parameters such as language or gesture patterns corresponding to each mode of the table shown in FIG. 4, synthetic pattern parameters such as relative position / size / orientation with respect to the face, voice data of sound effects, and the like. It goes without saying that the correspondent or the relay company of the relay station 50 may use the modified data such as the modified image data and the modified voice data that are independently created. In addition, it is preferable that the modification data used for the modification pattern provided by the relay company or the like be stored in advance in the server of the relay station 50, for example, the database 54 as a common data group and be held by the relay station.

In the present invention, when the transmission image is modified by using the modification pattern including the plurality of processing modules, the modification pattern is executed by sequentially executing the plurality of processing modules as shown in FIG. , Modification processing,
In this embodiment, the image composition process may be performed. That is, as shown in FIG. 8, the processing module 1 is executed in step 200 from the start of execution of the modification pattern, and step 210 is executed.
In step 290, the processing module 2 is executed, the processing modules 2 are sequentially executed, the processing module N is executed in step 290, and the execution of the modification pattern ends.

In the present invention, when executing a modification pattern including a plurality of processing modules as described above, the following 2
It can take one configuration. For example, as shown in FIG. 9, when a modification pattern including a plurality of processing modules is executed and reproduced in the image processing unit 52 of the relay station 50 (for example, refer to the image processing device 14 having the configuration shown in FIG. 2). Will be described as a typical example. The first configuration is the image processing unit 5
This is a form in which the programs of all the processing modules to be executed are stored as internal modules in the internal memory 56 in the storage unit 2, and all the processing modules can be executed independently. In this form, of course, it is preferable to retain all the modification data used for all the internal modules as internal data in the memory 56 of the image processing unit 52. This form is adopted when using an original processing module, its original program, original modification data, and the like. Since this form does not need to read the program from the database 54 or the like, the program of the processing module can be immediately executed, and therefore it can be preferably adopted. The amount of data that needs to be increased increases. Note that the processing control part of the memory 56 is a part that describes the processing order of all the processing modules (in this case, all internal modules) necessary for execution, the usage of the modified data, etc. This is a part called by the processing unit 52 and used for execution control.

In the second configuration, the common module group as described above is stored in the database 54 of the relay station 50, and the memory 56 of the image processing unit 52 of the relay station 50 stores:
The image processing unit 52 does not have a program of processing modules to be executed at all, and calls all the processing modules to be executed from the common module group in the database 54, or the memory 56 of the image processing unit 52. A part of the processing modules to be executed, for example, a program of only the processing modules that are not in the common module group is retained as an internal module, and the remaining specific processing modules are selected from the common module group in the database 54. This is a mode in which the image processing unit 52 is called and the programs of all the processing modules are executed. In this form, the programs 56 of all the processing modules are not stored in the memory 56 of the image processing unit 52 at all, or only a part thereof is stored, so that the amount of data to be stored is small. is there. For example, a basic or specialized image processing module such as face extraction is not created by a user such as a correspondent, and a processing module provided by a relay company is stored in, for example, the database 54 of the relay station 50. It can be used by calling it from the common module group.

In this embodiment, of course, it is preferable that the modification data used for the processing modules in the common module group is also stored in the database 54 as the common data group as described above. The modification data used for all the processing modules called from the common module group in the database 54 or the remaining processing modules that are not internal modules in the memory 56 of the image processing unit 52 are also imaged from the common data group in the database 54. It is preferable to call the processing unit 52 for use. It is preferable that the modification data used for the internal module held in the memory 56 of the image processing unit 52 is also held in the memory 56 as internal data. In addition, when all the processing modules to be executed are called from the common module group in the database 54, when a part is called from the internal module in the memory 56 and the rest are called from the common module group, the processing order of the processing modules is described. The processing control part is called from the memory 56 to the image processing part 52, and is used by the image processing part 52 for controlling the execution of all the processing modules.

In this embodiment, as described above,
As the modification pattern, a modification pattern image (image or mark) that emphasizes emotions is combined to form an image combination pattern, but the facial expression may be changed by morphing processing. For example, in the case of "anger" as the mode of the modification pattern, the eyes may be lifted by morphing processing. However, in order to enhance the entertainingness of the photograph, a rather cartoonish one as in the example shown in FIG. 4 is more interesting and effective than a realistic change of the facial expression. As described above, the modification pattern may not include the modification pattern image but may include the program itself for image-processing the transmission image.

In the above-described embodiment, the still image is targeted as the captured image to be transmitted, but the moving image may be targeted. The moving image may be a moving image such as an image displayed on the display screen of a mobile TV phone or a displayed image of a movie in movie editing, but in the case of these moving images, the image composition processing method itself is Basically, it may be performed in the same manner as the above-mentioned embodiment.

That is, the same image processing pattern as that of the above-described embodiment is registered in advance as a modification pattern in a microcomputer such as a portable videophone terminal (see reference numerals 30 and 40 in FIG. 1), and an image is also registered. Incorporate processing software. Then, in the case of a portable videophone, when a keyword registered in the voice of the conversation is detected during a call, the terminal performs a process of synthesizing a synthetic image of a mode corresponding to the keyword. And
The composite image is transmitted and displayed on the terminal on the other end of the call.

At this time, the display image itself is also a moving image, and the synthesized "sweat mark" as shown in FIG. It is effective to move it. In addition, by ensuring a sufficient calculation speed of the CPU, it becomes possible to perform real-time image processing while reproducing during a call using a portable videophone or during movie editing. Also in this embodiment, the image synthesizing process may be performed by the relay station (see reference numeral 50 in FIG. 1) of the portable videophone terminal (see reference numerals 30 and 40 in FIG. 1) as in the above-described embodiment.

In the above-mentioned embodiment, the voice data is used as the information for judging the emotional type of the person.
Instead of voice data, facial expression recognition or gesture recognition may be used to determine the type of emotion. Also in this case, since it is applied to a portable videophone or a videophone, it is possible to perform image synthesizing processing in real time. For facial expression recognition, an individual correspondence table of images showing emotional and emotional expressions and modes corresponding to the facial expressions is registered in each portable videophone terminal. Further, for gesture recognition, for example, this mode is used when one finger is raised, this mode is used when two fingers are raised, and so on. Telephone terminal (see reference numerals 30 and 40 in FIG. 1)
Register in.

Then, during the call, a face is extracted from the photographed image of the speaker in the portable videophone terminal, the type of facial expression is identified by pattern matching, and the facial expression that matches the specific facial expression registered in advance is detected. If so, the image processing of the image processing pattern of the mode corresponding to the registered facial expression is performed, and the processed image is transmitted to the terminal of the other party of the call and displayed. Alternatively, the same processing is performed even when it is detected by the gesture recognition that the speaker is making a specific gesture. Also in this embodiment, the image synthesizing process may be performed by the relay station (see reference numeral 50 in FIG. 1) of the portable videophone terminal (see reference numerals 30 and 40 in FIG. 1) as in the above-described embodiment.

Known techniques for facial expression recognition and gesture recognition are, for example, PRMU in the Technical Report of the Institute of Electronics, Information and Communication Engineers, November 18, 1999.
99-106 "Gesture Recognition Robust to Change of Movement Position" Yasutoshi Amada, Motoyuki Suzuki, Hideaki Goto, Shozo Makino (Tohoku Univ.), PRMU99-138 "Automation of facial contour extraction and facial expression recognition" Hiroshi Kobayashi・ Hisashinori Takahashi, Kosei Kikuchi (Tokyo Univ. Of Science), PRMU99-139 "Estimation of facial area and facial expression recognition using potential net" Hiroaki Bessho (National Institute of Image Information), Norio Iwai, Masahiko Yanaida (Osaka Univ.), PRMU
99-140 (Special Lecture) "Facial expression recognition research and image processing technology" Hiroshi Yamada (Nihon Univ./AIR), PRMU99-14
2 (Special Invited Paper) “Human Motion Analysis and Recognition for Interactions” by Masahiko Taniuchi, Norio Iwai (Osaka Univ.) And others, and these techniques can be suitably applied.

By the way, in the various embodiments described above, when the image synthesizing process is performed in real time,
It is preferable to correct the positional deviation of the composite pattern when it occurs. Since the cause of the positional displacement of the composite pattern is almost the failure of face extraction, when the positional displacement of the composite pattern is found on the display screen, the face in the captured image is designated by an electronic pen or the like. At this time, the outline of the face may be surrounded with an electronic pen, or both eyes may be connected with a line. Further, the position of the mouth or the like may be designated. Alternatively, a key operation may be used to specify the position to which the composite pattern is to be moved in parallel, the size adjustment amount, and the like.

After that, the face position candidate area is automatically corrected based on the corrected position and size of the combined pattern so as to match with the original combined pattern, and the face extraction processing is re-executed. As a result, even in the case of a moving image such as a mobile TV phone, the position and size of the composite pattern are corrected in real time according to the movement of the face. Therefore, the synthetic pattern is displayed at a predetermined position in the display image by automatically following the face, so that the entertainment of image display on the image display medium is improved.

In addition, the TV image is taken into the personal computer,
It is preferable to perform the synthesizing process or to allow the receiver to have the synthesizing process function in a digital television so that the customer can set the synthesizing pattern because the entertainment property can be further improved. The portable videophone system to which the method of the present invention is applied and the modification pattern selection / synthesis processing device used therein are basically configured as described above.

Next, a first embodiment of the present invention will be described with reference to FIG. In the portable videophone system described above (including the modification pattern selection / synthesis processing device 10 shown in FIG. 2), the present embodiment is a communication partner displayed on the image display unit of the own communication device during a call or communication. If you like the modification pattern in the transmission image of, you can copy this modification pattern and add it to your library. In the portable videophone system shown in FIG. 1, as described above, first, an image modification pattern in the terminal 30 or 40 with a photographing function (camera), for example, an image modification pattern according to a voice, a gesture, or a facial expression is displayed, for example. The table as shown in FIG. 4 is registered in the database 54 for each customer.

Next, as shown in FIG. 5, Mr. A and Mr. B use their terminals 30 and 40 to start communication via the relay station 50. Then, during communication, for example, as described above, the correspondent Mr. A to the communication partner Mr.
At 0, a composite image obtained by combining a captured image with a modified pattern image corresponding to a voice, a gesture, or a facial expression is received and displayed as a transmission image 43 on the display screen of the image display unit 44 of the terminal 40. At this time, the modified transmission image (composite image)
43, the modification pattern identification information (modification pattern ID)
Is added.

At this time, as shown in FIG.
If he wants to add the modification pattern of the transmission image 43 from the correspondent A displayed on the display screen of the image display unit 44 of his terminal 40 or the modification image to his own modification pattern, For example, by pressing a pattern copy key (not shown) or by inputting pattern copy data, the modification pattern of the transmission image 43 can be copied for oneself. That is, in the terminal 40, by pressing the pattern copy key or inputting the pattern copy data, copy request data for copying the decoration pattern is created, and the decoration pattern identification information is received from the transmission image 43 received from the correspondent A. Is cut out and included in the copy request data together with Mr. B's customer ID. It is preferable that the copy request data also includes the customer ID of Mr. A who is the owner of the modification pattern. The copy request data created in this way is sent from Mr. B's terminal 40 to the relay station 50.

In the relay station 50 which receives the copy request data from Mr. B's terminal 40, the processing section 52 searches the internal database 54 for the modification pattern identification information or the modification pattern using the modification pattern identification information and the customer ID. The relevant (hit) modification pattern is read from Mr.'s storage area and written in Mr. B's storage area. In this way, the relay station 50 copies the modification pattern data (program and reference data (image / voice)) between the corresponding customers, that is, from the storage area of Mr. A to the storage area of Mr. B. In this way, the modification pattern is additionally registered in Mr. B's registration data (library in the storage area).

Therefore, in this embodiment, the processing unit 52 of the relay station 50 functions as a reading unit and a copying unit of the modified pattern from the content with the modified pattern, and the database 54 serves as a registration unit of the copied modified pattern. The relay station 50 functions as a registration device of a modification pattern of a transmission image. If the modification pattern identification information is cut out from the transmission image 43, the timing of transmission of the copy request data from Mr. B's terminal 40 to the relay station 50 and the relay station 50 receiving the copy request data.
The timing of the copy processing of the modification pattern data does not need to be during a call or communication between both communication parties.

In the present embodiment, the person (A) who copied the modification pattern owned by himself (Mr. A) from another person (Mr. B).
He preferably makes some profit return according to the number of times the copy is made or the number of times the modified pattern is used during a call (communication). For profit return, for example,
Call charge reduction (discount, reduction), awards, etc. can be mentioned. At this time, the total count of the number of times the modification pattern is copied and the number of times the modification pattern is used is determined by the relay station 50 for each customer and each modification pattern.
It is preferable to manage in.

On the other hand, instead of returning profits, a copy of the modification pattern or the use of the modified modification pattern may be charged for each modification pattern or collectively. On the contrary, depending on the customer or the modification pattern, it is preferable that the copy prohibition can be set for each modification pattern or collectively. Note that it is also possible to copy the modification pattern between the third parties. Profit can be returned to the creator by the creator identification information attached to the modification pattern. For example, even when copying from Mr. B to Mr. C, the usage frequency of creator A is counted up.

Next, a second embodiment of the present invention will be described with reference to FIG. In this embodiment, when a special effect scene displayed on a display screen being broadcast on an image display device such as a television is viewed and the special effect scene is liked, this special effect scene is used as a modification pattern for registering one's own registered data (live data). Rally) can be additionally registered. As shown in FIG. 6, when the television viewer who is the owner of the terminal 30 looks at the special effect scene of the television broadcast image 62 and likes the special effect scene, the television (T
V) Make a copy request to the station 60 using the terminal 30. That is, the terminal 30 transmits to the television station 60 copy request information including a customer ID, frame identification information (frame ID) (program channel and time, etc.) of a special effect scene of the television broadcast image 62, a telephone number, and the like.

Upon receiving the copy request information from the customer, the television station 60 retrieves the data of the modification pattern in the corresponding special effect scene image by searching the database 66 from the frame identification information by the processing unit 64, and the customer ID
Relay station 50 of the terminal 30 derived from the telephone number and the telephone number
Then, the data of the modification pattern in the corresponding special effect scene image is transmitted as the image modification pattern data together with the customer ID. In the relay station 50, the processing unit 52
The received image modification pattern data is written, copied, and stored in the customer's storage area of the internal database 54. In this way, the modification pattern is additionally registered in the viewer registration data (library in the storage area).

At this time, the viewer, at the time of additional registration, acquires the database 5 with the modification patterns fetched from the television or the like.
Keywords (languages) and the like drawn from No. 4 may be uniquely assigned. In addition, when there is a charge for importing the modified pattern from the television or the like into the database 54, the call charge (including the communication charge) of the terminal 30 is added to the relay station 5.
Preferably, the payment is made to the television station 60 via 0.

In the present embodiment, the viewer's terminal 30
Although the copy request information is transmitted from the television station 60 to the television station 60, the present invention is not limited to this, and a copy request signal is transmitted from the viewer's terminal 30 to the relay station 50 to display the television broadcast image 62. You may also request a copy of the special effect scene. In this case, first of all, the viewer has his own terminal 30
Then, the above-mentioned copy request signal including the frame identification information of the special effect scene of the television broadcast image 62 is transmitted to the relay station 50. Communicate between the relay station 50 and the television station 60,
Information is exchanged between the frame identification information of the special effect scene, the image data of the special effect scene, and its processing program (image modification pattern data), and the image modification pattern data is stored in the customer's storage area of the database 54 in the relay station 50. take in. In this way, the modification pattern is additionally registered in the viewer registration data (library in the storage area).

When the television station 60 transmits the modification pattern data for which a copy request has been made to the relay station 50, it has already received a copy request from another viewer or the like and has already received the copy request.
If the modification pattern has already been transmitted to the device, only the identification information (ID) of the modification pattern need be transferred, not the modification pattern itself. For example, the television station 60 records the identification information of the transmitted modification pattern, and every time the copying request is received, the television station 60 searches for the identification information of the corresponding modification pattern and the modification not transmitted to the relay station 50. Only in the case of a pattern, the modification pattern data itself is transmitted to the relay station 50. Alternatively, for each copy request, first, the identification information of the modification pattern is transmitted from the television station 60 side to the relay station 50 side, and the modification pattern data is transmitted to the relay station 50 only when the modification pattern does not exist on the relay station 50 side. To transmit. By doing so, there is an effect that unnecessary data transmission processing can be omitted.

In the above embodiment, the special effect scene of the television broadcast image to be requested for copying is specified by the frame identification information (frame ID) such as the program channel and time, and the corresponding modification pattern is acquired and copied. But
The present invention is not limited to this, and any scene designation method may be used as long as the scene can be identified, and any method of obtaining the information on the modification pattern of the designated scene may be used. For example, Japanese Unexamined Patent Application Publication No. Hei 10-10
The scene designation method disclosed in Japanese Patent No. 171027 may be used. In this scene designation method, the broadcasting station side adds the scene identification information coded by combining the broadcasting station name, program name, date and time, and image frame number to each scene, and the TV receiver side It decodes the identification information. In this way, the decoded frame identification information of the designated scene may be transmitted to the broadcasting station to obtain the modification pattern data (information) of the scene.

Further, Japanese Patent Application No. 2001 related to the applicant's application
-322392 specification (Japanese Patent Laid-Open No. 2002-3
Like the scene designating method (disclosed in Japanese Patent No. 05717), the information attached to the scene may be directly received from the television to obtain the modification pattern itself, or the identification information of the modification pattern may be obtained. . Further, like the scene designating method proposed in the specification of Japanese Patent Application No. 2002-281462 filed by the present applicant, a camera shoots a broadcast scene displayed on a TV screen, sends the shot image to a broadcasting station, and broadcasts it. The station may identify the scene and obtain the modification pattern data (information) of the scene. In the present embodiment, the television broadcast may be a terrestrial wave, a satellite broadcast, or an internet broadcast, and these broadcasts may be moving images or still images. It may be a picture.

Next, a third embodiment of the present invention will be described with reference to FIG. In the present embodiment, as in the above-described first and second embodiments, the modification pattern synthesis process is performed by the processing unit 52 of the relay station 50, and the modification pattern is registered in the database data (database data of the database 54 in the relay station 50). Rather than performing this in the library), the modification pattern composition processing and the modification pattern registration are performed on the terminal 30 or 40 side. When the modification pattern combining process is performed on the terminal 30 or 40 side as described above, the copy request signal and the modification pattern data are exchanged between the two parties talking. Also in this case, as described above, in the portable videophone system shown in FIG. 1, the relay station 50 is provided between the terminals 30 and 40.
Communication (call) is performed via the, and the modification pattern of the transmission image is stored (registered) in the memories 38 and 48 of the customer terminals 30 and 40.

As shown in FIG. 7, during communication, for example, the terminal 40 of Mr. B, who is the communication partner, is the terminal 3 of Mr. A, who is the correspondent.
From 0, the terminal 30 receives a composite image in which a modified pattern image corresponding to a voice, a gesture, or a facial expression is combined with a captured image, and the composite image is displayed as a transmission image 43 on the display screen of the image display unit 44 of the terminal 40. At this time, modification pattern identification information (modification pattern ID) is added to the modification-processed transmission image (composite image) 43. At this time, as shown in FIG. 7, when the communication partner Mr. B wants to add the modification pattern of the transmission image 43 from the correspondent Mr. A to his registration data, for example, pressing a pattern copy key, or By inputting the pattern copy data, the communication partner Mr. B can request the correspondent Mr. A to copy the modified pattern of the transmission image 43. For example, in the processing unit 47 of the terminal 40, copy request data for copying the modification pattern is created, and modification pattern identification information is cut out from the transmission image 43 received from the correspondent Mr.
It is included in the copy request data together with his customer ID.
It is preferable that the copy request data also includes the customer ID of Mr. A who is the owner of the modification pattern. The copy request data created in this way is transmitted from the terminal 40 of Mr. B to the terminal 30 of Mr. A via the relay station 50.

Next, in the terminal 30 of Mr. A who has received the copy request data, the processing unit 37 uses the modification pattern identification information or the customer ID from the internal memory 38 to display the image of the corresponding modification pattern. Read data and its processing program (modification pattern data). The thus-read modification pattern data is returned from the terminal 30 of Mr. A to the terminal 40 of Mr. B via the relay station 50. Next, Mr. B's terminal 40 stores the returned modification pattern data in the internal memory 48. In this way, the modification pattern is additionally registered in the storage area of the memory 48 inside Mr. B's terminal 40 as registration data (library of storage area). The timing of transmission of the copy request data from the terminal 40 of Mr. B to the terminal 30 of Mr. A, the timing of reading the modification pattern data at the terminal 30 of Mr. A who received the copy request data, and the read modification. Mr. A's terminal 30 of pattern data
From B to Mr. B's return to the terminal 40, it is not necessary that both parties are talking or communicating.

In the present invention, the content with the modification pattern is not limited to the transmission image of the portable videophone or the broadcast image of the television described above, and any content can be used as long as the modification pattern can be obtained. For example, the modification pattern data is directly recorded together with the video data, and if the modification pattern data can be copied, or a predetermined access destination is recorded together with the video data,
The content may be a commercially available DVD or the like as long as it can access a predetermined access destination and copy the modification pattern data. In the latter case, a commercially available DVD can be driven by a DVD driver or the like to reproduce video data, read a predetermined access destination, access the predetermined access destination, and copy the modification pattern data.

Next, a hard copy medium such as a magazine may be used as the content with the modification pattern. For example, a customer (communicator) who is watching a hard copy medium such as a magazine in which a modification pattern image (sample image) and modification pattern identification information (pattern ID), or a sample of a voice keyword is further displayed on the modification pattern image (sample image) , The pattern ID of the favorite sample image is input by the mobile phone and transmitted to the relay station. At the relay station side, based on the input pattern ID, order the modified pattern data,
It may be additionally registered in the registration data of the customer in the database. Alternatively, the modification pattern data may be further transmitted from the relay station to the customer's mobile phone and registered on the customer's mobile phone side, that is, the communication terminal side.

Further, a predetermined scene in the game on the game machine, for example, a special effect scene is designated on the game machine,
The modified pattern data may be copied from the game machine. The modification pattern data may be copied to the memory of the communication terminal, or may be copied to the database of the relay station via the communication terminal. In addition, the modifying pattern prepared by the theme park may be received by the communication terminal while walking in the theme park or when passing through the gate. The modification pattern received by the communication terminal may be copied to the memory of the communication terminal, for example.
It may be copied to the database of the relay station via the communication terminal.

In the example described above, the transmission image such as the photographed person image is transmitted (transmitted / received) and the voice is transmitted (transmitted / received) by using the communication device having the image display unit. As a modification pattern that modifies, the type of person's emotion is determined from the communication voice data and the person image (facial expression, gesture, etc.) in the transmission image, and the type of emotion that corresponds to that emotion is emphasized. Although the pattern is selected, the present invention is not limited to this, and the modification pattern may be controlled by another method. For example, the modification pattern may be switched depending on the communication partner. In this case, different decoration patterns may be set for the same keyword depending on the communication partner, or displayed (ON) / not displayed (OFF).
May be set. Further, the modification pattern may be automatically switched according to the location or time zone of the user / the location or time zone of the communication partner.

Further, a modification pattern for 3D (three-dimensional) display may be used. In this case, image processing is performed according to the display function of the communication partner, and a two-dimensional display image or a three-dimensional display image is obtained. Further, the vibration function of the mobile phone may be combined with the control of the modification pattern. Furthermore, the selected modification pattern may be edited. In this case, the modification pattern is preferably edited by size, position, display time, appearance timing, and a plurality of pattern combinations (appearance order, timing, etc.).

When a relay company or a specialized company provides various modification patterns, the provider who provided the modification patterns is required to pay a predetermined fee according to the frequency of pattern usage. preferable. At this time, it is preferable to collect the charges from the customer directly by the relay company and distribute them separately. Also, while using a vendor-supplied modification pattern,
You may display the image which shows having been charged.

In the above-described example, as a modification pattern, a modification pattern image corresponding to the content of the call by voice data or the content of the captured image to be transmitted is combined with the image captured by the portable videophone terminal with a capturing function (image composition). Although the pattern) synthesized image is displayed on the terminal of the communication partner, the present invention can be applied not only to a call using voice data but also to a communication using character information. For example, a modification pattern is selected according to the content of the character information to be communicated, and the character information is combined as a modification pattern image with a portrait or a photographed image of the correspondent and a line drawing or other related image characterizing the zodiac or the correspondent, The composite image may be displayed on the terminal of the communication partner.

By the way, the communication device provided with the image display section to which the method of the present invention is applied is not particularly limited as long as it is provided with the image display means and the communication means for voice and / or character information. But good. For example, in addition to the above-mentioned mobile videophone terminal, a fixed-line phone (or base unit) with a display, a portable phone (or a handset) with a display, a videophone, a mobile phone with a display, a mobile phone with a display, and a display with a display There may be mentioned communication devices with a display such as a wireless device, and these communication devices include a photographing function, that is,
It is preferable to have a camera function. Further, a video camera with a communication function, a digital camera with a communication function, an image capturing device having a communication function of voice and / or character information such as a digital video movie with a communication function, and the like can be given. Further, the communication device including the image display unit may be a device in which the image display unit and the communication unit are integrated, or may be a unit configured separately and combined.

The relay station and the database also include a distributed system. Although the relay station is shown as a single building in the figure, it also means that the entire system in which a plurality of relay facilities and database facilities are connected by a network is shown. Further, the modification pattern data to be copied to each customer database may be data indicating the reference destination of the modification pattern original data, not the program and the image data itself. In this case, during image processing,
The data is quoted from the reference. This is effective in reducing the capacity in the database. Further, as the modification pattern, not only the image synthesis but also the sound effect may be output. In this case, the modification pattern data includes voice data. In addition, it is desirable that the customer can download the original modification pattern creating software from the relay company. Furthermore, as the modified image data to be combined, either a raster image (a format having information in pixel units) or a vector image (a format having information for drawing a figure etc.) may be used. An example of the vector image format is the SVG format (Nikkei Electronics magazine, September 23, 2002 issue,
p169-p178).

The method and apparatus for registering the modification pattern of the transmission image and the reproducing method and apparatus according to the present invention have been described in detail above with reference to various embodiments, but the present invention is not limited to these embodiments. Of course, various improvements and changes may be made without departing from the scope of the present invention.

[0095]

As described in detail above, according to the present invention,
From the contents with the modification pattern that modifies the transmission image displayed on the image display unit of the communication device during communication, you can copy your favorite one for yourself and register it in a freely usable state. From the image modification pattern of the other party, you can send request information to copy your favorite one for yourself, and copy the program and image data of the corresponding modification pattern, that is, the modification pattern data in the database of the relay station. You can process it or copy it to the memory in your own communication device and register it in a state that you can use it freely, so you can increase your own modification pattern data and save your own modification pattern data without any trouble. Can be abundant. Therefore, according to the present invention, in the image displayed on the image display unit during communication or call,
Particularly in the display of the face of a person, the content to be emphasized is automatically visualized and expressed according to the emotion, so that it becomes possible to greatly improve the entertainment property in the image expression.

Further, according to the present invention, in the case where the modification pattern copy processing is performed in the database of the relay station, the substantial processing on the communication device terminal side can be eliminated, so that the memory on the communication device terminal side is eliminated. The load of can be reduced. Furthermore, according to the present invention, it is possible to provide the person who has copied his or her modification pattern with a service according to the total number of times of copying, the total number of times of use of the modified pattern that has been copied, and so on. You can improve the enjoyment of creating patterns.

[Brief description of drawings]

FIG. 1 is a block diagram showing an outline of an embodiment of a portable videophone system to which the method of the present invention is applied.

FIG. 2 is a block diagram showing an outline of an embodiment of a modification pattern selection / synthesis device including an image processing device for implementing the image synthesis processing method implemented in the portable videophone system shown in FIG.

FIG. 3 is a flowchart showing an example of a processing flow of an image combination processing method executed by the modification pattern selection and combination device shown in FIG.

FIG. 4 is an explanatory diagram showing an example of an image modification pattern and an image combination pattern used in the method of the present invention.

FIG. 5 is an explanatory diagram illustrating a first embodiment of a method of registering a modification pattern of a transmission image according to the present invention.

FIG. 6 is an explanatory diagram illustrating a second embodiment of a method of registering a modification pattern of a transmission image of the present invention.

FIG. 7 is an explanatory diagram illustrating a third embodiment of a method of registering a modification pattern of a transmission image of the present invention.

FIG. 8 is a flowchart showing an example of a method of reproducing a modified pattern of a transmission image of the present invention.

FIG. 9 is a block diagram showing an embodiment of a reproduction device for a modified pattern of a transmission image of the present invention.

[Explanation of symbols]

10 Modification Pattern Selection and Synthesis Processing Device (Selection / Synthesis Device) 12 Information Input / Output Means 14 Image Processing Device 16 Database 18 Operation System 18a Keyboard 18b Mouse 20 Monitor 22 Image Recording Device 30, 40 (Mobile Videophone) Terminals 32, 42 Image sensor 34, 44 Image display unit 36, 46 Display frame 37, 47 Processing unit 38, 48 Memory 43 Transmission image 50 Relay station 52, 64 (Image) processing unit 54, 66 Database 60 Television (TV) station 62 Television (TV) )image

Continued front page    F-term (reference) 5C064 AA01 AB03 AB04 AC04 AC06                       AC08 AC15 AC16 AC18 AD08                       AD14 BA07 BB10 BC18 BC23                       BC25 BD02 BD08                 5C076 AA14 AA19 AA22                 5C082 AA00 BA12 BB01 BD02 CA56                       CB05 DA42 DA53 MM05

Claims (29)

[Claims] 1. A first communication device having an image display unit of a correspondent and a second communication device having an image display unit of a communication partner are communicating with each other via a relay station, and When registering a modification pattern for modifying the transmission image displayed on the image display unit, the content accompanied by the modification pattern is read, the modification pattern is copied from the read content, and the copied content is copied. A method of registering a modification pattern of a transmission image, characterized in that the modification pattern is registered in a state where it can be used by the correspondent or the first communication device. 2. The modification pattern registration of a transmission image according to claim 1, wherein the modification pattern includes at least one of image information and audio information and a program for reproducing the information, or a program for processing the transmission image. Method. 3. The method of registering a modification pattern of a transmission image according to claim 1, wherein the modification pattern modifies a related image related to the correspondent in the transmission image. 4. The copied modification pattern is registered in a database of the relay station or a memory of the first communication device in association with related information related to the correspondent or the first communication device thereof. The method of registering a modification pattern of a transmission image according to claim 1. 5. The content is a transmission image from the second communication device displayed on the image display unit of the first communication device in communication with the communication partner, and the copied modification. The pattern is a modification pattern that modifies a transmission image from the second communication device that is reproduced and displayed on the image display unit of the first communication device that is communicating with the communication partner. 5. A method of registering a modification pattern of a transmission image according to any one of 4 to 4. 6. The content is a transmission image from the second communication device displayed on the image display unit of the first communication device in communication with the communication partner, and the copied modification. The pattern is a modification pattern that modifies a transmission image from the second communication device that is being reproduced and displayed on the image display unit of the first communication device that is communicating with the communication partner. Registered in the database of the relay station or the memory of the first communication device in association with related information related to a person or its first communication device, or transmitted from the relay station to the first communication device The method for registering a modification pattern of a transmission image according to claim 3, wherein the modification pattern is registered and registered in the memory. 7. The modification pattern is registered as modification pattern data in advance in the database of the relay station in association with the related information, and the transmission displayed on the image display unit of the communication device. The identification number of the reproduced decoration pattern is added to the image, and the first communication device cuts out the identification number of the decoration pattern from the transmission image displayed on the image display unit, and cuts out the decoration. The relay station that transmits the copy request data including the pattern identification number to the relay station and receives the copy request data is associated with the related information related to the communication partner or the second communication device thereof, and the database is previously stored. Is searched for from the identification number of the decoration pattern, and the searched decoration pattern is searched for by the correspondent or the correspondent. Registering modification pattern of the transmitted image according to claim 6, in association with related information associated with the first communication device is additionally registered in the database. 8. The copied modification pattern is counted every time it is copied or used, and the owner of the second communication device is provided with a service according to the count number. 6. The method for registering a modification pattern of a transmission image according to 6 or 7. 9. The modification pattern is previously registered as modification pattern data in a memory of the second communication device, and the modification image reproduced in the transmission image displayed on the image display unit. A pattern identification number is added, and the first communication device cuts out the identification number of the modification pattern from the transmission image displayed on the image display unit, and makes a copy request including the identification number of the cut-out modification pattern. The second communication device, which transmits data to the second communication device and receives the copy request data,
The modification pattern registered in the memory is read from the identification number of the modification pattern, the read modification pattern data is returned to the first communication device, and the first communication device receives the modification pattern data. The method of registering a modification pattern of a transmission image according to claim 6, wherein the modification pattern data is additionally registered in the memory. 10. The modification pattern is used for combining with the related image to display the transmission image in order to modify the related image related to the correspondent in the transmission image, and the copied pattern. 7. The modification pattern is a modification pattern synthesized in a transmission image from the second communication device displayed on the image display unit of the first communication device in communication with the communication partner. 10. A method of registering a modification pattern of a transmission image according to any one of 9 to 9. 11. The content is a video scene that is being viewed by the correspondent and is being broadcast on an image display device, and the copied modification pattern is a video scene that is being viewed by the correspondent. The method for registering a modification pattern of a transmission image according to any one of claims 1 to 4, which is predetermined video information and / or audio information and a reproduction program thereof. 12. The content is a video scene that is being viewed by the correspondent and is being broadcast on an image display device, and the copied modification pattern is a broadcast scene that is being viewed by the correspondent. It is video information and / or audio information of a special effect scene, and a reproduction program thereof, wherein the copied modification pattern is associated with related information related to the correspondent or the first communication device thereof, and a database of the relay station. Alternatively, the method of registering a modification pattern of a transmission image according to claim 3, which is registered in a memory of the first communication device. 13. The modification pattern is registered in advance as modification pattern data in the database of the relay station in association with the related information, or a modification pattern is stored in a database of a source of the broadcast image. It is registered as data, the scene image identification number is added to the broadcast image on the image display device, and the broadcast image is supplied from the first communication device to the image display device. Copy request data including the identification number of the scene frame is transmitted to the source, and the supply source receiving the copy request data searches for the special effect scene image broadcast from the identification number of the scene frame, and the searched The modification pattern in the special effect scene image is related to the correspondent or the communication device as the modification pattern. Registering modification pattern of the transmitted image according to claim 12 additionally registered in the database of the repeater station in association with related information. 14. The modification pattern is previously registered as modification pattern data in the database of the relay station in association with the related information, or a modification pattern is stored in a database of a source of the broadcast image. It is registered as data, and a scene frame identification number is added to the broadcast image on the image display device, and a copy including the scene frame identification number from the first communication device to the relay station. The relay station which has transmitted the request data and has received the copy request data has transmitted the copy request data to the supply source, and the supply source having received the copy request data has been broadcast from the identification number of the scene frame. The special effect scene image is searched, and the modification pattern in the searched special effect scene image is set to the modification pattern. 13. The method of registering a modification pattern of a transmission image according to claim 12, wherein the modification pattern is additionally registered in a database of the relay station in association with related information related to the correspondent or its communication device as a code. 15. The modification pattern is used for combining the related image in the transmission image with the related image to display the transmission image in order to modify the related image associated with the correspondent, and is reproduced. The method of registering a modification pattern of a transmission image according to any one of claims 12 to 14, wherein the modification pattern is a modification pattern in a special effect scene image of the image being displayed on the image display device. 16. The content is a video recording medium that the correspondent is watching, and the copied modification pattern is a reproduced video that the correspondent reproduces and views the video recording medium. The method for registering a modification pattern of a transmission image according to any one of claims 1 to 4, which is video information and / or audio information therein and a reproduction program thereof. 17. The video recording medium is a video data recording medium having at least the video information recorded as video data, and the modification pattern is recorded as modification pattern data on the video data recording medium. The modification pattern registration of a transmission image according to claim 16, which is copied from a recording medium or is copied from the access destination using identification information of the access destination recorded in the video data recording medium. Method. 18. The video recording medium is a hard copy medium that displays at least the video information as a hard copy image, and includes, as the video information, image information of the modification pattern and identification information thereof, wherein the modification pattern is 17. The method of registering a modification pattern of a transmission image according to claim 16, wherein the modification pattern data is copied from a predetermined access destination using identification information of the modification pattern displayed on the hard copy medium. 19. The decoration pattern data transmits identification information of the decoration pattern from the first communication device to the relay station, and the relay station side determines the predetermined access destination based on the identification information of the decoration pattern. 19. The method of registering a modification pattern of a transmission image according to claim 18, wherein the modification pattern is copied from the file and registered in the database of the relay station in association with the correspondent. 20. The related image is a photographed image of the correspondent, and the decoration pattern is one of facial expressions or gestures of the correspondent extracted from the photographed image or voice data attached to the photographed image. The method of registering a modification pattern of a transmission image according to any one of claims 3 to 19, which represents a type of emotion determined based on any one or more pieces of information. 21. The method of registering a modification pattern of a transmission image according to claim 1, wherein the communication device is a communication device having a photographing function. 22. A first communication device having an image display unit of a correspondent and a second communication device having an image display unit of a communication partner are communicating with each other via a relay station, and Reading means for reading the content accompanied by a modification pattern for modifying the transmission image displayed on the image display section; copying means for copying the modification pattern from the content read by the reading means; A device for registering a modification pattern of a transmission image, comprising: a storage unit for registering and storing the modification pattern copied by the means in a state in which the correspondent or the first communication device can use the modification pattern. 23. A database of the relay station or the first communication device, wherein the storage means stores the copied modification pattern in association with related information related to the correspondent or the first communication device thereof. 23. The device for registering a modification pattern of a transmission image according to claim 22, which is a memory of. 24. The registration device of a modification pattern of a transmission image according to claim 22, wherein the registration device is a server of the relay station or the first communication device. 25. The modification pattern registration device for a transmission image according to claim 22, wherein the communication device is a mobile phone with a photographing function. 26. The modification pattern registered by the method of registering a modification pattern of a transmission image according to claim 1, is read by the first communication device, and the modification pattern of the first communication device is read. A method of reproducing a modified pattern of a transmission image, which is reproduced and displayed on the image display unit, the image display unit of the second communication device, or the image display device of the relay station. 27. The transmission image transmitted from the first communication device and displayed on the image display unit of the second communication device is modified by the modification pattern and displayed. Method for reproducing the modified pattern of the transmission image of. 28. The first communication device having the image display unit of the correspondent and the second communication device having the image display unit of the communication partner are communicating with each other via the relay station, and A first reading unit for reading the content accompanied by a modification pattern for modifying the transmission image displayed on the image display unit; and a copying unit for copying the modification pattern from the content read by the reading unit. Storing means for registering and storing the modified pattern copied by the copying means in a state in which the correspondent or the first communication device can be used; and the storing means provided in the first communication device. Second reading means for reading the modification pattern registered by the second reading means, and the modification pattern read by the second reading means for the image of the first communication device. A reproduction device for a modified pattern of a transmission image, comprising: a display unit; and a reproduction display unit that reproduces and displays the image on the image display unit of the second communication device or the image display device of the relay station. 29. The transmission image according to claim 28, wherein the second communication device displays the transmission image transmitted from the first communication device and modified by the modification pattern on the image display unit thereof. Reproduction device for modified pattern.