JP2009044510A - Mobile unit, server, and video reproduction method in mobile unit - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reproduce a reproduction video image in a desired mode at a receiving side while preventing a transmitting-side speaker from recognizing the reproduction when making a video telephone call using a mobile unit equipped with a function for sending, to the receiving side, both a "face video image" of the transmitting-side speaker photographed by one camera and a "scenery video image" photographed by another camera, in a field of view in front of the speaker. <P>SOLUTION: In a receiving-side mobile unit 130 which receives from a transmitting-side mobile unit a multiplexed video signal of both "face video image" of a transmitting-side speaker photographed by one camera (111) and the "scenery video image" photographed by another camera (112) in a field of view in front of the speaker, the two video images are demultiplexed (132), and both the video images are displayed on a video display (134) in a mode corresponding to setting in a video display mode setting section (135) capable of selectively setting a display mode as desired by a receiving-side speaker, so as not to enable the transmitting-side speaker to recognize the reproduction. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a mobile device, a server, and a video playback method in the mobile device.

In a videophone using a mobile device, it is possible to transmit a video of a camera provided in the mobile device to a communication partner, and it is one advantage that a videophone can talk while looking at each other's face.
However, it may not always be convenient for yourself to be faithfully transmitted to the communication partner, including the background.

  Considering such a realistic situation, where the videophone user (speaker on the sending side) does not want to show his / her surroundings to the other party, or where the videophone is mobile, where When the other party doesn't want to be realized by the other party, the speaker intentionally makes it difficult to see part of the background, or replaces part of his background with the created image so that the other party does not feel uncomfortable A videophone device has been proposed (see Patent Document 1).

Also, since the conversation tends to be monotonous only with the audio and video of the call, a videophone that can freely transform the speaker's face and background between the speakers and enhance the production effect has also been proposed. (See Patent Document 2).
On the other hand, two cameras are installed so that their imaging fields of view are reversed, one side is used to photograph the user side, the other camera is used to photograph the user's front field of view, and these two cameras are used. There has also been proposed a technique for transmitting video by superimposing captured images and displaying the transmitted video so that both videos can be separated and recognized on the receiving side (see Patent Document 3).

Furthermore, a technique has also been proposed in which two cameras are installed in a mobile phone so that their fields of view are in different directions as described above, and the images captured by both cameras can be synthesized and recorded. (See Patent Document 4).
In addition, when a part of the background of a person photographed with a camera is replaced with a created image in a mobile phone with a videophone function, the background of the person is recognized by detecting the outline of the person and recognizing the background area. A specific method has also been proposed in which the image is removed by a background removal device and replaced with a background image registered in advance (see Patent Document 5).

  On the other hand, in a videophone system using a user terminal equipped with a camera, a menu screen is provided from the transmission side in order to improve interest during a call, and a menu selection operation using this menu screen on the reception side accordingly In this case, a technique has also been proposed in which data corresponding to this selection is supplied from the transmission side to the reception side so that the reception-side user terminal displays a screen as desired (see Patent Document 6). .

JP 2000-175166 A (Summary, paragraphs 0003 to 0005, FIG. 3 etc.) JP 2004-72310 A (summary, paragraphs 0002 to 0004, paragraphs 0070 to 0071, paragraphs 0096 to 0101, FIG. 6, FIG. 37, etc.) Japanese Patent Laying-Open No. 2005-64681 (Summary, paragraphs 0005 to 0010, FIG. 2, etc.) JP 2004-180107 A (Abstract, paragraphs 0040 to 0041, FIG. 3, etc.) JP 2003-333556 A (Summary, paragraphs 0020 to 0021, FIG. 2, FIG. 3, etc.) JP 2006-345076 A (Abstract, paragraphs 0043 to 0053, FIG. 4, FIG. 5, FIG. 6, etc.)

  The above-mentioned patent document 1 discloses that a speaker on the transmission side leads the way of selecting his / her background video, and patent document 2 discloses a talk on both the transmission side and the reception side. It is disclosed that the speaker's face and background can be freely transformed between the speakers with the understanding from the beginning, but the receiver's speaker when receiving the video supplied from the transmitter However, there is no idea that the reproduction mode (that is, the display mode) can be selected on the initiative. The technique of Patent Document 5 that replaces the area corresponding to the background of the speaker with an existing image is the same as described above.

In Patent Document 3 and Patent Document 4, two cameras with different imaging fields of view are applied to simultaneously record each image regarding the speaker and its front field of view and obtain a composite image by both cameras. However, there is no idea that the display mode of the composite video can be selected according to the request of the speaker on the receiving side.
On the other hand, in the technique of Patent Document 6, it seems that the receiving speaker can take the lead in selecting the display mode of the received video, but this selection is performed according to the menu screen provided from the transmitting side. In principle, the degree of freedom of selection is constrained on the transmission side.

However, in a videophone communication performed by a mobile device, a video receiver in the position of the receiving speaker is an image sent from a video sender in the transmitting speaker position, for example, a transmitting speaker. Will be observed on the small screen of the mobile device, which is a composite of the video showing the scene and the landscape image developed in the front field of view of the speaker.
For this reason, on a display screen of a very limited size of the mobile device, a large part of the display screen may inevitably be occupied by a video that is not of interest to the speaker on the receiving side.

In such a case, there is a problem that it becomes difficult for a speaker on the receiving side to observe a desired image that is interesting, important, or has special value. The
For example, a receiving speaker wants to display a video of a landscape developed in the forward view of the speaker rather than the transmitting speaker's face with a sufficient display size, but depending on the transmitting speaker's face. This is the case when the central area of the display screen is largely occupied, and the image of the landscape to be seen appears small in the surrounding area.

  In order to deal with such problems, a request can be made from the receiving speaker to the transmitting speaker to place the face image in a small position on the screen with a secondary positioning. However, if it is allowed to make such a request without any hesitation, in some cases, the sender's speaker may be injured or feel uncomfortable. There is also a risk of causing new problems such as becoming.

  Further, even if the receiving speaker does not send a separate request to the transmitting speaker, the receiving speaker is arbitrarily obstructed by the positioning of the transmitting speaker as a secondary position. Even if it is possible to perform operations such as displaying at a position that does not appear, if the transmission side recognizes such an operation as it is, the transmission side talks in the same way as described above. Concerns that make people feel uncomfortable cannot be dispelled.

  The present invention has been made in view of such a situation, and displays a playback video in a desired mode on the receiving side while preventing the transmitting side from recognizing it. It is an object of the present invention to provide a mobile device, a server, and a video playback method in the mobile device that allow the user to observe the video as desired on the receiving side without damaging the pride of the speaker.

In order to solve the above problems, the present application proposes an apparatus and a method as listed below.
That is, one mobile device according to the present invention has one video signal acquired by one imaging unit arranged so that a speaker can be included in the imaging field of view, and the one imaging unit has a different imaging field of view. It is configured to multiplex other video signals acquired by other imaging means arranged so as to be separable by a predetermined signal multiplexing processing unit to obtain a multiplexed video signal A mobile device configured to receive the multiplexed video signal supplied via a communication network from a mobile device on the transmission side,
A multiplexed video signal receiving unit that receives the multiplexed video signal from the communication network, and a multiplexed video signal received by the multiplexed video signal receiving unit is separated into the one video signal and the other video signal. A signal separation unit, a video display unit that displays each video based on the one video signal and the other video signal separated by the signal separation unit, and the one video signal and the other video signal. And a video display operation unit for operating a display mode of each reproduced video on the video display unit without affecting the display on the mobile device on the transmission side.

According to the mobile device as described above, the multiplexed video signal generated by separably multiplexing the video signals respectively acquired by the one imaging unit and the other imaging unit having different imaging fields of view is generated. Then, when supplied from the mobile device on the transmission side through the communication network, the supplied multiplexed video signal is received by the multiplexed video signal receiving unit of the own device, and the one video signal and the Separated from other video signals.
Each video based on the separated one video signal and the other video signal is displayed on a video display unit. At the time of display, each reproduced video based on the one video signal and the other video signal is displayed. The display mode of the video display unit can be operated by the video display operation unit without affecting the display on the mobile device on the transmission side.

Therefore, it is possible to display the reproduced video in the desired mode set through the video display operation unit on the receiving side while being not recognized by the speaker on the transmitting side.
In addition, as one aspect of the mobile device as described above, the video display operation unit may select and operate a display area pattern of the one video signal and the other video signal in the video display unit. Composed.
By adopting such a configuration, the video display operation unit can be selected and operated arbitrarily by the receiving speaker, so that it is possible to adopt a display mode in a pattern that meets the receiving speaker's desire.

  In another aspect of the mobile device as described above, the signal separation unit is obtained by the one imaging unit so that the imaging field of view includes a face of a speaker who handles the transmitting-side mobile device. Multiplex generated by multiplexing a video signal and another video signal acquired by the other imaging means with the front visual field of the speaker as the imaging visual field, in a form that can be separated by the signal multiplexing processing unit When the video signal is received by the multiplexed video signal receiving unit of the own device, the received multiplexed video signal is separated into the one video signal and the other video signal, and the video The display operation unit operates a display mode in the video display unit of a reproduced video related to the speaker's face based on the one video signal and a reproduced video related to the front view of the speaker based on the other video signal. Configured to .

  By adopting such a configuration, the video display operation unit is operated as desired by the receiving speaker, for example, in front of the transmitting speaker that is another video of interest to the receiving speaker. The video of the landscape of the sender on the sender's face is displayed on the video display unit so that it occupies the entire area of the display screen for the time being. It is possible to adopt a display mode in which the image is only superimposed on a relatively narrow area.

  Further, as another aspect, the mobile device as described above is configured such that one imaging unit arranged to include a speaker in the imaging field of view and the one imaging unit have different imaging fields of view. A predetermined video composition control signal defines the other image pickup means disposed in the first image signal, the one video signal acquired by the one image pickup means, and the other video signal acquired by the other image pickup means. A video synthesis unit that obtains a synthesized video signal by performing synthesis processing in a mode, and generates the video synthesis control signal based on a video synthesis mode designation signal supplied from another device via a communication network and supplies the video synthesis control signal to the video synthesis unit And a video composition mode selection unit.

By adopting such a configuration, each image signal acquired by one imaging unit and another imaging unit having different imaging fields of view is defined by a predetermined video composition control signal in the video composition unit. The synthesized video signal is obtained by performing the synthesis process in the above manner.
In this synthesis process, the video synthesis mode selection unit generates the video synthesis control signal based on the video synthesis mode designation signal supplied from the other device via the communication network, and the generated video synthesis control signal is Supplied to the video composition unit.

  In this case, the video composition mode designation signal supplied from the other device via the communication network is generated, for example, in accordance with an operation desired by the receiving speaker in the other device, and is generated and automatically generated. The synthesized video signal generated by the synthesis process according to the video synthesis control signal based on the video synthesis mode designation signal supplied to the machine is sent from the own machine to the other machine and played back. As a result, the receiving speaker The display mode suitable for the desired is realized in the other machine.

  The mobile device according to still another aspect of the present invention is arranged such that one imaging means arranged so as to include a speaker in the imaging visual field and the one imaging means are different in imaging visual field. The multiplexed video signal is multiplexed by separating the other imaging means, the one video signal acquired by the one imaging means, and the other video signal acquired by the other imaging means into a separable form. A multiplexing processing unit to obtain, a video composition mode designation signal for defining a display mode when displaying each reproduced video based on the one video signal and the other video signal on a predetermined video display unit, and the multiplexing In response to the supply of the video signal, the supplied multiplexed video signal is separated into one video signal and another video signal, and the separated one video signal and the other video signal are separated from the video signal. Match the display mode defined by the composite mode specification signal It is characterized in that comprises a multiplexed video signal transmission unit that transmits to the server that the image synthesizing process so as to form a combined image signal.

  In the mobile device of the above aspect, the video composition mode designation signal defining the display mode when the reproduced video based on the one video signal and the other video signal are displayed together on a predetermined video display unit, and the multiplexing In response to the supply of the video signal, the supplied multiplexed video signal is separated into one video signal and another video signal, and the separated one video signal and the other video signal are separated from the video signal. It is assumed that it can be used in such a form that it can be used in a form connected to a server that performs video composition processing so as to form a composite video signal that conforms to the display mode specified by the composite mode designation signal, for example, through a wireless communication network. A function unit that executes processing for separating the multiplexed video signal into one video signal and another video signal, and the separated multiplexed video signal is defined by the video synthesis mode designation signal Each configuration is reduced corresponding to the functional unit for executing an image synthesizing process to form a compatible composite video signal to 示態 like.

The configuration is simplified as described above, and the image processing apparatus includes the one image pickup unit and the other image pickup unit. And another video signal obtained by photographing the person's front vision.
Further, a multiplexed video signal is obtained by multiplexing one video signal and another video signal in a separable form by a multiplexing processor.
The multiplexed video signal obtained as described above is transmitted from the multiplexed video signal transmission unit to a server having the above functions.
On the server side that receives such a multiplexed video signal, a composite video signal is generated by executing the above-described processing. The synthesized video signal can display the reproduced video in the display mode defined by the video synthesis mode designation signal.

Still another mobile device of the present invention is characterized in that one video signal acquired by one imaging means arranged so that a speaker can be included in the imaging visual field, and the one imaging means has a different imaging visual field. A communication network from a mobile device on the transmission side configured to obtain a multiplexed video signal that is multiplexed in a separable manner with other video signals acquired by other imaging means arranged to A display mode when the multiplexed video signal supplied via the video signal is received and each reproduction video based on the one video signal and the other video signal is displayed together on a predetermined video display unit Receiving a video composition mode designation signal, separating the received multiplexed video signal into one video signal and another video signal, and further separating the separated one video signal and the other video signal; Suitable for display modes defined by video composition mode specification signal A configured mobile as tied via the server and the communication network to the image synthesizing process to form a composite video signal which,
A synthesized video signal receiving unit that receives the synthesized video signal from the server, and a video synthesis operating unit that generates the video synthesis mode designation signal without affecting the display on the transmitting mobile device according to an external operation; A video composition mode designation signal transmitting unit that transmits the video synthesis mode designation signal generated by the video synthesis operation unit to the server.

  In the mobile device of the aspect as described above, one image signal acquired by one imaging unit arranged so that a speaker can be included in the imaging field of view and the one imaging unit differ in the imaging field of view. A communication network from a mobile device on the transmission side configured to obtain a multiplexed video signal that is multiplexed in a separable manner with other video signals acquired by other imaging means arranged to A display mode when the multiplexed video signal supplied via the video signal is received and each reproduction video based on the one video signal and the other video signal is displayed together on a predetermined video display unit A video composition mode designation signal is received (this reception may be reception of a video synthesis mode designation signal emitted from a mobile device that is a communication partner of the above-mentioned transmitting-side mobile device), and the received multiplexed video Signal to one video signal and other video signal A server that performs video composition processing of the separated video signal and the other video signal so as to form a composite video signal that conforms to a display mode defined by the video composite mode designation signal, and the communication network. It is comprised so that it may be tied.

The video compositing mode designation signal transmission unit of the own device displays on the mobile device on the transmission side in response to an external operation by the operator, for example, in the video compositing operation unit of the own device with respect to the server having the above-described function. The video composition mode designation signal generated without affecting is supplied.
The composite video signal receiving unit of the own device receives the supplied video synthesis mode designation signal and receives the synthesized video signal generated by the server so as to conform to the display mode defined by the video synthesis mode designation signal. Receive.
Therefore, the received composite video signal can display the playback video in a manner that matches the setting by the external operation on the video composition operation unit.

  In a server according to still another aspect of the present invention, one video signal acquired by one imaging means arranged so that a speaker can be included in the imaging visual field, and the one imaging means has an imaging visual field. Communication from a mobile device on the transmission side configured to obtain a multiplexed video signal that is multiplexed in a form that can be separated from other video signals acquired by other imaging means arranged differently A multiplexed video signal receiving unit that receives the multiplexed video signal supplied via the network, and each reproduced video based on the one video signal and the other video signal are displayed together on a predetermined video display unit A video composition mode designation signal receiving unit for receiving a video synthesis mode designation signal for defining a display mode, and the multiplexed video signal received by the multiplexed video signal receiving unit as one video signal and another video signal Before being separated Video synthesis processing for synthesizing one video signal and another video signal so as to form a synthesized video signal that conforms to a display mode defined by the video synthesis mode designation signal received by the video synthesis mode designation signal receiver And a section.

In the server as described above, in the multiplexed video signal receiving unit, one video signal acquired by one imaging unit arranged so as to include a speaker in the imaging field of view, and the one imaging unit Mobile device on the transmission side configured to obtain a multiplexed video signal that is multiplexed with other video signals acquired by other imaging means arranged so as to have different imaging fields of view The multiplexed video signal supplied via the communication network is received.
In the video composition mode designation signal receiving unit, a video synthesis mode designation signal for defining a display mode when displaying each reproduced video based on the one video signal and the other video signal on a predetermined video display unit. Receive.

Further, in the video composition processing unit, the multiplexed video signal received by the multiplexed video signal receiving unit is separated into one video signal and another video signal, and the separated one video signal and the other Are combined so as to form a composite video signal suitable for a display mode defined by the video composite mode designation signal received by the video synthesis mode designation signal receiving unit.
Since the above-mentioned server functions to cover the above-described processing in the video composition processing unit, it is possible to reduce the configuration corresponding to the functional unit in the video composition processing unit in the mobile device connected through the communication network. In addition, it opens up the possibility of being able to perform complex processing that cannot be performed by ordinary mobile devices.

On the other hand, the video reproduction method in the mobile device of the present invention includes one video signal acquired by one imaging means arranged so as to include one or a plurality of speakers in the imaging visual field, the one imaging means, Is a communication network from a mobile device on the transmission side that multiplexes other video signals acquired by other imaging means arranged to have different imaging fields of view into a separable form to obtain multiplexed video signals. A video playback method in a mobile device that receives the multiplexed video signal supplied via
A multiplexed video signal receiving step for receiving the multiplexed video signal from the communication network, and the multiplexed video signal received in the multiplexed video signal receiving step into the one video signal and the other video signal. A signal separation step of separating, a video display step of displaying each video based on the one video signal and the other video signal separated in the signal separation step, the one video signal and the other Video display operation step of operating the display mode in the video display step of each reproduced video based on the video signal without affecting the display on the transmitting mobile device.

In the video playback method in the mobile device as described above, the multiplexed video generated by separably multiplexing the video signals respectively acquired by one imaging means and other imaging means having different imaging fields of view. When the signal is supplied from the mobile device on the transmission side via the communication network, the supplied multiplexed video signal is received in the multiplexed video signal reception step.
The received multiplexed video signal is separated into the one video signal and the other video signal in a signal separation step.

Each video based on the one video signal and the other video signal separated in the signal separation step is displayed in a video display step.
In the video display operation step, the display mode of each playback video based on the one video signal and the other video signal in the display in the video display step without affecting the display in the mobile device on the transmission side Manipulate.
Accordingly, it is possible to display the reproduced video in a desired mode of the receiving speaker while preventing the transmitting speaker from recognizing it.

  According to the present invention, a playback video is displayed in a desired manner on the receiving side, while not being recognized by the transmitting speaker, thereby damaging the pride of the transmitting speaker. Rather, a mobile device is realized that can observe video according to the desire of the speaker on the receiving side.

Hereinafter, the present invention will be clarified by describing embodiments of the present invention in detail with reference to the drawings.
(First embodiment)
FIG. 1 is a functional block diagram showing the configuration of a mobile device as one embodiment of the present invention. The mobile device in FIG. 1 takes the form of a mobile phone with a camera configured to function as a terminal device of a videophone system.
The videophone is performed by two-way communication. In FIG. 1, for convenience of explanation, the configuration regarding each element when functioning as the mobile device 110 on the transmission side is depicted on the upper side of the wireless communication network 150 in the figure, and received. A configuration regarding each element in the case of functioning as a mobile station 130 on the side is depicted below the wireless communication network 150 in the figure.
As will be described in detail later with reference to the drawings, the above-described mobile device in the form of a camera-equipped mobile phone generally includes each element when functioning as the mobile device 110 on the transmission side and the mobile device on the reception side. Each element when functioning as 130 is configured to have both.

(Configuration of mobile device on transmission side)
The transmitting-side mobile device 110 includes an inner video input unit 111 that constitutes one imaging unit arranged so that a speaker can be included in the imaging field of view, and the imaging field of view is different from that of the one imaging unit. Provided in correspondence with each other so as to encode the video signals respectively acquired by the outer video input unit 112, which constitutes another imaging means, the inner video input unit 111, and the outer video input unit 112 described above. The video encoding units 113 and 114 are provided.
In these video encoding units 113 and 114, for example, a predetermined encoding process such as MPEG4 is executed (MPEG: Moving Picture Experts Group).

Furthermore, a multiplexing unit 115 that multiplexes each video signal encoded by the video encoding units 113 and 114 into a separable form to obtain a multiplexed video signal, and a multiplexed video obtained by the multiplexing unit 115 A signal (with audio also multiplexed as will be described later) is transmitted to the other mobile device through the wireless communication network 150, and when the own device functions as the receiving mobile device, the multiplexed video signal is transmitted from the other mobile device. The videophone transmission / reception processing unit 116 for receiving the message is provided.
In addition, an audio input unit 117 configured to include a microphone or the like for detecting the voice of the speaker is provided, and an audio signal that is an output of the audio input unit 117 is also supplied to the multiplexing unit 115 to be a multiplexed video signal. Is multiplexed.

(Configuration of receiving mobile unit)
On the other hand, the mobile station 130 on the receiving side receives the multiplexed video signal received by the videophone transmission / reception processing unit 131 and the videophone transmission / reception processing unit 131 for transmitting and receiving the multiplexed video signal. The video separator 132 functions as a signal separator that separates the video signal and separates the video signal and the above-described audio signal, and decodes one video signal and the other video signal separated by the video separator 132. Decoding processing unit 133 to be converted, video display unit 134 to display each video based on the decoded video signal, and each reproduced video based on the one video signal and the other video signal in video display unit 134 The video display mode designation unit 135 is provided for operating the display mode without affecting the display on the mobile device 110 on the transmission side.
Note that the audio signal separated by the video separation unit 132 is supplied to an audio output unit 136 including, for example, a speaker, and is output here as audio.

(Camera placement on mobile device on the transmitting side)
FIG. 2 is a diagram for explaining the arrangement of the inner video input unit 111 and the outer video input unit 112 described above (attachment status of the cameras 111a and 112a) in the mobile device 110. FIG.
The inner video input unit 111 constituting one imaging means in the mobile device 110 on the transmission side is displayed on the display unit 134 (although the illustration of the mobile device 110 on the transmission side is omitted in FIG. 1 described above, This is the same as that in the mobile station 130, and this reference numeral is attached for convenience in FIG. 2), and the transmitting speaker (its face) is imaged in the imaging field of view V1. So as to include the camera 111a mounted inward.

As described above, the camera 111a attached inward is suitable for photographing the speaker himself because he can shoot while looking at the liquid crystal screen which is the display unit 134 of the mobile device. In videophones, in order to display the sender's speaker (its face) on the other party, it is common to photograph using the inner camera 111a.
On the other hand, the outer video input unit 112 constituting the other imaging means described in relation to the mobile device 110 on the transmitting side is attached outward so as to have an imaging field of view V2 different from the inner video input unit 111 described above. The camera 112a is included.

In this example, the direction of the camera 112a of the outer video input unit 112 is set so as to capture the opposite direction to the camera 111a of the inner video input unit 111. When the speaker on the transmitting side (the face) is being photographed, the camera 112a of the outer video input unit 112 can generally photograph the front view of the speaker.
When a mobile device including the two cameras 111a and 112a arranged as described above is used as a video phone, an outer camera (outside It is common to switch to the camera 112a) of the video input unit 112 and shoot.

(Video signal multiplexing)
In the above configuration, the multiplexing unit 115 of the transmitting-side mobile device 110 multiplexes each video signal encoded by the video encoding units 113 and 114 into a separable form as described above. For example, in the case of 3G-324M, this multiplexing processing is performed in accordance with H.223 Annex B.
At this time, the control signal of H.245 is used in advance to define an index for distinguishing between two mobile cameras applied to videophones, which is the video from either of the two cameras (either the inner video or the outer video). Keep it.

FIG. 3 is a diagram illustrating a format of moving image data generated by the multiplexing process executed by the multiplexing unit 115.
As shown in FIG. 3, if the first bit of the moving image data is “1”, the video by the camera of the inner video input unit 111 (for example, it is highly likely that the face of the transmitting speaker is mainly included), “ If it is “0”, it is defined as a video by the camera of the outer video input unit 112 (for example, a probability that the scene developed mainly in the forward view of the transmitting speaker is high).
Multiplexed video data multiplexed in such a format as shown in FIG. 3 in which the multiplexing unit 115 can distinguish two videos (the audio signal is further multiplexed as described above) It is supplied to the videophone transmission / reception processing unit 116, and further transmitted from the videophone transmission / reception processing unit 116 to the designated destination (reception-side mobile device) via the wireless communication network 150.

The videophone transmission / reception processing unit 131 of the receiving-side mobile device 130 receives the videophone data transmitted from the transmitting-side mobile device 110 and supplies it to the video separating unit 132.
The video separation unit 132 captures the received data received from the videophone transmission / reception processing unit 201 with the video captured by the camera of the inner video input unit 111 and the camera of the outer video input unit 112 according to the multiplexing method of the transmission source. Separated into video.

  For example, in the case of 3G-324M, it is multiplexed in H.223 Annex B, and when the first bit of the moving image data is “1” at the transmission source, the video shot by the camera of the inner video input unit 111 (hereinafter referred to as appropriate) In the case of FIG. 3, which is “face image”), and in the case of “0”, the image is taken by the camera of the outer image input unit 112 (hereinafter referred to as “landscape image” as appropriate). Are separated into video data photographed by the camera of the inner video input unit 111 and video data photographed by the camera of the outer video input unit 112.

(Video display mode)
FIG. 4 is a diagram for explaining the display mode data that defines the video display mode held by the video display mode designating unit 135.
As understood with reference to FIG. 4, the combined pattern, the video size / position, and other processing of both videos related to the display of “face video” and “scenery video” on the video display unit 134 are as follows: It is defined by a plurality of display mode data held in the video display mode designating unit 135.
The receiving speaker performs an operation on an operation unit (to be described later) on the mobile device on the receiving side, so that any desired combination pattern, video size / position, and other processing can be displayed. Select by identifying data.

The display mode data thus selected is supplied to the video display unit 134, and the video by the synthesis pattern, video size / position, and other processing according to the request of the receiving speaker is displayed on the video display unit 134.
That is, the video display unit 134 performs video composition based on the display mode data from the video display mode instruction unit 135, the data of the “face video” and the data of the “view video” separated by the video separation unit 132. The video is displayed on the liquid crystal screen of the video display unit 134.
The composition pattern, the image size / position, and other processing itself as illustrated in FIG. 4 are general existing image processing techniques that are also applied to image composition related to television, image processing software, and the like.
FIG. 4 is an example, and it is possible to increase the composite pattern and the video special effect according to the capability of the mobile device.

(Configuration and operation of mobile device)
FIG. 5 is a functional block diagram of a mobile device having both functions of the transmission-side mobile device and the reception-side mobile device in FIG. In FIG. 1 described above, for convenience of explanation, a portion functioning as a transmitting-side mobile device (the portion described above as the transmitting mobile device 110) and a portion functioning as a receiving-side mobile device (as the receiving-side mobile device 130). The above-described part) is shown as each corresponding part of one mobile unit and another mobile unit, but the mobile unit applied to the videophone is a part that functions as a mobile unit on the transmission side as shown in FIG. And a portion functioning as a mobile device on the receiving side.

  In FIG. 5, the corresponding parts to those of FIG. 1 described above are shown with the same reference numerals. For example, the system controller 510 mainly composed of a microprocessor is connected to each corresponding function unit by a signal line (bus) 511 and manages these function units in an integrated manner, and has already been described at a predetermined timing. It is also configured to operate as a decoding / decoding function unit that executes the encoding and decoding processes.

That is, each of them can function in substantially the same manner as shown in FIG. 1, and the inner video input unit 111, the outer video input unit 112, the audio input unit 117, the multiplexing unit 115, and the videophone transmission / reception processing unit 131 (116). ), The video separation unit 132, the audio output unit 136, and the video display unit 134 are connected to the system controller 510 through the bus 511.
Further, an operation unit 512 including operation keys or a touch panel for receiving various external operations from the operator is connected to the system controller 510 through a bus 511.

Next, the operation of the mobile device as the first embodiment of the present invention having the configuration of FIG. 5 will be described with reference to FIG.
FIG. 6 is a sequence diagram showing the operation of the mobile device according to the embodiment of the present invention. In this description of the operation, as a mobile device that is used when making a videophone call and uses both the transmitting side and the receiving side as appropriate, the configuration shown in FIG. 5 is used. Is assumed to apply.
In operating the mobile device (FIG. 5) according to the present embodiment, first, each video signal (for example, “face” acquired by simultaneously capturing with each camera of the inner video input unit 111 and the outer video input unit 112 is used. Video "and" scenery video "are supplied to the video encoding function unit (corresponding to the video encoding units 103 and 104 in FIG. 1) of the system controller 510 (S601, S602).

The system controller 510 (its video encoding function) executes a predetermined encoding process based on MPEG4 or the like on the video signal supplied in S601 and S602 described above, and supplies the encoded video to the multiplexing unit 115. (S603).
Video signals of, for example, “face video” and “landscape video” from the cameras of the inner video input unit 111 and the outer video input unit 112 are multiplexed by the multiplexing unit 115, for example, in the case of 3G-324M, H.223. Multiplexed in Annex B.
In this multiplexing, multiplexing is performed in a format in which it is discriminated whether the image is taken by the camera of the inner video input unit 111 or the outer video input unit 112.
That is, for example, in the case of 3G-324M, it is converted according to H.223 Annex B, but by defining the protocol between both mobile stations in advance by the control information of H.245, refer to FIG. As described above, it is possible to determine which camera image is used according to whether the first bit of the moving image data portion is “1” or “0”.

The multiplexed data is supplied to the videophone transmission / reception processing unit 116 (in this case, the transmission function) having required functions for both transmission and reception (S604).
The videophone transmission / reception processing unit 116 (in this case, the transmission function) transmits the transmission data to the destination terminal (receiving-side mobile device) via the communication network (S605).
In the mobile device on the receiving side, the videophone data transmission / reception processing unit 131 (shown in the same functional block as the videophone transmission / reception processing unit 116 in FIG. 5) receives the videophone data (S605).
The videophone transmission / reception processing unit 131 (in this case, the reception function) transfers data relating to the video to the video separation unit 132 (S606).

Upon receiving the video data, the video separation unit 132 separates the video data from the cameras of the inner video input unit 111 and the outer video input unit 112 according to the method multiplexed by the multiplexing unit 115 in the transmitting mobile device. The audio data is also separated, and the separated audio data is transferred to the audio output unit 136.
Among the operation unit 512 and the system controller 510, a function unit corresponding to the video display mode designation unit 135 in FIG. For example, an instruction is given as to how the video signals of, for example, “face video” and “scenery video” from each camera of the outer video input unit 112 are combined and displayed (S607).

The video separation unit 132 supplies the video data separated into the video data of each camera of the inner video input unit 111 and the outer video input unit 112 to the video display unit 134.
Upon receiving this video data, the video display unit 134 synthesizes and displays the images from the cameras of the inner video input unit 111 and the outer video input unit 112 in a mode instructed by the operator from the operation unit 512 ( S608). Audio is output from the audio output unit 136 in synchronization with this display.

Note that among the operation unit 512 and the system controller 510, functional units corresponding to the video display mode designation unit 135 of FIG. 1 are the cameras of the inner video input unit 111 and the outer video input unit 112 as shown in FIG. Is stored in a form that can be specified in accordance with an operator's selection operation, which defines a plurality of display modes related to the video synthesis pattern, video size / position, and other processing.
The above-described functional units of the operation unit 512 and the system controller 510 read out data corresponding to the selection operation of the operator (recipient speaker) from the data thus held and supplies the data to the video display unit 134. The video display unit 134 realizes video display in a manner according to the desire of the operator (reception-side speaker).
It should be noted that the video display mode selection operation by the operator (speaker on the receiving side) can be accepted even during video reception and can be updated at any time.

Further, when an operation for designating a display mode on the operation unit 512 is not performed, the default setting in the mobile device is applied.
For example, the composite pattern on the display screen is divided vertically, and the video size / position is the video from the camera of the outer video input unit 112 (“landscape video”) on the upper side and the video (“face” from the camera of the inner video input unit 111 on the lower side. The video size ratio is 5: 5, and no other processing is applied.
In the video display unit 134, video synthesis processing is performed in accordance with an operation for designating a display mode on the operation unit 512.
Note that the data defining the display mode illustrated in FIG. 4 can hold data unique to each mobile device, and it is possible to increase composite patterns, video special effects, etc. according to the capabilities of the mobile device. It is.

According to the embodiment described above, it is possible to multiplex the images from the two cameras of the mobile device on the transmission side and transmit it to the other mobile device via video communication via the communication network. In the mobile device on the receiving side, the received videophone data is separated into a video by the camera of the inner video input unit 111 (“face video”) and a video by the camera of the outer video input unit 112 (“scenery video”). The receiving side speaker can select and observe the display mode according to the request of the receiving side, and the transmitting side speaker does not recognize the display mode selection status of the receiving side speaker. In principle, it does not occur that the selection of the display mode in the other speaker affects the identification of the transmitting speaker.
Therefore, the video sent from the speaker on the transmission side can be freely combined and displayed on the mobile terminal screen (video display screen of the mobile device) according to the intention of the receiver, and a relatively large display area is allocated to the desired video. Etc. can be selected to realize video communication with a higher sense of presence than in this type of conventional apparatus.

(Second Embodiment)
FIG. 7 is a functional block diagram showing another embodiment of the mobile device of the present invention. The mobile device in FIG. 7 takes the form of a mobile phone with a camera configured to function as a terminal device of a videophone system, similar to that in FIG.
In FIG. 7, for the sake of convenience of explanation, the configuration regarding each element when functioning as the mobile device 710 on the transmission side is depicted on the upper side of the wireless communication network 750 in FIG. A configuration regarding each element in the case of functioning as a mobile station 130 on the side is depicted below the wireless communication network 150 in the figure.
As will be described later, the above-described mobile device that takes the form of a camera-equipped mobile phone has, as a whole, each element when it functions as a mobile device 710 on the transmission side and each mobile device 730 when it functions as a mobile device 730 on the reception side It is configured to have both elements.

(Configuration of mobile device on transmission side)
The transmitting-side mobile device 710 has an inner video input unit 711 that constitutes one imaging means arranged so that a speaker can be included in the imaging visual field V1, and an imaging visual field V2 that is different from the one imaging means. And an outer video input unit 712 that constitutes another imaging means, and a video encoding unit 713 provided corresponding to each of the video signals respectively acquired by the outer video input unit 712. And 714.
In these video encoding units 713 and 714, for example, a predetermined encoding process such as MPEG4 is executed.

Furthermore, a video composition unit 715 for synthesizing each video signal encoded by these video encoding units 713 and 714 in a manner defined by a video synthesis control signal described later to obtain a composite video signal, and this video composition unit A synthesized video signal obtained by multiplexing the voice detected by the voice input unit 716 including the microphone for detecting the voice of the speaker, and the voice multiplexed by the multiplexing unit 717. Is transmitted to the other party's mobile device through the wireless communication network 750, and a videophone transmission / reception processing unit 718 is provided for receiving a composite video signal from the other party's mobile device when the own device functions as a receiving-side mobile device. .
In this embodiment, in particular, the control signal transmission / reception unit 719 configured to be able to exchange various control signals with the other party's mobile device through the wireless communication network 750 and the control signal transmission / reception unit 719 receive the control signal. There is provided a video composition mode selection unit 720 that generates the above-described video synthesis control signal based on a video synthesis mode designation signal among the control signals and supplies the video synthesis mode signal to the video synthesis unit 715.

(Configuration of receiving mobile unit)
On the other hand, the mobile device 730 on the receiving side separates the above audio signal from the synthesized video signal received by the videophone transmission / reception processing unit 731 and the videophone transmission / reception processing unit 731 for transmitting and receiving the synthesized video signal. Unit 732, a decoding processing unit 733 for decoding the synthesized video signal separated by the audio / video separating unit 732, a video display unit 734 for displaying a synthesized video based on the decoded synthesized video signal, and an audio / video An audio output unit 737 configured to include, for example, a speaker that outputs audio based on the above-described audio signal separated by the video separation unit 732 is provided.

In addition, a video display mode designating unit 735 for operating the above-described composite video signal display mode without affecting the display on the mobile device 710 on the transmission side is provided, and is generated according to the operation on the video display mode designating unit 735. The video composition mode designation signal to be transmitted is transmitted from the control signal transmission / reception unit 736 to the control signal transmission / reception unit 719 of the counterpart mobile unit 710 through the wireless communication network 750.
In the mobile device 730 on the receiving side in this example, the video display mode designating unit 735 does not directly act on the video display unit 734 of its own device by its output, A video compositing mode designation signal that defines the above-described video compositing mode is sent to the transmitting-side mobile device 710 by an operation according to the user's request. A video signal suitable for the display of the mode is synthesized in advance.

In the mobile device that has received the synthesized video signal thus synthesized, it is possible to perform an operation according to a desired situation in advance by reproducing it as it is without performing an operation such as separation of multiplexed video or designation of a display mode using the video. The display in the mode selected by can be obtained.
In the mobile device 710 on the transmission side in FIG. 7, regarding the arrangement of the inner video input unit 711 and the outer video input unit 712 (the installation state of those cameras), the inner side in the mobile device in FIG. The arrangement of the video input unit 311 and the outer video input unit 312 (installation status of those cameras) is the same as described.

The video display mode designation unit 735 of the mobile device 730 on the receiving side has display mode data that defines the same video display mode as described with reference to FIG.
As described above with reference to FIG. 4, the combined pattern, the video size / position, and other processing of both videos related to the display of “face video” and “scenery video” in the video display unit 734 are as follows: It is defined by a plurality of display mode data held in the video display mode designating unit 735.

In the case of the embodiment of FIG. 7, the plurality of display mode data held in the video display mode designating unit 735 form a video synthesis mode designation signal that defines the above-described video synthesis mode. A video composition mode designation signal is transmitted from the control signal transmission / reception unit 736 to the above-described control signal transmission / reception unit 719 of the counterpart mobile device 710 through the wireless communication network 750.
The video synthesis mode designation signal received by the control signal transmission / reception unit 719 is supplied to the video synthesis mode selection unit 720.
The video composition mode selection unit 720 generates the above-described video composition control signal based on the supplied video composition mode designation signal, and supplies the generated video composition control signal to the video composition unit 715.
In the video synthesizing unit 715, the video signals encoded by the video encoding units 713 and 714 described above are synthesized in a manner defined by the video synthesis control signal to obtain a synthesized video signal.

(Configuration and operation of mobile device)
FIG. 8 is a functional block diagram showing a configuration of a mobile device having both functions of the mobile device on the transmission side and the mobile device on the reception side in FIG. As described above, in FIG. 7, for convenience of explanation, a portion that functions as a transmitting-side mobile device (the portion described above as a transmitting mobile device 710) and a portion that functions as a receiving-side mobile device (movement on the receiving side). 8 is shown as corresponding parts of one mobile station and another mobile station. However, the mobile station of the present embodiment applied to a videophone can transmit as shown in FIG. Both a portion functioning as a mobile device on the side and a portion functioning as a mobile device on the reception side are provided.

In FIG. 8, the corresponding parts to those of FIG. 7 described above are shown with the same reference numerals.
For example, the system controller 810 mainly composed of a microprocessor is connected to each corresponding functional unit by a signal line (bus) 811 so as to comprehensively manage each functional unit, and at the predetermined timing by itself. It is also configured to operate as a decoding / decoding function unit that executes the encoding and decoding processes.

That is, each of them can function in substantially the same manner as shown in FIG. 7, and an inner video input unit 711, an outer video input unit 712, an audio input unit 716, a video composition mode selection unit 720, a multiplexing unit 717, a television set A telephone transmission / reception processing unit 718 (731), a video separation unit 732, a control signal transmission / reception unit 719 (736), an audio output unit 737, and a video display unit 734 are connected to the system controller 810 through a bus 811.
Furthermore, an operation unit 812 configured to include operation keys or a touch panel for receiving various external operations from the operator is connected to the system controller 810 through a bus 811.

Next, the operation of the mobile device as the second embodiment of the present invention having the configuration of FIG. 8 will be described with reference to FIG.
FIG. 9 is a sequence diagram showing the operation of the mobile device of FIG. In this description of the operation, as a mobile device used when making a videophone call and changing the positions of both the transmitting side and the receiving side as appropriate, the mobile unit is configured as shown in FIG. Is assumed to apply.
In operating the mobile device (FIG. 8) according to the present embodiment, first, each video signal (for example, “face” obtained by simultaneously capturing images with each camera of the inner video input unit 711 and the outer video input unit 712 is used. The video signals of “the video of the image” and “the video of the scenery”) are supplied to the video encoding function unit (corresponding to the video encoding units 713 and 714 in FIG. 7) of the system controller 810 (S901, S902).

The speaker on the receiving side performs an operation on the operation unit 812 to select (designate) a desired display mode when the video display mode is to be changed.
The function unit corresponding to the operation unit 812 and the image display mode specifying unit 735 in FIG. 7 in the system controller 810 generates a video composition mode specifying signal in response to an external operation by the operator, and the generated video composition mode specifying signal is displayed. The control signal is transmitted to the transmission / reception unit 736 (S903).

The control signal transmission / reception unit 736 transmits the video composition mode designation signal to the control signal transmission / reception unit 719 (in FIG. 8, the same functional block as the control signal transmission / reception unit 736) in the other party's mobile unit (that is, the transmission side mobile unit) (S904).
When receiving the video composition mode designation signal, the control signal transmission / reception unit 719 (736) in the transmitting-side mobile device transfers the video synthesis mode designation signal to the video synthesis mode selection unit 720 (S905).

The video synthesis mode selection unit 720 generates a video synthesis control signal based on the video synthesis mode designation signal supplied from the control signal transmission / reception unit 719 (736) and supplies the video synthesis control signal to the video synthesis unit 715 (S906).
This video composition control signal synthesizes video signals of, for example, “face video” and “scenery video” by the respective cameras of the inner video input unit 711 and the outer video input unit 712 in the video display unit 734. The mode of the video signal synthesis processing according to whether the video is displayed is specified.

Note that the display mode selection operation by the operator can be accepted even during video reception, and can be updated at any time. Further, when there is no input from the operator, the default setting in this mobile device is applied.
For example, the operator can specify that the composite pattern is vertically divided, the video size / position is the outer video on the upper side, the inner video on the lower side, the ratio is 5: 5, and no other processing is applied.

In a function unit corresponding to the video encoding units 713 and 714 in FIG. 7 in the system controller 810, video data from each camera of the inner video input unit 101 and the outer video input unit 102 is encoded and supplied to the video synthesis unit 715 ( S907).
In the video synthesis unit 715, the encoded video data obtained from the above-described functional unit of the system controller 810 is in a mode defined by the video synthesis control signal supplied from the video synthesis mode selection unit 720 as described above. It is synthesized into one video by synthesizing by video synthesis processing.

The synthesized video signal (data) synthesized by the video synthesizing unit 715 is further transmitted by the multiplexing unit 717 in which the audio signal related to the voice of the speaker generated by the audio input unit 716 is multiplexed. (S908).
In the multiplexing process in the multiplexing unit 717, the received composite video data is converted into transmission data in accordance with a predetermined multiplexing method. For example, in the case of 3G-324M, conversion is performed according to H.223 Annex B.
The data multiplexed as described above is transferred to the videophone transmission / reception processing unit 718 (shown in the same functional block as the videophone transmission / reception processing unit 731 in FIG. 8) (S909).

The videophone transmission / reception processing unit 718 transmits the transmission data to the mobile device on the receiving side via the communication network (S910).
In the mobile device on the receiving side, the videophone transmission / reception processing unit 731 receives the videophone data multiplexed as described above (S910).
The above-described videophone data received by the videophone transmission / reception processing unit 731 is transferred to the audio / video separation unit 732 (S911).
The audio / video separation unit 732 separates the data related to the audio and video in the multiplexed data described above, and is further decoded by the decoding processing function unit of the system controller 810 and supplied to the video display unit 734. (S912).

In the video display unit 734, the received operator (speaker) selects, as desired, by the operation from the operation unit 812 (video display mode designation unit 735 in FIG. 7) by using the decoded composite video. The video is displayed. Audio is output from the audio output unit 737 in synchronization with this display.
Also in the second embodiment described above, it is possible to synthesize and multiplex the images from the two cameras of the mobile device on the transmission side and transmit them to the counterpart mobile device via video communication via a communication network.

In this embodiment, in particular, in the mobile device on the receiving side, the video (“face video”) obtained by the camera of the inner video input unit 111 and the video (“face video”) acquired by the mobile device on the transmitting side ( When it is desired to change the display mode with “scenery video”), the operation unit 812 (video display mode designating unit 735 in FIG. 7) is operated to make the display mode desired.
By sending the video composition mode designation signal generated by this operation to the mobile device on the transmission side, a video synthesis control signal corresponding to the video synthesis mode designation signal is generated in the mobile device on the transmission side, and the generated video The video signals from both cameras are synthesized in a manner defined by the synthesis control signal.

Therefore, the composite video signal already synthesized in this way is supplied from the transmission-side mobile unit to the reception-side mobile unit, so that the reception-side mobile unit can display a desired display mode. The video is displayed on the screen.
In this case, in particular, the selection of the display mode of the receiving speaker is not recognized by the transmitting speaker, so that the selection of the display mode of the receiving speaker may affect the sentiment of the transmitting speaker. Nothing happens in principle.

In addition, a synthesized video signal (data) that can be obtained in a desired manner according to the receiver's free intention is obtained from the mobile device on the transmission side, and the mobile terminal screen (of the mobile device on the reception side) is obtained. Video display screen), and by selecting a relatively large display area for a desired video, etc., it is possible to realize video communication with a higher sense of presence than in a conventional device of this type. Can do.
Furthermore, compared with the case where all the screen data of each video acquired by both cameras of the transmitting side mobile device is transmitted, a composite video signal sufficient to reproduce one screen as a composite result. Since (data) can be obtained from the mobile device on the transmission side, there is an effect that the required amount of communication data can be relatively reduced.

(Third embodiment)
FIG. 10 is a functional block diagram showing the third embodiment of the present invention. The features of the embodiment of FIG. 10 can be easily understood by contrasting with the embodiment of FIG.
That is, in the embodiment of FIG. 10, the configuration of the mobile device on the reception side is substantially the same as that of the mobile device 730 on the reception side in the embodiment of FIG. Reference numerals are attached.
On the other hand, the configuration of mobile station 1010 on the transmission side is simplified as compared with mobile station 710 on the transmission side in the embodiment of FIG.

This is connected to the wireless communication network 1050 for the functions of the control signal transmission / reception unit 719, video composition mode selection unit 720, and video composition unit 715 of the mobile device 710 on the transmission side in the embodiment of FIG. Therefore, the configuration corresponding to each functional unit entrusted to the server 1100 is reduced.
A video composition mode designation signal for defining a video display mode on the video display unit 734 generated by the operation of the receiving speaker on the video display mode designation unit 735 of the mobile device 730 on the reception side is transmitted from the control signal transmission / reception unit 736. It is configured to be sent to the server 1100 through the wireless communication network 1050.

The server 1100 includes a control signal transmission / reception unit 719, a video composition mode selection unit 720 provided in the mobile device 710 on the transmission side in the embodiment of FIG. 7, as described with reference to FIGS. A data processing function corresponding to the function of each unit of the video composition unit 715 is provided.
For this reason, the mobile station 1010 on the transmission side has a configuration corresponding to each part of the mobile station 110 on the transmission side in the embodiment of FIG. In order to avoid duplication of explanation and facilitate understanding, the mobile station 1010 on the transmission side in FIG. 10 is assigned a code in the 1000s corresponding to the reference codes in the 100s in FIG. These numbers are shown so that they correspond with the corresponding parts in FIG.

  That is, the transmitting-side mobile device 1010 has a different imaging field of view from the inner image input unit 1011 that constitutes one imaging unit arranged so that a speaker can be included in the imaging field of view. The video signal acquired by the outer video input unit 1012, the inner video input unit 1011 and the outer video input unit 1012, which are arranged in the manner described above and constitute another imaging unit, is encoded. Video encoders 1013 and 1014 provided.

  Further, a multiplexing unit 1015 that multiplexes each video signal encoded by these video encoding units 1013 and 1014 into a separable form to obtain a multiplexed video signal, and a multiplexed video obtained by this multiplexing unit 1015 A videophone transmission / reception processing unit 1016 is provided for transmitting a signal to the other mobile device through the wireless communication network 1050 and receiving a multiplexed video signal from the other mobile device when the own device functions as a receiving mobile device. Yes.

In addition, an audio input unit 1017 configured to include a microphone or the like for detecting the voice of the speaker is provided, and an audio signal that is an output of the audio input unit 1017 is also supplied to the multiplexing unit 1015 to be multiplexed video signals. Is multiplexed.
From the transmitting-side mobile device 1010 having the above-described configuration, the multiplexed video in which the video signals from the two cameras of the inner video input unit 1011 and the outer video input unit 1012 are multiplexed and the audio is also multiplexed. A signal is supplied from the videophone transmission / reception processing unit 1016 to the server 1100 through the external wireless communication network 1050.

As described above, the server 1100 has the functions of the control signal transmission / reception unit 719, the video composition mode selection unit 720, and the video composition unit 715 included in the mobile device 710 on the transmission side in the embodiment of FIG. The data processing function is provided.
FIG. 13 is a functional block diagram of the server 1100. The server 1100 includes an interface unit 1100a having a function of the control signal transmission / reception unit 719 and a function related to video data transmission / reception processing of the videophone transmission / reception processing unit 718 (731) in the mobile device shown in FIG. A signal relating to the above-described video and a video composition mode designation signal are exchanged with the outside through the unit 1100a.

That is, the video composition mode designation signal supplied from the receiving-side mobile device 730 (its control signal transmission / reception unit 736) via the wireless communication network 1050 corresponds to the video composition mode selection unit 720 of FIG. 7 through the interface unit 1100a. The data is input to the composition mode selection unit 720a (transmission corresponding to S1209 described later).
Upon receiving the video synthesis mode designation signal, the video synthesis mode selection unit 720a generates a video synthesis control signal corresponding to the video synthesis mode designation signal and supplies it to the video synthesis unit 715a corresponding to the video synthesis unit 715 in FIG. To do.

On the other hand, a multiplexed video signal supplied from the transmitting-side mobile device 710 (its videophone transmission / reception unit 718) via the wireless communication network 1050 is input to the above-described video composition unit 715a through the interface unit 1100a (S1206 described later). Equivalent to transmission).
The video synthesizing unit 715a generates a synthesized video signal as described above for reproducing a video in a desired mode based on the video synthesis control signal.
The generated composite video signal is sent to the wireless communication network 1050 through the interface unit 1100a (transmission corresponding to S1210 described later).

Here, the video in the desired mode means that the speaker (operator) on the receiving side asks the video mode designating unit 735 before the video is displayed or while observing the displayed video. This is a video in a mode according to an operation for specifying a desired video display mode, and is performed with respect to the composite pattern of both videos, the video size / position, and other processing as exemplified with reference to FIG. The result of the selected operation is reflected.
In the example described above, an image having a display mode selected as desired in relation to the display of the “face image” and the “landscape image” on the image display unit 734 is displayed.

(Configuration and operation of mobile device)
FIG. 11 is a functional block diagram illustrating a configuration of a mobile device having both functions of the mobile device 1010 on the transmission side and the mobile device 730 on the reception side in FIG.
For convenience of explanation, in FIG. 10, a portion functioning as a transmitting-side mobile device (the portion described above as the transmitting mobile device 1010) and a portion functioning as a receiving-side mobile device (described above as the receiving-side mobile device 730. Are shown as corresponding portions of one mobile device and another mobile device, but the mobile device of the present embodiment applied to a videophone functions as a mobile device on the transmission side as shown in FIG. And a portion functioning as a mobile device on the receiving side.

In FIG. 11, the corresponding parts to those in FIG. 10 described above are shown with the same reference numerals.
For example, the system controller 1110 mainly composed of a microprocessor is connected to each corresponding functional unit by a signal line (bus) 1111 so as to comprehensively manage each functional unit, and at the predetermined timing by itself. It is also configured to operate as an encoding / decoding function unit that executes the encoding and decoding processes.

That is, each of them can function in substantially the same manner as shown in FIG. 10, and the inner video input unit 1011, the outer video input unit 1012, the audio input unit 1017, the multiplexing unit 1015, the videophone transmission / reception processing unit 731 (1016). ), An audio / video separation unit 732, a control signal transmission / reception unit 736, an audio output unit 737, and a video display unit 734 are connected to the system controller 1110 through the bus 1111.
Furthermore, an operation unit 1112 configured to include operation keys or a touch panel for receiving various external operations from the operator is connected to the system controller 1110 through the bus 1111. The operation unit 1112 includes a function unit corresponding to a function for receiving an input operation in the video display mode specifying unit 735 of FIG.

Next, the operation of the mobile device as the embodiment of the present invention having the configuration of FIG. 11 will be described with reference to FIG.
FIG. 12 is a sequence diagram showing the operation of the mobile device according to the embodiment of the present invention. In the description of this operation, as a mobile device that is used when making a videophone call and appropriately switches the positions of both the transmitting side and the receiving side, the configuration shown in FIG. 11 is used. Is assumed to apply.
In operating the mobile device (FIG. 11) according to the present embodiment, first, each video signal (for example, “face” obtained by simultaneously capturing images with the cameras of the inner video input unit 1011 and the outer video input unit 1012 is used. The video signals of “the video of the image” and “the video of the scenery”) are supplied to the video encoding function unit (corresponding to the video encoding units 1013 and 1014 in FIG. 10) of the system controller 1110 (S1201, S1202).

The system controller 1110 (its video encoding function) executes a predetermined encoding process based on MPEG4 or the like on the video signal supplied in S1201 and S1202, and supplies the encoded video to the multiplexing unit 1015. (S1203).
The video signals of, for example, “face image” and “landscape image” by the cameras of the inner image input unit 1011 and the outer image input unit 1012 are multiplexed by the multiplexing unit 1015, for example, H.223 in the case of 3G-324M. Multiplexed in Annex B.
In this multiplexing, multiplexing is performed in a format in which it is discriminated whether the image is captured by the camera of the inner image input unit 1011 or the outer image input unit 1012.

That is, for example, in the case of 3G-324M, it is converted according to H.223 Annex B, but by defining the protocol between both mobile stations in advance by the control information of H.245, refer to FIG. As described above, it is possible to determine which camera image is used according to whether the first bit of the moving image data portion is “1” or “0”.
The multiplexed data is supplied to a videophone transmission / reception processing unit 1106 (in this case, a transmission function) having necessary functions for both transmission and reception (S1204).
The videophone transmission / reception processing unit 1106 (in this case, the transmission function) transmits the transmission data to the server via the communication network (S1205).
The server receives the transmission data (videophone data) from the communication network (S1206).

On the other hand, the speaker on the receiving side performs an operation on the operation unit 1112 for specifying a desired display mode when the display mode of the video is to be changed.
The function unit corresponding to the video display mode designation unit 735 in FIG. 10 in the operation unit 1112 and the system controller 1110 generates a video synthesis mode designation signal in response to an external operation by the operator, and the generated video synthesis mode designation signal is displayed. The control signal is transmitted to the transmission / reception unit 736 (S1207).
The control signal transmission / reception unit 736 transmits this video composition mode designation signal to the wireless communication network (S1208).

The above-mentioned video composition mode designation signal is transmitted from the communication network to the server (S1209).
The server executes the synthesizing process for both video and audio on the multiplexed data related to both the video and audio received in S1205, based on the video synthesis mode designation signal received in S1209. Then, a synthesized video signal (which is also multiplexed with audio) as defined by the video synthesis mode designation signal is generated.
The server transmits the composite video signal generated as described above to the wireless communication network (S1210).
The composite video signal transmitted to the wireless communication network in S1210 is transmitted to the videophone transmission / reception unit 731 in the receiving-side mobile device (S1211).
The above videophone data received by the videophone transmission / reception processing unit 731 is transferred to the audio / video separation unit 732 (S1212).

The audio / video separation unit 732 separates the data related to the audio and video in the multiplexed data described above, and is further decoded by the decoding processing function unit of the system controller 810 and supplied to the video display unit 734. (S1213).
In the video display unit 734, the received operator (speaker) selects, as desired, by the operation from the operation unit 1112 (video display mode designation unit 735 in FIG. 10) by using the decoded composite video. The video is displayed. Audio is output from the audio output unit 737 in synchronization with this display.

As described above, also in the third embodiment described with reference to FIG. 10 to FIG. 12, the video from the two cameras of the mobile device on the transmission side is synthesized and multiplexed, and the other party mobile device is transmitted by videophone through the communication network. Can be sent to.
In the third embodiment, in particular, in the mobile device on the receiving side, the video (“face video”) obtained by the camera of the inner video input unit 1011 and the video (“face video”) acquired by the mobile device on the transmitting side ( When it is desired to change the display mode with “scenery video”), the operation unit 1112 (the video display mode designating unit 735 in FIG. 10) is operated to make the display mode desired.

The video composition mode designation signal generated by this operation is supplied to the server 1100 connected to the wireless communication network 1050, and the video from the two cameras described above acquired by the transmitting-side mobile device and supplied to the server is displayed. A composite video signal in a form that has already been combined in the server 1100 is supplied from the server 1100 to the mobile device on the receiving side. In the mobile device on the receiving side, video is displayed in the desired display mode.
Therefore, compared with the case where all the screen data of each image acquired by both cameras of the mobile device on the transmission side is transmitted to the mobile device on the reception side via the wireless communication network, the mobile device on the reception side is transmitted from the server. In the interval up to this point, it is only necessary to transmit a synthesized video signal (data) sufficient to reproduce one screen as a result of synthesis in the server, so that the required amount of communication data can be relatively reduced. There is an effect.

Also in this embodiment, the selection of the display mode in the receiving speaker is not recognized from the transmitting speaker, so the selection of the display mode in the receiving speaker has an influence on the sentiment of the transmitting speaker. Such a thing cannot occur in principle.
In addition, a synthesized video signal (data) that provides a desired display mode already synthesized in a form in line with the receiver's free will is obtained from the server, and the mobile terminal screen (video display screen of the mobile device on the receiving side) It is possible to display a video image with a higher sense of reality than in a conventional device of this type by selecting a relatively large display area for a desired video.

(Invention of video playback method in mobile equipment)
As can be easily understood from the description with reference to FIGS. 1 to 6, the above-described idea of the present invention constitutes the invention of the following video reproduction method in a mobile device.
In other words, one imaging signal acquired by one imaging means (111) arranged so that one or a plurality of speakers can be included in the imaging visual field and the one imaging means are made to have different imaging visual fields. From the mobile device (110) on the transmission side, which multiplexes (115) other video signals acquired by other imaging means (112) arranged in a separate form into a separable form and obtains multiplexed video signals, A video reproduction method in a mobile device (130) that receives the multiplexed video signal supplied via a communication network (150), comprising:
A multiplexed video signal receiving step for receiving the multiplexed video signal from the communication network (150) (131: S605), and the multiplexed video signal received in the multiplexed video signal receiving step is the one video signal; And a signal separation step of separating the signal into the other video signal (132: S606), and a video for displaying each video based on the one video signal and the other video signal separated in the signal separation step The display mode in the display step (134: S608) and the video display step of each playback video based on the one video signal and the other video signal is not affected by the display on the mobile device on the transmission side. And (135: S607) a video display operation step for operating the mobile phone.
Also in this method, it is possible to display the reproduced video in a desired mode of the receiving speaker while preventing it from being recognized by the transmitting speaker.

  In each of the above embodiments, as described with reference to FIG. 2, the example in which two cameras are provided has been described in detail. However, the camera (therefore, the video input unit including the camera and the video signal in the subsequent stage) are described. The system) is not limited to two (two systems), and is provided with more cameras having different imaging fields of view, and when the video is reproduced by them, as in the present invention, the display mode in the mobile device on the receiving side Of course, it may be configured so that it can be set as desired without being recognized by the speaker on the transmission side.

  The present invention is a function for sending both a “face image” of a transmitting speaker photographed by one camera and a “landscape image” of the front view of the speaker photographed by another camera to the receiving side. When making a videophone call using a mobile device equipped with a mobile phone equipped with a display device, the playback side displays a playback video in a desired manner on the receiving side so that it is not recognized by the transmitting side speaker. Even if "video" is handled as a secondary, it can be used to realize a mobile device that can observe video in accordance with the receiver's request without damaging the sender's pride. .

It is a functional block diagram showing the structure of the moving apparatus as the 1st Embodiment of this invention. It is a figure for demonstrating arrangement | positioning (the attachment condition of those cameras) of the inner side image input part and the outer side image input part in a mobile device. It is a figure which illustrates the format of the moving image data produced | generated by the multiplexing process performed in the multiplexing part in a mobile device. It is a figure for demonstrating the display mode data which prescribe | regulate the video display mode which the video display mode designation | designated part has in a mobile device. FIG. 2 is a functional block diagram of a mobile device having both functions of a transmission-side mobile device and a reception-side mobile device in FIG. 1. It is a sequence diagram showing operation | movement of the moving apparatus as embodiment of this invention. It is a functional block diagram showing the 2nd Embodiment of this invention. FIG. 8 is a functional block diagram illustrating a configuration of a mobile device having both functions of the mobile device on the transmission side and the mobile device on the reception side in FIG. 7. It is a sequence diagram showing operation | movement of the mobile apparatus of FIG. It is a functional block diagram showing the 3rd Embodiment of this invention. FIG. 11 is a functional block diagram illustrating a configuration of a mobile device having both functions of the mobile device on the transmission side and the mobile device on the reception side in FIG. 10. It is a sequence diagram showing operation | movement of the moving apparatus as embodiment of this invention. It is a functional block diagram of the server of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 110 ... Transmitting side mobile device 111 ... Inner video input unit 111a ... Inner video input unit camera 112 ... Outer video input unit 112a ... Outer video input unit camera 113 ... Video encoding unit 114 ... Video encoding unit 115 ... Multiplexing 116: Videophone transmission / reception processing unit 117 ... Audio input unit 130 ... Receiver-side mobile device 131 ... Videophone transmission / reception processing unit 132 ... Video separation unit 133 ... Decoding processing unit 134 ... Video display unit 135 ... Video display mode designation Unit 136 ... audio output unit 150 ... wireless communication network 510 ... system controller 511 ... bus 512 ... operation unit 710 ... transmission side mobile device 711 ... inner video input unit 712 ... outer video input unit 713 ... video encoding unit 714 ... video Encoding unit 715 ... Video synthesis unit 715a ... Video synthesis unit 716 ... Audio input unit 717 ... Multiplexing unit 718 ... Videophone transmission / reception processing unit DESCRIPTION OF SYMBOLS 19 ... Control signal transmission / reception part 720 ... Video | video synthetic | combination aspect selection part 720a ... Video | video synthetic | combination aspect selection part 730 ... Mobile station 731 of a receiving side ... Videophone transmission / reception processing part 732 ... Audio | voice / video separation part 733 ... Decoding processing part 734 ... Video | video Display unit 735 ... Video display mode designation unit 736 ... Control signal transmission / reception unit 737 ... Audio output unit 750 ... Wireless communication network 1010 ... Mobile device 1011 on transmission side ... Inner video input unit 1012 ... Outer video input unit 1013 ... Video encoding unit DESCRIPTION OF SYMBOLS 1014 ... Video encoding part 1015 ... Multiplexing part 1016 ... Videophone transmission / reception processing part 1017 ... Audio | voice input part 1050 ... Wireless communication network 1100 ... Server 1100a ... Interface part 1110 ... System controller 1111 ... Bus 1112 ... Operation part

Claims (8)

One video signal acquired by one imaging means arranged so that a speaker can be included in the imaging visual field, and another imaging arranged such that the one imaging means has a different imaging visual field From a mobile device on the transmission side configured to obtain a multiplexed video signal by multiplexing with another video signal acquired by the means in a form that can be separated by a predetermined signal multiplexing processing unit via a communication network A mobile device configured to receive the multiplexed video signal supplied by
A multiplexed video signal receiving unit that receives the multiplexed video signal from the communication network, and a multiplexed video signal received by the multiplexed video signal receiving unit is separated into the one video signal and the other video signal. A signal separation unit, a video display unit that displays each video based on the one video signal and the other video signal separated by the signal separation unit, and the one video signal and the other video signal. And a video display operation unit that operates a display mode of each playback video on the video display unit without affecting the display on the mobile device on the transmission side.   The video display operation unit is configured to select and operate a display area pattern of each playback video based on the one video signal and the other video signal in the video display unit. Item 2. The mobile device according to Item 1. The signal separation unit includes one video signal acquired by the one imaging unit so as to include the face of a speaker who handles the mobile device on the transmission side in the imaging visual field, and the front visual field of the speaker as the imaging visual field. The multiplexed video signal receiving unit of the own device receives the multiplexed video signal generated by multiplexing the other video signal acquired by other imaging means into a form that can be separated by the signal multiplexing processing unit. The received multiplexed video signal is separated into the one video signal and the other video signal,
The video display operation unit is configured to display, in the video display unit, a playback video related to the speaker's face based on the one video signal and a playback video related to the front view of the speaker based on the other video signal. The mobile device according to any one of claims 1 to 2, wherein the mobile device is configured to be operated.   One imaging means arranged so that a speaker can be included in the imaging field, the other imaging means arranged so that the imaging field is different from the one imaging means, and the one imaging means A video synthesizing unit that obtains a synthesized video signal by synthesizing the one video signal obtained by the other imaging signal and the other video signal obtained by the other imaging unit in a mode defined by a predetermined video synthesis control signal; A video synthesis mode selection unit that generates the video synthesis control signal based on a video synthesis mode designation signal supplied from another device via a communication network and supplies the video synthesis control signal to the video synthesis unit; Mobile machine to do. One imaging means arranged so that a speaker can be included in the imaging field, the other imaging means arranged so that the imaging field is different from the one imaging means, and the one imaging means A multiplexing processing unit that multiplexes one video signal acquired by the second imaging signal acquired by the other imaging unit into a separable form to obtain a multiplexed video signal;
Received are a video composition mode designating signal for defining a display mode when each playback video based on the one video signal and the other video signal is displayed together on a predetermined video display unit, and the multiplexed video signal. The multiplexed video signal supplied is separated into one video signal and another video signal, and the separated one video signal and the other video signal are defined by the video composition mode designation signal. And a multiplexed video signal transmitter for transmitting to a server for video synthesis processing so as to form a synthesized video signal suitable for the display mode. One video signal acquired by one imaging means arranged so that a speaker can be included in the imaging visual field, and another imaging arranged such that the one imaging means has a different imaging visual field The multiplexed video signal supplied via a communication network from a mobile device on the transmission side configured to obtain a multiplexed video signal that is multiplexed in a form that can be separated from other video signals obtained by the means And receiving a video composition mode designation signal that defines a display mode when each playback video based on the one video signal and the other video signal is displayed together on a predetermined video display unit, The received multiplexed video signal is separated into one video signal and another video signal, and the separated one video signal and other video signal are displayed by the video composition mode designation signal. Create a composite video signal that matches the mode A configured mobile as tied via the server and the communication network to the video synthesis process,
A synthesized video signal receiving unit that receives the synthesized video signal from the server, and a video synthesis operating unit that generates the video synthesis mode designation signal without affecting the display on the transmitting mobile device according to an external operation; And a video composition mode designation signal transmitting unit that transmits the video synthesis mode designation signal generated by the video synthesis operation unit to the server.   One video signal acquired by one imaging means arranged so that a speaker can be included in the imaging visual field, and another imaging arranged such that the one imaging means has a different imaging visual field The multiplexed video signal supplied via a communication network from a mobile device on the transmission side configured to obtain a multiplexed video signal that is multiplexed in a form that can be separated from other video signals obtained by the means And a video composition mode designation signal for defining a display mode when each playback video based on the one video signal and the other video signal is displayed together on a predetermined video display unit. A video composition mode designation signal receiving unit for receiving the multiplexed video signal, and the multiplexed video signal received by the multiplexed video signal receiving unit is separated into one video signal and another video signal, and the separated one video signal The video signal and other video signals are And a video composition processing unit that performs composition processing so as to form a composite video signal that conforms to a display mode defined by the video composition mode designation signal received by the composition mode designation signal receiving unit. . One video signal acquired by one imaging means arranged so that one or a plurality of speakers can be included in the imaging visual field and the one imaging means are arranged so that the imaging visual fields are different. Receives the multiplexed video signal supplied via the communication network from a mobile device on the transmission side, which multiplexes the other video signal acquired by other imaging means into a separable form to obtain the multiplexed video signal. A video playback method in a mobile device,
A multiplexed video signal receiving step for receiving the multiplexed video signal from the communication network, and the multiplexed video signal received in the multiplexed video signal receiving step into the one video signal and the other video signal. A signal separation step of separating, a video display step of displaying each video based on the one video signal and the other video signal separated in the signal separation step, the one video signal and the other And a video display operation step of operating the display mode in the video display step of each playback video based on the video signal without affecting the display on the transmitting mobile device. Video playback method for mobile devices.

Images