JP4994646B2 - Communication terminal, communication system, and communication terminal display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication terminal and a display method thereof, and a display method of the communication terminal which can adaptively and optimally update the size and position of a display image area depending on conditions, makes reallocation of the display image area continuous, enables a receiver side to immediately display the image at a large size, and enables transmitter and receiver sides to reproduce the image at the same timing. <P>SOLUTION: The communication terminal 10 has functions of calculating a display magnification factor of a screen based on line segments connecting between centers of the screen, the thickness of a reference shape and the volume of a sound, optimally forming a plurality of screens on an image surface by controlling the movement of the screen and new generation thereof based on this display magnification factor, adding a different session from the same terminal as communication information as low order information of the same terminal, determining a display area concerning information on the lowest position as a display area to be interested for an display area size for displaying an image and controlling the display area size in a corresponding display area. <P>COPYRIGHT: (C)2007,JPO&amp;INPIT

Description

  The present invention relates to a communication terminal such as a mobile phone and a display method thereof, and more particularly to a communication terminal and communication system capable of multipoint communication, and a display method of the communication terminal.

  A representative example of multipoint communication is a video conference system. In the video conference system, a plurality of terminals are connected via an MCU (Multi-point Control Unit). The MCU divides and synthesizes image data sent from a large number of terminals on one screen, and transmits it to each terminal together with audio data, thereby realizing a video conference connecting multiple points.

Basically, when dividing and synthesizing the images of each site into one image,
(1) When one image is equally divided (for example, 4 divisions, 9 divisions),
(2) Taking one large image area and dividing the remaining area into equal parts;
(For example, 6 divisions).

In the case of (1), images from bases connected by MCU are synthesized using the same area.
In the case of (2), a large area is allocated to the talking base, and images from the remaining bases are assigned to the remaining equally divided areas to be combined.

  In any case, since the video conference system uses a large-screen monitor, even if the images of a plurality of locations are divided and combined into one image, the size of the image showing each location is sufficiently large. As long as you are alone, the face of that person will not be difficult to recognize. A video conference system is disclosed in, for example, Patent Documents 1 and 2 and the like.

1A to 1E are diagrams showing examples of display screens of a terminal such as a personal computer (PC) at the time of multilevel communication in a general video conference system.
In the example of FIG. 1, the screen 1 is divided into a predetermined frame (rectangular) window.
For example, the screen 1 is formed of one large window (square) 2 and a plurality of small windows (squares) 3-1 to 3-5, and displays a speaker on the large window 2.
In this case, since the window size and the number of divisions are fixed and the captured image is displayed as it is, the size of the face varies depending on the shooting state.

  In the case of a general PC window control, it is possible to freely change the window size and select the window by dragging the window with the mouse.

By the way, mobile communication terminals such as mobile phones are not only for voice calls, but have become more sophisticated year by year, such as mail, web access, games, cameras, videophones, media players, radios, and televisions.
At present, a videophone in a mobile communication terminal is selected to connect by a videophone when a call is made.

  However, with the progress of support for packet communication, VoIP that supports packet communication is used as the voice call itself, and the camera is activated during the call and switched to voice and video calls, and vice versa. The main usage is to stop and make only voice calls. In addition, send documents (ex. Mail), address data, images (still images, moving images), audio, etc. saved on your terminal to the other party on the call, )) You can watch at the same time or watch the website at the same time.

As described above, when the mobile communication terminal is converted to IP due to high functionality, it is possible to communicate with a plurality of other parties (including a server) at the same time.
In this case, it is necessary to handle a plurality of screens with one terminal. As a method of handling a plurality of screens, there is a method of handling pages in a page turning manner (such as with a PDA).
Japanese Patent Laid-Open No. 06-141310 Japanese Patent Application Laid-Open No. 06-141311

By the way, for example, in a portable IP-TV phone, since the screen size is small, when a call with images is made by a plurality of people, the size of each person's face becomes small.
In order to be able to change the size when the user moves the window like a PC window, the screen size and operation keys are limited and difficult.
In addition, when multiple people talk at the same level, the screen is not compatible.

Furthermore, when the screen size is set according to the volume, there is a problem that a useless space is increased or the entire screen cannot be displayed in the screen.
Specifically, when the volume of all the screens is small, a small screen floats on the screen. On the other hand, when the volume of all screens is maximum, there is a problem that the total area of the display screen exceeds the screen area.
On the other hand, by locally normalizing the total area to the screen area, it is possible to fit within the screen, but the rearrangement (movement) of the screen position at the next time becomes discontinuous.

Further, when the photographer on the transmission side changes the subject by panning the camera or the like, he wants to show the subject to the other party on the reception side. At that time, the photographer is not always talking. For this reason, you have to speak something after you panic. For example, if you are sending an image to choose what to purchase, the photographer cannot keep talking.
However, the received image itself needs to be a screen with a size that can be seen continuously during that time.

Furthermore, the file sent during the call is a file to be noticed during the conversation, and the displayed and reproduced content needs to be a screen that can be visually recognized.
When shifting from a two-party call to a three-party call in a conventional connection, a terminal connected to both parties changes the medium connected to both parties and operates as media mixing. Screen control is performed by a terminal that performs media mixing.
That is, the terminal that displays the screen cannot be controlled. In this case, if synchronized playback of files (images, sounds) and web access are performed together, it is necessary to change media accordingly.

  It is an object of the present invention to adaptively and optimally display the size and position of the display image area (screen) according to the situation such as the volume level and the number of display image areas (screens) to be displayed without the user's operation. In addition, the rearrangement (movement) of the display image area becomes continuous, and the receiving side can instantly make the image appear larger, and it is played back at the same timing on the transmitting side and the receiving side. An object of the present invention is to provide a communication terminal, a communication system, and a display method for the communication terminal.

A first aspect of the present invention is a communication terminal capable of transmitting and receiving image data and audio data, and capable of reproducing received image data and audio data. The display unit displays an image, and the display unit displays the image data and audio data. Control means capable of displaying images by forming a plurality of display areas, and mutually controlling the sizes of images to be displayed based on the reception volume, and the control means can be used in different sessions from the same terminal. involved Ri, the display area that are displayed on the display means, for controlling the display area to control the size.

  Preferably, the control means minimizes other display areas.

A second aspect of the present invention is a communication system capable of transmitting and receiving image data and audio data, and performing communication between a plurality of communication terminals capable of reproducing received image data and audio data, wherein the communication terminal includes: A display means for displaying an image; and a control means for forming a plurality of display areas to be displayed on the display means and capable of displaying the image, and for mutually controlling the size of the image to be displayed based on the reception volume. and, wherein, Ri involved in another session from the same terminal, the display area that are displayed on the display means, for controlling the display area to control the size.

  Preferably, with regard to file display and playback, when the transmission side requests synchronous playback, the file is transferred and the playback start time information is sent, and the transmission side and reception side are in accordance with the playback start time. , Play the file.

  Preferably, the apparatus has a calling function and an imaging function, and displays the presence or absence of streaming transmission of captured images based on the operation of the imaging function during a call. In addition, when a playback request for streaming is sent and the receiving side permits playback, streaming delivery is started based on the playback permission, and the control means on the receiving side generates a new screen and performs reception streaming. Reproduce.

  Preferably, it has a call function and an imaging function. When a still image is captured by the imaging function during a call, the presence / absence of transmission of a still image is displayed. First, when an image playback request is sent and the receiving side permits playback, the image is sent based on the playback permission, and the control means on the receiving side generates a new display area, and receives the received image. Reproduce.

  Preferably, when the reproduction display area is closed on the transmission side, a message to that effect is sent to the reception side, the display area reproducing the information corresponding to the closed display area is closed, and the corresponding information is deleted. To do.

  Preferably, the control means sets the color and thickness of the outer frame of the display area to the same color in the display areas belonging to the same simultaneous call group on the screen.

  Preferably, when there is an incoming call, the control means reads an image corresponding to the called party number from the memory and newly generates an image on the display area, and the display area sets the color of the outer frame for incoming calls. The assigned color is changed, the outer frame is increased or decreased in amplitude, and the color is varied or blinked.

  Preferably, the control means is configured such that the size of the display area for displaying the image captured by the own terminal does not depend on the voice input by the voice input means of the own terminal, and is a constant size. For the behavior calculation, the outer frame of the display area is treated as a wall in the outer frame of the screen.

  Preferably, the control means performs control so that the color of the outer frame of the display area that is transmitting audio acquired by the terminal is different from the color of the outer frame of the display area that is not transmitting audio. .

  Preferably, the control means controls the coding rate of the transmission image based on the display area size on the receiving side at the previous time.

  Preferably, according to the display area size on the receiving side that receives and reproduces the same image, encoding is performed according to the size with the largest occupation ratio on the screen, and all of the I picture, P picture, and B picture are encoded. For the size smaller than the transmitted size, I picture and P picture excluding B picture are sent out from the coded data, and for smaller size, the P picture, B from the coded data are sent out. Only the I picture excluding the picture is transmitted, and for smaller sizes, the data obtained by excluding the P picture and B picture from the encoded data and thinning out the I picture is transmitted.

  Preferably, when the display area size is within a range that does not generate visually uncomfortable noise by upsampling, the control means treats the same size.

  Preferably, the control means handles a change in coding size accompanying a change in display area size in units of I pictures.

  Preferably, in accordance with the change in the display area size, the change in the coding size is obtained by sampling the predicted image to the current size and performing inter-frame prediction, and similarly sampling and decoding the predicted image on the receiving side.

According to a third aspect of the present invention, there is provided a communication terminal display method capable of transmitting and receiving image data and audio data and capable of reproducing the received image data and audio data, and forming a plurality of display areas to be displayed on the display means. and displaying an image by having the steps of controlling the mutual size of the image to be displayed on the basis of the received sound volume, a, Ri involved in another session from the same terminal, that is displayed on the display means The display area is controlled as a display area for controlling the size.

According to the present invention, the size and position of the display image area (screen) are adaptively and optimally according to the situation such as the volume level and the number of display image areas (screens) to be displayed without the user's operation. In addition, the display image area can be rearranged (moved) continuously, and even with different shapes, it can be arranged in an optimum size.
Also, when the sender sends an image file that the sender wants to show, the sender can immediately add that image (screen without any additional operation) (and the sender does not speak). (Size) can be shown larger.
Also, by transmitting the information of the reproduction start time together with the file, it becomes possible to reproduce at the same timing on the transmission side and the reception side.
In the case of two to three parties, it is possible to issue an authentication key to both sides from a terminal connected to both sides, and to immediately connect based on this.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  2 and 3 are diagrams illustrating a configuration example of the mobile communication terminal according to the embodiment of the present invention. FIG. 2 is a block diagram illustrating an encoding apparatus, and FIG. 3 is a block diagram illustrating a decoding apparatus. .

  The mobile communication terminal 10 includes an encoding device 20 serving as a transmission source and a decoding device 30 serving as a reception side, and is configured to be capable of multipoint communication.

The encoding device 20 has a function of transmitting the encoded audio data and image data to the network as a packet by adding instruction information for the receiving terminal, image top and bottom information, and the like.
The instruction information of the transmission source added to the audio data and the image data includes information for identifying the transmission source of the instructed image (for example, IP address, MAC address) and position information indicating the position on the received image. Including.
The encoding device 20 serving as a transmission source generates corresponding instruction information, screen information, and volume information when the transmission source has a screen (the screen will be described in detail later) at the indicated position on the screen. It has a function to send to the other party in communication.

  The encoding apparatus 20 in FIG. 3 encodes an audio input unit 201 including a microphone, an image input unit 202 such as a digital camera, an operation unit 203 capable of key input and the like, and audio data input by the audio input unit 201. The audio encoding processing unit 204, the image encoding processing unit 205 that encodes the image data input from the image input unit 202 and cut out into a predetermined area, and receives the top and bottom of the captured image based on the top and bottom information associated with the captured image Detected by a top / bottom correction unit 206 that corrects the screen to match the top / bottom of the display screen (terminal screen), a face area detection unit 207 that detects and extracts a face area from the captured image, and a face area detection unit 207. A screen discriminating unit 208 that discriminates a screen (display image area to be displayed) to be used based on the face area and generates screen information. Based on the input information of the operation unit 203 and the input volume measuring unit 210 that generates the volume information by measuring the input volume by the voice input unit 201 by cutting out the corresponding area from the received image based on the determination of the lean determination unit 208. A terminal control unit 211 that controls the terminal, a control information generation unit 212 that generates control information including instruction information, top and bottom information, screen information, volume information, and the like based on instructions from the terminal control unit 211, and stores images and videos A storage unit 213, encoded audio data and image data, control information, a transmission packet generation unit 214 that generates image / video data read from the storage unit 213 based on an instruction from the terminal control unit 211 as a transmission packet; In addition, the transmission packet generated so that wireless communication with the network can be performed via the network is performed on the other party's terminal or A network interface (I / F) 215 for transmitting to the server.

The decoding device 30 has a function of reproducing audio data and image data transmitted from the encoding device 20 of the communication partner (transmission source) and received via the network.
For example, when performing multipoint communication, the decoding device 30 selects a screen (a display area whose size is controlled) to use an image including a face as a specific area based on the control information of the received image. It has a function to display and emit sound.
The decoding device 30 has a function to divide in a circular (concept including an ellipse) window from which a dead zone is eliminated when the screen is displayed.
The reason for the division into the circular (elliptical) windows is as follows.
In general, the screen is divided into rectangles. The human face is basically oval, and the four corners of the rectangle are dead zones. This dead zone results in a narrow (small) area for displaying the face.
Therefore, in this embodiment, it is configured so as to be divided by a circular (elliptical) window from which this dead zone is eliminated.
Further, the decoding device 30 has a function of displaying a multi-screen, calculates a screen display magnification based on a line segment connecting the centers of the screens, a thickness of a reference shape, and a loudness of the sound. By controlling the movement and new generation of the screen based on the above, it has a function of optimally forming a plurality of screens on the screen.
The communication terminal of this embodiment adds another session from the same terminal as communication information as lower information of the same terminal, and the display area size for displaying an image pays attention to the display area related to the information located at the lowest position. It has a function of determining a display area to be controlled and controlling the corresponding display area.
Specific processing will be described in detail later in association with the drawings.

  The decoding device 30 of FIG. 3 is a network interface that receives packets including audio data, image (video) data, control information and instruction information, screen information, volume information, and the like that are wirelessly communicable with a network and transmitted from a transmission source. (I / F) 301, operation unit 302 capable of key input, etc., and packet received by network interface 301 are analyzed, and voice data, image data, transmission source address, control information (top and bottom information, instruction information, etc.) are extracted. The received packet analysis unit 303, the audio decoding processing unit 304 that decodes the audio data extracted by the reception packet analysis unit 303, the video decoding processing unit 305 that decodes the video data extracted by the reception packet analysis unit 303, and the video decoding process. Video data decoded by the unit 305, transmission source address, control information, screen information Display image control unit 306 that controls the size and display form of a screen (display window) to be displayed based on the information, size information, and top and bottom information, and a volume correction unit that corrects the volume of the audio decoded by the audio decoding processing unit 304 307, an audio output unit 308 such as a speaker that produces sound with the volume corrected by the volume correction unit 307, an image correction unit 309 that corrects an image whose size and display form are controlled by the display image control unit 306, and an image correction unit 309. A display unit (image output unit) 310 such as an LCD for displaying an image via the terminal, and a local terminal control unit 311 for giving control information (top and bottom information) to the display image control unit 306 based on input information from the operation unit 302 .

  The encoding device 20 and the decoding device 30 can share the operation units 203 and 302, the network interfaces 215 and 301, the terminal control unit 211, and the own terminal control unit 311.

  Hereinafter, a more specific configuration and function of the display image control unit 306, which is a characteristic part of the present embodiment, and a specific configuration and display mode example of the screen will be described in order.

  The display image control unit 306 in FIG. 3 is based on the control information analysis unit 3061 that extracts screen information, size information, top and bottom information, and instruction information based on the control information supplied from the received packet analysis unit 303, and based on the screen information. A masking processing unit 3062 that masks the video decoded by the video decoding processing unit 305, a display magnification calculation determination unit 3063 that calculates a display magnification of a screen (display image area) to be displayed based on the size information and the screen information. , A reduction / enlargement processing unit 3064 for reducing / enlarging the image after the masking process according to the display magnification calculated by the display magnification calculation determination unit 3063, and a display position according to the display magnification and the top / bottom information calculated by the display magnification calculation determination unit 3063. Obtained by the display position calculation unit 3065 to be calculated and the display position calculation unit 3065 And having a mapping processing unit 3066 for mapping the resulting image on the display unit 310 Scaling processing unit at a position on 3064.

The display image control unit 306 of this embodiment has a vibration state determination function that determines whether or not one or more screens are in a state of continuously moving in a local region.
This vibration state determination function does not change the number of screens in a certain period (n), does not change the sound volume of each screen, and further changes the position of the screen, and the fluctuation is below the threshold (Pthresh0). If the fluctuation of the display magnification of the screen is less than or equal to the threshold (Rthresh0), it is determined that the vibration state is present, and the arrangement in a state where the square sum value (R) of the display magnification during the period is the largest during the vibration state. Secure to.

Further, the display image control unit 306 of the present embodiment has a stable state determination function for determining whether or not the screen is in a stable state.
This stable state determination function has no fluctuation in the number of screens in a certain period (m), no fluctuation in the sound volume of each screen, and the fluctuation in the screen position is below a threshold (Pthresh1). If the change in display magnification is less than or equal to the threshold (Rthresh1), it is determined that the state is stable.
In addition, the stable state judgment function does not change the number of screens in a certain period, does not change the sound volume of each screen, and the screen position fluctuation is less than the threshold (Pthresh1), and the screen display When the square sum ratio of magnifications is equal to or less than the threshold (Rthresh2), it is determined that the local stable state is established.

In addition, the display image control unit 306 of the present embodiment has a stirring processing function that prompts rearrangement of the screen position when it is determined to be stable by the screen stable state determination function.
This agitation treatment function has a temporary center on the screen, the display magnification at each center is not selected first, and the position where the smallest value is acquired is the center of the screen for agitation, for a certain period of time Stirring is performed by creating, updating, and extinguishing the screen for stirring.
When the number of executions (S) of the stirring process exceeds the threshold value (Sthresh), it is determined that the state is stable and the stirring process is stopped.

The screen whose size and display form are controlled by the display image control unit 306 of this embodiment is displayed as a multi-screen that displays a plurality of screens on one screen.
Hereinafter, multi-screen display control will be described focusing on the processing of the display magnification calculation determination unit 3063 and the display position calculation unit 3065 of the display image control unit 306 of the present embodiment.

In the display magnification calculation determination unit 3063 of this embodiment, the screen is associated with the center position coordinates (P (i)) indicating the display position of the screen, the reference shape (Unit (i)) indicating the shape of the screen, and the screen. Display sound (V (i)) and display magnification (R (i)) when the screen is displayed on the screen, and the display magnification (R (i)) is the center position coordinate of the surrounding screen (P (j)) connecting line segment (L (i, j)), thickness of reference shape on that line segment (Lm (i, j), Lm (j, i)), and voice The smallest value is selected from the temporary display magnifications (R (i, j)) calculated based on the sizes (V (i), V (j)).
In the display magnification calculation determination unit 3063, the audio volume (V (k) = 0) and thickness (Lm (k, i) = 0) are set at a point that is perpendicular to the screen boundary from the center of the screen. The display magnification (R (i, k)) is calculated.
Further, the screen moves to a position where the display magnification (R (i)) is maximized.
The screen generates the center of the new screen at the position where the display magnification (R (k)) is the largest.
The reference shape has the same area.
Furthermore, a separation line is drawn between the screens formed in the reference shape, and an area separated by the separation line is set as a new screen.

  Next, the calculation of the display magnification of the screen whose size and display form are controlled by the display image control unit 306 according to the present embodiment, the calculation of the generation position of the new screen, the calculation of the movement position of the screen, etc. will be described more specifically. To do.

  As shown in FIG. 4, each screen 40 has a reference shape (Unit). The screen 40 on the screen of the display unit 310 displays the reference shape (Unit) enlarged or reduced according to the display magnification (R).

Calculation of display magnification (R) :
The display magnification calculation determination unit 3063 is a distance (L (i, j)) between the centers of the screen i and the screen j, and a thickness (Lm (i, j,) in the reference shape (Unit) from the center of each screen to the direction. j), Lm (j, i)), and based on the received audio volume (V (i), V (j)) in the content displayed on each screen, from screen j on screen i The display magnification (R (i, j)) is calculated as follows.

  The display magnification between the surrounding screens is calculated, and the smallest value among the display magnifications is set as the actual display magnification (R (i)) as shown in the following equation.

Calculation of new screen generation position :
The display magnification calculation determination unit 3063 places a temporary center on the screen and calculates the display magnification (Rmin) at each center. Of each display magnification (R), the position having the largest value is set as the center position for generating a new screen.

  The center (P (k)) satisfying this condition is set as the center position of the new screen.

Calculation of screen movement position :
Each screen calculates the display magnification (R) at each position within a certain distance (set I) from the current (t) position, and the position having the largest value among the display magnifications is calculated at the next time (t + Δt ).

  Move to the center (P (t + Δt)) that satisfies this condition.

  The screen position moves on the screen over time. For this reason, in the generation of a new screen, it is not necessary to perform calculation for all empty positions on the screen. That is, even if a new generation position is determined for some points on the screen and the positions are arranged based on the result, the display magnification moves to the position where the display magnification is the highest as time passes. Thereby, it becomes possible to reduce the calculation load in generation.

Since the positional relationship of the screen changes at any time, it is necessary to calculate the thickness (Lm (*)) of the reference shape in the display magnification (R (*)) calculation with respect to the direction at that time.
The thickness can be calculated by drawing a digital straight line from the center to the target direction (for a complicated shape). However, this increases the calculation load. For this, by referring to a table in which the thickness for each angle is calculated in advance for each reference shape, it is possible to reduce the calculation load when calculating the display magnification.

Wall treatment on all sides of the screen :
Each screen 40 calculates the display magnification (R) according to the following calculation rule between the four walls.
As shown in FIG. 5, the point dropped vertically from the center of the screen to the wall is taken as the center of the calculated wall, and the line segment between the centers (L (i, k)) and the thickness in the reference shape (Unit) ( Lm (i, k), Lm (k, i)) and the magnitude (V (i), V (k)) of the voice received by the screen are calculated. At this time, the sound volume on the wall is calculated as (V (k) = 0) and the thickness of the reference shape (Lm (k, i) = 0) in the same manner as the calculation of the display magnification (R) described above. I do.
When calculating the display magnification (R (i)) for each screen 40, the display magnification with the wall (R (i, k)) is the same as the display magnification with the surrounding screen (R (i, j)). And the display magnification (R (i)) when actually displaying the smallest value among these

An example in which the ratio of the sound volume (V (0), V (1)) is changed on the screen (S (0), S (1)) whose reference shape is an ellipse is shown in FIGS. Shown in (C).
6A to 6C, from the left, the audio volume ratio (V (0): V (1)) is 1: 1, 2: 1, 3: 1. Thus, the screen size can be adaptively changed according to the volume.

7A to 7 (A) to 7 (A) to 7 (A) to 7 (A) to 7 (A) to 7 (A) to 7 (A) to (E) in which the number of screens formed on the screen is increased or decreased. C).
7A to 7C, the numbers of screens are 2, 3, and 4 from the left.
In this way, it is possible to adaptively change the screen size according to the number of screens and form all the screens in the screen.

In a screen (S (0), S (1), S (2), S (3)) with an elliptical reference shape, the number of screens formed on the screen is increased or decreased, and the sound of one of these screens is recorded. FIGS. 8A to 8C show examples in which the size is double the size of the sound of other screens.
8A to 8C, the number of screens is 2, 3, and 4 from the left, and the audio volume ratio (V (0): V (1)) is 2: 1. (V (0): V (1): V (2)) is 2: 1: 1, ratio (V (0): V (1): V (2): V (3)) is 2: 1: This is the case of 1: 1.
In this way, it is possible to adaptively change the screen size according to the number of screens and form all the screens in the screen. This is an example in which a person shown on the screen (S (0)) is speaking, and when one person is speaking in this way, only that person's screen will depend on its size. The screen size can be adaptively enlarged / reduced.

In the screens S (0), S (1), S (2), S (3)) having an elliptical reference shape, the number of screens formed on the screen is increased or decreased, and the sound volume of one of the screens is increased. FIGS. 9A to 9C show examples in which the sound volume of other screens is halved.
9A to 9C, the number of screens is 2, 3, and 4 from the left, and the audio volume ratio (V (0): V (1)) is 2: 1. (V (0): V (1): V (2)) is 2: 1: 2, ratio (V (0): V (1): V (2): V (3)) is 2: 1: This is the case of 2: 2.
In this way, it is possible to adaptively change the screen size according to the number of screens and form all the screens in the screen. This is an example where a person other than the person shown on the screen (S (0)) is speaking, and in this way, even when multiple persons are speaking, the screen is adaptively adapted to the situation. It becomes possible to enlarge / reduce the size.

FIG. 10A shows an example in which the reference shape includes an ellipse (S (oval)), a circle (S (circle)), and a rectangle (S (rectangle)) (sound volume is equal). Shown in (D).
10A to 10D, from the left, a case where a rectangle and an ellipse, a circle and an ellipse, a circle and a rectangle, and a lower part are a circle, an ellipse, and a rectangle are shown.
Regardless of the shape of the reference shape, by setting the area to be equal, each screen adaptively adjusts the screen size, and if the audio volume is equal, each screen size is visually displayed equally It becomes possible to do.

Furthermore, in this embodiment, since the dead zone outside the screen is reduced, as shown in FIGS. 11 (A) to (C), an area separation line (bold line) is formed between the screens. The area based on the separation line is the display area of each screen. This makes it possible to divide and use the screen to the maximum while adapting to the increase / decrease in the number of screens and the increase / decrease in the sound volume of each screen.
In the example of FIGS. 11A to 11C, the audio volume ratio (V (0): V (1)) is 2: 1 and the ratio (V (0): V (1): V (2). )) Is 2: 1: 1 and the ratio (V (0): V (1): V (2): V (3)) is 2: 1: 1: 1.

  Next, a display form example will be described.

As shown in FIG. 12, when the area of the face area is a certain value or more, the received image is determined as a “person image”. When the area of the face area is equal to or smaller than a certain value, the received image is determined as a “non-person image”.
When it is determined that the image is a “person image”, a circular screen is used. When it is determined that the image is a “non-person image”, a quadrangular screen is used. Even for a “non-human image”, the display size is changed in accordance with the sound pressure from the transmission source.

  As shown in FIG. 13, when there are a plurality of areas determined as faces, when the number is equal to or less than a certain value, priority is given to the larger area, and when the number is equal to or more than a certain value, Display on a square screen.

As shown in FIG. 14, when the cutout area is a face that protrudes from the received image (when protruding), the area that can be cut out is estimated from the motion vector, and the circular screen (window) is only visible. To do.
If the ratio of the face image remaining on the screen to the estimated size of the face area is below a certain value, the face is excluded.

  Next, multipoint communication processing using the communication terminal 10 having the above functions will be described.

In this embodiment, as communication information, another session from the same terminal is added as lower information of the same terminal, and the screen size should be noted for the screen related to the information located at the lowest (added last). Judge as a screen and set it to the corresponding screen size. Minimize other screens.
Regarding the file display and playback, when the transmission side requests synchronous playback, the file is transferred and the playback start time information is transmitted, and the transmission side and the reception side send the file according to the playback start time. Reproduce.
When a call is activated, the camera is activated (such as an operation) to display whether the captured image is streamed by the camera. If the operator permits it, a streaming playback request is sent to the other party who is currently talking When the receiving side permits reproduction, streaming distribution is started based on this, and the receiving side generates a new screen and reproduces the received streaming.
In addition, when a still image is taken with a camera during a call, the presence / absence of transmission of the still image is displayed. When the operator permits, an image playback request is sent to the other party who is currently talking, and the receiving side plays it. Is permitted, the image is transmitted based on this, and the receiving side generates a new screen and reproduces the received image.
When the playback screen is closed on the transmission side, a message to that effect is sent to the reception side, the screen reproducing the information corresponding to the closed screen is closed, and the corresponding information is deleted.

  The first group that is talking simultaneously with the number of terminals (number N (N ≧ 1)), the second group that is talking simultaneously with the number of terminals (number M (M ≧ 1)), and a call connection with both If both terminals are grouped into one group, the caller information (telephone number, IP address, MAC address, other identifier that can identify the terminal, etc.), authentication key, An incoming call request composed of an encryption key is sent out, and the other group is composed of destination information (telephone number, IP address, MAC address, other identifier that can identify a terminal, etc.), authentication key, and encryption key. The terminal that has sent the call request and received the call request makes a call based on the information described in the call request, and the terminal that has received the call request receives the call request. (Caller information, authentication key, No. or determined to match the key), if they match, the connection is made.

On the screen, the screens belonging to the same simultaneous call group have the same color and thickness of the screen outer frame.
When an incoming call is received, an image corresponding to the number of the called party is read from the memory and newly generated on the screen. The screen changes the color of the outer frame for incoming calls, and the outer frame has a large and small amplitude. In addition, the color is varied (or flashing).
The size of the screen that displays the image captured by the camera of its own terminal does not depend on the sound input by the microphone of its own terminal, and is set to a certain size. The outer frame is treated in the same way as the wall in the outer frame of the screen.

Also, darken the color of the outer frame of the screen that is sending out the sound acquired by the microphone of its own terminal, and lighten the color of the outer frame of the screen that is not sending out the sound (and change the thickness) .
Based on the screen size on the receiving side at the previous time, the coding rate of the transmission image is controlled.
Also, according to the screen size of the receiving side that receives and reproduces the same image, it is encoded according to the size with the largest occupation ratio on the screen, and all (I picture, P picture, B picture) are sent out. For smaller sizes, I and P pictures excluding B pictures are sent from the encoded data.
Further, for small sizes, only I pictures excluding P pictures and B pictures are transmitted from the encoded data. For smaller sizes, P pictures and B pictures are excluded from the encoded data, Data with I picture thinned out is transmitted.

Further, when the screen size is within a range that does not generate visually unpleasant noise (for example, an area ratio of 2 times or less) by upsampling, the screen size is handled as the same size.
Coding size variation accompanying screen size variation is handled in I-picture units.
As the screen size fluctuates, the coding size fluctuates by up-sampling (or down-sampling) the predicted image to the current size and performing inter-frame prediction. On the receiving side, the predicted image is similarly up-sampled (or down-sampled). Sampling) and decoding.

  The multipoint communication process using the communication terminal 10 having the above functions will be specifically described with reference to FIGS.

  FIG. 15 is a diagram showing an example of basic communication processing according to the present embodiment (S101 to S108).

In the present embodiment, as described above, as communication information, another session from the same terminal is accompanied as lower information of the same terminal. The size of the screen related to the lowest-level information (added last) is controlled as a screen to be noted, and the other screens are set to the minimum size.
Regarding file playback, when (synchronous) playback is requested on the transmission side, the playback start time when the data capacity that can be played back is transmitted in order to transfer the same file and synchronize the playback timing. The transmission side and the reception side start reproduction according to the reproduction start time.
When the playback screen is closed on the transmission side, the playback screen is also closed on the reception side. Release the memory reserved for communication information.
When playback is stopped on the transmitting side and the receiving side, information indicating that is sent to other terminals that have also started up the playback screen, and on the terminal that received the information, the playback is stopped on the playback screen. The message is displayed and playback is stopped.
At the start of the call, voice call (VoIP) is basically used, and the video captured from the camera is transmitted as another session based on the operation of the terminal.
Accompanying the telephone number or address (identifier that identifies the other party), information on whether or not to establish a session for starting the camera and sending a captured video at the same time is activated. Processing of video transmission at the time.
If the file is a file such as music, a playback application is displayed on the screen. The screen size does not depend on the volume.

  FIG. 16 is a diagram showing an example of communication processing when a website is opened (S111 to S118).

When sharing (or sending) is selected while the website is open, the website address is sent out.
When the receiving side receives the website, it links the information for accessing the website as the attached information of the same session, and displays the accessed website on a new screen. If it is the last information attached, set the size of the screen for displaying the website to the size corresponding to the audio ratio. Other screens should be the smallest size.
When a sound file is transmitted, the sound file and the reproduction timing are transmitted, and the transmission side and the reception side reproduce the sound file in accordance with the reproduction timing (when synchronous reproduction is requested).
On the receiving side, when there is no attached information such as an image or information to be displayed for the sound file, the playback software is displayed on a new screen in a size that allows operation of the playback software. Other screens should be the smallest size.

  FIG. 17 is a diagram showing an example of communication processing when the receiving side refuses to send a website address (S121 to S125).

When the receiving side rejects the sending of the website address, the sending side saves an image of the currently displayed website and tries to send it again as an image.
When image reception is permitted on the receiving side, the image is transmitted, and the image is reproduced on a new screen on the receiving side.

As for the sending and receiving of image files, audio files, websites, etc., the sender's address and the associated connection port are confirmed, and the authentication key and encryption key are exchanged. By doing so, it eliminates the playback of files (including streaming) and websites sent from unrelated terminals.
As a result, it is possible to eliminate information distribution such as advertisements and solicitations from the related terminals to the unspecified majority. In addition, even if the other party is making a voice call, it is possible to prevent reception of information that is estimated to be unnecessary by enabling the transmission request to be rejected.
Note that the screen size can be controlled by selectively selecting a screen on the transmission side and the reception side.

  FIG. 18 is a diagram illustrating an example of a screen display form during communication processing.

The outer frame of the screen is the same color for the screens belonging to the group that is talking together.
For different groups, the outer frame of the screen is basically a different color.
In the case of a group that does not send audio, the outer frame of the screen that belongs to the group has a color (for example, lightened) or a thickness different from normal so that it can be clearly seen that no audio is being transmitted.
In addition, the image on the screen is changed to light-colored or foggy image.
If there is an incoming call, the screen is changed so that the incoming call is clearly identified. For example, the screen size is increased, the vibration is reduced, and the color of the outer frame is blinked.
Participation in a third party's call is performed from a terminal communicating with both parties. A call request (an identifier for identifying a call destination, an authentication key) is sent to an existing group, and a call request (an identifier for identifying a call destination, an authentication key) is sent to a newly participating terminal. ).
The newly joining terminal calls the described call destination according to the call request. The existing terminal performs an incoming call operation for a call from the new terminal, and connects when the authentication keys match.
If the authentication key is different, disconnect immediately. When the connection is completed, the terminal that prompted the connection determines that the new terminal has completed participation based on the connection completion information from the new participation terminal. In the existing terminal, participation in the group is completed based on the authentication key in the incoming call operation.
When an image captured by the camera of its own terminal is displayed on the screen, the size of the screen is set to a constant size without depending on sound. In addition, the color and thickness of the outer frame of the screen are set so that it can be clearly seen that the video is from the terminal itself. In the calculation of the size of other screens, the outer frame of the own screen is handled in the same manner as the wall.

  FIG. 19 is a diagram illustrating an example of communication processing for controlling the coding rate of a transmission image based on the screen size information on the reception side of the previous time.

By controlling the coding rate of the transmission image based on the screen size information on the receiving side at the previous time, it is possible to prevent the total amount of codes received at the receiving terminal from becoming extremely large.
Depending on the screen size on the receiving side, it is classified into two or more sizes. Encode according to the largest size. For small sizes, the encoded data is thinned out and transmitted.
For example, for the largest size, I picture (intra-frame coding), P picture (inter-frame forward prediction coding), and B picture (inter-frame bi-directional prediction coding) are all transmitted.
On the other hand, the code amount to be transmitted is reduced by thinning out the B picture for the next lower size and by thinning out the B picture and the P picture for the smaller size. Also, the size of the image to be encoded is not adjusted to the maximum size of the screen on the receiving side, but can be visually interpolated on the maximum size screen by upsampling the maximum size. The size is suppressed to a small size (for example, it is possible to reduce aliasing due to upsampling by suppressing upsampling to an area ratio of 2 times or less).

Actually, when transmitting a face image, encoding efficiency can be improved by encoding only the face area of all captured images and reducing unnecessary areas for transmission of the background. It becomes possible.
In other words, in the image to be encoded, the face becomes large and the background with a large amount of change is reduced, so that it is possible to reduce the high frequency component that increases the code amount and consequently reduce the code amount. .
As a result, it is possible to generate encoded data to be transmitted to individual terminals without increasing the burden of encoding processing on the transmission side, and to increase the encoded data capacity received at each terminal. It is possible to suppress the increase in the load on the network.

In FIG. 19 (1), the user of the user terminal UA is talking alone, and the screen size of the user terminal UA in the user terminals UB and UC is almost the same.
For this reason, the user terminal UA sends an encoded image in accordance with this size.
On the other hand, the screen of the user terminal UB is moderately large at the user terminal UA, whereas it is very small at the user terminal UC. For this reason, it sends out as image data for the user terminal UA and the user terminal UC. Since the user terminal UA is larger, encoding is performed accordingly. To the user terminal UC, encoded data is transmitted only for I pictures (or further, I pictures are also thinned out).

In the case of FIG. 19 (3), the screen size of the user terminal UA in the user terminals UB and UC is smaller when it is determined that visual discomfort (due to noise) does not occur even when up-sampled and reproduced. Send encoded data according to the size.
Similarly, regarding the screen size of the user terminal UC in the user terminals UA and UB, similarly, if it is determined that there is no visual discomfort (due to noise) even when upsampling and playback are performed, the smaller size is used. Encoded data is sent in time.

When the size of the playback screen is within a range in which the difference can be absorbed by upsampling (ex. Area ratio is within 2 times), the size is the same, and encoding processing is performed in accordance with the smaller screen size.
The screen size is changed in units of I picture (intra-frame coding) in accordance with the encoding method associated with the screen size variation.
In addition, when there is a variation in screen size in a P picture (interframe forward prediction encoding), the predicted image is upsampled or downsampled according to the current size-changed image, and the interframe prediction process is performed during this period. Do. On the receiving side, similarly, the predicted image is decoded by upsampling or downsampling.

As described above, according to the present embodiment, the encoding device 20 serving as the transmission source transmits the corresponding instruction information, screen information, and volume information when the transmission source has a screen at the designated position on the screen. The decoding device 30 has a function of displaying a multi-screen, a line connecting between the centers of the screen, a thickness of a reference shape, and a voice size. The screen display magnification is calculated based on the display magnification, and the screen movement and new generation are controlled based on the display magnification, so that a plurality of screens are optimally formed on the screen. In the other session, it is added as lower information of the same terminal, and the display area size for displaying an image is the display area to which attention should be paid to the display area related to the information at the lowest position. It determined that, since it has a function of controlling the corresponding display area, the size of the screen, the size of the volume, and in accordance with the number of screens, can be adaptively be varied in size.
In addition, there is an advantage that the movement of the screen becomes continuous, and even when the shapes are different, it can be arranged in an optimum size.
Also, when the sender sends an image file that the sender wants to show, the sender can immediately add that image (screen without any additional operation) (and the sender does not speak). (Size) can be shown larger.
Also, by transmitting the information of the reproduction start time together with the file, it becomes possible to reproduce at the same timing on the transmission side and the reception side.
In the case of two to three parties, an authentication key is issued to both sides from a terminal connected to both sides, and it is possible to immediately connect based on this.

It is a figure which shows the example of a display screen of terminals, such as a personal computer (PC) at the time of multipoint communication in a general video conference system. It is a figure which shows the structural example of the portable communication terminal which concerns on embodiment of this invention, Comprising: It is a block diagram which shows an encoding apparatus. It is a figure which shows the structural example of the portable communication terminal which concerns on embodiment of this invention, Comprising: It is a block diagram which shows a decoding apparatus. It is a figure for demonstrating the calculation process of a display magnification. It is a figure for demonstrating the wall process of a screen four directions. It is a figure which shows the example which changed the ratio of the magnitude | size (V (0), V (1)) of an audio | voice in the screen (S (0), S (1)) which makes an elliptical reference | standard shape. It is a figure which shows the example which increased / decreased the number of screens formed on a screen in the screen (S (0), S (1), S (2), S (3)) which makes an oval reference shape. In a screen (S (0), S (1), S (2), S (3)) with an elliptical reference shape, the number of screens formed on the screen is increased or decreased, and the sound of one of these screens is recorded. It is a figure which shows the example which doubled the magnitude | size of the audio | voice of the other screen. In the screens S (0), S (1), S (2), S (3)) having an elliptical reference shape, the number of screens formed on the screen is increased or decreased, and the sound volume of one of the screens is increased. It is a figure which shows the example which doubled the magnitude | size of the audio | voice of the other screen. It is a figure which shows the example when a reference | standard shape has mixed ellipse (S (oval)), circle | round | yen (S (circle)), and a rectangle (S (rectangle)) (the magnitude | size of an audio | voice is equal). It is a figure which shows the example which forms the separation line (bold line) of an area between each screen, and makes the area based on the said separation line into the display area of each screen. It is a figure for demonstrating screen display control, Comprising: It is explanatory drawing of the process according to the area of a face area. It is a figure for demonstrating screen display control, Comprising: It is explanatory drawing of a corresponding | compatible process when there exist multiple areas by which the face determination was carried out. It is a figure for demonstrating screen display control, Comprising: It is explanatory drawing of a corresponding | compatible process when a cut-out area protrudes from a received image. It is a figure showing an example of basic communication processing concerning this embodiment. It is a figure which shows the example of a communication process at the time of opening the web site. It is a figure which shows the example of a communication process when the receiving side refuses with respect to address transmission of a website. It is a figure which shows the example of a display form of the screen at the time of a communication process. It is a figure which shows the example of a communication process which controls the encoding rate of a transmission image based on the screen size information of the receiving side of the previous time.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Portable communication terminal, 20 ... Encoding apparatus, 201 ... Audio | voice input part, 202 ... Image input part, 203 ... Operation part, 204 ... Voice encoding process part, 205. ..Image encoding processing unit, 206... Top and bottom correction unit, 207... Face area detection unit, 208... Screen determination unit, 209. 211: Terminal control unit, 212: Control information generation unit, 213 ... Storage unit, 214 ... Transmission packet generation unit, 215 ... Network interface (I / F), 30 ... Decoding Device 301 network interface (I / F) 302 operation unit 303 received packet analysis unit 304 audio decoding processing unit 305 video decoding processing unit 306 ··table Image control unit, 307... Volume correction unit, 308 .. Audio output unit, 309... Image correction unit, 310... Display unit (image output unit), 311. Control information analysis unit, 3062 ... Masking processing unit, 3063 ... Display magnification calculation determination unit, 3064 ... Reduction / enlargement processing unit, 3065 ... Display position calculation unit, 3066 ... Mapping.

Claims (21)

A communication terminal capable of transmitting and receiving image data and audio data and capable of reproducing received image data and audio data,
Display means for displaying an image;
Control means for forming a plurality of display areas to be displayed on the display means and capable of displaying an image, and controlling the sizes of images to be displayed based on reception volume;
The control means includes
Ri involved in another session from the same terminal, the display area that are displayed on the display unit, a communication terminal for controlling the display area to control the size. The communication terminal according to claim 1, wherein the control means minimizes other display areas. When there is an incoming call, the control means reads out an image corresponding to the number of the called party from the memory and newly generates an image on the display area, and the display area uses a color assigned to the outer frame for the incoming call. The communication terminal according to claim 1, wherein the outer frame has a large and small amplitude, and the color varies or blinks. The control means does not depend on the voice input by the voice input means of the terminal itself, and the size of the display area for displaying the image captured by the terminal is set to a constant size, for behavior calculation of other display areas. The communication terminal according to any one of claims 1 to 3, wherein an outer frame of the display area is handled in the same manner as a wall in the outer frame of the screen. The control means controls so that the color of the outer frame of the display area that is transmitting audio acquired by the terminal is different from the color of the outer frame of the display area that is not transmitting audio. 4. The communication terminal according to any one of 4. The communication terminal according to any one of claims 1 to 5, wherein the control unit controls a coding rate of a transmission image based on a display area size on a reception side at a previous time. A communication system capable of transmitting and receiving image data and audio data and performing communication between a plurality of communication terminals capable of reproducing received image data and audio data,
The communication terminal is
Display means for displaying an image;
Control means for forming a plurality of display areas to be displayed on the display means and capable of displaying an image, and controlling the sizes of images to be displayed based on reception volume;
The control means includes
Ri involved in another session from the same terminal, the display area that are displayed on the display unit, a communication system for controlling a display area to control the size. Regarding the file display and playback, when the transmission side requests synchronous playback, the file is transferred and the playback start time information is transmitted, and the transmission side and the reception side send the file according to the playback start time. The communication system according to claim 7 for reproduction. Call function,
An imaging function;
When a call is made, the presence or absence of streaming transmission of the captured image is displayed based on the operation of the imaging function, and if permitted, a streaming playback request is sent to the currently talking destination, and the receiving side permits playback. 9. The communication system according to claim 7, wherein streaming control is started based on the reproduction permission, and the control unit on the receiving side generates a new screen and reproduces the received streaming. Call function,
An imaging function;
When a still image is taken with the imaging function during a call, the presence / absence of transmission of the still image is displayed. If permitted, an image playback request is sent to the other party who is currently talking, and the receiving side permits playback. 9. The communication system according to claim 7, wherein, in a case where an image is transmitted, an image is transmitted based on the reproduction permission, and the control unit on the receiving side generates a new display area and reproduces the received image. 8. When the reproduction display area is closed on the transmission side, a message to that effect is sent to the reception side, the display area reproducing the information corresponding to the closed display area is closed, and the corresponding information is deleted. The communication system according to any one of 1 to 10. The communication system according to claim 7, wherein the control means makes the display area belonging to the same simultaneous call group have the same color and thickness of the outer frame of the display area on the screen. When there is an incoming call, the control means reads out an image corresponding to the number of the called party from the memory and newly generates an image on the display area, and the display area uses a color assigned to the outer frame for the incoming call. The communication system according to any one of claims 7 to 12, wherein the outer frame has a large and small amplitude, and the color varies or blinks. The control means does not depend on the voice input by the voice input means of the terminal itself, and the size of the display area for displaying the image captured by the terminal is set to a constant size, for behavior calculation of other display areas. The communication system according to any one of claims 7 to 12, wherein the outer frame of the display area is handled in the same manner as a wall in the outer frame of the screen. The control means performs control so that the color of the outer frame of the display area that is transmitting audio acquired by the terminal is different from the color of the outer frame of the display area that is not transmitting audio. 14. The communication system according to any one of 14. The communication system according to any one of claims 7 to 15, wherein the control unit controls a coding rate of a transmission image based on a display area size on a reception side at a previous time. Depending on the display area size on the receiving side that receives and reproduces the same image, it is encoded according to the size with the largest occupation ratio on the screen, and all of the I picture, P picture, and B picture are transmitted,
For sizes smaller than the sent size, send out I and P pictures excluding B pictures from the encoded data,
For even smaller sizes, send only I pictures excluding P and B pictures from the encoded data,
The communication system according to any one of claims 7 to 15, wherein for a smaller size, data obtained by excluding P pictures and B pictures from the encoded data and thinning out I pictures is transmitted. The communication system according to claim 17, wherein the control means treats the display area size as the same size when the display area size is a range that does not generate visually unpleasant noise in the upsampling. 19. The communication system according to claim 17, wherein the control unit handles a change in coding size accompanying a change in display area size in units of I pictures. The encoding size change is performed by sampling the predicted image to the current size and performing inter-frame prediction, and similarly sampling and decoding the predicted image on the receiving side in accordance with the display area size change. The communication system according to any one of the above. A display method of a communication terminal capable of transmitting and receiving image data and audio data and capable of reproducing received image data and audio data,
Forming a plurality of display areas to be displayed on the display means and displaying an image;
Mutually controlling the size of the image to be displayed based on the reception volume,
Ri involved in another session from the same terminal, the display area that are displayed on the display unit, the display method of a communication terminal for controlling the display area to control the size.

Images