JP2007201938A - Communication terminal, communication system, and communication method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To quickly and simply transmit an arbitrary still image with a communication terminal for transmitting and receiving a time-varying image in real time. <P>SOLUTION: If image data input in a frame memory 122 is only the still image, a motion vector of the whole macro block is made zero, and the still image is made an I picture. Then, there is no need to transmit B, P pictures. Thus, the still image can be relatively finely transmitted to the communication terminal 1 of the counterpart as a moving image even when a transmission bandwidth of a network 10 narrows. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a communication terminal, a communication system, and a communication method, and more particularly to a terminal, a system, and a method that provide bidirectional communication using images or sounds.

  As a method for compressing moving images and audio, there is an “MPEG” method. In MPEG video compression, an image is divided into 16 × 16 pixel rectangular blocks called macroblocks, and an area (reference area) similar to the macroblock to be compressed is extracted from the temporally preceding and following images. Then, the spatial distance and direction (motion vector) from the reference area and the difference information between the reference area and the area to be compressed are calculated, and these pieces of information are converted into DCT (discrete cosine transform) and variable length coding. The amount of data is reduced by using it to compress into a bitstream and reducing temporal redundancy. However, since an image compressed by the difference information cannot be expanded without an image to be referred to, in order to cope with a use in which the bitstream is expanded from the middle (hereinafter referred to as “random access”), It is necessary to periodically provide images that do not refer to images. This image is called an “I picture” (Intra Picture). The next image is compressed using the I picture as a reference image, and the subsequent images are further compressed using the already compressed image as a reference image.

  Among the images to be compressed using the reference image, a “P picture” (Predictive-coded Picture) in which only the previous image in time is used as a reference image and a “B picture in which images before and after in time are used as reference images” "Picture" (Bidirectionally predictive-coded Picture). A P picture can be a reference image of another image, like an I picture. In MPEG, a GOP (Group Of Pictures) can be configured from an I picture and a series of image aggregates that are directly and indirectly related thereto. A GOP header is added to the GOP. GOP is used as a unit of random access.

  In compression using a B picture, in order to use temporally previous and subsequent images as reference images, processing is performed to switch the order of input images and the order of images on the bitstream obtained as a result of compression.

In moving picture compression by MPEG, the compression efficiency is further increased by a method called skipping. In B picture and P picture coding, if the macroblock of the current picture being coded is the same as the macroblock of the previous picture in time, the data is sent as a skipped macro. Do not. In decoding, the same image data as the image data decoded from the immediately preceding I picture or P picture is restored.
Japanese Patent Publication 2001-103429 Japanese Patent Publication No. 2001-224028 Japanese Patent Publication No. 2002-10214 Japanese Patent Publication No. 8-12612

  By the way, a plurality of videos are mutually exchanged in real time between terminals connected via a communication network such as that found in a TV phone or a TV conference system, and a plurality of videos received by the terminals are displayed on one screen. In this case, it is convenient to be able to select a still image prepared in advance as a video to be transmitted from one's own terminal to the other terminal. However, if still image data itself (such as a JPEG format file) is sent each time it is selected, the amount of data Temporarily increases and puts pressure on the transmission of other images. Also, if the still image data itself is sent, an unnecessary still image may remain on the other party, or it may not be desired to leave it.

  An object of the present invention is to send an arbitrary still image quickly and easily in a communication terminal that transmits and receives a moving image in real time.

  On the other hand, if video data (including moving images and still images) that are separately input from a plurality of systems to the own terminal is sent to the counterpart terminal in an equal transmission band, various adverse effects are expected.

  For example, if the transmission band of a video that is displayed relatively large is small, the fineness of the screen is impaired. Although the frame rate may be lowered in order to prioritize the image quality, it is not preferable because the movement becomes stiff and difficult to see.

  On the other hand, even if a relatively large transmission band for a video to be displayed is ensured, the other party will eventually be resized to a small size, so sending fine data is meaningless.

  Another object of the present invention is to appropriately allocate a band for transmitting a plurality of videos from a limited transmission band in a real-time moving image transmitting / receiving terminal.

  In order to solve the above-described problem, a communication terminal according to the present invention includes an input unit that inputs a desired still image, a developing unit that develops the still image input to the input unit, and a still image developed on the developing unit. A reference image and an encoding unit that encodes a compressed moving image that is restored to a one-frame moving image by a difference image having a zero motion vector with respect to the reference image, and an encoded compressed moving image encoded by the encoding unit A transmission unit for transmitting to a desired partner.

  The communication terminal transmits a desired still image, and a compressed moving image that is decoded with a difference image having a motion vector of 0 using the still image as a reference image, to a desired counterpart. The counterpart communication terminal can decode the still image as a frame moving image from the received reference image and difference image.

  The communication terminal includes a selection unit that selects a desired still image input system from one or a plurality of still image input systems, and the input unit inputs a desired still image from the still image input system selected by the selection unit. You may make it do.

  Regardless of the type of input source, regardless of the type of the input source, a frame moving image whose motion vector is zero is displayed on the other side regardless of the input source type. Sent to the communication terminal in real time. For this reason, even if the input source of the still image by the selection unit is switched irregularly, the frame moving image transmitted to the other party's communication terminal is switched immediately following this, and consequently displayed on the other party's communication terminal. The still image to be displayed is switched quickly.

  The communication terminal further includes an operation unit that receives a manual input operation, and an input system display unit that displays a list of still image input systems, and the selection unit is configured to input a still image input system displayed on the input system display unit. From this list, an input system arbitrarily designated by an input operation to the operation unit may be selected.

  In order to solve the above-described problem, a communication terminal according to the present invention includes a plurality of encoding units that individually encode videos input from a plurality of input systems, and a plurality of encoding units encoded by a plurality of encoding units. A display area that can specify each display area on the display screen of the communication terminal of the other party of a plurality of images transmitted by the transmission unit to the other party's communication terminal Ratio of display area of each of the plurality of images indicated by the display area specifying signal received by the display area specifying signal received by the display area specifying signal receiving unit, the band estimating unit for estimating the transmission band of the network, and the display area specifying signal receiving unit In general, a bandwidth allocation unit that determines allocation of transmission bands for each of a plurality of videos within the range of the transmission band estimated by the bandwidth estimation unit, and a plurality of bandwidths within the range of the transmission band allocated by the bandwidth allocation unit It comprises a coding control unit that controls the amount of coding a plurality of picture by the encoding unit.

  In this way, it is possible to allocate a transmission band according to the size of the video display area on the counterpart terminal from the limited transmission band. A relatively large band can be allocated to a large display video and transmitted to the other party for fine display. In addition, a relatively small band can be allocated to the video displayed in a small size and transmitted to the other party to be displayed roughly.

  The communication terminal further includes an allocation table storage unit that stores an allocation table in which transmission band allocation according to a display area ratio of each of a plurality of videos is associated with a transmission band to be estimated by the band estimation unit The bandwidth allocation unit refers to the allocation table of the allocation table storage unit, and determines transmission band allocation according to each allocation of a plurality of videos associated with the transmission band estimated by the bandwidth estimation unit. Also good.

  The bandwidth estimation technique is arbitrary. For example, a protocol “RTCP (RTP Control Protocol)” that performs RTP (Real-Time Transport Protocol) session control, such as “Band Adaptive Rate Control Method” (BARC). A technique for monitoring the status of the network can be used.

  It is preferable that the transmission band allocation of each of the plurality of videos stored in the allocation table exceeds a predetermined lower limit value.

  If video transmission bands are assigned in strict accordance with the display area ratio, it is expected that when the estimated transmission band is small, the image quality of the video with the smaller area ratio will deteriorate beyond visual perception. For this reason, the transmission band with the smaller display area is assigned to exceed the predetermined lower limit value.

  The display area specifying signal received by the display area specifying signal receiving unit includes information for specifying a display mode that defines an area for displaying each of a plurality of videos displayed by the counterpart communication terminal, and the band allocating unit displays the display area. The display mode set by the communication terminal of the other party is specified from the display area specifying signal received by the specific signal receiving unit, and each area according to the area ratio of the areas where each of the plurality of videos is displayed in the specified display mode The allocation of the transmission band of the video displayed on the screen may be determined.

  In the display mode, the arrangement of each of a plurality of videos in part or all of a display area in which one screen is equally divided by a predetermined area may be defined.

  A communication system according to the present invention includes an input unit that inputs a desired still image, a developing unit that develops a still image input to the input unit, a still image developed in the developing unit as a reference image, and a reference image An encoding unit that encodes a compressed moving image that is restored to a one-frame moving image with a difference image having a motion vector of zero, and a transmission unit that transmits the compressed moving image encoded by the encoding unit. A transmitting communication terminal, a receiving unit that receives a compressed moving image encoded from the transmitting communication terminal, a decoding unit that decodes the encoded compressed moving image received by the receiving unit, and a decoding unit A receiving-side communication terminal including: a developing unit that develops the decoded compressed moving image into a one-frame moving image; and a display unit that displays the one-frame moving image developed by the developing unit as a still image.

  The communication system according to the present invention includes a plurality of encoding units that individually encode images input from a plurality of input systems, and a transmission that transmits a plurality of images encoded by the plurality of encoding units via a network. And a transmission side communication terminal including a bandwidth estimation unit that estimates a transmission band of the network, a reception unit that receives a plurality of videos encoded from the transmission side communication terminal via the network, and a reception unit received Receiving unit comprising: a decoding unit that decodes a plurality of encoded videos; a developing unit that develops the plurality of videos decoded by the decoding unit; and a display unit that displays the plurality of videos developed by the developing unit It is related with a communication terminal.

  In this communication system, the receiving communication terminal includes a display area specifying signal transmitting unit that transmits a display area specifying signal that can specify a display area in each display unit of a plurality of videos transmitted from the transmitting communication terminal. The transmission side communication terminal includes a display area specifying signal receiving unit that receives a display area specifying signal from the receiving side communication terminal, and each of the plurality of videos indicated by the display area specifying signal received by the display area specifying signal receiving unit. In accordance with the ratio of the display area, a bandwidth allocating unit that determines the allocation of each transmission band within the transmission band estimated by the band estimator, and a transmission band allocated by the band allocating unit. An encoding control unit that controls the amount of encoding of the plurality of videos by the plurality of encoding units.

  The communication method according to the present invention includes a step of inputting a desired still image, a step of expanding the input still image, a difference image having the expanded still image as a reference image and a motion vector with respect to the reference image being zero. A step of encoding a compressed moving image that is restored to a one-frame moving image; and a step of transmitting the encoded compressed moving image to a desired counterpart communication terminal.

  In addition, the communication method according to the present invention includes a step of individually encoding videos input from a plurality of input systems, a step of transmitting a plurality of encoded videos to a counterpart communication terminal, and a counterpart communication terminal. Receiving a display area specifying signal capable of specifying each display area on the display screen of the communication terminal of the other party of the plurality of images transmitted to the network, estimating a transmission band of the network, and receiving the display area Determining the transmission band allocation of each of the plurality of videos within the estimated transmission band within approximately the ratio of the display area of each of the plurality of videos indicated by the specific signal; and for each of the plurality of videos Controlling the amount of encoding of the plurality of videos within the allocated transmission band.

  According to the present invention, a desired still image is transmitted to a desired counterpart as a compressed moving image that is decoded with a difference image having the still image as a reference image and a motion vector of zero. The counterpart communication terminal can decode the still image as a frame moving image from the received reference image and difference image.

  In addition, according to the present invention, it is possible to assign a transmission band corresponding to the size of the video display area from a limited transmission band. A relatively large band can be assigned to a large image to be displayed in detail. In addition, a relatively small band can be allocated to an image displayed in a small size so that the image is roughly displayed.

  FIG. 1 is a block diagram of a video / audio communication system according to a preferred embodiment of the present invention. In this system, a communication terminal 1a and a communication terminal 1b having an equivalent configuration are connected via a network 10 such as the Internet, and transmit and receive video and audio.

  The communication terminal 1a and the communication terminal 1b have the same configuration, and the difference between them is only to distinguish the communication partner of the network. In the following description, all or part of the roles of both may be interchanged. Note that you can. If it is not necessary to distinguish the two as communication partners of the network, they may be collectively referred to as the communication terminal 1.

  The network 10 includes, for example, broadband networks such as ADSL, optical fiber (FTTH), and cable television, narrow band networks such as ISDN, IEEE 802.802 such as UWB (Ultra Wide Band) and Wi-Fi (Wireless Fidelity). It consists of xx-compliant wireless communication.

  In the present embodiment, it is assumed that the network 10 is a best-effort network in which it is not guaranteed whether a predetermined bandwidth (communication speed) can always be secured. In the network 10, the nominal maximum bandwidth may be substantially limited due to various factors such as the distance between the telephone office and the home, the communication speed between the ADSL modem, the increase or decrease in traffic, and the communication environment of the other party of the session. . Often the rms value is less than a fraction of the nominal value. The bandwidth of the network 10 is expressed in bits per second (bps). For example, the nominal bandwidth of FTTH is generally 100 Mbps, but in practice, it may be limited to several hundred kbps.

  The connection path between the communication terminal 1a and the communication terminal 1b is designated by the switchboard server 6 constituted by a SIP (Session Initiation Protocol) server using a network address (such as a global IP address), a port, and an identifier (such as a MAC address). To do. Information relating to the user of the communication terminal 1 such as name and e-mail address and information relating to the connection of the communication terminal 1 (account information) are stored in the account database (DB) 8 a and managed by the account management server 8. The account information can be updated / changed / deleted from the communication terminal 1 connected to the account management server 8 via the Web server 7. The Web server 7 also serves as a mail server that transmits mail and a file server that downloads files.

  The communication terminal 1a is connected to the microphone 3a, the camera 4a, the speaker 2a, and the monitor 5a, and the video captured by the camera 4a and the sound collected by the microphone 3a are transmitted to the communication terminal 1b via the network 10. The communication terminal 1b is also connected to the microphone 3b, the camera 4b, the speaker 2b, and the monitor 5b, and can similarly transmit video and audio to the communication terminal 1a.

  The video and audio received by the communication terminal 1b are output to the monitor 5b and the speaker 2b, and the video and audio received by the communication terminal 1a are output to the monitor 5a and the speaker 2a, respectively. The microphone 3 and the speaker 2 may be integrated as a headset.

  FIG. 2 is a block diagram showing a detailed configuration of the communication terminal 1.

  An audio input terminal 31, a video input terminal 32, an audio output terminal 33, and a video output terminal 34 are provided on the outer surface of the communication terminal 1, and are connected to the microphone 3, the camera 4, the speaker 2, and the monitor 5, respectively. .

  The external input terminal 30-1 is an IEEE 1394 input terminal, and receives input of moving image / still image / audio data in accordance with the DV system and other specifications from the digital video camera 70. The external input terminal 30-2 receives a still image input from the digital still camera 71 according to the JPEG specification and other specifications.

  The audio signal input to the audio data converting unit 14 from the microphone 3 connected to the audio input terminal 31 and the color difference signal generated by the NTSC decoder 15 are encoded with CH1 encoded by a high-quality encoder such as an MPEG4 encoder. The data is digitally compressed and encoded by the unit 12-1 and converted into stream data (content data in a format that can be distributed in real time). This stream data is called CH1 stream data.

  Still image or moving image downloaded from the web content server 90 by the web browser module 43, which is a data input source by the switcher 78, still image or moving image from the digital video camera 70, still image or moving image from the digital still camera 71 Image, the moving image downloaded by the streaming module 44 from the streaming server 91, or a moving image or a still image from the recording medium 73 (hereinafter, these image input sources are used to input video contents such as the digital video camera 70). Audio signal or digital video camera downloaded from the streaming server 91 by the streaming module 44 as a data input source by the switcher 78 An audio signal including audio from 0 (hereinafter, these audio input sources may be abbreviated as audio input sources of the digital video camera 70 or the like) is composed of a high image quality encoder such as an MPEG4 encoder. The CH2 encoder 12-2 performs digital compression encoding and converts the stream data. This stream data is called CH2 stream data.

  The CH2 encoding unit 12-2 has a function of converting a still image input from the digital video camera 70 or the like into a moving image and outputting the moving image. Details of this function will be described later.

  The combining unit 51-1 creates stream data (combined stream data) obtained by combining the CH1 stream data and the CH2 stream data, and outputs the stream data to the packetizing unit 25.

  The combined stream data is packetized by the packetizer 25 and temporarily stored in the transmission buffer 26. The transmission buffer 26 sends the packet to the network 10 at a certain timing via the communication interface 13. For example, when a moving image of 30 frames per second is captured, the transmission buffer 26 has a capability of storing and transmitting data of one frame in one packet.

  In the present embodiment, even if a decrease in the transmission band of the network 10 is estimated, the transmission frame rate is not reduced, that is, the frame is not thinned out. This is to prevent the movement of the image from becoming jerky and not smooth.

  The video / audio data separating unit 45-1 separates video data and audio data from the multiplexed data input from the external input terminal 30-1.

  The moving image data or still image data separated by the video / audio data separation unit 45-1 is decoded by the moving image decoder 41 or the still image decoder 42, respectively, and then stored in the video buffer 80 at predetermined time intervals as frame images. Temporarily stored. Note that the number of frames per second (frame rate) stored in the video buffer 80 needs to match the frame rate (for example, 30 fps (frame per second)) of the video capture buffer 54 described later.

  The audio data separated by the video / audio data separation unit 45-1 is decoded by the audio decoder 47-2 and then temporarily stored in the audio buffer 81.

  The NTSC decoder 15 is a color decoder that converts an NTSC signal input from the camera 4 into a luminance signal and a color difference signal. The NTSC signal is separated into a luminance signal and a carrier color signal by a Y / C separation circuit. The signal is demodulated by a color signal demodulating circuit to generate a color difference signal (Cb, Cr).

  The audio data converting unit 14 converts the analog audio sound signal input from the microphone 3 into digital data and outputs the digital data to the audio capture buffer 53.

  A switcher (switching circuit) 78 controls the input video to the video buffer 80 according to the control of the control unit 11, as a moving image or a still image of the digital video camera 70, a still image from the digital still camera 71, and a recording medium by the media reader 74. Switching to one of the moving image or the still image read from 73.

  The synthesizing unit 51-2 synthesizes the video from the video content input source such as the digital video camera 70 and the moving image frame image decoded from the CH1 decoding unit 13-1 and the CH2 decoding unit 13-2. The composite image is output to the video output unit 17. The composite image obtained in this way is displayed on the monitor 5.

  The counterpart communication terminal 1 individually streams the video data encoded by the CH1 encoding unit 12-1 and the video data encoded by the CH2 encoding unit 12-2 by the streaming circuit 22, respectively. The stream data encoded by the encoding unit 12-1 is decoded by the CH1 decoding unit 13-1, and the stream data encoded by the CH2 encoding unit 12-2 is decoded by the CH2 decoding unit 13-2 into moving images or sounds, respectively. And output to the combining unit 51-2.

  The synthesizing unit 51-2 receives the video of the camera 4, that is, the own video, the decoded moving image of the CH1 decoding unit 13-1, that is, the counterpart video, and the decoded moving image of the CH2 decoding unit 13-2, that is, the video content. Then, the image is resized and combined so as to fit in the display area on the display screen of the monitor 5. Resizing is performed in accordance with display mode switching input from the remote controller 60.

  FIG. 3 shows an example of an arrangement of images displayed on the monitor 5. As shown in this figure, the video (input video) of the camera 4 of the other party's communication terminal 1 is input to the monitor 5 in the first display area X1 and the video content of the digital video camera 70 etc. of the other party's communication terminal 1 is input. The originally input video (video content) is displayed in the second display area X2, and the video (self video) input from the own camera 4 is displayed in the third display area X3.

  The images arranged in the first display area X1 to the third display area X3 are not limited to those shown in this figure, and are switched according to the display mode setting described later.

  In addition, a content menu M that lists video information input sources such as the digital video camera 70 for the switcher 78 of one's own and other information, and a message & information display area Y that displays various messages and notifications are each in one screen. The images are reduced so as to fit and are displayed in non-overlapping areas.

  In this figure, each display area X1 to X3 in one display screen is divided and displayed according to a predetermined area ratio, but this screen division method can be variously modified. In addition, it is not always necessary to display all of the plurality of images at the same time in one screen. The display mode is switched according to a predetermined operation of the remote controller 60, and only the own image, only the other image, only the image content, or a part of them is combined. May be displayed. The display mode will be described later.

  In the content menu M, any item can be selected by operating the remote controller 60. The control unit 11 performs control to switch the video content input source using the switcher 78 in accordance with an item selection operation of the remote controller 60. Thereby, the video to be displayed as the video content can be arbitrarily selected. Here, when the “Web server” item is selected, the Web content acquired by the Web browser module 43 from the Web content server 90, and when the “Content server” item is selected, the streaming content acquired by the streaming module 44 from the streaming server 91 is “DV”. "" Is the video content from the digital video camera 70, the "Still" item is the video from the digital still camera 71, and the "Media" item is the video read from the recording medium 73. .

  The CH1 encoding unit 12-1 sequentially compresses and encodes audio capture data from the microphone 3 supplied from the audio capture buffer 53 in accordance with the MPEG method or the like. The encoded voice data is packetized by the packetizing unit 25 and stream-transmitted to the counterpart communication terminal 1.

  The CH2 encoding unit 12-2 converts either the audio from the streaming module 44 or the audio from the digital video camera 70 (the audio input source of the digital video camera 70 or the like), which is the audio input source by the switcher 78, into MPEG. Compression encoding is performed according to a method or the like. The encoded voice data is packetized by the packetizing unit 25 and stream-transmitted to the counterpart communication terminal 1.

  The CH1 decoding unit 13-1 decodes the audio data encoded by the CH1 encoding unit 12-1. The CH2 decoding unit 13-2 decodes the audio data encoded by the CH2 encoding unit 12-2.

  The synthesizing unit 51-2 synthesizes the audio data decoded by the CH1 decoding unit 13-1 and the audio data decoded by the CH2 decoding unit 13-2, and sends the synthesized audio data to the audio output unit 16. Output. In this way, the sound collected by the microphone 3 of the other party's communication terminal 1 and the sound obtained from the digital video camera 70 or the like connected to the other party's communication terminal 1 are reproduced by the own speaker 2.

  The band estimation unit 11 c estimates the transmission band from the jitter (fluctuation) of the network 10.

  The encoding control unit 11e changes the video transmission bit rate of the CH1 encoding unit 12-1 and the CH2 encoding unit 12-2 according to the estimated transmission band. That is, if it is estimated that the transmission band is reduced, the video transmission bit rate is decreased, and if it is estimated that the transmission band is increased, the video transmission bit rate is increased. By doing so, it is possible to prevent packet loss from occurring due to packet transmission exceeding the transmission band, and to perform smooth stream data transmission according to changes in the transmission band.

  Specific band estimation by the band estimation unit 11c may be performed as follows, for example. When an RTCP packet (RTCP SR) of SR (Sender Report) type is received from the communication terminal 1b of the other party, the number of lost RTCP SRs is obtained by counting the sequence number of the sequence number field in the header of the RTCP SR packet. calculate. Then, an RR (Receiver Report) type RTCP packet (RTCP RR) in which the number of losses is described is transmitted to the communication terminal 1 of the other party. RTCP RR also describes the time from reception of RTCP SR to transmission of RTCP RR (referred to as response time for convenience).

  When the communication terminal 1b of the other party receives RTCP RR, RTT (Round Trip Time) that is the time obtained by subtracting the response time from the time from the RTCP SR transmission time to the RTCP RR reception time is calculated. Also, referring to the number of RTCP SR transmission packets and the number of RTCP RR losses, (loss number) / (number of transmission packets) = packet loss rate within a regular period is calculated. This RTT and packet loss rate are used as a communication status report.

  The interval at which monitoring packets are sent is considered to be appropriate once every 10 seconds to several tens of seconds. However, there are many cases where the network state cannot be accurately grasped by estimation based on one monitoring packet attempt. The estimation accuracy increases if the estimation is performed separately and taking the average or the like. If the number of monitoring packets is increased, it itself becomes a factor for narrowing the bandwidth, so it is preferable to keep it at 2-3% of the total communication amount.

  In addition to the above description, a communication status report can be obtained by using various QoS (Quality of Service) control techniques for the bandwidth estimation unit 11c.

  Note that the bit rate of audio encoding may be changed according to the estimated transmission band. However, since the audio transmission band has a lower contribution ratio to the band than video, there is no problem even if it is fixed.

  Packets of stream data received from other communication terminals 1 via the communication interface 13 are temporarily stored in the reception buffer 21 and then output to the streamer 22 at a fixed timing. The fluctuation absorbing buffer 21a of the reception buffer 21 adds a delay from the reception of the packet to the start of reproduction for continuous reproduction even if the transmission delay time of the packet fluctuates and the arrival interval varies. The streaming device 22 reconstructs the packet data into stream reproduction data.

  The CH1 decoding unit 13-1 and the CH1 decoding unit 13-2 are video / audio decoding devices configured by an MPEG4 decoder or the like.

  The display control unit 11d controls the synthesizing unit 51-2 according to the screen switching signal input from the remote controller 60, and the video data (CH1 video data) decoded by the CH1 decoding unit 13-1 and the CH2 decoding. The video data (CH2 video data) decoded by the unit 13-2, the video data input from the NTSC decoder 15 (self video), and the video data (video content) input from the video buffer 80 are synthesized in whole or in part. Are output (combined output), or any one of the video data is output without being combined with the other (through output). The video data output from the combining unit 51-2 is converted into an NTSC signal by the video output unit 17 and output to the monitor 5.

  4 to 9 illustrate screens of the monitor 5 displaying the synthesized video data. These screens are sequentially switched by a display mode switching operation by the remote controller 60.

  FIG. 4 shows a screen display of the monitor 5 when the synthesizing unit 51-2 outputs only the video data (self video) from the camera 4 to the video output unit 17 without synthesizing with the other video data. On this screen, only the video (self video) taken by the user's own camera 4 is displayed in full screen.

  FIG. 5 shows the monitor 5 when the synthesizing unit 51-2 outputs only the video data (partner video) from the CH1 decoding unit 13-1 to the video output unit 17 without synthesizing with the other video data. The screen display is shown. On this screen, only the video (partner video) taken by the other party's camera 4 is displayed in full screen.

  In FIG. 6, the synthesizing unit 51-2 synthesizes the video data (partner video) from the CH1 decoding unit 13-1 and the video data (self video) from the own camera 4 and outputs the synthesized video data to the video output unit 17. The screen display of the monitor 5 in the case of having performed is shown. On this screen, the partner video and the self video are displayed in the display areas X1 and X3, respectively.

  FIG. 7 shows the video data (video content) from the video data (video content) from the CH2 decoding unit 13-2, the video data from the own camera 4 by the synthesizing unit 51-2. The screen display of the monitor 5 when the data (self video) is synthesized and output to the video output unit 17 is shown. On this screen, the partner video is resized and displayed so as to fit within the display area X1, the video content within the display area X2, and the self video within the display area X3. In addition, the display areas X1 and X3 maintain a predetermined area ratio such that the display area X1 is larger than the display area X3.

  FIG. 8 shows the video data (video content) from the video data (video content) from the CH2 decoding unit 13-2, the video data from the own camera 4 and the video data (video content) from the CH2 decoding unit 13-1. The screen display of the monitor 5 when the data (self video) is synthesized and output to the video output unit 17 is shown. On this screen, the video content is displayed in the display area X1, the partner video is displayed in the display area X2, and the self video is displayed in the display area X3.

  FIG. 9 shows the monitor 5 when the combining unit 51-2 outputs only the video data (video content) from the CH2 decoding unit 13-2 to the video output unit 17 without combining with the other video data. The screen display is shown. Only video content is displayed on this screen.

  FIG. 10 shows an example of the area ratio of the display areas X1 to X3. Here, the screen with a screen ratio of 4: 3 is equally divided into nine tiles, the area of the display area X1 is 4 tiles, and the areas of the display areas X2 and X3 are 1 tile. The area of the content menu display area M is 1 tile, and the area of the message / information display area is 2 tiles.

  When a screen switching signal is input from the remote controller 60, the communication terminal 1b transmits a control packet indicating that the screen switching signal has been input to the communication terminal 1a via the network 10. A similar function is also provided in the communication terminal 1a.

  The encoding control unit 11e displays the display areas X1, X2, or X3 of the monitor 5 of the counterpart communication terminal 1 according to the area ratio of the display areas X1, X2, or X3 identified by the control packet received from the counterpart communication terminal 1. The CH1 code is allocated so that the transmission band of the video (which can be specified by the control packet) is displayed within the range of the estimated transmission band so that the data fits within the allocated transmission band (packet overflow does not occur). The quantization circuit 117 of the encoding unit 12-1 and the CH2 encoding unit 12-2 is controlled.

  Note that the audio data decoded by the CH1 decoding unit 13-1 and the CH2 decoding unit 13-2 is converted into an analog audio signal by the audio output unit 16 and output to the speaker 2. If necessary, the voice data input from the user's own digital video camera 70 or the like and the voice data included in the content data can be synthesized by the synthesis unit 51-2 and output to the voice output unit 16.

  The communication interface 13 is provided with a network terminal 61, which is connected to the network 10 by being connected to a broadband router, an ADSL modem, or the like by various cables. One or more network terminals 61 are provided.

  Note that when the communication interface 13 is connected to a router having a firewall or NAT function (network address translation, which performs mutual conversion between a global IP address and a private IP address), the SIP communication terminals 1 cannot be directly connected to each other. It is recognized by those skilled in the art that so-called NAT traversal) occurs. In order to minimize the delay in video / audio transmission and reception by directly connecting the communication terminals 1, STUN technology using STUN (Simple Traversal of UDP through NATs) server 30 and NAT traversal by UPnP (Universal Plug and Play) server It is preferable to implement the function in the communication terminal 1.

  The control unit 11 performs overall control of each circuit in the communication terminal 1 based on operation inputs from the operation unit 18 including various buttons and keys or the remote controller 60. The control unit 11 is configured by a calculation device such as a CPU, and the own display mode notification unit 11a, the other party display mode detection unit 11b, the band estimation unit 11c, the display control unit 11d, the encoding control unit 11e, and the operation specific signal transmission unit. Each function of 11f is realized by a program stored in the storage medium 23.

  An address for uniquely identifying each communication terminal 1 (not necessarily synonymous with a global IP address), a password necessary for the account management server 8 to authenticate the communication terminal 1, and a startup program for the communication terminal 1 are in a power-off state. It is stored in a non-volatile storage medium 23 that can hold data. The program stored here can be updated to the latest version by the update program provided from the account management server 8.

  Data necessary for various processes of the control unit 11 is stored in a main memory 36 constituted by a RAM that temporarily stores data.

  The communication terminal 1 is provided with a remote control light receiving circuit 63, and a remote control light receiving unit 64 is connected to the remote control light receiving circuit 63. The remote control light receiving circuit 63 converts an infrared signal incident on the remote control light receiving unit 64 from the remote control 60 into a digital signal and outputs the digital signal to the control unit 11. The control unit 11 controls various operations in accordance with the digital infrared signal input from the remote control light receiving circuit 63.

  The light emission control circuit 24 controls light emission, blinking, and lighting of the LED 65 provided on the outer surface of the communication terminal 1 under the control of the control unit 11. A flash lamp 67 can also be connected to the light emission control circuit 24 via a connector 66, and the light emission control circuit 24 also controls light emission / flashing / lighting of the flash lamp 67. The RTC 20 is a built-in clock.

  FIG. 11 is a block diagram showing a main configuration common to the CH1 encoding unit 12-1 and the CH2 encoding unit 12-2. The CH1 encoding unit 12-1 and CH2 encoding unit 12-2 (may be collectively referred to as the encoding unit 12) include an image input unit 111, a motion vector detection circuit 114, a motion compensation circuit 115, a DCT 116, a quantization A circuit 117, a variable length encoder (VLC) 118, an encoding control unit 11e, a static block detection unit 124, a static block storage unit 125, and the like are provided. This apparatus partially includes a configuration of an MPEG video encoding apparatus that combines motion compensation prediction encoding and compression encoding by DCT.

  The image input unit 111 stores the video (only the moving image of the camera 4, only the moving image or still image input from the digital video camera 70, or the moving image and still image thereof) stored in the video capture buffer 54 and the video buffer 80. A moving image composed of a composite image) is input to the frame memory 122.

  The motion vector detection circuit 114 detects the motion vector by comparing the current frame image represented by the data input from the image input unit 111 with the previous frame image stored in the frame memory 122. This motion vector detection is performed by dividing the input current frame image into a plurality of macro blocks, and appropriately searching the macro block to be searched within the search range set on the previous frame image in units of individual macro blocks. By repeating the error calculation while moving, the macro block that is most similar to the macro block to be searched (the macro block with the smallest error) is searched from the search range, the amount of deviation between the macro block and the macro block to be searched, and The direction of deviation is taken as the motion vector for the searched macroblock. Then, by synthesizing the motion vector obtained for each macroblock in consideration of the error for each macroblock, a motion vector that minimizes the prediction difference in predictive coding can be obtained.

  The motion compensation circuit 115 generates motion prediction image data by performing motion compensation on the prediction reference image based on the detected motion vector, and outputs the data to the subtractor 123. The subtractor 123 generates difference data representing a prediction difference by subtracting a predicted image represented by data input from the motion compensation circuit 115 from a current frame image represented by data input from the image input unit 111. To do.

  A DCT (discrete cosine transform) unit 116, a quantization circuit 117, and a VLC 118 are sequentially connected to the subtractor 123. The DCT 116 orthogonally transforms the difference data input from the subtractor 123 for each arbitrary block, and outputs the result. The quantization circuit 117 quantizes the difference data input from the DCT 116 after the orthogonal transformation in a predetermined quantization step. And output to VLC118. A motion compensation circuit 115 is connected to the VLC 118, and motion vector data is also input from the motion compensation circuit 115.

  The VLC 118 encodes the difference data that has undergone orthogonal transformation and quantization using a two-dimensional Huffman code, and also encodes input motion vector data using a Huffman code, and multiplexes both. Then, the variable length encoded moving image data is output at a rate determined based on the encoding bit rate output from the encoding control unit 11e. The variable-length encoded moving image data is output to the packetizing unit 25 and transmitted as a packet to the network 10 as image compression information. The encoding amount (bit rate) of the quantization circuit 117 is controlled by the encoding control unit 11e.

  The data structure of the encoded moving image data created by the VLC 118 has a hierarchical structure, and the block layer, macroblock layer, slice layer, picture layer, GOP layer, and sequence layer are arranged from the lower order.

  The block layer is composed of DCT blocks that are units for performing DCT. The macroblock layer is composed of a plurality of DCT blocks. The slice layer is composed of a header part and one or more macroblocks. The picture layer is composed of a header part and one or more slice layers. A picture corresponds to one screen. The GOP layer includes a header part, an I picture that is a picture based on intra-frame coding, and a P and B picture that are pictures based on predictive coding. The I picture can be decoded only with its own information, and the P and B pictures require previous or previous pictures as predicted pictures and are not decoded alone.

  Also, at the beginning of the sequence layer, GOP layer, picture layer, slice layer, and macroblock layer, an identification code having a predetermined bit pattern is arranged, and the encoding parameters of each layer are stored following the identification code. A header part is arranged.

  The macroblock included in the slice layer is a set of a plurality of DCT blocks, and is obtained by dividing a screen (picture) into a lattice shape (for example, 8 pixels × 8 pixels). The slice is formed by, for example, connecting the macro blocks in the horizontal direction. When the screen size is determined, the number of macroblocks per screen is uniquely determined.

  In the MPEG format, the slice layer is one variable length code sequence. A variable-length code sequence is a sequence in which a data boundary cannot be detected unless the variable-length code is decoded. When decoding the MPEG stream, the header part of the slice layer is detected to find the start point and end point of the variable length code.

  Here, if the image data input to the frame memory 122 is only a still image, the motion vectors of all macroblocks are zero, and decoding can be performed using only I pictures. Then, it is not necessary to send B and P pictures. For this reason, even if the transmission bandwidth of the network 10 is narrowed, the still image can be sent to the communication terminal 1 of the other party as a moving image with relatively high precision.

  Even if the image data input to the frame memory 122 is a composite image of a still image and a moving image, the motion vector of the macroblock corresponding to the still image is zero, and that portion is not sent as a skipped macro. That's it.

  If the image data input to the frame memory 122 is only a still image, the frame rate may be reduced and the code amount of the I picture may be increased instead. Thereby, a still picture without movement can be displayed finely.

  Regardless of the type of the input source, regardless of whether the input source of the still image is switched to the Web browser module 43, the digital video camera 70, the digital still camera 71, or the media reader 73 by the switcher 78 of the own communication terminal 1a, A frame moving image in which a macroblock corresponding to a still image has a motion vector of zero is transmitted in real time to the counterpart communication terminal 1b. For this reason, even if the input source of the still image by the switcher 78 is switched irregularly in the own communication terminal 1a, the frame moving image transmitted to the communication terminal 1 of the other party is quickly switched following this, and the result Thus, the still image displayed on the other party's communication terminal 1b is also quickly switched.

  Next, operations performed between the communication terminal 1a and the communication terminal 1b will be described with reference to the flowchart of FIG.

  First, when a screen switching signal is input from the remote controller 60, the own display mode notifying unit 11a of the communication terminal 1b specifies a control packet (display mode specifying) that specifies a display mode set according to the input screen switching signal. Packet) to the communication terminal 1a (B1).

  The other party display mode detection unit 11b of the communication terminal 1a determines whether or not the display mode specifying packet has been received from the communication terminal 1b (A1). If it is determined that the display mode specifying packet has been received, the display mode set by the communication terminal 1b is specified. (A2).

  Further, the communication terminal 1b periodically transmits the monitoring packet RTCP RR regardless of the display mode setting by the operation of the remote controller 60 (B2).

  The communication terminal 1a receives the monitoring packet RTCP RR from the sequential communication terminal 1b (A3).

  The bandwidth estimation unit 11c of the communication terminal 1a estimates the current bandwidth of the network 10 based on the monitoring packet RTCP RR (A4).

  The display control unit 11d of the communication terminal 1a refers to the band allocation table corresponding to the specified display mode, and determines the allocated band for each video displayed in each display area in the display mode (A5).

  FIG. 13 shows a bandwidth allocation table corresponding to the “PoutP screen (content dialogue (1))” display mode (corresponding to the screen of FIG. 7) as an example of the bandwidth allocation table. In this table, of the estimated bands, the band allocated to the display area X1 (4 tiles) having a relatively large area and the band allocated to the display area X2 or X3 (1 tile) having a relatively small area. Are associated with each band to be estimated. Although how to allocate the band to the two display areas X1, X2 (or X3) within the estimated band is arbitrary, it preferably corresponds to the area ratio of the display areas X1, X2 (or X3). . In this way, it is possible to increase the amount of transmission information of video displayed in the large display area X1 and display it finely, and it is not necessary to reduce the amount of transmission information of video displayed in the small display area X2 and display it in detail. The amount of video transmission can be saved, and a limited transmission band can be used effectively.

  However, if the band is strictly associated with the area ratio of the display area, when the band is lowered, the information amount of the small display area X2 or X3 is extremely reduced, and the viewing may not endure. For this reason, the allocated bandwidth is set so that the transmission bandwidth does not become lower than the predetermined lower limit value. In this table, the allocated bandwidth of the display area X2 or X3 is set so as not to fall below “128k”.

  The encoding control unit 11e changes the transmission bit rate of the video displayed in the display area X1, X2 or X3 according to the transmission band assigned to the display area X1, X2 or X3, and within the assigned transmission band. The quantization circuit 117 of the CH1 encoding unit 13-1 and the CH2 encoding unit 13-2 is controlled (A6) so that the data can be accommodated in (so that no packet overflow occurs).

  That is, the video data encoded by the CH1 encoding unit 13-1 is displayed in the display area X1 and the video data encoded by the CH2 encoding unit 13-2 is displayed in the display area X2 by the screen switching signal of the remote control 60 of the communication terminal 1b. Alternatively, if it is instructed to be displayed on X3, the transmission bit rate of CH1 encoding unit 13-1 is controlled to be within the transmission band assigned to display area X1, and transmission of CH2 encoding unit 13-2 is performed. The bit rate is controlled to be within the transmission band assigned to the display area X2 or X3.

  Alternatively, the screen switching signal of the remote controller 60 of the communication terminal 1b instructs that the video data encoded by the CH1 encoding unit 13-1 is displayed in the display area X2 or X3, and the CH2 encoding unit 13-2 If it is instructed that the encoded video data is displayed in the display area X1, the transmission bit rate of the CH1 encoder 13-1 is controlled to be within the transmission band assigned to the display area X2 or X3, and The transmission bit rate of the CH2 encoding unit 13-2 is controlled so as to be within the transmission band assigned to the display area X1.

  Packets of video data whose bit rate is controlled in this way are sequentially sent to the communication terminal 1b. The communication terminal 1b streams the packets and reproduces and displays the video data in a display area corresponding to the set display mode.

  Since the transmission band changes from time to time, a series of processing including packet transmission band estimation (A3) to transmission bit rate control (A6) is periodically repeated. Control packet detection (A1) and display mode specification (A2) are also periodically repeated.

  Since the configurations of the communication terminal 1a and the communication terminal 1b are equivalent, the above operation is established even if the roles of both are exchanged. By performing the transmission bit rate control as described above independently at the communication terminal 1a and the communication terminal 1b, bi-directional video data transfer can be performed even in a network having an asymmetric transmission band such as ADSL. It can be optimized according to fluctuations.

Block diagram of the video / audio communication system according to the first embodiment. Block diagram of communication terminal The figure which shows an example of the screen displayed on the monitor 5 Conceptual diagram of full screen self-image display mode Conceptual diagram of full screen partner video display mode Conceptual illustration of PoutP screen (normal dialogue) display mode PoutP screen (content dialogue (1)) display mode conceptual diagram PoutP screen (content dialogue (2)) display mode conceptual diagram Conceptual illustration of full screen (content dialogue (3)) display mode Conceptual illustration of tiles that define the display area Detailed block diagram of the encoder Flow chart showing operation of communication terminal Conceptual diagram of communication bandwidth allocation table

Explanation of symbols

11a: own display mode notification unit, 11b: counterpart display mode detection unit, 11e: encoding control unit

Claims (12)

An input unit for inputting a desired still image;
A developing unit that expands the still image input to the input unit;
An encoding unit that encodes a compressed moving image that uses the still image expanded in the expansion unit as a reference image and is restored to a one-frame moving image using a difference image with a zero motion vector with respect to the reference image;
A transmitting unit that transmits the compressed compressed video encoded by the encoding unit to a desired counterpart communication terminal;
A communication terminal comprising: A selection unit that selects a desired still image input system from one or a plurality of still image input systems;
The communication terminal according to claim 1, wherein the input unit inputs a desired still image from a still image input system selected by the selection unit. An operation unit that accepts manual input operations;
An input system display unit for displaying a list of input systems of the still image;
The communication terminal according to claim 2, wherein the selection unit selects an input system arbitrarily designated by an input operation to the operation unit from a list of input systems of still images displayed on the input system display unit. A plurality of encoding units for individually encoding videos input from a plurality of input systems;
A transmission unit that transmits a plurality of videos encoded by the plurality of encoding units to a communication terminal of the other party;
A display area specifying signal receiving unit for receiving a display area specifying signal capable of specifying each display area on the display screen of the other communication terminal of the plurality of videos transmitted to the other communication terminal by the transmitting unit;
A bandwidth estimation unit for estimating a transmission bandwidth of the network;
The display area specifying signal receiving unit approximately follows the display area ratio of each of the plurality of videos indicated by the display area specifying signal, and each of the plurality of videos is within the transmission band estimated by the band estimating unit. A bandwidth allocation unit that determines transmission bandwidth allocation;
An encoding control unit that controls an amount of encoding of a plurality of videos by the plurality of encoding units within a range of a transmission band allocated by the band allocation unit;
A communication terminal comprising: An allocation table storage unit for storing an allocation table in which transmission band allocation according to a ratio of display areas of each of a plurality of videos and transmission bands to be estimated by the band estimation unit are associated;
The bandwidth allocating unit refers to an allocation table in the allocation table storage unit, and determines transmission band allocation according to each allocation of a plurality of videos associated with the transmission band estimated by the band estimation unit. 4. The communication terminal according to 4.   The communication terminal according to claim 5, wherein the transmission band allocation of each of the plurality of videos stored in the allocation table exceeds a predetermined lower limit value. The display area specifying signal received by the display area specifying signal receiving unit includes information for specifying a display mode that defines an area for displaying each of a plurality of images displayed by the counterpart communication terminal,
The band allocating unit specifies a display mode set in the communication terminal of the other party from the display area specifying signal received by the display area specifying signal receiving unit, and each of a plurality of videos is displayed in the specified display mode. The communication terminal according to any one of claims 4 to 6, wherein an allocation of a transmission band of a video displayed in each area is determined in accordance with an area ratio of the area.   The communication terminal according to claim 7, wherein the display mode defines an arrangement of each of the plurality of videos in a part or all of a display area in which one screen is equally divided by a predetermined area. An input unit for inputting a desired still image, a developing unit for developing the still image input to the input unit, a still image developed in the developing unit as a reference image, and a motion vector for the reference image is zero A transmission-side communication terminal comprising: an encoding unit that encodes a compressed moving image that is restored to a one-frame moving image by a difference image; and a transmission unit that transmits the compressed moving image encoded by the encoding unit. When,
A receiving unit that receives the encoded compressed moving image from the transmission-side communication terminal, a decoding unit that decodes the encoded compressed moving image received by the receiving unit, and a decoding unit that decodes the encoded moving image A receiving-side communication terminal comprising: a developing unit that develops the compressed compressed video into the one-frame moving image; and a display unit that displays the one-frame moving image developed by the developing unit as the still image;
A communication system comprising: A plurality of encoding units that individually encode videos input from a plurality of input systems; a transmission unit that transmits a plurality of videos encoded by the plurality of encoding units via a network; and transmission of the network A transmission side communication terminal comprising a band estimation unit for estimating a band, a reception unit for receiving the plurality of encoded videos from the transmission side communication terminal via the network, and the code received by the reception unit A decoding unit that decodes the plurality of converted videos, a development unit that develops the plurality of videos decoded by the decoding unit, and a display unit that displays the plurality of videos developed by the development unit A communication system including a receiving communication terminal,
The receiving-side communication terminal includes a display area specifying signal transmitting unit that transmits a display area specifying signal that can specify a display area of each of the plurality of videos transmitted from the transmitting-side communication terminal. ,
The transmitting communication terminal includes a display area specifying signal receiving unit that receives the display area specifying signal from the receiving communication terminal, and a plurality of videos indicated by the display area specifying signal received by the display area specifying signal receiving unit. A bandwidth allocation unit that determines allocation of transmission bands for each of the plurality of videos within a transmission band range estimated by the bandwidth estimation unit according to a ratio of each display area, and a transmission allocated by the bandwidth allocation unit And a coding control unit that controls an amount of coding of the plurality of videos by the plurality of coding units within a band range. Inputting a desired still image;
Expanding the input still image; and
Encoding a compressed moving image such that the developed still image is a reference image and is restored to a one-frame moving image by a difference image having a zero motion vector with respect to the reference image;
Transmitting the encoded compressed video to a communication terminal of a desired counterpart;
Including a communication method. Individually encoding video input from a plurality of input systems;
Transmitting a plurality of encoded videos to the communication terminal of the other party;
Receiving a display area specifying signal capable of specifying each display area on the display screen of the counterpart communication terminal of the plurality of videos transmitted to the counterpart communication terminal;
Estimating the transmission bandwidth of the network;
Determining the transmission band allocation of each of the plurality of videos within the estimated transmission band in accordance with a ratio of the display areas of each of the plurality of videos indicated by the received display area specifying signal;
Controlling the amount of encoding of the plurality of videos within a transmission bandwidth allocated to each of the plurality of videos;
Including a communication method.

Images