JP4768250B2 - Transmission device, reception device, transmission / reception device, transmission method, and transmission system - Google Patents

Description

  The present invention relates to a video transmission system that encodes a video signal input on the transmission side and transmits the encoded image data via a network, and more particularly, changes in characteristics of the input video and display on the reception side. The present invention relates to a video transmission system that performs optimal encoder control in response to changes in the situation and fluctuations in the effective bandwidth of a network.

  2. Description of the Related Art In recent years, video transmission systems that encode and transmit video data in real time using a network such as the Internet have been developed with the progress of digital encoding technology for content such as images and broadband network technology.

  For example, a video signal input from a video capture device such as a camera is received, digitally encoded by a video encoder, and transmitted as transmission data to a transmission path.

  The video data digitized by digital encoding of the video signal has an enormous amount of data and often does not fit in the network bandwidth. For this reason, a method of compressing and encoding digitized data is used. As an international standard system for such digital encoding technology, ISO MPEG-1 (Moving Picture Coding Experts Group-1), MPEG2, MPEG4, ITU-T H.264, etc. 261, H262, and H263.

  On the other hand, as a data transmission method, in recent years, the number of systems that packetize and transmit / receive data is increasing due to the spread of the Internet / intranet. (User Datagram Protocol / Internet Protocol) is often used.

  The Internet and the like are connected through a myriad of networks, and it is usually not known which network is in what kind of situation. Also, since the amount of data flowing there changes from moment to moment, a mechanism for determining how much data can be communicated in real time is necessary.

  Therefore, in a system that transmits video data packets and audio data packets in real time, an application that carries on UDP by using a packet format called RTP (Real-time Transport Protocol) that adds time information to the packet and transmits the packet. Is increasing. In RTP, by adding the packet sequence number and time information in the RTP header on the transmission side, video and audio are reproduced using the correct time information on the reception side, or the order is transmitted over the network using the packet number. It is now possible to determine the exchanged packets and detect packet loss.

  Further, by using RTCP (Real-time Transport Control Protocol), the packet loss rate and delay information detected on the receiving side can be notified to the transmitting side. How to use the RTCP notification information depends on the application, and is not determined by the standard.

  However, even if RTP is used, UDP that does not retransmit is used, and data compressed by a video data compression technique such as MPEG is very vulnerable to data loss and error mixing. Therefore, it is necessary to control the transmission bit rate even if the video data quality is lowered so that the image does not occur.

  In general, as shown in FIG. 14, a countermeasure for congestion of the transmission path is taken such that the transmission bit rate is controlled while monitoring the congestion state of the transmission path based on the RTCP notification information received on the transmission side. The transmission bit rate is controlled by adjusting the output data amount of the encoding process of the video encoder.

  In addition to the purpose of countermeasures for congestion of the transmission path, the control method of the encoding process includes a control method that takes into account the response to the size of the video data displayed on the receiving side, the number of displayed images, and the like. As a technique corresponding to a change in the size of video data to be displayed, a change in the number of displayed images, and the like, there is a technique described in Patent Document 1. In Patent Document 1, information on current video display is managed as a mode setting state table on the reception side, and video display mode information when the video display mode of the display video is changed is notified to the transmission side. By recognizing it on the transmission side and changing the video encoding process as necessary, the encoding when the motion is required or the image quality is required is finely controlled.

JP 2001-69472 A

  However, in the technique described in Patent Document 1, since the transmission bit rate is uniquely determined by the display state on the reception side, congestion occurs and the video on the reception side deteriorates depending on the network state. Alternatively, even if the effective bandwidth of the network is sufficiently large and higher-quality video transmission is possible, this state is not reflected in the transmission bit rate control, and the display video on the receiving side remains low quality. .

  An object of the present invention is to perform video transmission optimal for display quality on the receiving side in accordance with the transmission bandwidth.

  In the present invention, the effective bandwidth of the network between video transmission / reception devices and the change in the display status of the video reception device are monitored, and the values of parameters (frame rate, image capture size) used for encoding are optimized according to the characteristics of the video encoder. The display video quality of the video receiver is controlled using a method of controlling the encoder of the video transmitter using this optimization parameter value.

  According to an aspect of the present invention, there is provided a video transmission device that encodes a video signal and transmits encoded data based on the encoded video signal to a reception side, and includes encoding parameters for encoding the video signal. A video transmission apparatus configured to set based on at least one of a reception state of a reception unit that receives the encoded data acquired from the reception side and a display state of a display unit that displays the encoded data Is provided.

  A transmission bit rate (data rate output from the encoder) that avoids congestion between the video transmission / reception devices is calculated, and an encoding parameter (capture size, frame rate, etc.) of the encoder of the video transmission device is controlled. When this parameter is determined, the display status (display size, display device capability, display viewer's preference, etc.) of the video decoded and displayed by the video receiver is reflected. Thereby, transmission of video quality suitable for the display screen state on the reception side can be performed on the transmission side within the effective band range between transmission and reception.

  The display status is a display screen size based on the encoded data of the display means. By notifying the transmission side of the display screen size from the reception side, transmission of video quality reflecting the reception video display screen size of the reception side can be performed on the transmission side.

  The display status is information that uniquely specifies a display capability of the display means or a display device used for the display means. From the reception side, the display device name or the corresponding output resolution of the display device is notified to the transmission side. Thereby, transmission of video quality reflecting the capability of the reception video display device on the reception side can be performed on the transmission side.

  The display status is an instruction on the receiving side, including an instruction to increase / decrease the frame rate of the encoded data displayed by the display means, or an instruction to increase / decrease the capture size of the encoded data. It is characterized by that. Request the frame rate and capture size from the receiving side to the transmitting side. As a result, transmission of video quality reflecting image quality control instructions for video quality on the reception side can be performed on the transmission side.

  The display status is an instruction on the receiving side, and is a request for increasing / decreasing the number of display colors of encoded data displayed by the display means. The receiving side notifies the transmitting side of a request to increase / decrease the number of display colors of the currently displayed video. Thus, transmission of video quality that reflects in real time a request to increase / decrease the number of display colors of the display video on the reception side can be performed on the transmission side.

  The setting of the encoding parameter is characterized in that the display status or the reception status and the encoding parameter of the encoder are set with reference to association data. As a feature of MPEG moving image compression data, the magnitude of the motion of the original video signal can be known from the data size of each frame, so that priority is given to image quality or frame rate according to the magnitude of the motion of the video signal. As a result, when the receiver display screen is a call video, it has a medium definition and a medium frame rate, when it is a video playback video, it has a low definition and a high frame rate, and when it is a still image, it has a high resolution. Transmission of video quality in accordance with the video characteristics of the reception side can be performed on the transmission side, such as transmission of video with definition and low frame rate.

  A frame data size measuring unit that measures a data size of each frame of the encoded data; and setting an encoding parameter for encoding the video signal based on a measurement result of the frame data size measuring unit. Features. Video characteristics: Call video = “Small screen fluctuation and frame rate required”, Video playback video = “Big screen fluctuation and frame rate required”, Still image = “High screen fluctuation but no frame rate required” And the receiving side display video characteristics currently being displayed are notified to the transmitting side. As a result, when the receiver display screen is a call video, it has a medium definition and a medium frame rate, when it is a video playback video, it has a low definition and a high frame rate, and when it is a still image, it has a high resolution. Transmission of video quality in accordance with the video characteristics of the reception side can be performed on the transmission side, such as transmission of video with definition and low frame rate.

  Input device detecting means for detecting the input device of the video signal is further included, and an encoding parameter for encoding the video signal is set based on a relationship between the detected image variation of the input device and a frame rate. It is characterized by that. The call camera is the video input terminal 1, the video playback device is the video input terminal 2, and the video input terminal is divided according to the characteristics of the video equipment, and the encoding parameter of the input video is switched for each terminal. As a result, when transmission of video quality matching the encoder input video characteristics is performed on the transmission side, the video characteristic detection method can be detected by the connection location with the video input device.

  According to another aspect of the present invention, there is provided a video transmission method for transmitting encoded video data to a video receiving side, the step of transmitting the video data to the video receiving side based on an initial value of an encoding parameter. Obtaining reception-side data relating to at least one of a reception status of the reception means for receiving the encoded data and a display status of the display means for displaying the encoded data; and the encoding based on the reception-side data There is provided a video transmission method comprising updating parameters and transmitting image data encoded based on the updated encoding parameters.

  In addition, there is provided a receiving-side data transmission method for a video transmitting side that transmits encoded video data, the step of receiving video data from the video transmitting side based on an initial value of an encoding parameter, and the received video data There is provided a video reception method including a step of transmitting reception side data relating to at least one of a reception state and a display state of display means for displaying the encoded data to a video transmission side.

  According to the video transmission system of the present invention, the video suitable for the video encoder capability of the video transmission device and the display capability of the video reception device even if the effective bandwidth of the network between the video transmission / reception devices and the display status of the video reception device vary. Transmission is possible.

  A video transmission / reception technique according to an embodiment of the present invention will be described below with reference to the drawings. In the following description and drawings, the same parts are denoted by the same reference numerals and names, and detailed description thereof will not be repeated.

  FIG. 1 is a diagram showing a configuration example of a communication transmitting / receiving apparatus and an input / output device using a video transmission system according to an embodiment of the present invention. As shown in FIG. 1, the system according to this embodiment is provided with a video transmission terminal 103 (provided in a home A101 in the figure) for users to create and distribute video data in real time between homes and offices. And a video receiving terminal 108 (provided in the home B102 in the figure) for receiving and displaying the video data. The transmission / reception terminals 103 and 108 are connected via a network. The video transmission terminal 103 is connected to a communication camera 104, a still image playback device 105, and a video video playback device 106. The video receiving terminal 108 is connected to a display device (TV) 109 that outputs video.

  The video transmission terminal 103 receives the transmission start instruction from the video reception terminal 108 and starts transmission of video data. In an application such as a TV phone, the other party is called to establish a connection between the transmission video receiving terminals 103 and 108, and then a video transmission start procedure is performed. Regardless of whether a transmission start instruction is received from the video reception device, transmission may be started by user input or video input detection.

  As shown in FIG. 1, an audio input device 107 such as a microphone or an audio playback device is connected to the video transmission terminal 103, and an audio output device 110 such as a speaker that outputs audio is connected to the video reception terminal 108, so that real time It may be configured to send and receive audio data.

  In this embodiment, a so-called Internet is assumed as the network between the transmission / reception apparatuses. However, the network is not limited to the Internet. For example, it may be a private network that is not connected to the Internet, or a wireless public network such as a mobile phone or PHS. Although a bidirectional communication path is required at least between the video transmission device and the video reception device, it is not necessary to use the same physical layer between the transmission / reception devices as in the embodiment.

  FIG. 2 is a functional block diagram showing a configuration example of the video transmission terminal 103 according to the present embodiment. Referring to FIG. 2, the video transmission terminal 103 includes a video input unit 201 that receives a video signal from a video input device such as a camera or a video player, digital encoding of the video signal, and compression of the encoded data. A video encoding unit 202 that performs data transmission packetization, a packet transmission unit 203 that performs data packetization, a network interface 204 that communicates with an external video reception device, a packet reception unit 212 that decodes received data from a received packet, A display mode information acquisition unit 210 that analyzes the display mode information acquired from the received data and extracts an instruction to change the video display size and motion priority of the video receiving device, and a video receiving terminal that analyzes the network information acquired from the received data 108 (FIG. 1) to collect network information to extract packet loss rate, delay time, etc. Unit 209, the effective bandwidth calculation unit 208 that calculates the effective bandwidth of the transmission / reception period from the packet loss rate, delay time, etc. of the video receiving terminal 108 (FIG. 1) and detects the variation of the effective bandwidth, the characteristics of the video encoder, the transmission / reception An encoding parameter database 206 in which encoding parameters are registered according to the effective bandwidth of the period and the display mode of the video receiving terminal, and the display mode of the video receiving terminal 108 (FIG. 1) with reference to the encoding parameter database 206 Including connection management between the database reference unit 207 for obtaining the coding parameters suitable for the effective bandwidth of the network, the coding parameter switching unit 205 for switching the coding parameters of the video encoder, and the video receiving terminal 108 (FIG. 1). And a control unit 211 that controls all of these means.

  Alternatively, as shown in FIG. 3, the video transmission terminal 103 includes an audio input unit 301 that receives an audio signal from an audio input device such as a microphone or an audio playback device, and audio that performs digital encoding of the audio signal and compression of the encoded data. The encoding unit 302 may be included. By transmitting audio synchronized with video to the receiving side in real time, an application such as live broadcasting can be realized. Furthermore, an application such as a TV phone can be realized by bidirectionally transmitting real-time video / audio.

  The video transmission terminal 103 preferably includes a timer 304. The timer 304 has a function of synchronizing the video transmission / reception terminals, whereby the reception side can rearrange the received packets, detect the loss of the packets, and correctly measure the delay time. The timer is preferably configured to automatically correct at the correct time at least once a day. There are a plurality of known methods, such as a method of receiving and correcting a standard radio wave, a method of correcting using an Internet NTP (Network Time Protocol), a method of correcting using a Simple Network Time Protocol (SNTP). However, you can adopt any of them.

  As illustrated in FIG. 3, the video transmission terminal 103 may further include a database reference parameter temporary storage unit 303 for temporarily storing an effective bandwidth calculated from display mode information and network information. The database reference parameter temporary storage unit 303 can speed up the process by not referring to the same encoding parameter many times when a plurality of similar requests arrive, and the encoding parameter of the video encoder. It is also used to prevent instability of the transmission bit rate due to frequent switching.

  FIG. 4 is a diagram showing an example of an encoding parameter table registered in the encoding parameter database 206 (FIG. 3). As shown in FIG. 4, the items in each row are encoding parameters such as effective bandwidth, display screen size, motion priority, capture size / frame rate, etc., in order from the left. When the display mode information received from the video receiving terminal or the calculated effective bandwidth is applicable to the value of each item of the table, the value of the encoding parameter is applied.

  For example, the effective bandwidth = “1M-900 kbps”, the display screen size = “640 × 480”, and the motion priority = “medium” line has an effective bandwidth calculated from the network information received from the video receiving device from 1 M to 900 kbps. When the display screen size in the display mode information received from the video receiver is 640 × 480, the motion priority is medium, the encoding parameter is capture size = 360 × 270, and the frame rate is 10 fps. Is shown. The “motion priority” represents the user preference on the video receiving terminal side, and is described here in three levels, “high”, “medium”, and “low”. It may be described with a specific frame rate value. In addition, the display screen size describes only one value “640 × 480”, but a plurality of values such as “640 × 480, 720 × 480” are described and corresponds to one of these values. If so, the value of the encoding parameter may be applied.

  The encoding parameters referred to from the table shown in FIG. 4 are the users who use the video receiving device such as the display screen size and the motion priority in addition to the transmission bit rate of the encoded video data being within the range of the effective band. Encoding parameters that match the performance of the video encoder are registered so that optimal video encoding can be performed according to the display mode that reflects the user's preference.

  For example, a display screen size = “800 × 600” row has a lower frame rate than a row of “display screen size = 600 × 480” having the same effective bandwidth and motion priority, and the display device on the video receiving terminal side. Instead of sacrificing the operational feeling of the image displayed on the screen, the display definition is enhanced by increasing the capture size. This is because the video data size to be encoded depends on the frame rate and the capture size, the video data size is limited, and if one of these values is prioritized, the other value is sacrificed. This is because it must be done.

  Similarly, the line with motion priority = “medium” is displayed on the display device on the video receiving terminal side with a lower capture size than the line with motion priority = “high” for the same effective bandwidth and display screen size. Instead of sacrificing the display fineness of the displayed video, the frame rate is increased to increase the operational feeling of the video displayed on the display device on the video receiving terminal side.

  Further, since the encoding parameter is registered with a value within the performance range of the video encoder, the maximum capture size of the video encoder even when the display screen size is as very large as 1800 × 1350 with an effective bandwidth of 1 Gbps, for example. Is 1200 × 900 and the maximum frame rate is 30 fps, the value is taken. Similarly, if the display screen size is as small as 128 × 96 even if the effective bandwidth is sufficiently large, if the minimum capture size of the video encoder is 176 × 144 and the minimum frame rate is 1 fps, video encoding is performed with that value. Do. Further, when the video data that does not match the display size cannot be displayed on the receiving side, the video encoding may be stopped.

  FIG. 5 is a flowchart showing the flow of processing executed in the video transmission terminal. Reference is made to FIGS. 1 to 4 as appropriate. As shown in FIG. 5, when the process is started (Start), first, in step S501, the video transmission terminal 103 connects to the video reception terminal 108 and starts transmitting information. In step S502, encoding parameters (frame rate and capture size) used for video encoding are set to predetermined values, and video encoding of the input video signal received by the video input unit 201 is started in the video encoding unit 202. To do. In step S503, the packet transmission unit starts packetizing the video encoded data created by the video encoding unit 202, starts transmission from the network interface 204 to the video receiving device 108, and enters the video transmission state after step S504. move on.

  When a packet is received from the video receiving terminal 108 in step S504 during video transmission, the packet reception unit 212 analyzes the packet data. If it is network information, the received packet data is obtained as network information. If the packet data is display mode information, the received packet is sent to the display mode information acquisition unit 210, and then the process proceeds to step S506.

  In step S505, the network information acquisition unit 209 extracts the packet loss rate and delay time measured by the video reception device 108 from the received network information, and the effective bandwidth calculation unit estimates and calculates the effective bandwidth between the transmission and reception devices. The value of the effective bandwidth is sent to the database reference parameter temporary storage unit 303, and the process proceeds to step S507. In step S506, the display size or motion priority is extracted from the received display mode information, and the extracted display size or motion priority value is sent to the database reference parameter temporary storage unit 303, and the process proceeds to step S507.

  In step S507, if the effective bandwidth value received in the database reference parameter temporary storage unit 303, or the display size or motion priority value is different from the stored effective bandwidth, display size, or motion priority, a new one is created. Update and store the value. Thereafter, in step S508, it is determined whether or not the storage has been updated, and the effective bandwidth, display size, and motion priority value stored only when there is an update are sent to the database reference unit, and the process proceeds to step S509. In step S509, the encoding parameter database 206 is referred to using the effective bandwidth, display size, and motion priority values received by the database reference unit 207 as database reference parameters, and the capture size and frame rate values obtained by reference are referred to. The data is sent to the encoding parameter switching unit 205. The encoding parameter switching unit 205 performs video encoder control using the received value as an encoding parameter used for the video encoder. Thereafter, in step S510, the packet receiving unit 212 or the like is monitored to check whether or not video transmission is in progress. If video transmission is in progress (Yes), the process returns to step S504. If transmission is not in progress, transmission ends (end). .

  In step S507 and S508, a means for temporarily storing the database reference parameter is used as a speed-up means for eliminating the need to refer to the same encoding parameter many times. Only when the stored effective band value and the newly calculated value are significantly different from each other, the process proceeds to the encoding parameter database reference means in step S509. If the value does not change significantly, step S509 is skipped and step S510 is skipped. It may be possible to return to the video transmission state.

  Communication control protocols such as RTCP and SIP (Session Initiation Protocol) described above may be used for the start and end of communication. When SIP is used, it is necessary to start video transmission of RTP endpoints (address and port numbers at both ends) and codec information using SDP (Session Description Protocol) defined in RFC 2327 in the SIP message. Information is exchanged between the transmitting and receiving devices.

  Next, switching of video encoder parameters according to the video characteristics of the video input source will be described. The size of the video data to be encoded is greatly affected by the difference in the video characteristics of the transmission video, such as the video with a large time fluctuation and the video with a small time fluctuation. Since the difference in characteristics is not taken into consideration, an enormous transmission bit rate is required when transmitting an image whose entire screen is greatly fluctuated in time as compared with the case of transmitting a moving image that is as fine as the same.

  The video data encoded from the video signal in the video encoder has a data size that varies greatly depending on the characteristics of the input video signal. When the input video signal is a moving image, the time variation of the video signal is large instead of decreasing the number of camera pixels. When the input video signal is a still image, the time variation of the video signal is increased instead of increasing the number of camera pixels. Is small. Also, video data of moving images varies greatly depending on the device being shot. For example, a portable camera device such as a digital video camera or a video playback device that plays back processed content such as editing a video video player. Video data obtained by video encoding the input video signal of the camera often fluctuates in time over the entire image of the captured video, but a camera device used at a fixed position of the communication camera or surveillance camera covers the entire image of the captured video. Fluctuates with time. Therefore, it is effective to optimize the encoding parameters used for video encoding for each input device for input video signals having different characteristics.

  FIG. 6 is a diagram illustrating a configuration example of the video transmission terminal 103 that identifies the input devices 104, 105, and 106 from the input terminals 601, 602, and 603 of the input device. As shown in FIG. 6, the video transmission terminal 103 is provided with a video input terminal 601 of the communication camera 104, a video input terminal 602 of the still image playback device 105, and a video input terminal 603 of the video video playback device 106. Yes. When the communication camera 104 is connected to the video transmission terminal 103, the user looks at the labels 604, 605, and 606 corresponding to the video input terminals 601, 602, and 603, respectively. The communication camera 104 is connected to the input terminal 601 with the label 604. The video transmission terminal prepares an input path for each input terminal, and the input device detection unit 607 specifies a video input destination device based on the position of the input path.

  FIG. 7 is a functional block diagram of a video transmission terminal in which a function for determining an encoding parameter for each input device is added to the video transmission terminal 103 according to the present embodiment. As shown in FIG. 7, in addition to the block diagram of FIG. 2, the video transmission terminal 103 detects input devices from which a video signal is input as a video input device such as a camera or a video player. Part 607 is included. The input device detection method includes a method of determining input devices by providing input terminals for each device, a method of allowing a user to select and input an input device using an input switching button such as a remote control, and a device-specific method in an input video signal. A method of detecting a signal can be used.

  FIG. 8 is a diagram illustrating an example of an encoding parameter table in which the types of input devices registered in the encoding parameter database are added to items. The items in each row shown in FIG. 8 are coding parameters such as an effective bandwidth, a display screen size, a motion priority, an input device, a capture size / frame rate, in order from the left. In addition to the display mode information received from the video receiving terminal and the calculated effective bandwidth, the value of the encoding parameter to be applied is determined according to the type of input device detected at the video transmitting terminal.

  For example, the line of effective bandwidth = “1M-900 kbps”, display screen size = “640 × 480”, motion priority = “medium”, input device = “communication camera” is from the network information received from the video receiving terminal. When the calculated effective bandwidth is in the range of 1M to 900 kbps, the display screen size in the display mode information received from the video receiving terminal is 640 × 480, the motion priority is medium, and the input device is a communication camera, It shows that the encoding parameter is capture size = 360 × 270 and the frame rate is 10 fps.

  FIG. 9 is a flowchart showing the flow of processing executed by the program at the video transmission terminal. As shown in FIG. 9, when the process is started (Start), the presence / absence of a system input is detected in step S901 instead of the process of step S504 in FIG. 5, and the step is performed when the input is detection of a video input device. Proceeding to step S902, the type of the input device is specified in step S902, and the process of using this input device specification result as a reference parameter for referring to the encoding parameter database in step S507, step S508, and step S509 is added to FIG.

  As described above, in this embodiment, the video characteristics are determined by specifying the input device and the encoding parameters of the video encoder are controlled. However, the data size of the video encoded varies greatly depending on the characteristics of the input video. Thus, the switching of the input video characteristics may be detected by monitoring the temporal variation of the transmission data size. For example, when video encoding is performed according to MPEG4, the data size of an I frame increases and the data size of a P frame decreases when a still image is input, but the difference between the I frame and the P frame data size of a moving image increases as a still image. Absent. There is also a method for determining the input video characteristics from the difference in data size between specific frames.

  Next, the video receiving terminal will be described. FIG. 10 is a block diagram illustrating a configuration example of the video receiving terminal used in the present embodiment. As shown in FIG. 10, the video receiving terminal 108 includes a network interface 1001 that communicates with an external video transmitting terminal, a packet receiving unit 1002 that decodes received data from a received packet, and decoding and decompressing received video data. A video decoding unit 1003 that performs video processing, a video output unit 1004 that outputs a video signal from the decoded video data to a display device, a motion priority switching instruction for improving the smoothness of a received video operation during display, and reception during display The user interface 1008 that receives an input such as a display size switching instruction such as a reduction in the video screen size from an input device such as a remote control, and the display mode on the receiving side is switched from the movement priority switching instruction or the display size switching instruction. A display mode information creating unit 1007 for creating display mode information for transmitting a message to the transmission side, and a received packet Management of network information creation unit 1006 that measures packet loss rate, delay time, etc., creates network information, packet transmission unit 1009 that performs display packetization and network information transmission packetization, and video transmission terminal 103 And a control unit 1005 that controls all of these means, including a display control for switching control of video output during display from a motion priority switching instruction and a display size switching instruction.

  As shown in FIG. 11, the video receiving terminal 108 includes an audio decoding unit 1101 that decompresses received audio data and decodes the audio data, and an audio output unit 1102 that outputs an audio signal to an audio output device such as a speaker. You may go out. By receiving audio synchronized with video from the transmission side in real time, live broadcasting based on processing by an application can be realized.

  FIG. 12 is a flowchart showing a flow of processing executed based on a program in the video receiving terminal. This will be described with reference to FIG. As shown in FIG. 12, when processing is started (Start), first, in step S1201, video encoding of received video data and output of a video signal to a display device such as a TV are started, and a received video display state in step S1202 is started. Proceed to The video transmission terminal 103 is connected and transmission is started. In step S1202, reception of the video data packet from the video transmission terminal 103 is started through the network interface 1001, and the video data packet received by the packet receiving unit 1002 is passed to the video decoding unit 1003. In step S1203, the video decoding unit 1003 analyzes the frame rate and capture size of the video data. In step S1204, the video data is decoded based on the analyzed frame rate and capture size of the video data, and the video signal generated by the decoding is output from the video output unit 1004 to the connected display device. . Thereafter, the process proceeds to the video transmission state after step S1205.

  When a video receiving terminal receives system input while displaying received video, the video decoding unit analyzes whether the system input is network information including values such as packet loss rate and RTT measured from received packets. If it is network information, the process proceeds to step S1206, and if it is a change notification, the process proceeds to step S1208.

  In step S1206, network information for notifying the transmission side of values such as the packet loss rate and RTT measured by the network information creation unit is created, and the process proceeds to step S1215.

  In step S1208, if it is determined from the analysis result of the video data that the capture size of the video data has been changed, the process proceeds to step S1213. If it is determined that the frame rate of the video data has been changed, the process proceeds to step S1214. In step S1213, the output of the video signal is adjusted by changing the enlargement / reduction ratio of the display output size according to the changed capture size. Similarly, in step S1214, the output of the video signal is adjusted by changing the decode frame rate in accordance with the changed frame rate. After making these changes, the process returns to the received video display state in step S1215 and again enters an input waiting state.

  In step S1207, when the video receiving terminal receives the user input, if it is determined that the user input is a motion priority switching instruction indicating contents such as raising / lowering the video display operation feeling during display, the process proceeds to step S1209. If it is determined that the screen configuration switching instruction is such that the display area of the received video in the screen is changed from the entire screen to a part of the screen, the process proceeds to step S1210.

  In step S1209, the motion priority value for which a switching instruction has been given is acquired. Here, the value of the motion priority is a value that reflects user preference indicating how much priority is given to the video encoding data size of the display video in a trade-off relationship with the frame rate of the display video. Here, as an example, values such as motion priority = high, medium, and low are used. For example, when a user wants to view a video that gives priority to the operational feeling over the video display detail level, the user can input a user instruction so that the motion priority is high by pressing a button input on the remote control or the main body. Communicating the user's will to The motion priority value may be a frame rate such as 15 fps. In step S1210, the size of the received video display area is acquired from the screen configuration designated by the user, and the process advances to step S1211. There are multiple screen configurations for displaying received video, such as a screen that displays received video in full screen, and a screen that displays multiple contents simultaneously in addition to the displayed received video. Is different. In addition, the user may input a value such as display size = 640 × 480, and the screen configuration may be controlled according to the input value.

  In step S1211, the display mode information creation unit 1007 creates display mode information from the motion priority value acquired through steps S1209 and S1210 or the received video display size, and the process advances to step S1212. Here, the motion priority and the received video display size are illustrated in the display mode information, but in order to convey the display capability of the video receiving terminal to the video transmitting terminal, the model information of the display device and the decoder, Display capability information such as resolution may be included. When detecting the display capability of these display devices, output terminal information such as composite, S terminal, and D terminal may be used for output from the video receiving terminal to the display device. Control information obtained from the display device when the D terminal is used may be included as display mode information. Further, when the display size of the video data received by the video receiving terminal and the video data output to the display device cannot be matched, the display mode information may be automatically created. Further, color priority information may be included in order to convey the preference of the user of the video receiving terminal to the video transmitting terminal.

  In step S <b> 1212, when display mode information or network information is created in the transmission packet unit, the transmission packet unit converts the display packet into transmission packets, which are transmitted from the network interface 104 to the video transmission terminal 103. In order to notify the status of the receiving device such as display mode information or network information, the above-described communication control protocol such as RTCP or SIP may be used.

  In the video receiving terminal of the present embodiment described above, when the size of the received video display area is changed in response to an instruction to change the screen configuration from the user in step S1210, the capture size is changed on the transmission side in this embodiment. Change the size of the display area of the received video data by enlarging / reducing the video data before executing the capture size change on the transmission side until the video data after the capture size change on the receiving side is received and displayed I do.

  When the enlarged display of the received video data is performed by changing the size of the received video display area, the detail level of the display image is greatly impaired and a display image with poor quality is presented to the user. Therefore, it is not necessary to change the screen configuration until the video data after execution of the capture size change is received on the receiving side.

  Or, although the screen configuration is changed, the received video data of the display area size before the change is displayed in a predetermined area in the received video display area after the configuration change, and the remaining area is used as a display area for other contents such as wallpaper. May be used.

  Alternatively, a transmitter terminal transmits and receives a still image with a large capture size or a video with a large capture size and a low frame rate at regular intervals (for example, at intervals of several tens of seconds) or at the start of connection. The still image and the video may be displayed until the video data after the capture size change is received on the receiving side after buffering.

  As described above, according to the video transmission system according to the embodiment of the present invention, even if the effective bandwidth of the network between the video transmission / reception devices and the display status of the video reception device vary, the video encoder capability of the video transmission device and the video reception device There is an advantage that video transmission suitable for display capability becomes possible. In the embodiment, the present invention has been described by way of an example of a system that controls a video signal encoding method in a video transmission terminal. However, an audio signal may be encoded in the same manner as a video signal. At this time, the encoding method used for audio encoding is based on the ETSI AMR, ITU-T GG, or the like based on the audio reproduction capability of the video receiving terminal, the user's preference, or the audio input path to the transmission device. 711, ITU-T G.I. 723.1, ITU-TG 726, ITU-TG 729 or the like.

  In the present embodiment, the system for distributing the video / audio signal from the transmitting terminal at home A to the receiving terminal at home B by real-time encoding has been described, but the same applies from the transmitting device at home B to the receiving device at home A. It is also possible to apply to a two-way communication system that performs distribution. In this way, it can be used as a two-way real-time communication system such as a videophone.

  FIG. 13 is a diagram illustrating an example of a system configuration in which such a transmission / reception apparatus is installed in each of the home A and the home B to perform bidirectional communication. Each home A and B is provided with an input / output device for audio and video, and information such as video and audio can be transmitted and received from either home A or B.

    INDUSTRIAL APPLICABILITY The present invention can be used for a system that transmits video, music, and the like via a network.

1 is a system configuration diagram showing a configuration of a video transmission system in an embodiment of the present invention. It is a block diagram of the video transmission apparatus in one Example of this invention. It is a block diagram of the video transmission apparatus provided with the audio | voice transmission part, the timer, and the database reference parameter temporary storage part in one Example of this invention. It is an encoding parameter table of the video encoder of the video transmission apparatus in one Example of this invention. It is a flowchart which shows the control action in the control part of the video transmission apparatus in one Example of this invention. 1 is a device configuration diagram of a video transmission device that detects a video input device for each video input terminal according to an embodiment of the present invention; FIG. It is a block diagram of a video transmission apparatus provided with an input device detection unit in one embodiment of the present invention. It is an encoding parameter table of the video encoder of the video transmission apparatus which determines an encoding parameter for every input device in one Example of this invention. It is a flowchart which shows the control operation | movement in the control part of the video transmission apparatus which determines an encoding parameter for every input device in one Example of this invention. It is a block diagram of the video receiver in one Example of this invention. 1 is a block diagram of a video receiving apparatus including an audio transmission unit according to an embodiment of the present invention. It is a flowchart which shows the control operation | movement in the control part of the video receiver in one Example of this invention. 1 is a system configuration diagram showing a configuration of a bidirectional video transmission system in an embodiment of the present invention. It is a block diagram of the video transmission system which performs encoder control.

Explanation of symbols

DESCRIPTION OF SYMBOLS 103 Image transmitter 104 Communication camera 105 Still image reproducing device 106 Video image reproducing device 108 Video receiving device 109 Display device 201 Video input unit 202 Video encoding unit 203 Packet transmitting unit 204 Network interface 205 Encoding parameter switching unit 206 Encoding parameter Database 207 Database reference unit 208 Effective bandwidth calculation unit 209 Network information acquisition unit 210 Display mode information acquisition unit 211 Control unit 212 Packet reception unit 301 Audio input unit 302 Audio encoding unit 303 Database reference parameter temporary storage unit 304 Timer 607 Input device detection unit 1001 Network interface 1002 Packet receiving unit 1003 Video decoding unit 1004 Video output unit 1005 Control unit 1006 Network Over click information creation unit 1007 display mode information forming section 1008 user interface 1009 packet transmission unit 1101 audio decoding unit 1102 sound output unit

Claims (14)

Send data based on video or audio signal to the receiver,
Update the encoding parameter of the video or audio signal based on the packet loss rate or delay time detected in the receiving means for receiving the data acquired from the receiving side and the display mode of the output means for outputting the data A transmitting device that,
Means for encoding the video signal; and input device detection means for detecting an input device of the video signal;
When the input device is detected by the input device detection means , the transmission apparatus updates the encoding parameter for encoding the video signal according to the detected type of the input device . The update of the encoding parameter is performed by obtaining a new encoding parameter obtained by referring to an encoding parameter database that holds a correspondence relationship between the detected input device and the encoding parameter. The transmission device according to claim 1, wherein The transmission apparatus according to claim 1, further comprising: a plurality of input terminals corresponding to devices, wherein the input device detection unit specifies a type of the input device based on a connection location of the input terminals. . The transmission apparatus according to claim 1, wherein the input device detection unit identifies the type of the input device based on a difference in data size between frames of the encoded data. The transmission apparatus according to claim 1, wherein the display mode is a video display size obtained by decoding the data of the output unit. The transmission apparatus according to claim 1, wherein the display mode is a resolution of a display device of the output unit. 2. The transmission apparatus according to claim 1, wherein the display mode is a frame rate of a display video obtained by decoding the data of the output unit . The encoding parameter for encoding the video signal is updated with reference to an effective bandwidth of the network calculated based on a packet loss rate or a delay time received at the receiving side. 8. The transmission device according to any one of 1 to 7 . The program which operates a computer in order to perform the function of the transmitter of any one of Claim 1-8 . A computer-readable recording medium on which the program according to claim 9 is recorded. Receiving data as claimed in any one of claims 1 to 8, a receiving apparatus having an output unit, to said transmitting device, transmitting and the display manner as the packet loss rate or the delay time A receiving apparatus. The program which operates a computer in order to perform the function of the receiver of Claim 11 . Means a transmission device according to any one of claims 1 to 8, with means as a receiving device according to claim 11, transceiver characterized in that it comprises both. A transmission system comprising: the transmission device according to any one of claims 1 to 8; and the reception device according to claim 11 .

Images