JP2006304182A - Stream data generating method, video conference system, stream data generating device, and stream data compositing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stream data generating method with which a video conference system can be realized even in a video signal other than a video signal which conforms to a variable length format. <P>SOLUTION: Processing executed by a terminal constituting the video conference system comprises the steps of: receiving the input of a video signal from a camera (S1110); storing the video signal in a reception buffer (S1120); reading the video signal from the reception buffer (S1130); encoding areas other than an area (effective area) allocated to a terminal itself as ineffective areas (S1140); extracting data corresponding to an effective area in compressed data generated by encoding (S1150); storing the extracted data in a transmission buffer (S1160); and reading the compressed data from the transmission buffer and outputting the compressed data to a communication line (S1170). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to video signal processing. More specifically, the present invention relates to a stream data generation method, a video conference system, a stream data generation apparatus, and a stream data synthesis apparatus.

  There is a so-called video conference system as a usage form of video / audio signal communication via a communication line. In this system, a plurality of video signals are combined, and the combined signals are distributed to each terminal constituting the system. A user of each terminal, that is, a participant in a video conference can view the video displayed and the audio output based on the signal. Here, techniques for synthesizing video signals or encoding video signals are disclosed, for example, as follows (see Patent Documents 1 to 3 and Non-Patent Document 1).

For example, according to the technique disclosed in Non-Patent Document 1, the format length of the display position can be calculated at high speed by a high-speed calculation algorithm. Further, since the processing can be realized as processing by software, it can be realized at a PC (Personal Computer) level. Therefore, since no special hardware is required, a video conference system can be realized at a low cost.
JP 7-298263 A JP-A-9-214970 Japanese Patent Laid-Open No. 10-234043 Matsushita Electric Industrial Co., Ltd. Multimedia System Laboratory, “Developing Multi-screen MPEG Compositing Technology”, December 9, 1998

  In the reproduction of moving images, since the amount of data to be processed is large, the capacity of the processing device provided in the terminal to be reproduced is restricted. For example, according to the technique disclosed in Non-Patent Document 1, since only a video signal based on a display position format extended to a variable length can be synthesized, there is a problem that versatility of a video conference system is lacking. In addition, since a processing device capable of executing high-speed processing is required, there is a possibility that the cost reduction of the hardware configuration is limited.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to generate stream data capable of realizing a video conference system even when a video signal other than a video signal based on a variable-length format is processed. Is to provide a method.

  Another object of the present invention is to provide a stream data generation method capable of preventing a delay in video display processing during operation of a video conference system.

  Another object of the present invention is to provide a video conference system that operates even on video signals other than video signals based on a variable length format.

  Another object of the present invention is to provide a stream data generating apparatus capable of generating stream data capable of preventing delay of video display processing during operation of a video conference system.

  Another object of the present invention is to provide a stream data generating apparatus that generates stream data for realizing a video conference system even when a video signal other than a video signal based on a variable length format is processed.

  Another object of the present invention is to provide a stream data synthesizing apparatus capable of generating stream data capable of preventing delay of video display processing during operation of a video conference system.

  In order to solve the above-described problem, according to one aspect of the present invention, a stream data generation method includes a step of generating stream data in a plurality of terminals electrically connected to a communication line. Each of the plurality of terminals includes display means for displaying a video in a display area based on the video signal. The step of generating stream data includes a step of receiving an input of a video signal, a step of preparing region information for identifying a predetermined region in the display region, and a step of encoding the video signal based on the region information. The generation step of generating video data for displaying video based on the video signal in a partial area, the step of generating data for transmission of the video data, and the communication of the data for transmission And outputting to the line. The generation method further includes a step of synthesizing the video signal based on the data output from each terminal in the video signal synthesis device electrically connected to the communication line. The step of synthesizing the video signal includes a step of receiving each stream data output by each of the plurality of terminals via the communication line. Each stream data includes video data encoded in accordance with a predetermined position for displaying a video in a part of the display area. The position differs depending on each of the plurality of stream data. The synthesizing step includes a synthesizing step of generating display data for displaying video based on each video data in a display area based on a plurality of stream data, and transmission of the generated display data And a step of outputting the generated data for transmission to a communication line. A method of combining stream data includes a step of receiving data generated based on data for transmission from a video signal synthesizer in each of a plurality of terminals, and a method of converting received data in each of a plurality of terminals. The method further includes a step of generating display data by executing decoding processing based on the above, and a step of causing the display means to display an image based on the display data in each of the plurality of terminals.

  Preferably, the generation step includes an encoding step for generating video data by encoding a video signal corresponding to a partial area.

  Preferably, in the encoding step, encoding is executed by using a video signal corresponding to a partial area as a target of motion compensation.

  Preferably, the synthesizing step extracts each video data from each of the plurality of stream data, and synthesizes the extracted video data based on a predetermined position with respect to the extracted video data. Generating display data.

  According to another aspect of the present invention, a video conference system includes a generation device electrically connected to a communication line. The generation apparatus includes a display unit that displays a video in a display area, a storage unit that stores area information for specifying a predetermined area in the display area, an input unit that receives an input of a video signal, A generation means for generating video data for displaying video based on the video signal in a partial area by encoding the video signal based on the area information, and transmission for generating data for transmitting the video data Data generating means and output means electrically connected to the communication line and outputting data for transmission to the communication line. The video conference system further includes a video signal synthesizer electrically connected to the communication line. The video signal synthesizer includes input means for receiving a plurality of stream data inputs via a communication line. Each of the plurality of stream data includes video data encoded in accordance with a predetermined position for displaying video in a part of the video display area. The position differs depending on each of the plurality of stream data. The video signal synthesizing device is configured to generate display data for displaying video based on each video data in the display area based on the plurality of stream data, and display data generated by the synthesizing unit. Transmission data generating means for generating data for transmission and output means for outputting the generated data for transmission to a communication line are included. The generation device generates data for display from a receiving unit that receives data generated based on data for transmission from a communication line and a decoding process based on the data received by the receiving unit. And a control means for causing the display means to display an image based on the display data.

  According to another aspect of the present invention, the stream data generation apparatus includes a display unit that displays video in a display area, and a storage unit that stores area information for specifying a predetermined area in the display area. And an input means for receiving an input of the video signal; a generating means for generating video data for displaying the video based on the video signal in a partial area by encoding the video signal based on the area information; An output means electrically connected to the communication line and outputting video data to the communication line; a receiving means for receiving data generated based on the data for transmission from the communication line; and a receiving means Decoding means for generating display data by executing decoding processing based on the data received by the control, and control for displaying video on the display means based on the display data And a stage.

  Preferably, the generation unit includes an encoding unit that generates video data by encoding a video signal corresponding to a partial area.

  Preferably, the encoding unit performs encoding using a video signal corresponding to a partial area as a target of motion compensation.

  Preferably, the video data includes only data for displaying a video only in a partial area.

  Preferably, the stream data generation device further includes a changing unit that changes the area information stored in the storage unit based on an external input.

  Preferably, the stream data generation apparatus further includes an instruction input unit that receives an input of region information from the outside. The changing means changes the area information stored in the storage means to the area information input via the instruction input means.

  Preferably, the receiving means receives data including area information from the communication line. The changing means changes the area information stored in the storage means to area information included in the received data.

  According to another aspect of the present invention, the stream data synthesizing apparatus includes input means for receiving a plurality of stream data inputs via a communication line. Each of the plurality of stream data includes video data encoded in accordance with a predetermined position for displaying video in a part of the video display area. The position differs depending on each of the plurality of stream data. A stream data synthesizing device includes a synthesizing unit that generates display data for displaying video based on each video data in a display area based on a plurality of stream data, and display data generated by the synthesizing unit Transmission data generating means for generating data for transmission of the data, and output means for outputting the generated data for transmission to the communication line.

  Preferably, the synthesizing unit synthesizes the extracted video data based on a predetermined position with respect to the extracted video data and an extraction unit that extracts each video data from each of the plurality of stream data. And generating means for generating display data.

  Preferably, each of the plurality of stream data includes time data for specifying a time at which each stream data is generated. The synthesizing unit generates display data based on the time data of each of the plurality of stream data.

  Preferably, the synthesizing unit generates display data without decoding each video data.

  Preferably, each of the plurality of stream data includes specific data for specifying an area in which a video based on each video data is displayed.

  Preferably, the synthesizing unit generates display data based on the specific data.

  According to the stream data generation method according to the present invention, a video conference system can be realized even with a video signal other than a video signal whose processing target is based on a variable length format.

  According to the stream data generation method of the present invention, it is possible to prevent a delay in video display processing during the operation of the video conference system.

  According to the video conference system according to the present invention, a video signal other than a video signal based on a variable length format is processed.

  According to the stream data generation device of the present invention, it is possible to prevent delay of video display processing during operation of the video conference system.

  According to the stream data generating apparatus of the present invention, a video conference system can be realized even with a video signal other than a video signal whose processing target is based on a variable length format.

  According to the stream data synthesizing apparatus according to the present invention, it is possible to generate stream data that can prevent a delay in video display processing during operation of the video conference system.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  A video conference system according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing the system configuration of the video conference system 10.

  The video conference system 10 includes terminals 100a, 100b, 100c, and 100d, and a video signal synthesizer 1200. Each terminal is referred to as a terminal 100 when collectively referred to. The video signal synthesizer 1200 and each terminal are connected to each other via a communication line. The terminal 100 includes a camera 110, a microphone 120, a monitor television 130, and a stream data generation device 400. The stream data generating apparatus 400 includes an MPEG (Moving Picture Experts Group) encoder 500 and an MPEG decoder 460. A signal output from the camera 110 is input to the MPEG encoder 500. A signal output from the microphone 120 is input to the MPEG encoder 500. A signal output from the MPEG decoder 460 is input to the monitor television 130. A signal output from the MPEG encoder 500 is transmitted to the video signal synthesis apparatus 1200 via a communication line. The MPEG decoder 460 receives a signal transmitted from the video signal synthesis device 1200.

  Video signal synthesizer 1200 includes a multipoint connection controller 1220 that receives a signal input via a communication line, and a transmission unit 1240 that can transmit the signal via the communication line. A signal output from the multipoint connection controller 1220 is input to the transmission unit 1240.

  Here, the configuration of signals communicated between each terminal and the video signal synthesis apparatus 1200 will be described with reference to FIG. FIG. 2A is a diagram conceptually showing the structure of a stream output from terminal 100a arranged at point A. FIG. 2B is a diagram conceptually showing the configuration of the stream output from terminal 100b arranged at point B. FIG. 2C is a diagram conceptually showing the configuration of the stream output from terminal 100c arranged at point C. FIG. 2D is a diagram illustrating a configuration of a stream output from terminal 100d arranged at point D. FIG. 2 (E) is a diagram conceptually showing the structure of a stream transmitted from video signal synthesis apparatus 1200 to each terminal.

  As shown in FIG. 2A, the signal output from the terminal 100a includes regions 210 to 214. The area 210 stores a header. In the area 212, video / audio data A is stored. The video / audio data includes data for displaying video and data for outputting audio. In the area 214, data for aligning the data size of the signal output from the terminal 100a with a predetermined size is stored. The header includes data items determined in advance for realizing the video conference system 10 according to the present embodiment. The video / audio data A includes data based on video captured by the camera 110 included in the terminal 100 a and audio data acquired by the microphone 120. As will be described later, the video / audio data A is compression-encoded in advance so as to display video in an area allocated to each terminal constituting the video conference system 10.

  As illustrated in FIG. 2B, the signal output from the terminal 100b includes regions 220 to 226. The area 220 stores a header. This header includes the same items as the data items included in the header stored in the area 210. The area 222 has the same size as the data size of the area 212. In the area 222, data for displaying a video is not stored. In the area 224, video / audio data B is stored. In the area 226, data for displaying a video is not stored. The area 226 stores data for aligning the data size of the signal output from the terminal 100b with a predetermined size. Here, the predetermined size is the same as the predetermined size for the size of the signal output from the terminal 100a.

  As shown in FIG. 2C, the signal output from the terminal 100c includes regions 230 to 236. In the area 230, a header is stored. This header includes the same items as the data items included in the header stored in the area 210. The area 232 has a size equal to the sum of the data size of the area 222 and the data size of the area 224. In the area 234, video / audio data C is stored. In the area 236, data for displaying a video is not stored. The area 236 stores data for aligning the data size of the signal output from the terminal 100c with a predetermined size. Here, the predetermined size is the same as the predetermined size for the size of the signal output from the terminal 100a.

  As shown in FIG. 2D, the signal output from terminal 100d includes regions 240-246. In the area 240, a header is stored. This header includes the same items as the data items included in the header stored in the area 210. The area 242 does not store data for displaying video. In the area 242, valid video data is not stored. However, since the coordinates of the macroblock corresponding to the region 242 exist in the frame (because it is not valid but is encoded as the presence of video data), another valid video data is included. A frame is synthesized by replacing between the same macroblock addresses of the frame. The macro block will be described later. In the area 246, video / audio data D is stored.

  As shown in FIG. 2E, the signal output from the video signal synthesis device 1200 includes regions 250 to 258. The header is stored in area 250. This header includes data items predetermined for realizing the video conference system. This data item includes data for specifying the transmission source (that is, the video signal combining device 1200) of the signal and data for specifying the transmission destination (that is, the terminals 100a, 100b, 100c, and 100d). When the video signal synthesizer 1200 outputs a signal to the communication line based on such a header, the signal is duplicated in accordance with the number of terminals designated as transmission destinations, and the signal after each duplication is communicated. It is sent to each terminal via the line.

  The area 252 stores video data transmitted from the terminal 100a. That is, the video data stored in the area 252 is the same as the video data stored in the area 212. The area 254 stores video data transmitted from the terminal 100b. This video data is the same as the video data stored in the area 224 shown in FIG. The area 256 stores video data transmitted from the terminal 100c. This video data is the same as the video data stored in the area 234 shown in FIG. The area 258 stores video data transmitted from the terminal 100d. This video data is the same as the video data stored in the area 246 shown in FIG.

  The signals shown in FIGS. 2A to 2D are shown as having a predetermined data size, but the configuration of each signal output from each terminal is shown in FIG. (A)-It is not restricted to what is shown in Drawing 2 (D). For example, the signal output from the terminal 100a may have only the area 210 and the area 212.

  Next, with reference to FIG. 3, an image displayed based on the signals shown in FIGS. 2A to 2E will be described. FIG. 3A to FIG. 3E are diagrams showing one mode of screens displayed on the monitor television 130 by the terminals 100a, 100b, 100c, and 100d, respectively.

  Referring to FIG. 3A, when terminal 100a displays a video based on the signal shown in FIG. 2A, video A is displayed in area 310. Here, the area 310 is an area set in advance when the video conference system is realized. This area is set according to the number of terminals constituting the video conference system 10. For example, when four terminals participate in the video conference system, the display area of the monitor television 130 is set to be divided into four. Each area in that case can specify an area where an image is actually displayed by associating the area with a position assigned to a terminal that can participate in the system in advance. Further, this area can be changed based on an external input as will be described later.

  Referring to FIG. 3B, when terminal 100b displays a video based on the signal shown in FIG. 2B, video B is displayed in area 320 of monitor television 130 of terminal 100b. .

  Referring to FIG. 3C, when terminal 100c displays a video based on the signal shown in FIG. 2C, video C is displayed in area 330 of monitor television 130 provided in terminal 100c. The

  Referring to FIG. 3D, when terminal 100d displays a video based on the signal shown in FIG. 2D, video D is displayed in area 340 of monitor TV 130 provided in terminal 100d. The

  3E, when each terminal 100 displays an image based on the signal shown in FIG. 2E, the image A is an area 312 in the monitor television 130 of each terminal. Is displayed. This display position is the same as the display position shown in FIG. Video B is displayed in area 322. The position where this video is displayed is the same as the display position of video B shown in FIG. Video C is displayed in area 332. The position where this video is displayed is the same as the display position (that is, the region 330) shown in FIG. Video D is displayed in area 342. The position where this video is displayed is the same as the video display position (ie, region 340) shown in FIG.

  In this manner, each terminal transmits a signal for displaying a video as shown in FIGS. 3A to 3D to the video signal synthesis device 1200, thereby FIG. A signal for realizing display of the displayed video is received. A configuration for realizing such communication will be described below.

  With reference to FIG. 4, a configuration of stream data generation apparatus 400 according to the present embodiment will be described. FIG. 4 is a block diagram showing a hardware configuration of stream data generation apparatus 400. The stream data generating apparatus 400 includes a video interface 410, an audio interface 420, a memory 430, an MPEG encoder 500, a transmission unit 440, a reception unit 450, an MPEG decoder 460, a video interface 470, and an audio interface 480. Including.

  The video interface 410 is connected to the camera 110. The video interface 410 receives a video signal output from the camera 110. This video signal is sent to the MPEG encoder 500.

  The audio interface 420 is connected to the microphone 120. The audio interface 420 receives an audio signal output from the microphone 120. The audio interface 420 sends the signal to the MPEG encoder 500 as a digital signal.

  The memory 430 stores data for realizing video display shown in any of FIGS. 3A to 3D for each terminal. This data structure will be described later (FIG. 6).

  The MPEG encoder 500 displays the video at a predetermined position on the monitor television 130 based on the signal output from the video interface 410, the signal output from the audio interface 420, and the data stored in the memory 430. Video data is generated. The MPEG encoder 500 sends the generated video data to the transmission unit 440.

  The transmission unit 440 generates data to be transmitted to the video signal synthesis device 1200 based on the video data from the MPEG encoder 500. Specifically, the transmission unit 440 adds a header having a predetermined data item to the video data to generate data for transmission. This data item has, for example, the network address of the video signal synthesis device 1200 that is the destination of the data for transmission. When the transmission unit 440 generates data for transmission, it transmits the data to the communication line.

  The receiving unit 450 receives video data for video conferencing from the video signal synthesis device 1200 via a communication line. This video data includes video data generated by other terminals in addition to the video data generated by the MPEG encoder 500. Receiving unit 450 sends the data to MPEG decoder 460.

  The MPEG decoder 460 decodes the data received by the receiving unit 450. The signal generated by the decoding process includes a signal for displaying video (hereinafter referred to as video signal) and a signal for outputting audio (hereinafter referred to as audio signal). The video signal is sent to the video interface 470. The audio signal is sent to the audio interface 480.

  The video interface 470 outputs the signal to the monitor television 130. Thereby, the monitor television 130 displays a video (FIG. 3E) configured for the video conference.

  The audio interface 480 outputs an audio signal to a speaker (not shown). The said speaker outputs the audio | voice based on the signal according to the output of an image | video.

  The MPEG encoder 500 will be described with reference to FIG. FIG. 5 is a block diagram showing a hardware configuration of MPEG encoder 500.

  The MPEG encoder 500 includes a reception buffer 510 that receives an input of a video signal and an audio signal, an encoding circuit 520 that executes a predetermined compression encoding process on each input signal, and a compressed signal. An extraction circuit 530 for extracting data for transmission from the data and a transmission buffer 540 for temporarily storing the extracted data are included.

  With reference to FIG. 6, the data structure of the terminal according to the present embodiment will be described. FIG. 6 is a diagram illustrating an aspect of data storage in memory 430.

  Memory 430 includes areas 610-630. Data for specifying an area defined in the monitor television 130 is stored in the area 610. Data for representing each area is stored in area 620. Data indicating whether video can be displayed in each area is stored in area 630. For example, the first display area is defined as coordinates (10, 10) to (40, 50). The first display area is set to be usable in the first terminal 100a. The second display area is defined as coordinates (10, 50) to (40, 90). This area is set to be unusable for video display. Similarly, with respect to the third display area and the fourth display area, the respective coordinates and whether or not video can be displayed are set.

  The data structure of the memory 430 shown in FIG. 6 is that in the terminal 100a. In the data structure of the terminal 100b, the second display area is set to “usable” instead of the first display area. For the terminal 100c, the third display area is set as “usable”. The specific form of the display area on the monitor television 130 is not limited to that shown in FIG. The data shown in FIG. 6 may be stored in advance at the start of the video conference system, or may be changeable by input via a keyboard (not shown) or other input device. Or you may set according to the number of the terminals connected to the said system. In this case, data for identifying the area so that the number of connected terminals is detected based on a login operation at a terminal to be newly connected and the display area is subdivided based on the detection result. (For example, coordinates) may be calculated.

  Here, the coordinates shown in FIG. 6 will be described with reference to FIG. FIG. 7 is a diagram conceptually showing coordinates on the monitor television 130.

  As shown in FIG. 7, in the monitor television 130, coordinates (10, 10) to (80, 90) are set in advance as areas where video can be displayed. This definition of coordinates corresponds to the definition for specifying the region shown in FIG. Therefore, when each terminal executes the video display process based on the coordinates shown in FIG. 6, the monitor television 130 displays the video in an area corresponding to the coordinates.

  Next, the relationship between video data for a video conference system and a display area will be described with reference to FIG. FIG. 8 is a diagram showing a corresponding region of video data to be displayed when video is displayed on the monitor television 130 with respect to the terminal 100a.

  Regarding the terminal 100a, the video display area included in the monitor television 130 includes an area 810 and an area 820. An area 810 is an area for displaying a video based on video data generated by the camera 110 included in the terminal 100a. This region is identified by a dot as shown in FIG. The area 820 is an area reserved exclusively for not performing display by the camera 110. The area 820 is used to display a video based on video data transmitted from other terminals (for example, the terminals 100b to 100d).

  Next, the configuration of an MPEG stream transmitted from each terminal 100 to the video signal synthesis device 1200 will be described with reference to FIG. FIG. 9A shows the structure of the picture layer of the MPEG stream. FIG. 9B is a diagram conceptually showing the configuration of slice layers included in the picture layer. FIG. 9C conceptually shows a configuration of a macroblock (MB) layer included in the slice layer. FIG. 9D is a diagram showing the relationship between the reference video area and the reference area based on the MPEG stream data.

  As shown in FIG. 9A, the picture includes a picture header 910, a picture type 920, a motion vector search range 930, and a slice layer 940. As shown in FIG. 9B, the slice layer 940 includes a slice header 950, a slice position 960, and an MB layer 970. As shown in FIG. 9C, the MB layer 970 includes an MB address 972, an MB type 974, a motion vector 976, and MB code data 978. The MB code data 978 is video data compressed and encoded by the MPEG encoder 500.

  Referring to FIG. 9D, the reference video area 980 includes a slice 962. The slice 962 is identified based on the slice position 960. The slice position 960 corresponds to, for example, a vertical line number. Slice 962 includes a macroblock (MB) 964. The macro block 964 is specified by the MB address 972. The reference area 990 relates to a motion compensation reference and includes a valid range in which the reference is valid and an invalid range in which the reference is not valid. For example, in the overlapping portion of the reference video area 980 and the reference area 990, the motion vector can be detected by the macro block 964. On the other hand, in the invalid range (hatched portion) in the reference area 990, the motion vector detection by the macroblock 964 is prohibited.

  With reference to FIG. 10, the stream data transmitted from each terminal according to the present embodiment to video signal combining apparatus 1200 will be described. FIG. 10A is a diagram conceptually showing a stream transmitted from terminal 100 a to video signal synthesis apparatus 1200. FIG. 10B is a diagram conceptually showing a stream transmitted from terminal 100b to video signal synthesis apparatus 1200. FIG. 10C is a diagram conceptually showing a stream transmitted from terminal 100 c to video signal synthesis apparatus 1200. FIG. 10D is a diagram conceptually showing a stream transmitted from terminal 100d to video signal synthesis apparatus 1200.

  As shown in FIG. 10A, the stream transmitted by the terminal 100a includes a picture header PH, a slice header SH, and a slice layer. The slice layer is divided into, for example, 480 layers from “SL1” to “SL480”. Each slice layer includes macroblocks (MB) from MB0 to MB21 and macroblocks from MB22 to MB43. MB0 to MB21 in the slice layers SL1 to SL240 correspond to the display data 1010 for realizing the display in the area 310, as shown in FIG.

  As shown in FIG. 10B, the stream transmitted by the terminal 100b includes a picture header PH, a slice header SH, and a slice layer. The slice layer is divided into, for example, 480 layers from “SL1” to “SL480”. Each slice layer includes macroblocks (MB) from MB0 to MB21 and macroblocks from MB22 to MB43. MB22 to MB43 in the slice layers SL1 to SL240 correspond to display data 1020 for realizing display in the region 320, as shown in FIG.

  Referring to FIG. 10C, the stream transmitted by terminal 100c similarly includes picture header PH, slice header SH, and display data 1030. The data structure of the stream is the same as that shown in FIG. The display data 1030 is data for realizing display in the area 330 as shown in FIG. Referring to FIG. 10D, the stream transmitted by terminal 100d similarly includes picture header PH, slice header SH, and display data 1030. The data structure of the stream is the same as that shown in FIG. The display data 1040 is data for realizing display in the area 340 as shown in FIG. A mode in which the streams are combined will be described later (FIG. 14).

  With reference to FIG. 11, the control structure of terminal 100 according to the present embodiment will be described. FIG. 11 is a flowchart showing a procedure of processing executed by MPEG encoder 500 of terminal 100.

  In step S 1110, MPEG encoder 500 receives an input of a video signal from camera 110. In step S 1120, MPEG encoder 500 stores the video signal in reception buffer 510. In step S1130, MPEG encoder 500 reads the video signal stored in reception buffer 510. In step S1140, encoding circuit 520 encodes the video signal using an area other than the area (effective area) allocated to terminal 100 as an invalid area. When executing this encoding process, the encoding circuit 520 further adds the time when the process was performed as a time stamp.

  In step S1150, extraction circuit 530 extracts data corresponding to the effective area from the compressed data generated by the encoding of encoding circuit 520. In step S1160, MPEG encoder 500 stores the data extracted by extraction circuit 530 in transmission buffer 540. In step S 1170, MPEG encoder 500 reads data from transmission buffer 540 and sends the data to transmission unit 440. The transmission unit 440 generates data for transmission based on the data and outputs it to the communication line. The stream data output in this way is transmitted to the video signal synthesis apparatus 1200.

  With reference to FIG. 12, video signal synthesis apparatus 1200 constituting video conference system 10 according to the present embodiment will be described. FIG. 12 is a block diagram showing a hardware configuration of video signal synthesis device 1200.

  The video signal synthesis device 1200 includes a reception unit 1210, a multipoint connection controller 1220, a transmission unit 1240, and an audio packet synthesis unit 1230. The multipoint connection controller 1220 includes an effective stream extraction unit 1222 and a stream synthesis unit 1224. The receiving unit 1210 is electrically connected to the communication line. The receiving unit 1210 receives stream data from each terminal via the communication line. The receiving unit 1210 extracts a video signal and an audio signal from the stream data. The receiving unit 1210 sends the video signal to the multipoint connection controller 1220. The receiving unit 1210 also transmits the audio signal to the audio packet synthesizing unit 1230.

  An effective stream is extracted by the effective stream extraction unit 1220 from the video signal input to the multipoint connection controller 1220. Here, the effective stream refers to video data necessary for displaying video on the monitoring 130 of each terminal. The valid stream extraction units 1222 are included in the multipoint connection controller 1220 by the number of terminals that can participate in the video conference system 10. Each video data extracted by each valid stream extraction unit 1222 is input to the stream synthesis unit 1224. The stream combining unit 1224 combines each video data. This synthesis is performed so that the videos do not overlap based on, for example, area information included in the header of stream data from each terminal. The stream synthesis unit 1224 sends the video data generated by such synthesis to the transmission unit 1240.

  The audio signal acquired by the reception unit 1210 is sent to the audio packet synthesis unit 1230. The voice packet synthesizer 1230 synthesizes voice signals from the terminals. The combined signal is sent to the transmission unit 1240.

  The transmission unit 1240 synthesizes the video signal from the stream synthesis unit 1224 and the audio signal from the audio packet synthesis unit 1230. The transmission unit 1240 adds a header for configuring a transmission packet to the data generated by the synthesis. The transmission unit 1240 transmits a transmission packet to the communication line. As a result, the signal output from the video signal synthesizer 1200 is transmitted to each terminal 100a to 100d.

  With reference to FIG. 13, a control structure of video signal synthesis apparatus 1200 according to the present embodiment will be described. FIG. 13 is a flowchart showing a procedure of processing executed by video signal synthesis device 1200.

  In step S1310, the reception unit 1210 of the video signal synthesis device receives each compressed data from each terminal via the communication line. In step S1320, video signal synthesizer 1200 stores each data in a reception buffer (not shown). In step S1330, video signal synthesizer 1200 reads data from the reception buffer.

  In step S1340, video signal synthesizing device 1200 refers to the time stamp included in each read data and synthesizes data included in the same time zone in order to synchronize data from each terminal. This process is executed in the stream synthesis unit 1224. At this time, the voice signal is similarly synthesized by the voice packet synthesis unit 1230. Each synthesized signal is sent to transmission section 1240.

  In step S1350, transmission section 1240 generates stream data by adding a transmission address to the data generated by each synthesis. Here, the transmission address refers to the network address of each terminal participating in the video conference system. Since each terminal is specified by this address, a video of a participant in the video conference system is distributed to each terminal. In step S1360, transmission unit 1240 transmits stream data based on the transmission address.

  Here, with reference to FIG. 14, the stream data transmitted from the video signal synthesis device 1200 will be described. FIG. 14 schematically shows the structure of stream data.

  The stream data includes a picture header PH, a slice header SH, and display data 1010, 1020, 1030, and 1040. Display data 1010 includes macroblocks based on video data generated by terminal 100a, as shown in FIG. Similarly, the display data 1020 includes a macro block shown in FIG. Display data 1030 includes macroblocks for video data generated by terminal 100c. Display data 1040 includes macroblocks for video data generated by terminal 100d.

  Note that the stream data is not subjected to decoding processing for macroblocks included in each slice layer. That is, the stream data is data generated by combining the video data included in the data received by the video signal synthesizer 1200 as they are. Therefore, data to be transmitted to each terminal is generated in response to reception of video data from each terminal, so that the occurrence of delay for decoding processing is suppressed.

  When each terminal receives the stream data shown in FIG. 14, each monitor TV 130 displays a video (for example, a user who participates in the video conference system) captured by the camera 110 of another terminal in each area. indicate.

  As described above, according to the terminal according to the present embodiment, with respect to data for displaying video for realizing the video conference system, the compression encoding processing is performed in order to display video captured by the terminal. It is executed so as to be data corresponding to a predetermined area. Each terminal transmits the data generated by the compression encoding process to the video signal synthesis apparatus 1200 that synthesizes video data from each terminal in order to realize a so-called other-point connection controller. The video signal synthesizer 1200 generates synthesized data for the video conference system by extracting and synthesizing video data from each terminal. That is, the decoding process and the compression process of the video data are not executed for the difference in the generation of the composite data. Therefore, in the video signal synthesizer 1200, processing delay for generating synthesized data is suppressed. As a result, the user of each terminal, that is, a participant in the video conference can smoothly view the video and audio.

<Modification>
Hereinafter, modifications of the embodiment of the present invention will be described. In the above-described embodiment, the stream data transmitted from the video signal synthesis device 1200 to each terminal is configured to include the same video data. However, the configuration of data transmitted from the video signal synthesis device 1200 is not limited to that. For example, the stream data transmitted to each terminal may be data that does not include video data generated at each terminal. That is, since each terminal has its own video data generated by each MPEG encoder 500, it is not necessary to receive the same data from the video signal synthesis device 1200. In this case, therefore, the process for returning the video data transmitted from the specific terminal back to the transmission source in the video signal synthesis apparatus 1200 can be omitted.

  Note that each terminal and the video signal synthesis device for realizing the video conference system according to this modification have the same hardware configuration as that of the terminal 100 and the video signal synthesis device 1200 described above. Their functions are the same. Therefore, detailed description thereof will not be repeated here.

  Here, with reference to FIG. 15, data transmitted in the video conference system according to the present modification will be described. FIG. 15A is a diagram showing a configuration of data transmitted to terminal 100a by video signal synthesis apparatus 1200 according to this modification. FIG. 15B is a diagram illustrating a configuration of data that the video signal synthesis device transmits to terminal 100b. FIG. 15 (C) is a diagram illustrating a configuration of data that video signal synthesizing apparatus 1200 transmits to terminal 100c. FIG. 15D is a diagram illustrating a configuration of data that video signal synthesis apparatus 1200 transmits to terminal 100d.

  Referring to FIG. 15A, stream data 1510 includes areas 1511 to 1514. An area 1511 stores a header. The area 1512 stores video data generated by the terminal 100b. The area 1513 stores video data generated by the terminal 100c. The area 1514 stores video data generated by the terminal 100d. The header stored in area 1511 has the same items as the header (FIG. 2) stored in area 250, for example.

  When such data is transmitted from the video signal synthesis device 1200 to the terminal 100a, the video data generated by the terminal 100a is not transmitted. Therefore, the amount of communication between the terminal 100a and the video signal synthesis device 1200 can be reduced. The terminal 100 a that receives such stream data 1510 receives the video data A held by itself based on the time stamp included in the header of the area 1511, and the video data B to D stored in the stream data 1510. Are decoded by the MPEG decoder 460. The monitor television 130 displays video corresponding to each video data in each area based on the decoded data.

  As shown in FIG. 15B, the stream data 1520 transmitted from the video signal synthesis device 1200 to the terminal 100b includes regions 1521 to 1524. The header is stored in area 1521. Video data A generated by the terminal 100 a is stored in the area 1522. Video data C generated by the terminal 100c is stored in the area 1523. Video data generated by the terminal 100d is stored in an area 1524. That is, the video data B generated by the terminal 100b is not transmitted from the video signal synthesis device 1200 to the terminal 100b. As a result, the communication amount between the terminal 100b and the video signal synthesis device 1200 is reduced by the data amount of the video data B.

  Referring to FIG. 15C, stream data 1530 transmitted from video signal synthesis apparatus 1200 to terminal 100c includes areas 1531 to 1534. The header is stored in area 1531. Video data A generated by the terminal 100 a is stored in the area 1532. Video data B generated by the terminal 100 b is stored in the area 1533. The video data D generated by the terminal 100d is stored in the area 1534. That is, the video data C generated by the terminal 100c is not transmitted from the video signal synthesis device 1200 to the terminal 100c. As a result, the communication amount is reduced by a data amount corresponding to the video data C.

  Further, referring to FIG. 15D, stream data 1540 transmitted from video signal synthesis apparatus 1200 to terminal 100d includes regions 1541 to 1544. The header is stored in area 1541. The video data A generated by the terminal 100a is stored in the area 1542. The video data B generated by the terminal 100b is stored in the area 1543. The video data C generated by the terminal 100c is stored in the area 1544. In this case, the video data D generated by the terminal 100d is not included in the data transmitted from the video signal synthesis device 1200 to the terminal 100d.

  When the terminal 100d receives the stream data 1540, the video data D generated by itself and the video data A to C included in the stream data 1540 are combined to generate display data. The monitor television 130 displays videos taken by the terminals 100a to 100d in the respective areas based on the data generated in this way.

  As described above, according to the video conference system according to the present modification, each terminal configuring the system receives distribution of stream data that does not include video generated by itself from the video signal synthesis device. Each terminal synthesizes video data generated by another terminal included in the received stream data and video data created by itself. In this way, the amount of data communication on the communication line that constitutes the video conference system can be reduced, so that video display can be realized quickly.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a block diagram showing the system configuration | structure of the video conference system 10 which concerns on embodiment of this invention. FIG. 2 is a diagram conceptually showing a configuration of signals communicated between each terminal shown in FIG. 1 and a video signal synthesis device 1200. It is a figure showing the one aspect | mode of the screen which the terminal 100a, 100b, 100c, 100d displays on the monitor television 130. FIG. It is a block diagram showing the hardware constitutions of the production | generation apparatus 400 of the stream data concerning this Embodiment. 2 is a block diagram showing a hardware configuration of MPEG encoder 500. FIG. FIG. 11 is a diagram illustrating an aspect of data storage in memory 430. It is a figure which represents the coordinate in the monitor television 130 notionally. When terminal 100a displays a video on monitor television 130, it is a diagram showing a corresponding region of video data to be displayed. 3 is a diagram illustrating a configuration of an MPEG stream transmitted from each terminal 100 to a video signal synthesis device 1200. FIG. It is a figure which represents notionally the structure of the stream data transmitted with respect to the video signal synthesis apparatus 1200 from each terminal which concerns on this Embodiment. It is a flowchart showing the procedure of the process which the MPEG encoder 500 of the terminal 100 which concerns on this Embodiment performs. It is a block diagram showing the hardware constitutions of the video signal synthesizer 1200 which comprises the video conference system 10 which concerns on this Embodiment. It is a flowchart showing the procedure of the process which the video signal synthesis apparatus 1200 which concerns on this Embodiment performs. FIG. 3 is a diagram schematically showing a configuration of stream data transmitted from a video signal synthesis device 1200. It is a figure showing the structure of the data transmitted in the video conference system concerning this modification.

Explanation of symbols

  10 video conference system, 100a, 100b, 100c, 100d terminal, 110 camera, 120 microphone, 130 TV monitor, 400 stream data generator, 410, 470 video interface, 420, 480 audio interface, 430 memory, 440, 1240 transmission 450, 1210 receiver, 460 MPEG decoder, 500 MPEG encoder, 510 receive buffer, 520 encoding circuit, 530 extraction circuit, 540 transmission buffer, 1200 video signal synthesizer, 1220 multipoint connection controller, 1222 effective stream extraction Part, 1224 stream composition part, 1230 voice packet composition part.

Claims (17)

A plurality of terminals electrically connected to a communication line, each of which includes a step of generating stream data, each of the plurality of terminals including display means for displaying a video in a display area based on a video signal; The step to generate
Receiving a video signal input;
Preparing region information for specifying a predetermined region in the display region;
Generating the video data for displaying the video based on the video signal in the partial area by encoding the video signal based on the area information;
Generating data for transmission of the video data;
Outputting the data for transmission to the communication line,
In the video signal synthesizer electrically connected to the communication line, the video signal synthesizer further comprises the step of synthesizing the video signal based on the data output from each of the terminals, and the step of synthesizing the video signal comprises:
Receiving each stream data output from each of the plurality of terminals via the communication line, wherein each stream data is preliminarily displayed in order to display a video in a part of the display area. Including video data encoded according to a predetermined position, the position being different according to each of the plurality of stream data,
Based on the plurality of stream data, a synthesis step for generating display data for displaying each video based on the video data in the display area;
Generating data for transmission of the generated display data;
Outputting the generated data for transmission to the communication line,
In each of the plurality of terminals, receiving data generated based on the data for transmission from the video signal synthesizer;
In each of the plurality of terminals, generating display data by performing a decoding process based on the received data;
Each of the plurality of terminals further comprises a step of displaying a video on the display means based on the display data.   The stream data generation method according to claim 1, wherein the generation step includes an encoding step of generating the video data by encoding a video signal corresponding to the partial area.   The stream data generation method according to claim 2, wherein the encoding step executes the encoding using a video signal corresponding to the partial area as a target of motion compensation. The synthesis step includes
Extracting each of the video data from each of the plurality of stream data;
The step of generating the display data by respectively combining the extracted video data based on a predetermined position with respect to the extracted video data. How to generate stream data. Comprising a generating device electrically connected to a communication line, the generating device comprising:
Display means for displaying video in the display area;
Storage means for storing area information for specifying a predetermined area in the display area;
An input means for receiving an input of a video signal;
Generating means for generating video data for displaying the video based on the video signal in the partial area by encoding the video signal based on the area information;
Transmission data generating means for generating data for transmission of the video data;
Output means electrically connected to the communication line and outputting data for transmission to the communication line;
The video signal synthesis device further comprising a video signal synthesis device electrically connected to the communication line,
Input means for receiving a plurality of stream data inputs via the communication line, each of the plurality of stream data corresponding to a predetermined position for displaying a video in a part of a video display area. Encoded video data, and the position differs according to each of the plurality of stream data,
Based on the plurality of stream data, combining means for generating display data for displaying each video based on the video data in the display area;
Transmission data generating means for generating data for transmission of display data generated by the combining means;
Output means for outputting the generated data for transmission to the communication line;
The generator is
Receiving means for receiving data generated based on the data for transmission from the communication line;
Decoding means for generating display data by executing a decoding process based on the data received by the receiving means;
The video conference system further comprising: control means for displaying video on the display means based on the display data. Display means for displaying video in the display area;
Storage means for storing area information for specifying a predetermined area in the display area;
An input means for receiving an input of a video signal;
Generating means for generating video data for displaying the video based on the video signal in the partial area by encoding the video signal based on the area information;
An output means electrically connected to a communication line for outputting the video data to the communication line;
Receiving means for receiving data generated based on the data for transmission from the communication line;
Decoding means for generating display data by executing a decoding process based on the data received by the receiving means;
A stream data generating apparatus, comprising: control means for displaying video on the display means based on the display data.   The stream data generation apparatus according to claim 6, wherein the generation unit includes an encoding unit that generates the video data by encoding a video signal corresponding to the partial area.   The stream data generation device according to claim 7, wherein the encoding unit performs the encoding using a video signal corresponding to the partial area as a target of motion compensation.   The stream data generation device according to claim 6, wherein the video data includes only data for displaying a video only in the partial area.   The stream data generation device according to claim 6, further comprising a changing unit that changes area information stored in the storage unit based on an input from the outside. It further comprises an instruction input means for receiving area information input from the outside,
The stream data generation device according to claim 10, wherein the changing unit changes the region information stored in the storage unit to the region information input via the instruction input unit. The receiving means receives data including the area information from the communication line,
The stream data generation device according to claim 10, wherein the changing unit changes the region information stored in the storage unit to region information included in the received data. Input means for receiving a plurality of stream data inputs via a communication line, each of the plurality of stream data corresponding to a predetermined position for displaying a video in a part of a video display area Including encoded video data, and the position differs according to each of the plurality of stream data;
Based on the plurality of stream data, combining means for generating display data for displaying each video based on the video data in the display area;
Transmission data generating means for generating data for transmission of display data generated by the combining means;
An apparatus for synthesizing stream data, comprising: output means for outputting the generated data for transmission to the communication line. The synthesis means includes
Extraction means for extracting each of the video data from each of the plurality of stream data;
14. The stream data according to claim 13, further comprising: generation means for generating the display data by combining the extracted video data based on a predetermined position with respect to the extracted video data. Synthesizer. Each of the plurality of stream data includes time data for specifying a time at which each of the stream data is generated,
15. The stream data synthesizing apparatus according to claim 13, wherein the synthesizing unit generates the display data based on the time data of each of the plurality of stream data.   The stream data synthesizing apparatus according to claim 13, wherein the synthesizing unit generates the display data without decoding the video data. Each of the plurality of stream data includes specific data for specifying an area in which video based on the video data is displayed,
The stream data synthesizing apparatus according to claim 13, wherein the synthesizing unit generates the display data based on the specific data.

Images