JP2005252898A - Reconstruction program for encoded data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video encoded data synthesizer capable of synthesizing encoded data composed of data units, having irregular data lengths. <P>SOLUTION: A video-encoded data synthesizer 30 comprises a determination portion 31 for analyzing a data unit of video encoded data and determining whether the data unit straddles a boundary of the screen, a reconstruction portion 32 for reconstructing the data unit so as not to straddle over the screen boundary, when it is determined that the data unit straddles over the boundary, and a synthesis portion 33 for generating a synthetic screen from a plurality of reconstructed encoded data. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an encoded data reconstruction program for reconstructing encoded data before being used in screen composition processing.

Conventionally, there is a video conference system configured by connecting a video conference terminal and a multipoint control unit (hereinafter referred to as MCU) via a network.
By transmitting and receiving video images via a network, a video conference using such a conference system becomes possible.

In recent years, reductions in conference and business trip costs have been promoted in each company, and from such a background, further widespread use of video conference systems is expected.
The MCU has a function of synthesizing video images of conference participants sent from each point and distributing the combined video image to each point. By referring to the display screen of the MCU, it is possible to confirm the state of the participants participating in the conference from each point.

In this MCU, low delay and high quality are required for video processing for reasons such as eliminating the unnaturalness of conversation in a conference using a video conference system.
As normally performed in such a video conference system, in order to reduce the amount of communication data, the video conference terminal at each point performs encoding based on a moving image encoding method, and the encoded data is stored in the MCU. Sent to The MCU that has received the encoded data decodes the video encoded data, and then performs image size change processing such as enlargement / reduction as necessary, and requires the images that have been resized or not resized. The number of points is combined, and the combined image is encoded and distributed to each point.

  The series of processing from decoding of encoded data on the MCU side to re-encoding is required to be executed with low delay. However, it is essential to install dedicated hardware (hardware codec) on the MCU side due to the need for high computing power for encoding processing, which contributes to an increase in cost.

  In order to solve this problem, other processing methods have been adopted. For example, the moving picture encoding method H.264 developed by ITU-T. In H.261, GOB (Group Of Blocks) is used in data units. In GOB, each frame does not have data extending from the right end to the left end of the screen, so it can be combined with other video data.

  In the following Patent Document 1, as shown in FIG. In the coding scheme of H.261, an example is shown in which frame images for four points configured with GOB as a data unit, that is, images # 0, # 1, # 2, and # 3 are combined into one image. Such a synthesis method can be applied to an encoding format (for example, H.261) in which consistency between a rectangular area indicated by a data unit (for example, GOB) and a screen rectangle is ensured.

  However, the compression efficiency is H.264. If the data unit (video packet) is not consistent with the screen rectangle, as in the MPEG4 format, which has recently been widely used, for reasons such as being superior to H.261, The above synthesis method is not applicable.

  Patent Document 1 listed below also discloses a method in which the synthesis process can be started in the MCU even when video data is not collected for all points. That is, when an instruction to synthesize an image at each point is given at a predetermined timing, the data at that point is treated as a P picture for a point for which data for one frame is not prepared, Dummy data (GOB0) indicating that there is no change is inserted.

However, a motion vector in a GOB may point to a macroblock in an adjacent GOB. In such a case, when the above-described method for inserting dummy data is applied, the image is disturbed at the receiving terminal until the macroblock data is refreshed by the intra-frame encoded data (I picture). End up.
Japanese Patent Laid-Open No. 7-222131, “Screen Synthesis System and Method for Multipoint Conference”

  An object of the present invention is to reconstruct video encoded data for reconfiguring data units of video encoded data so as to be able to synthesize encoded data consisting of data units having irregular data lengths. Is to provide a program.

  Another object of the present invention is to provide a video encoded data synthesizing apparatus capable of synthesizing encoded data composed of data units having an irregular data length.

  An encoded data reconstruction program according to an aspect of the present invention is a program for causing a computer to perform a process of reconfiguring encoded data before being used in a screen synthesis process. Analysis processing to determine whether or not to cross the screen boundary, and when it is determined that the data unit crosses the screen boundary, the data unit is re-established so as not to cross the boundary. An encoded data reconstruction program that causes the computer to execute a reconstruction process to be configured.

Since the data unit of the video encoded data may have an irregular data length, it may cross the screen boundary depending on the case. Using the encoded data reconstruction program of the present invention, the data unit is reconstructed so as not to cross the screen boundary. As a result, the compositing side can perform the compositing as if the fixed-length data unit aligned with the screen boundary is handled.

  In the encoded data reconstruction program of the present invention, the data unit is reconstructed so as not to cross the screen boundary. As a result, the compositing side can perform the compositing as if the fixed-length data unit aligned with the screen boundary is handled.

  In the present invention, since the stream composition method is adopted, it is not necessary to perform “decoding, composition, and re-encoding”, the processing load of the apparatus is very light, and it is easy to process by software. is there.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In the present embodiment, the video data transmitted to the MCU is encoded on the video conference terminal side using the MPEG-4 encoding method. In MPEG-4, as shown in FIG. 1, a macro block (16 horizontal pixels × 16 vertical pixels: Macro Block, hereinafter referred to as MB) is used as a basic unit of data. A line in which MBs are arranged in a horizontal line by the width of the screen is called a macro block line (MBL).

  A data unit when the encoded data is handled in the stream format is called a video packet (hereinafter referred to as VP). In MPEG-4, the number of MBs included in a VP is not determined. For example, as shown in FIG. 1, some MBLs include several lines, or in some cases, less than 1 MBL. is there. From this relationship, the VP boundary and the MBL boundary usually do not coincide.

FIG. 2 is a block diagram showing the configuration of the video encoded data synthesis system of this embodiment.
2, the encoded video data synthesis system 10 is constructed by the video conference terminal 20 ₁ to 20 _N and the encoded video data synthesis device 30 is connected via a network.

Each video conference terminal includes encoders 21 _{1 to} 21 _N. Each encoding unit performs an encoding process on an image captured from the video conference terminal.
The image data encoded by the encoding unit of the video conference terminal is transmitted to the video encoded data synthesizing device 30 via the network. The determination unit 31 in the encoded video data synthesis apparatus 30 analyzes the received video packet and determines whether the video packet straddles the macroblock line.

  The packet reconstruction unit 32 divides the video packet at the boundary of the macroblock line when the video packet straddles the macroblock line. As a result, as shown in FIG. 3, the reconstructed video packet does not cross the macroblock line.

  Although not certain from FIG. 3, there may be a case where a macroblock line that is divided into two or more video packets remains even after the dividing process. The determination unit 31 preferably determines whether a video packet in the macroblock line can be integrated with respect to the macroblock line divided into two or more video packets.

When the integration is possible, the packet reconstruction unit 32 performs an integration process on the video packets in the macroblock line.
The encoded data (video packet) from each point is reconstructed through the processes described above, and then input to the packet combining unit 33 in FIG.

The packet combining unit 33 performs necessary combining processing, and the encoded data (video packet) after the combining is sent to a predetermined point.
When a video conference is performed at five points and an image other than the own point is received and displayed at each point, the packet combining unit 34, for example, displays four points other than the own point at the upper left, upper right, and lower left. Place it in the lower right and synthesize.

FIG. 4 is a schematic configuration diagram of a video packet encoded by the encoding unit of FIG.
As shown in FIG. 4, the video packet holds a macro block number (hereinafter referred to as MBnum) in the header portion. This MBnum indicates the position information of the video packet in the frame.

FIG. 5 is a flowchart showing a video packet reconstruction process performed in cooperation with the determination unit and the packet reconstruction unit in FIG.
In FIG. 5, first, variables lenVP and lenRest indicating the packet length of the video packet are initialized to 0 in step S101.

  In step S102, a video packet is acquired. This acquisition result is set in the variable vp. In step S103, it is determined whether a video packet is set in the variable vp as a result of the acquisition process in step S102. If no video packet is set in the variable vp, a series of processing ends.

  If a video packet is set in the variable vp, the process proceeds to step S104. In step S104, the length of the video packet set in the variable vp is set in the variable lenVP. In step S105, it is determined whether a video packet is set in the variable rest with reference to the variable lenRest.

If no video packet is set in the variable rest, the process proceeds to step S106. In step S106, the variable lenVP indicates the MBL indicating the length of the macroblock line. It is determined whether LEN or higher.

In step S106, LenVP ≧ MBL If it is LEN, the process proceeds to step S107. In step S107, MBL The video packet of variable VP is divided by the length of LEN, and the remaining video packets are set in variable rest. In addition, the length of the video packet set in the variable rest is set in the variable lenRest. In step S108, the header information and the like are corrected and output for the remaining video packets set in the variable rest.

Thereafter, the process proceeds to step S109. In step S109, lenRest ≧ MBL It is determined whether it is LEN. lenRest ≧ MBL If it is LEN, in step S110, the video packet set in the variable rest is reset to the variable vp, and the variable lenRest = 0 is set. Then, the process returns to step S104.

On the other hand, in step S109, lenRest <MBL If it is LEN, the process returns to step S102 to obtain a new video packet. The newly acquired video packet is set in the variable vp.

  In any case, the process of step S104 is performed on the video packet of the variable vp. Then, in step S105 again, it is determined whether a video packet is set in the variable rest.

In step S106, LenVP <MBL If it is LEN, the process proceeds to step S113, where the video packet set in the variable vp is reset in the variable rest. Also, the length of the video packet is set in the variable lenRest. Then, the process returns to step S102.

  On the other hand, if a video packet is set in the variable rest in step S105, the process proceeds to step S111, where the video packet set in the variable vp and the video packet set in the variable rest can be combined. Is determined.

  If it is determined in step S111 that it can be combined, the process proceeds to step S112, where the video packet set in the variable vp and the video packet set in the variable rest are combined, and the combined video packet is combined. To the variable vp. Then, the process returns to step S104.

  On the other hand, if it is determined in step S111 that the packets cannot be combined, the video packet set in the variable rest has been reconstructed, so the process proceeds to step S114 and is set in the variable rest. Output video packets.

In step S115, the sum of the variable lenRest and the variable lenVP is MBL. It is determined whether it is greater than LEN.
In step S115, the sum of the variable lenRest and the variable lenVP is MBL. Greater than LEN, ie lenRest + lenVP> MBL If it is determined as LEN, the process proceeds to step S116.

In step S116, the video packet set in the variable vp is (MBL (LEN-lenRest), and the header information of the first half of the divided video packet is corrected and output. In the latter half of the divided video packet, the header information and the like are corrected and set in the variable rest. The packet length of the latter half is set in the variable lenRest. Then, the process proceeds to step S109.

In step S115, the sum of the variable lenRest and the variable lenVP is MBL. LEN or less, that is, lenRest + lenVP ≦ MBL If it is determined as LEN, the process proceeds to step S117.
In step S117, the video packet set to the variable vp is reset to the variable rest. The packet length of this video packet is set in the variable lenRest. Thereafter, the process returns to step S102.

  For example, the video packet shown on the left side of FIG. 6 is reconstructed into the video packet shown on the right side of FIG. 6 by the series of processes described above. That is, by performing such reconstruction processing, there are no video packets straddling the macroblock line, and the packet length is limited to the length of one macroblock line at the longest.

In the following, the process for determining whether two video packets can be combined will be described in more detail. This process corresponds to the determination process in step S111 of FIG.
Note that in the present embodiment, the above-described determination of whether or not to combine is performed based on the quantization value stored in the video packet. Here, the quantized value is an encoding used when division is performed on a coefficient obtained by subjecting each macroblock to discrete cosine transform (DCT), as is normally done in the MPEG system. Parameter. As a result of this division, a quantized DCT coefficient is obtained. By performing quantization, the amount of encoding can be reduced.

  In each video packet, the quantization value is stored as follows. That is, a reference value for the quantization value of each macroblock in the video packet is stored in the header portion of the video packet. In each macroblock, the quantization value is stored as difference information from the quantization value of the previous macroblock.

  In the present embodiment, the quantized value of the first macroblock is used as the reference value. Because of this relationship, the difference information stored in the first macroblock is originally “0”, but in order to further increase the compression efficiency, “difference information is stored for the macroblock of“ difference “0” ”. "Do not do". Then, within one video packet, by making the difference between quantized values of consecutive macroblocks fall within a predetermined range (here, ± 2 range), absolute values are obtained for all macroblocks. It is possible to increase the compression efficiency as compared with the case of storing.

More specifically _{, let} Q _{(m, n)} be the quantized value of the nth macroblock in the mth video packet in a certain frame. In the header portion of the m-th video packet, the reference value of the quantization value for the m-th video packet, for example, the quantization value of the first macroblock, is set as the parameter quant. Stored in scale. Each macroblock stores a parameter dquant indicating a difference from the quantized value of the previous macroblock. That is, the quantized value of the first macroblock in the mth video packet is given by the following equation (1).

Q _{(m, 1)} = quant scale (1)
The quantized values of the subsequent macroblocks are given by the following equation (2).
Q _{(m, n)} + dquant = Q _{(m, n + 1)} (2)
Here, dquant = {− 2, −1, 1, 2}.

In order to be able to combine the m-th video packet and the m + 1-th video packet according to the above equation (2), the quantization value of the last macroblock of the m-th video packet and the first of the m + 1-th video packet The difference from the quantization value of the macroblock must be within a predetermined range (here, a range of ± 2). That is, if the m-th video packet is composed of last macroblocks, the following conditional expression (3):
| Q _{(m, last)} −Q _{(m + 1,1)} | ≦ 2 (3)
Is established, the mth video packet and the m + 1th video packet can be combined.

  In the present embodiment, from the viewpoint of increasing the compression efficiency as described above, in the case of corresponding to dquant = 0, that is, in two consecutive macroblocks, the quantized value of the subsequent macroblock is the previous one. Therefore, if the difference is zero, the difference information is not stored in the subsequent macroblock. However, this does not limit the expression of difference information in the present invention. For example, in the above case, even when the difference information = 0, it is possible to store information “0” in each macroblock. Further, difference information from the reference value stored in the header part may be stored in each macroblock. That is, it is possible to deal with any case where the quantized value of each macroblock is expressed by difference information based on the reference value stored in the header part. In this case, in the combination determination process, when each macroblock in a continuous video packet existing in one macroblock line can be expressed by difference information based on the reference value of the header part, the video packets are combined. .

  FIG. 7 is a flowchart of the combination determination process. This determination process is performed by the determination unit in FIG. 2, and is a more detailed flowchart for the determination process in step S111 in FIG.

  First, when it is determined in step S109 in FIG. 5 that a video packet is set in the variable rest, in step S201 in FIG. 7, each macroblock in the video packet set in the variable rest is scanned, The quantized values in the macroblock are calculated sequentially. That is, for example, the quantization value for the first macroblock is given by the above equation (1), and for the subsequent macroblocks, the quantization value is sequentially obtained by repeatedly using the above equation (2). In step S202, the quantization value of the last macroblock of the video packet set in the variable rest obtained in step S201 is stored in the variable MBlast.

  In the state where the control is shifted to step S201, the video packet immediately after the video packet set in the variable rest is set in the variable vp. Therefore, in the subsequent step S203, the quantized value of the first macroblock of the video packet set in the variable vp is stored in the variable MBnew.

  In step S204, it is determined whether the video packet set in the variable rest and the video packet set in the variable vp can be combined. If | MBlast−MBnew | ≦ 2, it is determined that the combination is possible, and the process proceeds to step S112 in FIG. If | MBlast-MBnew |> 2, it is determined that the connection is impossible, and the process proceeds to step S113 in FIG.

FIG. 8 is a diagram illustrating a case where it is determined that the connection is impossible in the flowchart of FIG.
In FIG. 8, since the quantized value Q (1,1) = 10 of the first macroblock of the video packet VP # 1 and dquant = −2 for the second macroblock, Since the quantized value Q (1,2) = Q (1,1) + dquant = 10 + (− 2) = 8, and dquant = −1 for the third macroblock, the quantum It is shown that the quantization value Q (1,3) = Q (1,2) + dquant = 8 + (− 1) = 7. Further, it is indicated that the quantization value Q (1, n) = Q (1, n−1) + dquant = 12 for the last (n-th) macroblock of the video packet VP # 1. .

On the other hand, it is shown that the quantization value Q (2, 1) = 15 of the first macroblock of the video packet VP # 2 subsequent to the video packet VP # 1.
In the example of FIG. 8, | Q (1, n) −Q (2,1) | = | 12−15 | = 3> 2, so that the video packet VP # 1 and the video packet VP # 2 cannot be combined. It is determined.

FIG. 9 is a diagram for explaining a case where it is determined that combining is possible in the flowchart of FIG.
In FIG. 9, the quantized value Q (1,1) = 10 of the first macroblock of the video packet VP # 1, and the last (nth) macroblock of the video packet VP # 1 It is shown that the quantized value Q (1, n) = Q (1, n−1) + dquant = 12.

On the other hand, it is shown that the quantization value Q (2, 1) of the first macroblock of the video packet VP # 2 subsequent to the video packet VP # 1 is 11.
In the example of FIG. 9, | Q (1, n) −Q (2,1) | = | 12−11 | = 1 ≦ 2, so that the video packet VP # 1 and the video packet VP # 2 can be combined. It is determined.

  Note that the lower half of FIG. 9 shows the combined video packet. The quantized value (= 11) of the first macroblock of the subsequent video packet VP # 2 was expressed as difference information from the reference value of the header portion of VP # 2 before combining. It is expressed as difference information (= −1) from the quantization value (= 12) of the last macroblock of the packet VP # 1.

  In the above description, the quantized value is expressed based on difference information from the reference value, and a plurality of video packets included in one macroblock line are determined based on whether or not the differential expression is possible. Although it has been determined whether or not to combine, the quantized value is not necessarily expressed by the difference information. For example, each quantized value can be expressed by an absolute value. In that case, a plurality of video packets included in one macroblock line are combined without going through the above-described combination determination process.

  FIG. 10 and FIG. 11 are diagrams in which macroblock numbers are specifically described with respect to the example shown in FIG. FIG. 10 shows an example of data before reconstruction, and FIG. 11 shows an example of data after reconstruction.

  In FIG. 10, the image size is a horizontal 11 × vertical 9 macroblock. This image is composed of six video packets of VP # 1, VP # 2, VP # 3, VP # 4, VP # 5, and VP # 6. VP # 1 is composed of nine macroblocks from MBnum = 0 to MBnum = 8. VP # 2 is composed of 18 macro blocks from MBnum = 9 to MBnum = 26. VP # 3 is composed of 25 macroblocks from MBnum = 27 to MBnum = 51. VP # 4 is composed of seven macroblocks from MBnum = 52 to MBnum = 58. VP # 5 is composed of 13 macro blocks from MBnum = 59 to MBnum = 71. VP # 6 is composed of 27 macro blocks from MBnum = 72 to MBnum = 98.

  Table 1 below shows quantization values (one example) for the first and last macroblocks of these video packets VP # 1 to VP # 6.

  Referring to Table 1, the difference between the quantization values of the last macroblock of VP # 1 (MBnum = 8) and the first macroblock of VP # 2 (MBnum = 9) is given by | 11−8 | = 3 Since the condition (3) is not satisfied, the connection is impossible. Similarly, in VP # 2 and VP # 3, the difference in quantization value is given by | 10-11 | = 1, so that the coupling is possible. In VP # 3 and VP # 4, the difference in quantization value is | 10−12 | = 2 so that the coupling is possible. In VP # 4 and VP # 5, the difference between the quantization values is given by | 9−8 | = 1, so that the coupling is possible. VP # 5 and VP # In 6, the difference between the quantized values is given by | 8-12 | = 4, so that the coupling is impossible.

In the following, an outline of the fact that the data after reconstruction shown in FIG. 11 is obtained by performing the processing shown in the flowchart of FIG. 5 or 7 on the data example before reconstruction in FIG.
First, in step S102, the video packet VP # 1 is acquired. This acquisition result is set in the variable vp. In step S104, the length of the video packet (VP # 1) set to the variable vp = 9 is set to the variable lenVP. In this case, no video packet is set in the variable rest, so that the process proceeds from step S104 to step S106. In this case, LenVP (= 9) <MBL Since LEN (= 11), the process advances from step S106 to step S113.

  In step S113, the video packet set to the variable vp is reset to the variable rest, and the packet length (= 9) of the video packet VP # 1 is also set to the variable lenRest. Then, the process returns to step S102. In step S102, the next video packet VP # 2 is acquired and set in the variable vp. In step S104, the length of the video packet (VP # 2) set to the variable vp = 18 is set to the variable lenVP.

  In this case, since the video packet is set in the variable rest, the process proceeds from the determination step of step S105 to the determination step of step S111. Referring to Table 1 above, since there is a difference of | 11−8 | = 3 at the boundary between the video packets VP # 1 and VP # 2, in step S111, it is determined that combining is impossible, and the process proceeds to step S114. . In step S114, the video packet set in the variable rest is output. This output video packet becomes VP # 1 (after reconfiguration).

In this case, lenRest + lenVP = 9 + 18> MBL Since LEN = 11, the process proceeds from step S115 to step S116, where the video packet set in the variable vp is MBL. Divide by LEN-lenRest = 11−8 = 3. As a result, the header information and the like are corrected and output for the three macroblocks from the beginning of the video packet set in the variable vp. This output video packet becomes VP # 2 (after reconfiguration).

  In the latter half of the divided video packet, the header information and the like are corrected and set in the variable rest. The packet length 18-3 = 15 of the latter half is set in the variable lenRest. Then, the process proceeds to step S109.

In this case, lenRest = 15 ≧ MBL Since LEN = 11, the process proceeds from step S109 to step S110, where the video packet set in the variable rest is reset to the variable vp, and the variable lenRest = 0 is set. Then, returning to step S104, the length of the video packet set to the variable vp = 15 is set to lenVP.

Since the variable lenRest = 0 is set in step S110, “no rest” occurs, so the process proceeds from step S105 to step S106. And lenVP = 15 ≧ MBL Since LEN = 11, the process proceeds to step S107. In step S107, the MBL in the video packet set in the variable vp LEN = 11 minutes are output. This output video packet becomes VP # 3 (after reconfiguration). The remaining video packets are set in the variable rest. This remaining video packet length = 15-11 = 4 is set in the variable lenRest. In step S108, header correction or the like is performed on the video packet set in the variable rest. Then, the process proceeds to step S109.

In this case, lenRest = 4 <MBL Since LEN = 11, the process returns from step S109 to step S102, where a new video packet VP # 3 is acquired and set in the variable vp. In step S104, the video packet length = 25 set in the variable vp is stored in the variable lenVP.

By continuing such processing, video packets VP # 1 (after reconstruction) to VP # 11 (after reconstruction) are sequentially output as shown in FIG.
As described above, when the reconstruction process is performed on the packet, the video composition unit performs the composition process based on the reconstructed packet.

FIG. 12 is a diagram for explaining this composition processing.
In FIG. 12, it is assumed that two screens are arranged side by side on the composite screen, as in the case where one screen is obtained by combining four screens (frames).

In FIG. 12, a frame (No. 1) is shown in the upper right. In the upper left, a frame (No. 2) is shown. Both frames have already undergone reconstruction processing.
In the terminal that receives the composite screen, in order to correctly decode the composite screen, starting from the upper left, in the order of upper left → upper right → lower left → lower right, until the lower right, it will be a serial number, The macroblock number of each video packet is reassigned. That is, in this case, the macroblock number in each video packet is rewritten so that two screens are multiplexed for each macroblock line.

  As described above, in the present embodiment, since the boundary of the macroblock line is aligned with the boundary of the video packet by the packet reconstructing unit, video by header analysis and rewriting of the macroblock number is performed without performing decoding processing. Multiplexing in units of packets enables screen composition by the composition unit.

The synthesized screen on which the macro block number has been reassigned is received on the terminal side. Then, it is decoded and reproduced in the decoding unit.
FIG. 13 is a diagram for explaining the synthesis timing for synthesizing the screen.

In FIG. 13, synthesis is performed by setting a cycle earlier than the frame rate of the encoded video data received from the terminal at each point as the synthesis timing.
If data from all points is not collected at a certain synthesis timing, encoded data that indicates repetition of the image of the previous frame is inserted for the points that are not arranged. In addition, when data from all points is available, synthesis is performed using these data.

  In the example of FIG. 13, data to be transmitted is input from each point for 5 frames per second, and synthesis is performed at a rate of 7 times per second. That is, frames are received at 5 Hz intervals, and synthesis is performed at 7 Hz intervals.

  At the synthesis timing t1, data for four points are prepared, and an image is synthesized using these data. On the other hand, at time t2 at which 1/7 second has elapsed from time t1, reception of the next encoded data is completed at points 2 and 4, but reception of the next encoded data is received at points 1 and 3. Is not complete. In such a case, for the points 1 and 3, the image at the previous time t1 is repeatedly used. That is, for the points 1 and 3, encoded data indicating that there is no change from the previous image is generated and used.

Thereby, even when the arrival timing of the data from each point is not synchronized, the data at each point can be combined.
In the above description, the determination unit and the packet reconstruction unit are provided on the video encoded data synthesis apparatus side. However, the determination unit and the packet reconstruction unit may be provided on the video conference terminal side. In this case, after the captured image is encoded on the video conference terminal side, the processing of the flowcharts of FIGS. 5 and 7 is performed, and the video packet is reconstructed on the terminal side. In this case, the reconfiguration process can be omitted on the synthesizing apparatus side, and the processing load is reduced.

  In the above description, the packet reconstruction process is performed after the encoding process. However, the packet reconstruction process and the encoding process may be performed in parallel on the terminal side. In this case, for example, the packet reconstruction process shown in FIG. 5 is executed in parallel with the encoding process. In this way, the video packet included in the encoded data does not cross the screen boundary. Also in this case, the reconfiguration process can be omitted on the synthesizing apparatus side, and the processing load is reduced. Note that a reconstruction / encoding unit that performs such reconstruction / encoding processing includes a macroblock generator that generates macroblock data from a captured image, and a concatenation unit that connects the generated macroblocks. And a determination unit that determines whether the length of the linked macroblocks exceeds the number of macroblocks constituting the screen size, and when it is determined that the length exceeds the number of macroblocks of the screen size, A video packet generation unit to be generated and an encoded data generation unit to generate encoded data from the plurality of generated video packets.

  Note that the encoding unit of the video conference terminal performs encoding so that the motion vector of each MB points only to the inside of the frame (screen) based on a predetermined profile, for example, a simple profile. May be. That is, in the motion detection process, when the motion vector points outside the reference image frame, the next candidate may be searched for and a vector facing in the frame may be selected. In this way, the processing load on the packet synthesizing unit 33 in the video encoded data synthesizing device 30 can be greatly reduced, and the macroblock can be obtained by the intra-frame encoded data (I picture) as described above. It is possible to prevent the image from being disturbed until the data is refreshed.

  In a video conference, considering that the image portion corresponding to the background of the conference participant has almost no motion (in many cases, it is stationary), in the encoding process, the inner side of the screen is added to the motion vector. It is also possible to remove the above restriction of making it go.

  FIG. 14 is a configuration example of a video encoded data synthesis system. In this system, a video encoded data synthesizing device capable of synthesizing and sending images to N points is provided. Further, in this example, an image of the own point is also displayed at each point, mixed with images from other points.

  The above-described determination processing and packet reconstruction processing of the present invention can be realized as hardware and also as software. When implemented as software, the program is loaded from the storage medium to the internal memory and executed.

FIG. 15 is a diagram illustrating an example of a storage medium in which a program for performing determination processing and packet combining processing according to the present embodiment is stored.
As shown in FIG. 15, the storage medium includes a portable storage medium 46 that is removable from a medium driving device 47 such as a CD-ROM, a flexible disk (or may be an MO, a DVD, a removable hard disk, etc.). In addition, a storage means (database or the like) 42 in an external device (server or the like) transmitted via the network line 43, a memory (RAM or hard disk or the like) 45 in the main body 44 of the information processing device 41, or the like is included.

The program of the present embodiment is loaded from the storage medium to the memory 45 in the main body 44 and executed.
The present invention may have the following configuration.

(Supplementary note 1) In a program for causing a computer to execute processing for reconstructing encoded data prior to being used in screen composition processing,
A determination process for analyzing the data unit of the video encoded data and determining whether to cross the boundary of the screen;
When it is determined that the data unit crosses the screen boundary, a reconstruction process for reconfiguring the data unit so as not to cross the boundary;
A program for reconstructing encoded data, characterized in that the computer is executed.

  (Supplementary note 2) The encoded data reconstruction program according to supplementary note 1, wherein the encoded video data is encoded such that a motion vector indicates only the inside of the screen.

  (Additional remark 3) In the said determination process, it is determined whether a data unit crosses the boundary of a screen based on the positional information of the data unit of the said encoded data, and the screen size stored in the memory | storage means of the said computer. A program for reconstructing encoded data as set forth in appendix 1, wherein:

(Supplementary note 4) The encoded data reconstruction program according to supplementary note 1, wherein the reconstruction process further includes a process of combining those that can be combined within the screen size.
(Supplementary Note 5) The encoded data reconstruction program according to Supplementary Note 1, wherein the reconstruction process further includes a process of combining a plurality of data units existing in the screen size.

(Supplementary Note 6) The data unit includes a header part and a plurality of basic units. The header part stores a reference value of an encoding parameter value, and the encoding parameter value in each basic unit is It is expressed by difference information determined based on the reference value,
The reconfiguration process further includes a combination determination process for determining whether or not the data units can be combined at the boundary of the data units when there are a plurality of data units within the screen size,
The combination determination process combines the data units when the basic units in the continuous data units existing in the screen size can be expressed by the difference information based on the reference value of the header part. The reconstructed program of encoded data according to appendix 3.

(Supplementary note 7) The encoding parameter value of each basic unit determined based on the reference value is expressed as difference information from the encoding parameter value of the previous basic unit in each basic unit,
In the combination determination process, in a continuous data unit existing within the screen size, the encoding parameter value of the first basic unit of the subsequent data unit is the encoding of the last basic unit of the previous data unit. The encoded data reconstruction program according to appendix 6, wherein the data units are combined when they can be expressed as difference information from the parameter values.

(Supplementary note 8) The encoded data reconstruction program according to supplementary note 1, wherein the data unit is a video packet.
(Supplementary note 9) The encoded data reconstruction program according to supplementary note 6, wherein the data unit is a video packet, and the basic unit is a macroblock.

(Supplementary Note 10) In a program for causing a computer to execute a process of encoding and reconstructing image data before being used in the screen composition process,
Processing to generate basic unit data from captured images;
Processing for concatenating the generated basic units;
A process of determining whether the length of the linked basic units exceeds the number of basic units constituting the screen size stored in the storage means of the computer;
When it is determined that the data size exceeds the process of generating a data unit from the number of basic units of the screen size,
Processing for generating encoded data from the plurality of generated data units;
A program for encoding / reconstructing image data, characterized in that the computer is executed.

(Supplementary Note 11) A determination unit that analyzes a data unit of video encoded data and determines whether to cross a screen boundary;
A reconfiguration unit configured to reconfigure the data unit so that the data unit does not cross the boundary when it is determined that the data unit crosses the screen boundary;
A video encoded data synthesizing device comprising: a synthesizing unit that generates a synthesized screen from a plurality of reconstructed encoded data.

(Additional remark 12) The said synthetic | combination part is started by a periodic timing,
At a predetermined synthesis timing, when the video encoded data from all the points is not complete, the synthesis unit displays information indicating that the previous frame is repeated for the non-aligned point by one frame. 9. The video encoded data synthesizing device according to appendix 8 , wherein the synthesizing is performed by inserting a portion.

(Supplementary Note 13) In an encoding / reconstructing unit provided in a terminal that transmits video encoded data to a video encoded data synthesizer,
A basic unit generator for generating basic unit data from the captured image;
A connecting part for connecting the generated basic units;
A determination unit for determining whether the length of the connected basic units exceeds the number of basic units constituting the screen size;
A data unit generator that generates a data unit from the number of basic units of the screen size when it is determined that
An encoded data generation unit that generates encoded data from the plurality of generated data units;
An encoding / reconstructing unit comprising:

  The present invention can be applied to a stream synthesizing method in which decoding / encoding processing is not performed on the video encoded data synthesizing apparatus side.

It is a figure explaining the data unit at the time of handling encoded data in a stream format. It is a block diagram which shows the structure of the video coding data synthetic | combination system of this embodiment. It is a figure explaining a mode that the boundary of a video packet is reconfigure | reconstructed It is a figure which shows the structure of a video packet. It is a flowchart which shows the reconstruction process of a video packet. It is a figure which shows the example of data before and behind a reconstruction process. It is a flowchart of a joint determination process. It is a figure explaining the case where it determines with a coupling | bonding impossible in the flowchart of FIG. It is a figure explaining the case where it determines with combining possible in the flowchart of FIG. It is the figure which described the macroblock number concretely with respect to the data before reconstruction shown by FIG. It is the figure which described the macroblock number concretely with respect to the data after the reconstruction shown by FIG. It is a figure explaining a synthetic | combination process. It is a figure explaining the synthetic | combination timing which synthesize | combines a screen. 1 is a configuration example of a video encoded data synthesis system. It is a figure which shows the example of a storage medium. It is a figure explaining the conventional screen composition.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Video encoding data synthesis system 20 Video conference terminal 21 Encoding part 30 Video encoding data synthesis apparatus 31 Judgment part 32 Packet reconstruction part 33 Packet composition part 41 Information processing apparatus 42 Storage means 43 Network 44 Apparatus main body 45 Memory 46 Possible Portable storage medium 47 Medium drive device

Claims (5)

Prior to being used in screen composition processing, in a program for causing a computer to execute processing for reconstructing encoded data,
A determination process for analyzing the data unit of the video encoded data and determining whether to cross the boundary of the screen;
When it is determined that the data unit crosses the screen boundary, a reconstruction process for reconfiguring the data unit so as not to cross the boundary;
A program for reconstructing encoded data, characterized in that the computer is executed.   In the determination process, it is determined whether or not the data unit crosses a screen boundary based on the position information of the data unit of the encoded data and the screen size stored in the storage unit of the computer. The encoded data reconstruction program according to claim 1, wherein:   2. The encoded data reconstruction program according to claim 1, wherein the reconstruction process further includes a process of combining those that can be combined within the screen size. A determination unit that analyzes the data unit of the video encoded data and determines whether to cross the boundary of the screen;
A reconfiguration unit configured to reconfigure the data unit so that the data unit does not cross the boundary when it is determined that the data unit crosses the screen boundary;
A video encoded data synthesizing device comprising: a synthesizing unit that generates a synthesized screen from a plurality of reconstructed encoded data. The synthesizing unit is activated at a periodic timing,
At a predetermined synthesis timing, when the video encoded data from all the points is not complete, the synthesis unit displays information indicating that the previous frame is repeated for the non-aligned point by one frame. 5. The encoded video data synthesizing apparatus as claimed in claim 4, wherein the synthesizing is performed by inserting a part.

Images