JP3880597B2 - Multimedia information editing device - Google Patents

Description

  The present invention relates to a method for encoding each moving image data when a plurality of moving image data are simultaneously reproduced as a single video, and an editing method for generating a multi-screen synthesized moving image for any combination of moving image data, The present invention relates to a multimedia information editing apparatus using the.

  Conventionally, a composite code sequence obtained by dividing a moving image screen into a plurality of regions and concatenating the code sequences encoded for each region is a code sequence generated by performing encoding without dividing the screen into regions. As a method for achieving the same syntax, there is one disclosed in JP-A-7-298263.

FIG. 51 is a block diagram showing the outline of the operation.
First, an input moving image is divided into a plurality of areas in units of frames by a screen divider. At this time, the screen divider determines the division method based on the screen division control signal from the division controller. Here, the frame is vertically divided into three, and the divided image signals are divided image signals 77, 78, and 79 in order from the left area of the original video, and are input to the encoders 62, 63, and 64, respectively. The Each encoder receives the size of the divided frame and the encoding condition from the division controller, encodes the input divided image signal, and outputs the encoded image signals as code sequences 80, 81, and 82 to the code sequence synthesizer, respectively. After that, the division controller outputs a code sequence synthesis signal for transmitting the frame division method instructed to the screen divider to the code sequence synthesizer, and the code sequence synthesizer receives the code sequence synthesis signal from each encoder. Are combined in the order of the code strings so as to have the same syntax as that in the case where the frame is not divided into frames, and a synthesized code string is finally output.
JP 7-298263 A

  In Japanese Patent Application Laid-Open No. 7-298263, an image signal that was originally one video screen is divided into image signals of a plurality of areas according to instructions from a division controller, and then each image signal is MPEG-encoded with separate encoders. Processing is performed, and finally, output code strings of a plurality of encoders are arranged so as to have an original syntax.

  At this time, each encoder receives the size of the divided frame and the encoding condition from the division controller, so that the image signal encoded by each encoder encodes which area of the original video screen. In other words, the MPEG encoding process is performed after knowing in which area of the screen the code string output by the encoder itself is finally displayed.

  However, when displaying a reduced moving image (hereinafter, “thumbnail moving image”) in an arbitrary image area of a multi-screen, the position to display is not known at the time of performing MPEG encoding processing on each thumbnail moving image.

  Therefore, at the time of synthesizing a multi-screen, a process for designating a display position must be performed on each encoded thumbnail moving image (hereinafter referred to as an encoded thumbnail moving image).

  Incidentally, for example, in the MPEG standard (for reference, FIG. 52 shows a part of the syntax of a video stream in MPEG), the horizontal position at which a slice is displayed when decoded and reproduced is the head of the slice. The variable length code bit macroblock_address_increment (hereinafter referred to as MB_Addr_Inc) of the macroblock. When the value of the MB_Addr_Inc part is changed to arrange the encoded thumbnail video at an arbitrary position, the number of bits of the MB_Addr_Inc part that is a variable length code changes. As a result, when the multi-screen video is synthesized (if it is simply applied to MPEG as the process for specifying the display position for the encoded thumbnail video), the change occurs for all the code sequences following the MB_Addr_Inc part. As a result, a bit shift operation corresponding to the number of bits generated occurs, and a huge amount of calculation is required.

  FIG. 53 shows an example of this phenomenon. As shown in FIG. 53 (a), when the encoded thumbnail moving image A is arranged at the lower right of the multi-screen of 2 × 2 in the vertical direction, the code string of the encoded thumbnail moving image A changes as shown in FIG. 53 (b). The vertical position when the slice is viewed from the entire moving image is specified by the last 1 byte of the slice_start_code part.

  On the other hand, as described above, the horizontal position of the slice is specified by MB_Addr_Inc. The horizontal position of the slice has been changed from 1 to 5 as a result of synthesis, but when MB_Addr_Inc = 1 is represented by a variable-length code, it is 1 bit, and when MB_Addr_Inc = 5 is represented by a variable-length code, 4-bit " 0010 ", and after synthesis, an extra bit string of 3 bits is required. Therefore, as shown in FIGS. 53B and 53C, the code strings after MB_Addr_Inc after synthesis are arranged in a state shifted to the right by an extra 3 bits. Usually, a computer and a storage device normally used in the computer are accessed in byte units. Therefore, as shown in FIG. 53 (c), bit operations are required for all of the first macro block after MB_Addr_Inc.

  Similarly, in the MPEG standard, there is no guarantee that the code strings between adjacent macro blocks are separated in byte units (byte alignment), and only after the code string of the last macro block of the macro block group constituting the slice, Only byte alignment is performed by next_start_code () in FIG. If you try to realize the arrangement of the encoded thumbnail video at an arbitrary position by arranging the macroblocks in the desired order, the macroblocks crossing byte boundaries will be linked together, and the code string will be decoded as a correct image It becomes impossible.

  Further, as described above, byte alignment by padding bits is performed on the code string of the last macroblock of the slice. When another macroblock code sequence is concatenated after the macroblock code sequence to which padding bits are added, the decoder does not have padding bits in the code sequences of adjacent macroblocks unless they are at the end of a slice. As a result, the code string is decoded. For this reason, the decoder cannot decode a correct image due to padding bits in the code string.

  FIG. 54 shows an example of this phenomenon, in which a multi-screen of 1 × 2 is synthesized using the encoded thumbnail moving images A and B of FIG. 54 (a). When this synthesizing process is realized by arranging the macro blocks constituting each encoded thumbnail video at an arbitrary position in the multi-screen, the arrangement of each macro block in the multi-screen is as shown in FIG. As shown in the bottom of FIG. 54 (c), the code string of the slice header portion of the slice M1 is first arranged, and the code string is subsequently arranged in the order of the macroblocks A1, A2, B1, and B2. Continue.

  At this time, if the last macroblock A2 of the slice A1 is byte-aligned by padding bits, the code sequence of the slice M1 is in a state where a padding bit sequence is inserted between the macroblocks A2 and B1, and the decoder The image cannot be correctly decoded.

  The present invention solves the above-described problems in a system that displays thumbnail images encoded in various standards at arbitrary positions on a multi-screen. For example, in the case of the MPEG standard, different sizes are provided. It is an object to simultaneously display thumbnail images on the same multi-screen, to divide the thumbnail image obtained by encoding the multi-screen, and to reconstruct the multi-screen in an arbitrary order.

Therefore, in the present invention,
In consideration of the case where it is undecided where the reduced / encoded image before editing is placed in the edited image, encoding is performed so that the position information can be easily changed during encoding. In this case, when the position information needs to be changed, the change can be easily performed, thereby producing an effect that various images can be edited.

And in the case of the MPEG standard,
The MPEG code string that has already been converted into a multi-screen composite video is separated into a plurality of original encoded thumbnail images, and the separated encoded thumbnail images can be placed again at arbitrary positions. It is possible to reconstruct a program list of a combination / order desired by a viewer from a multi-screen image code string such as a predetermined multi-screen program list such as broadcasting.

  Further, it is necessary that the variable length data (image) encoding means for generating the encoded image to be input to the variable length data (image) editing means performs special processing for realizing the above-mentioned speeding up. Therefore, a general variable length data (image) encoding means for performing an encoding process based on the image encoding format can be used.

  In addition, it is possible to increase the speed of editing processing between general encoded images based on the image encoding format.

As described above, in the present invention,
When editing variable-length data (moving images), the captured variable-length data (moving images) is reduced, encoded, (stored), edited, and decoded in the order of editing. The method of (arrangement for display) requires a decoding process for the edited image, but since the decoding process is performed once after the editing, the time required for the editing process is minimized. Of course, it can be suppressed,
In consideration of the case where it is undecided where the reduced / encoded image before editing is placed in the edited image, encoding is performed so that the position information can be easily changed during encoding. When it is necessary to change the position information at that time, the change can be easily performed. Therefore, various images can be edited (reduced images before editing can be placed and displayed at any position on the multi-screen after editing). ) Can be performed.

And in the case of the MPEG standard,
The MPEG code sequence that has already become a multi-screen composite video is separated into a plurality of original encoded thumbnail videos, and the separated encoded thumbnail images can be placed again at arbitrary positions. There is an effect that it is possible to reconstruct a program list of a combination / order desired by a viewer from a multi-screen image code string such as a predetermined multi-screen program list such as broadcasting.

  Further, it is necessary that the variable length data (image) encoding means for generating the encoded image to be input to the variable length data (image) editing means performs special processing for realizing the above-mentioned speeding up. Therefore, a general variable length data (image) encoding means for performing an encoding process based on the image encoding format can be used. In addition, it is possible to increase the speed of editing processing between general encoded images based on the image encoding format.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, this invention is not limited to these embodiments at all, and can be implemented in various modes without departing from the scope of the invention.

(First embodiment)
First, a first embodiment in the present invention will be described.
FIG. 2 is a block configuration diagram showing the moving image synthesis apparatus according to the present embodiment.
Here, 201 is a moving image composition server for accumulating and managing a plurality of encoded thumbnail videos and performing multi-screen composition processing, and 202 is a moving image capturing means for capturing a moving image to be combined into a multi-screen composition into a moving image composition server. , 203 is a moving image reduction unit that reduces the captured moving image to a predetermined size and generates a thumbnail moving image, and 204 is a moving image encoding unit that MPEG-encodes the thumbnail moving image to generate an encoded thumbnail moving image, 205. Is a working memory for temporarily storing a slice code string when rewriting position information, 206 is an accumulating unit for accumulating and storing encoded thumbnail videos, and 207 is a multi-screen video that synthesizes a plurality of encoded thumbnail videos. A moving image synthesizing means for generating an image code string, 208 a reproduction terminal for reproducing and displaying a multi-screen moving image, and 209 a multi-screen A moving image list generating unit that generates a combination / order list of moving images to be generated; 210, a moving image decoding unit that decodes the synthesized multi-screen moving image; and 211, a moving image that displays the decoded moving image on a screen. Image display means.

The moving image composition server 201 and the playback terminal 208 are computers such as a personal computer and a workstation on which a general operating system operates, and include a moving image capturing unit 202, a moving image reducing unit 203, and a moving image encoding unit. 204, a moving image synthesizing unit 207, a moving image list generating unit 209, a moving image decoding unit 210, and a moving image display unit 211 are each a program that operates under a general-purpose or dedicated operating system in the computer, or Dedicated hardware. (Note that FIG. 1 is not specific to the open system of the server 201 and the terminal 208, but shows a wide general configuration.)
Hereinafter, a method of moving image encoding processing and multi-moving image combining processing in this moving image combining apparatus will be described.

First, the encoding process of moving image data will be described.
The moving image data to be subjected to multi-screen composition is captured in advance by the moving image capturing unit 202 into the moving image combining server 201, and is converted into a thumbnail moving image reduced to a predetermined size by the moving image reducing unit 203. . Further, the thumbnail moving image is subjected to MPEG encoding processing in a format that can be combined with the multi-screen by the moving image encoding unit 204, and is stored in the storage unit 206 as an encoded thumbnail moving image for multi-screen combining.

FIG. 3 shows a processing flow until the moving image encoding means 204 converts a thumbnail moving image into an MPEG-encoded encoded moving image that can be combined with multiple screens.
In step 301, image data for one frame of thumbnail video is read out,
In step 302, the image data for one frame is divided into N in the horizontal direction,
In step 303, the slice header code string of slice n is output as a part of the encoded thumbnail video code string,
In step 304, slice n is divided into M macroblocks,
In step 305, MB_Addr_Inc of the macro block m is output as a part of the encoded thumbnail video code sequence,
In step 306, the macroblock m is MPEG-encoded (DCT, quantization, variable-length encoding), and the code string is output as a part of the encoded thumbnail video code string.
Step 307 checks if all macroblocks in slice n have been encoded,
In step 308, the next_start_code () process in FIG.
In step 309, it is checked whether encoding has been completed for all N slices;
In step 310, it is checked whether all the frames of the thumbnail video have been processed. Here, the variable m is used for identifying the macroblock being processed, and the variable n is used for identifying the slice.

  FIG. 4 shows an example in which each frame image of the thumbnail moving image is divided into small blocks when the moving image encoding unit 204 encodes the thumbnail moving image as an encoded thumbnail moving image (steps 302 and 304).

  FIG. 5 shows the code output by the moving image encoding unit 204 so that multi-screen synthesis can be performed in units of slices when each slice of the thumbnail moving image is subjected to MPEG encoding processing in steps 303, 305, 306, and 308. An example of a column is shown. 5 is based on the slice and macroblock division example shown in FIG.

  As shown in FIGS. 5A and 5E, in step 303, when a slice header code string of slice n is output, slice_start_code is output with the last 1 byte as a value “n”.

  Furthermore, as shown in FIG. 5B, when the multi-screen composition is performed in units of slices, in order to compensate for the missing bits in preparation for the case where the number of bits of MB_Addr_Inc changes due to the change of the horizontal coordinate of the slice. The supplement code string is output in advance as extra_bit_slice and extra_information_slice. This supplementary code sequence uses the part specified in the MPEG standard of FIG. 52, and the decoder can normally decode the encoded thumbnail video having such a code sequence as an MPEG video sequence. It is.

  In the example of FIG. 5, 8 sets of extra_bit_slice and extra_information_slice are output as supplementary code strings, and a total of 72 bits are output with all bits set to “1” (extra_bit_slice and extra_information_slice are 8 sets) The reason will be described later.)

  Also, as shown in FIG. 5C, in step 305, a value 1 is designated for MB_Addr_Inc of the first macroblock of the slice, and a 1-bit variable-length code string “1” representing the value 1 is output. . The code string in part (d) of FIG. 5 is a code string obtained by performing general MPEG coding processing (DCT, quantization, motion prediction, variable length coding, and byte alignment processing) in steps 206 and 208. It is.

  As a supplement, when multi-screen composition is performed in units of slices, a part necessary as an encoded thumbnail moving image is a code string part below the slice layer as shown in FIG. However, by outputting a code sequence (sequence header, GOP header, picture header, etc.) necessary for an MPEG video sequence in addition to the code sequence of the slice part, an encoded thumbnail video composed of these code sequences can be converted into a single unit. It can be decoded and reproduced and displayed as an MPEG video sequence.

  In the above description of FIGS. 5A, 5C, and 5E, the values of the slice_start_code and MB_Addr_Inc portions are set to values that allow the encoded thumbnail video to be decoded and reproduced and displayed as a single MPEG video sequence. Actually, any value may be used during encoding.

Furthermore, as shown in the description of FIG. 5B, in order to prepare for the case where the number of bits of MB_Addr_Inc changes due to the change in the horizontal coordinate of the slice, an extra_bit_slice and a code string for compensating for the missing bits are provided. Although it is output in advance as extra_information_slice, a code string that can be compensated for a change in the number of bits in the MB_Addr_Inc part, such as macroblock_staffing, may be output in advance.

  Through the above-described process, the moving image encoding unit 204 outputs an encoded thumbnail moving image code string that can be combined into multiple screens in units of slices.

  Next, processing for synthesizing a plurality of encoded thumbnail moving images as a multi-screen moving image will be described.

  FIG. 6 shows an example of the encoded thumbnail video stored in the storage means 206. As shown in FIG. 6, the storage means 206 stores five encoded thumbnail moving images A, B, C, D, and E. At this time, each encoded thumbnail video is horizontally divided into two slices, and each slice is composed of three macro blocks.

  The moving image list generation unit 209 generates a combination / order list for the encoded thumbnail moving images stored in the storage unit 206. For example, as an example of the moving image list generation unit 209, list generation based on the hit rate ranking of search keys in the moving image search system can be mentioned. Here, the description will be made assuming that the moving image list generating unit 209 issues a request to the moving image combining unit 207 so as to synthesize a multi-screen of 2 × 2 as shown in FIG.

FIG. 8 shows a processing flow from the moving image synthesizing unit 207 until a plurality of encoded thumbnail videos are synthesized on a multi-screen and the MPEG code string is output.
In step 801, initialization processing for multi-screen composition is performed,
In step 802, an MPEG video sequence header code string is output so that the multi-screen composite video can be decoded as an MPEG video sequence.
In step 803, a GOP header and a picture header code string are output in order to enable decoding as an MPEG video sequence,
In step 804, a slice code string is read out from each encoded thumbnail video, and in step 805, position information for displaying the read slice at a target location in the multi-screen is rewritten,
In step 806, the counter value for scanning the encoded thumbnail video horizontally is checked,
In step 807, the slice count value used in steps 904 and 905 is checked,
In step 808, the counter value for scanning the encoded thumbnail video vertically is checked,
In step 809, it is checked whether to finish multi-screen composition.
Do each.

When the multi-screen composition as shown in FIG. 7 is requested, the moving image composition means 207 first performs the following initialization process. That is, since the screen is a multi-screen with two vertical and two horizontal lines, the variable X indicating the number of screens in the horizontal direction is set to 2, and the variable Y indicating the number of screens in the vertical direction is set to 2. Also, a variable N indicating the number of slices of each encoded thumbnail video displayed in each screen area of the multi-screen is initialized to 2, and a variable M indicating the number of macroblocks constituting each slice is initialized to 3. Further, a two-dimensional array variable V indicating which encoded thumbnail video is arranged at which position on the multi-screen (the first subscript indicates the horizontal screen area of each screen area, and the second is the vertical direction. Screen area)
V [1,1] = B, V [2,1] = D, V [1,2] = E, V [2,2] = A
Initialize to.

  FIG. 9 shows the loop processing in the loops 810, 811 and 812 in the processing flow of FIG. 8 in C language. The variable x is the horizontal position of the screen being processed, and similarly the variable y is vertical. The variable n uses the slice number of the encoded thumbnail moving image being processed as the loop counter value. Steps 804 and 805 are performed in the innermost loop of these three loops.

FIG. 10 shows a slice code string read process performed by the moving image synthesizing unit 207 in step 804. In step 804, the code string of the slice portion specified by the variable n of the frame specified by the variable f is read from the encoded thumbnail moving image specified by the variable V [x, y] (FIG. 12 (a)). .
For example, when x = 2, y = 2, and n = 1, in step 804, the code string of slice 1A is read. The read slice code string is stored in the synthesis work memory in which the head address of the buffer is pointed by the variable Buf (FIG. 12B).

  According to the MPEG standard, the code sequence of the slice layer is defined such that the code sequence starts from a byte boundary, and the end of the code sequence is also terminated in a byte-aligned state. Outputs a code string. Therefore, all the slice code strings from the beginning to the end can be read in byte units and stored in the synthesis work memory.

The slice display position information rewriting process performed by the moving image synthesizing unit 207 in step 805 is performed as follows.
The moving image synthesizing unit 207 calculates at which coordinate in the multi-screen the slice code string in the synthesis work memory is displayed by the following equation.
Horizontal position: (x-1) * M + 1
Vertical position: (y-1) * N + n
In the example of the slice 1A, since x = 2, y = 2, and n = 1, the horizontal position = 4 and the vertical position = 3.

FIG. 11 shows an example of processing for rewriting the vertical position information of the slice 1A in the composition work memory. FIG. 11A shows a state in which the code sequence of slice 1A is stored in the synthesis work memory in step 804, and FIG. 11B further shows slice_start_code including a code sequence of slice vertical position information. The part is displayed in detail in bit units. Since the vertical position information of the slice is indicated by the fourth byte of slice_start_code, that is, buf [3], in step 805, in order to set the vertical position to 3,
buf [3] = 3
Perform the substitution process. FIG. 11C shows a code string of the slice_start_code part after the vertical position information by the substitution process is rewritten.

  Similarly, FIG. 12 shows a processing example of rewriting the horizontal position information of the slice 1A in the composition work memory. FIG. 12A shows a state in which the code string of slice 1A has been read into the synthesis work memory in step 804, and FIG. 12B further shows details in bit units. The new horizontal position (MB_Addr_Inc) of slice 1A is 4, but when horizontal position 4 is represented by a variable-length code, it becomes “0011” having a 4-bit length. The moving image encoding means 204 for generating the encoded thumbnail moving image outputs MB_Addr_Inc to “1”, that is, 1 bit length as shown in FIG. Therefore, the moving image synthesizing unit 207 compensates for the missing 3 bits from the compensation code string portion. First, as shown in FIG. 12C, a thinning process is performed for the same number of bytes as the value of the missing bits (3 bytes in the example of slice 1A are insufficient).

Next, after outputting a 10-byte code string from buf [0] to buf [9] that was not the target of the thinning process, as shown in FIGS. 12 (c) to 12 (d), Processing to rewrite new horizontal position information,
buf [13] = buf [13] & 0x01 + 0xc6
To do. buf [13] & 0x01 performs a filter operation so that the lower 1 bit of buf [13] is not rewritten. After rewriting the horizontal position, all subsequent slice code sequences are output from buf [13] so that the code sequence of the thinned byte part is not output. After outputting all slice code strings, the synthesis work memory is set to an empty state.

  Through the above-described process, the moving image synthesizing unit 207 completes the rewriting process of the variable-length code bit portion indicating the horizontal position information of the slice by performing a few simple calculations, and performs the slice code string unit. A multi-screen moving image code string obtained by multi-screen synthesis is output. The code sequence of the multi-screen moving image output by the moving image synthesizing unit 207 is decoded as a moving image by the moving image decoding unit 210 and the video is displayed by the moving image display unit 211.

  The reason why the above-described extra_bit_slice and extra_information_slice are 8 sets will be described below.

  extra_bit_slice has a length of 1 bit and extra_information_slice has a length of 8 bits. The code length of the combination of the two is 9 bits. By thinning out 1 byte (= 8 bits long) from the 9-bit supplement code string portion, 1 bit is left. This 1 bit can be allocated to make up for the missing bits when the number of bits in the MB_Addr_Inc part changes.

  Since 1 bit can be compensated if 1 byte is thinned out for each set, 8 bits can be compensated by thinning 7 bytes from 8 pairs. If nine or more supplementary bits are required, the number of bytes to be thinned out can be reduced each time. For example, if 14 bits are to be compensated, 5 bits can be compensated by thinning out 5 bytes first. Next, a total of 14 bits that can be compensated can be obtained by further assigning a set of 9 bits as the fillable bits.

  Therefore, by setting extra_bit_slice and extra_information_slice to 8 sets, it is possible to compensate for an arbitrary number of insufficient bits, and by taking the minimum value of 8 sets, the bit rate of the moving image can be increased by the supplement code sequence. It can be kept to a minimum.

As described above, in the present embodiment,
A moving image capturing unit that captures a moving image to be combined into a multi-screen composition, a moving image reducing unit that reduces the captured moving image to a predetermined size to generate a thumbnail moving image, and a thumbnail moving image that is MPEG-encoded and encoded. Moving picture encoding means to be generated, storing means for storing and storing MPEG-encoded encoded thumbnail moving pictures, and moving picture for generating a multi-screen moving picture code string by combining multiple encoded thumbnail moving pictures stored in the storing means Image synthesizing means, synthesizing work memory used by the moving image synthesizing means for temporarily storing the slice code sequence, moving picture list generating means for generating a list of combinations / orders of the moving pictures constituting the multi-screen, and synthesis A moving image decoding means for decoding the multi-screen moving image code string obtained by the above and a moving image display means for displaying the decoded moving image on the screen. For example,
When the moving image encoding means changes the number of bits of the MB_Addr_Inc portion due to the change of the horizontal coordinate of the slice, it outputs in advance a supplementary code string for compensating for the missing bits, and the moving image synthesis means A multimedia information editing apparatus for generating a multi-screen moving image code sequence is realized by replenishing the missing bits from the compensation code sequence portion and rewriting the value of MB_Addr_Inc.

  This eliminates the bit shift operation for all subsequent code strings due to the change in the number of bits during multi-screen video composition. Specifically, it simply calculates the rewriting process of the variable length code bit part indicating the horizontal position information of the slice. Can be completed only by performing several times. As a result, a multi-screen moving image can be generated at high speed, and its practical effect is great.

More specifically,
The moving image encoding means has a function of outputting eight sets of extra_bit_slice and extra_information_slice as a supplement code sequence as a supplement code sequence,
The moving image synthesizing means can perform any number of missing bit compensation and macroblock_escape code sequence generation from the compensation code sequence, and can rewrite the variable length code bit portion indicating the display position of the slice to an arbitrary value. Its practical effect is great.

In the present embodiment, the case where the horizontal position is 4 has been described with respect to the position arrangement of slices, but the same rewriting process can be performed for other arrangement positions. FIGS. 13, 14, and 15 show MB_Addr_Inc rewrite processing when the horizontal positions to be arranged are 7, 13, and 37, respectively. When the arrangement position is 34 or more, an 11-bit code string called macroblock_escape must be output. In this case as well, by changing numerical values such as the number of thinned bytes and filter operation as shown in FIG. It is possible to obtain the same effect.

In the present embodiment,
The part indicating the horizontal position of the slice code string is the variable length code bit MB_Addr_Inc, and the bit shortage for rewriting the value of MB_Addr_Inc is obtained by simple byte decimation for the supplement code string that has been output to the slice header part in advance. The series of processing to be supplemented is described by taking the MPEG video sequence standard as an example, but this can also be applied to standards other than MPEG.

  That is, when a certain code string is decoded into an image image, the position information is expressed in a standard in which position information indicating where the code string is displayed in the image image is given by variable length code bits. When an area that can output an extended code string that does not affect the image image at the time of decoding is defined as the standard in the front, rear, or both parts of the indicated code string part, that area is used as a supplementary code string By doing so, the same effect can be obtained for standards other than MPEG.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. Note that the same numbers are assigned to those already described in the above-described embodiment, and the description thereof is omitted.

  FIG. 16 is a block configuration diagram showing the moving image composition apparatus in the present embodiment. Here, reference numeral 1601 denotes a moving picture encoding means for encoding a thumbnail moving picture by MPEG to generate an encoded thumbnail moving picture, and 1602 an encoding work memory for temporarily holding a code string generated by the MPEG encoding, Reference numeral 1603 denotes moving image combining means for combining a plurality of encoded thumbnail moving images to generate a multi-screen moving image code string.

  Hereinafter, a method of moving image encoding processing and multi-moving image combining processing in this moving image combining apparatus will be described.

First, the encoding process of moving image data will be described.
FIG. 17 shows a processing flow until the moving image encoding means 1601 converts a thumbnail moving image into an MPEG-encoded encoded thumbnail moving image that can be combined with multiple screens.
In step 1701, the slice header code string of slice n is output as part of the encoded thumbnail video code string,
In step 1702, as an initialization process of the encoding work memory, the contents of the encoding work memory are emptied,
In step 1703, the macroblock m is MPEG-encoded (DCT, quantization, variable-length encoding), and a code string generated by the encoding is output into the encoding work memory.
In step 1704, byte alignment processing is performed on the code string in the encoding work memory,
In step 1705, the code string in the coding work memory is output as a part of the coded thumbnail video code string,
Do each.

  FIG. 18 shows an example of a slice header code string in step 1701. FIGS. 18A and 18B are examples of code strings output by general MPEG encoding means. As described above, the head of the macro block 1 is across the byte boundary. In view of this, the moving image encoding means 1601 in the present embodiment outputs a supplementary code string consisting of two sets of extra_bit_slice and extra_information_slice, a total of 18 bits, as shown in FIG. By outputting the supplement code string in this way, the head of the macro block code string is properly started from a byte boundary.

  Since the supplementary code string is output using the part specified in the MPEG standard, the decoder can normally decode the encoded thumbnail video having such a code string as an MPEG video sequence. It is. In FIG. 18C, “1111 1111” is output as extra_information_slice, but other values may be used.

  As described above, by performing the processing in step 1701, it can be ensured that the macroblock code string constituting the slice starts from the byte boundary position.

Now, the operation of the encoding work memory 1602 will be described.
FIG. 19 shows the operation of the encoding work memory 1602. As a result of the processing in step 1702, the contents of the encoding work memory 1602 become empty (FIG. 19A). Incidentally, the encoding work memory is also a part of the computer, and its read / write access is performed in units of bytes. Therefore, the variable b is used to represent the effective bit length in the byte. As shown in FIG. 19 (b), since the effective code string is 35 bits, the extra 3 bits obtained by dividing 35 by 8 are expressed using some bits of the fifth byte of the work buffer 1902. Will do. Therefore, setting the variable b to 3 represents the effective bit length of the fifth byte. FIG. 19C shows a change in the contents in the encoding work memory when a 7-bit code string '1111 111' is further added to the encoding work memory from the state of FIG. 19B. It is.

  As described above, the encoding work memory uses the variable b to hold the code string in bit units.

  FIG. 20 shows the contents of the encoding work memory 1602 when the MPEG encoding process (DCT, quantization, motion prediction, variable length encoding) performed by the moving image encoding unit 1601 is performed in steps 1703 and 1704. It is shown. As shown in FIG. 20A, the content of the encoding work memory in the initial state is empty due to the processing in step 1702.

  Next, the code string obtained by the MPEG encoding process for the macroblock m in step 1703 is added to the encoding work memory, so that the content of the encoding work memory becomes as shown in FIG. In FIG. 20B, since the value of the variable b is 5, it indicates that the end of the macroblock straddles the byte boundary by 5 bits. Therefore, by inserting macroblock_stuffing at the beginning of the macroblock code string, the end of the macroblock is brought into contact with the byte boundary. As shown in FIG. 20B, since macroblock_stuffing is an 11-bit long code string, every time one macroblock_stuffing is inserted at the beginning of the encoding work memory, 11 code strings already in the encoding work memory are displayed. It is possible to shift backward by a bit. If the byte boundary extends over only 5 bits, if one macroblock_stuffing is inserted, the variable b becomes 0, and the end of the macroblock can be byte-aligned (FIG. 20 (c)).

The code string in which macroblock_stuffing is inserted at the beginning of the macroblock is correct as an MPEG video sequence as shown in FIG.

  FIG. 21 shows the relationship between the value of the variable b before inserting macroblock_stuffing and the number of macroblock_stuffing to be inserted at that time.

  As described above, the macroblock code string is brought into a state in which the head and the tail are in contact with the byte boundary in the encoding work memory. In step 1705, the moving image encoding unit 1601 outputs the macroblock code sequence in the encoding work memory as a part of the encoded thumbnail moving image code sequence.

  As a supplement, when multi-screen synthesis is performed in units of macroblocks, a necessary part of the encoded thumbnail video is a macroblock code sequence in which the beginning and end of the code sequence are in contact with a byte boundary. . In step 1701, the slice header code string is output. However, the slice header code string is not necessarily output in the code string of the encoded thumbnail moving image.

  Also, by outputting a code string (sequence header, GOP header, picture header, etc.) necessary as an MPEG video sequence, it is possible to decode the encoded thumbnail video as a single MPEG video sequence and to display and reproduce it. .

  Next, processing for synthesizing a plurality of encoded thumbnail moving images as a multi-screen moving image will be described. As in the first embodiment, it is assumed that the moving image list generating unit 209 issues a request to the moving image combining unit 1603 so as to synthesize a multi-screen of 2 × 2 as shown in FIG. .

FIG. 22 shows a processing flow until the moving image synthesizing unit 1603 synthesizes a plurality of encoded thumbnail moving images into a multi-screen and outputs the MPEG code string.
In step 2201, in order to be able to decode the multi-screen synthesized moving image as an MPEG video sequence, the header code strings of the GOP header, picture header, and slice header are output,
In step 2202, a macroblock code string is read from each encoded thumbnail video,
In Step 2203, the read macroblock code sequence is output as a multi-screen video code sequence,
In step 2204, the value of the macroblock count m is checked,
Do each.

  FIG. 23 shows the loop processing in the loops 810, 811, 812, 2205 in the processing flow of FIG. 22 in C language, and the variable x indicates the horizontal position of the screen being processed and the variable y Is the vertical position, variable n is the slice number of the encoded thumbnail video that is being processed, and variable m is the macroblock number of the encoded thumbnail video that is being processed as the loop counter value.

  FIG. 24 shows the composite image processing order for one frame of the multi-screen in this embodiment. The macro block code sequence of each encoded thumbnail video in step 2202 and the output of the multi-screen video code sequence in step 2203 by the loop processing of FIG. 23 are raster scan orders (in numerical order enclosed in parentheses in FIG. 24). ). That is, a multi-screen moving image code string is generated by sequentially connecting the read macroblock code strings in a raster scan order.

  As described above, the moving image encoding unit 1601 generates the macro block code sequence such that the beginning and end of the macro block code sequence do not cross the byte boundary, so that the moving image combining unit 1603 It is possible to generate a code sequence of a desired multi-screen synthesized moving image simply by connecting block code sequences in a raster scan order, and it is possible to increase the speed of multi-screen synthesized moving image generation.

  In the processing flow shown in FIG. 22 of the present embodiment, one picture in the MPEG video sequence of the multi-screen synthesized moving image is composed of only one slice, but the flow composed of a plurality of slices. Similarly, it is possible to generate a code sequence of a desired multi-screen synthesized moving image simply by concatenating macroblock code sequences in a raster scan order. At this time, each slice header code string is based on the supplement code string as shown in FIG. 18C applied in step 1701 so that each slice header code string and the macroblock code string are properly connected to each other at byte boundaries. It is necessary to perform byte alignment processing.

As described above, in the present embodiment,
A moving image capturing unit that captures a moving image to be combined into a multi-screen composition, a moving image reducing unit that reduces the captured moving image to a predetermined size to generate a thumbnail moving image, and a thumbnail moving image that is MPEG-encoded and encoded. Moving image encoding means to be generated; encoding work memory used by the moving image encoding means to temporarily store a macroblock code string; and storage means for accumulating and storing MPEG-encoded encoded thumbnail videos. A video synthesizing unit that synthesizes a plurality of encoded thumbnail videos stored in the storage unit to generate a multi-screen video code sequence, and a video list generation unit that generates a list of combinations / orders of the videos configuring the multi-screen. , A moving picture decoding means for decoding a multi-screen moving picture code string obtained by synthesis, and a moving picture display for displaying the decoded moving picture on the screen And a stage,
The encoding work memory has means for storing and holding the code string in bit units,
The moving image encoding means is such that the beginning and end of the macroblock code sequence are in contact with the byte boundary by inserting the supplement code sequence,
A moving image synthesizing unit that generates a multi-screen moving image code sequence by arranging the macroblock code sequences in the raster scan order so as to constitute a target multi-screen is realized. is there.

  As a result, multi-screen moving image synthesis by the moving image synthesizing apparatus can be performed simply by arranging the macroblocks in order, and the speed of multi-screen synthesized moving image generation can be increased.

(Third embodiment)
Next, a third embodiment of the present invention will be described. Note that the same numbers are assigned to those already described in the above-described embodiment, and the description thereof is omitted.

  FIG. 25 is a block diagram showing a moving image composition apparatus according to the present embodiment. Here, reference numeral 2501 denotes a layout table that holds multi-screen layout information of each area when the multi-screen is divided into macroblock sizes, and 2502 denotes a layout that creates the layout table upon request from the moving image list generation unit. A table creation unit 2503 synthesizes a plurality of encoded thumbnail videos having different image sizes to generate a multi-screen moving image code sequence, and a moving image synthesis unit 2504 forms a multi-screen having different sizes of the screen areas. This is a moving image list generating means for generating a combination / order list of moving images.

FIG. 26 shows a processing flow until the moving picture synthesizing means 2501 synthesizes the encoded thumbnail moving images having different image sizes into a multi-screen and outputs the MPEG code string.
In step 2601, a layout table is created as an initialization process for multi-screen composition.
In step 2602, it is checked whether or not the synthesis process is performed on the counter values x and y,
In step 2603, it is checked whether to output a skip macroblock code string in the counter values x and y.
In step 2604, a skip slice code string is output,
In step 2605, a slice code string is read from the encoded thumbnail video indicated by the layout table T [x, y],
In step 2606, the loop counter value x is incremented by 1,
Do each.

FIG. 27 shows an example of a multi-screen composed of encoded thumbnail videos having different image sizes in the present embodiment. Here, description will be made assuming that the moving image list generation unit 209 requests the moving image synthesis unit 2501 to synthesize a multi-screen as shown in FIG. In FIG. 27, the image size of each encoded thumbnail moving image is a size in which the following number of macro blocks are arranged in the horizontal and vertical directions (the size of a general macro block is 16 pixels * 16 pixels).
Encoded thumbnail video A: (5, 4)
Encoded thumbnail video B: (2, 3)
Encoded thumbnail video C: (2, 2)
Encoded thumbnail video D: (2, 2)
Encoded thumbnail video E: (3, 2)
Each encoded thumbnail moving image is encoded by the moving image encoding unit 204 and stored in the storage unit 206 as described in the first embodiment.

  When a multi-screen composition as shown in FIG. 27 is requested, the moving image composition means 2501 performs the following initialization process using the layout table creation means 2502 in step 2601. That is, the variable X indicating the number of macroblocks in the horizontal direction of the multi-screen moving image is set to 7, and the variable Y indicating the number of macroblocks in the vertical direction is set to 6. For each array element of the layout table 2501 represented as a two-dimensional array T [x, y] (the first subscript indicates a horizontal area when a multi-screen is divided into areas with a macroblock size, The second indicates a vertical area), and if the area corresponding to the coordinates (x, y) is an area that does not display a moving image, the symbol S is displayed, and the area corresponding to the coordinates (x, y) is When the screen area constituting the screen is other than the left end, the symbol N is used. When the area corresponding to the coordinates (x, y) is the left end of each screen area constituting the multi-screen, the encoded thumbnail video is displayed. And a symbol consisting of a vertical coordinate value when the top of the screen area is set to coordinate 1.

  FIG. 28 shows the contents of the layout table 2501 when the multi-screen composition shown in FIG. 27 is performed. In FIG. 28, the identification ID of the encoded thumbnail moving image corresponds to the A, B, C, D, and E portions of each cell, and the vertical coordinate value corresponds to the numeric portion of each cell.

  FIG. 29 represents the loop processing in the loops 2607 and 2608 in the processing flow of FIG. 26 in C language.

  In step 2602, when the value of T [x, y] is the symbol N, the moving image synthesis unit 2501 performs no processing. In step 2603, when the value of T [x, y] is the symbol S, the moving image synthesizing unit 2501 outputs a skip slice code string.

  FIG. 30 shows an example of a slice code string that is skipped by one macroblock. The moving image synthesizing unit 2501 outputs the 6 bytes shown in FIG. 30 as a code string, thereby skipping a region for one macroblock, thereby creating a region where no moving image is displayed. it can.

  In step 2605, the identification ID and vertical coordinate value of the encoded thumbnail video are obtained from the symbol indicated by the value of T [x, y]. The identification ID of the encoded thumbnail moving image is used to identify the encoded thumbnail moving image in the storage unit 206, and the vertical coordinate value is used as the number of the slice code string to be read from the encoded thumbnail moving image. In step 805, the slice code string read in step 2605 is subjected to the display position information rewriting process described in the first embodiment so as to be displayed at the position of coordinates (x, y).

  By repeating the processing in the loops 2607 and 2608, the moving image synthesizing unit 2501 rewrites slice_start_code and MB_Addr_Inc of the slice code string read from the encoded thumbnail video sequentially indicated by T [x, y], and the resulting code The sequence is output as a multi-screen video code sequence.

As described above, in the present embodiment,
The moving image synthesizing apparatus generates moving image list generating means for generating a combination / order list of multi-screens composed of encoded thumbnail videos having different screen sizes, and each region when the multi-screen is divided into regions by the macro block size. By including a moving image synthesizing unit having a layout table that holds multi-screen layout information and a layout table creating unit that creates the layout table, the moving image synthesizing unit can determine which encoding at each coordinate of the multi-screen. It is determined by referring to the layout table whether the slice code string of the thumbnail moving image should be processed. In the layout table, as the multi-screen layout information, the coordinates of the multi-screen are stored in which part of each screen area of the multi-screen, and what is the encoded thumbnail video arranged in the screen area. Let

  Through such a process, the moving image synthesizing unit 2501 can synthesize a plurality of encoded thumbnail videos having different image sizes, and a multi-screen moving image code in which encoded thumbnail videos having different screen sizes are mixed. It becomes possible to provide a column, and its practical effect is great.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. Note that the same numbers are assigned to those already described in the above-described embodiment, and the description thereof is omitted.

  FIG. 31 is a block configuration diagram showing a moving image synthesis apparatus according to the present embodiment. Here, 3101 is a moving image capturing means for capturing a multi-screen moving image code string into a moving image composition server, and 3102 extracts a moving image code string for each screen area portion from the captured multi-screen moving image code string. Multi-screen moving image dividing means 3103, a screen layout table creating means 3103 for creating a screen layout table that holds the size and position information of each area portion of the multi-screen moving image and multi-screen, and 3104 A moving image re-encoding means 3105 for converting a moving image code string into a code string (encoded thumbnail moving image) that can be re-synthesized in another multi-screen, and a re-encoded encoded thumbnail moving image and The moving image synthesizing means 3106 for synthesizing the encoded thumbnails in the accumulating means and generating the code sequence of the multi-screen moving image Moving image list generation means for generating a combined-order list screen is.

  First, a process for dividing the code sequence of each screen area portion from the input multi-screen video code sequence will be described.

In FIG. 32, the multi-screen moving image dividing unit 3102 extracts the moving image code sequence of each screen area part constituting the multi-screen from the input multi-screen moving image code sequence, The processing flow until output as a column is shown.
In Step 3201, it is checked whether or not layout information indicating the screen configuration of the multi-screen is included in the multi-screen moving image data code string.
In step 3202, if layout information is included, a screen layout table is created,
In step 3203, a code sequence for one frame is read from the multi-screen moving image data code sequence,
In step 3204, each slice code string is cut out by searching for slice_start_code (see FIG. 38A).
In step 3205, the position information of the cut slice is extracted,
In step 3206, the slice code string that has been cut out is output to the moving image re-encoding unit 3104 together with the identification ID of the encoded thumbnail, the slice position information in the multi-screen, and the slice position information in each screen area.
In step 3207, it is checked whether all slices in the code string for one frame have been processed,
In step 3208, it is checked whether all frames of the input multi-screen moving image data have been processed.
Do each.

FIG. 33 shows the format of the screen layout table.
The moving image horizontal size and moving image vertical size are the size of the input multi-screen moving image data in units of macroblocks, the number of screens is the number of screen regions constituting the multi-screen, and the identification ID of the screen m is each screen region m The identification ID of the encoded thumbnail moving image arranged in the part, the position information of the screen m, the coordinate position of the upper left corner of the screen region m, the horizontal size and the vertical size of the screen m store the size of the screen region m, respectively.

  FIG. 34 shows an example of a screen layout table created when multi-screen video data having the screen configuration shown in FIG. 27 is input. Hereinafter, a case where the input multi-screen moving image data has a screen configuration as shown in FIG. 27 will be described.

  The screen layout table is created by the screen layout table creating means, and in step 3201, in the extended code area or user data area of each header portion (sequence header, GOP header, picture header) of the multi-screen moving image data code string. It is checked whether the screen layout table is encoded and output. If it is output, it is created by reading the code string in step 3202.

  In order to output the screen layout table in the multi-screen moving image code sequence, the moving image synthesis apparatus that generates and outputs the multi-screen moving image code sequence performs a process of outputting the screen layout table in the code sequence. For example, the moving image composition unit 2503 in the third embodiment converts the layout table shown in FIG. 28 into the screen layout table in FIG. 34, and encodes and outputs the screen layout table.

In step 3204, each slice code string is cut out by searching for slice_start_code indicating the head of the slice. Position information (slice position information in the multi-screen) of the slice code string that has been cut out is obtained in step 3205.
Horizontal position = MB_Addr_Inc value of the first macroblock Vertical position = obtained by referring to the value of the fourth byte of slice_start_code.

  In step 3206, the multi-screen moving image dividing unit 3102 obtains the identification ID of the encoded thumbnail and the slice position information in each screen area from the position information of the slice code string in the multi-screen and the moving image. Output to the re-encoding means 3103.

  FIG. 35 shows the identification ID of the encoded thumbnail and the slice position information in each screen area obtained from the screen layout table shown in FIG. 34 and each position information example. Note that FIG. 35 is arranged from the top in the order of processing in the processing flow of FIG.

  Next, a process of converting the slice code string divided and extracted by the multi-screen moving image dividing unit 3102 into a code string that can be re-synthesized in another multi-screen will be described.

FIG. 36 shows a moving image re-encoding unit 3103 that converts a slice code sequence from the slice code sequence passed from the multi-screen moving image dividing unit 3102, the position information in the multi-screen, and the position information in the screen area, The example of the conversion process to the code string which can be recombined in a multiscreen is illustrated. Here, the value that the moving image encoding unit 3104 passes from the multi-screen moving image dividing unit 3102 is:
Slice code sequence: Code sequence shown in FIGS. 36 (a) and 36 (b) Position information on multi-screen: (5, 6)
Position information in the screen area: (1,2)
Identification ID: E
It is as follows. That is, FIG. 36 converts the second slice code string of the encoded thumbnail video E arranged at the coordinates (5, 6) in the multi-screen video shown in FIG.

FIG. 36 (a) shows a slice code string delivered from the multi-screen moving image dividing unit 3102, and FIG. 36 (b) is a detailed display in bit units. Since the vertical position information of the slice code string is stored in the fourth byte of slice_start_code, that is, buf [3], the value of buf [3] is set to the vertical position '2' indicated by the position information in the screen area. for,
buf [3] = 2
Perform the substitution process.

  Further, the horizontal position of the slice code string needs to be rewritten from 5 to 1, but is a variable length code bit MB_Addr_Inc portion. Therefore, when the horizontal position 5 is represented by a variable length code, it is “0010”, and when the horizontal position 1 is represented by a variable length code, it is “1”. Therefore, by rewriting the horizontal position from 5 to 1, 3 bits remain in the MB_Addr_Inc portion, and this is absorbed by inserting 3 bytes of padding bytes as shown in FIG. Then, as shown in the parts (c) to (d) of FIG. 36, a rewrite operation for setting the horizontal position to 1 is performed.

  The slice code string that has undergone the slice position information rewriting process as described above is sent to the storage means 206 or the direct moving picture synthesis means 3104 as a code string of the encoded thumbnail video E that can be synthesized in units of slices. It is used for composition as a screen composition animation.

As described above, in the present embodiment,
Moving image capturing means for capturing a multi-screen moving image; multi-screen moving image dividing means for extracting a slice code string of each screen area portion from the captured multi-screen moving image code string; and a size of each screen area of the multi-screen Screen layout table creating means for creating a screen layout table that retains correspondence with position information, and moving image re-encoding that performs a conversion process into a slice code string that can be re-synthesized in another multi-screen Means, and a moving image synthesizing unit that re-synthesizes the re-encoded slice code sequences to generate a multi-screen moving image code sequence having a different screen configuration,
The moving image dividing means extracts the position information from the header part of the extracted slice code string and refers to the screen layout table using this position information, so that the identification ID of the corresponding encoded thumbnail video, Position information is derived, and in response to the result of the derivation, the moving image re-encoding means converts the position information of the slice code sequence from the multi-screen to the screen area, thereby converting the slice information. It is an object of the present invention to provide an image synthesizing apparatus that performs conversion processing of a code string into a code string that can be recombined in another multi-screen.

By going through the above process,
It is possible to generate a multi-screen moving image obtained by reconfiguring the multi-screen moving image into a different screen configuration. In particular, the moving image re-encoding unit 3103 is the moving image encoding unit 204 described in the first embodiment. Since the slice code string in the same format as the encoded thumbnail video output by is output, the output slice code string can be re-synthesized in units of slices by the moving image synthesis means, and its practical effect is large.

(Fifth embodiment)
Next, a fifth embodiment of the present invention will be described. Note that the same numbers are assigned to those already described in the above-described embodiment, and the description thereof is omitted.

FIG. 37 shows a block diagram of the moving image composition apparatus in the present embodiment. Here, 3701 is an image encoding means for performing an encoding process based on an image encoding format for an input image and outputting an encoded image,
3702 is an accumulation means for accumulating and storing the encoded image;
3703 is an image synthesis means for generating a multi-screen image stream with at least one encoded image as an input,
3704 is an image decoding means for decoding a multi-screen image stream and generating and displaying a multi-screen image image;
It is.

The operation of the moving image composition apparatus in the present embodiment will be described below.
In this embodiment, since the MPEG format is used as an example of the image encoding format of the input image, the image encoding means 3701 is the same as the moving image encoding means 204 in the first embodiment, The same description applies to the internal operation. In other words, the description is given where the input image is a thumbnail moving image and the encoded image is an encoded thumbnail moving image. Therefore, the image encoding unit 3701 outputs the encoded image compressed and encoded in the MPEG format to the storage unit 3702 according to the description in the first embodiment.

  The storage unit 3702 is the same as the storage unit 205 in the first embodiment, and the same description applies to the internal operation thereof. The accumulation unit 3702 accumulates and stores the encoded image, and at this time, an image ID that can be uniquely identified is assigned to the encoded image for accumulation management. FIG. 10A shows a state in which five encoded images are accumulated, and image IDs of A, B, C, D, and E are assigned to the respective encoded images for accumulation management. . Based on these image IDs, an arbitrary encoded image can be extracted from the storage unit 3702. Each encoded image has the same configuration as the encoded thumbnail as shown in FIG.

  The image synthesizing unit 3703 takes out at least one encoded image from the storage unit 3702 and performs image synthesis processing to generate and output a multi-screen image stream. Note that the image composition means 3703 in the present embodiment is the same as the moving image composition means 207 in the first embodiment, and the same description applies to the internal operation thereof.

  In the present embodiment, a case where a multi-screen image as shown in FIG. 38 is generated by the image composition means 3703 will be described. That is, the image synthesizing unit 3703 displays each encoded image stored in the storage unit 3702 for each region obtained by dividing the multi-screen image into a total of four regions divided into two vertically and two horizontally. A multi-screen image like this is synthesized.

  In order to display a multi-screen image as shown in FIG. 38, the multi-screen image stream output from the image synthesis means 3703 must be a code string sequence as shown in FIG. Therefore, the image synthesizing unit 3703 performs the code string generation process shown in FIG. 39 through the steps shown in FIG. Since the multi-screen image stream generated by this code string generation process is a correct code string in the MPEG format, the image decoding means 3704 can perform MPEG reproduction.

  As described above, by configuring the moving image synthesizing apparatus according to the present embodiment, an enormous bit shift operation or the like can be performed for the code length change of the information generated by changing the information value defined by the variable length code. The information value can be changed without requiring processing, and the speed of multi-screen image stream generation processing can be increased.

  At this time, since the code sequence of the multi-screen image stream generated by the image synthesizing means is correct as the MPEG format, the image decoding means for decoding the MPEG code string or the image decoding device (generally, Decoding and playback / display are possible with an MPEG decoder).

  Note that the input image to be input to the image encoding means in this embodiment may be either a still image or a moving image. In the case of a still image, the processing in the present embodiment (refer to the first embodiment for the description of the operation of the image encoding means) is that the input image is only one frame in the processing portion at the frame level. This can be explained by applying the case.

  Further, the input image may be previously encoded in some image encoding format. In this case, the image encoding means performs an image encoding format conversion process on the input image.

  In this embodiment, the MPEG format is used as the image encoding format of the input image and the multi-screen image stream. However, the image encoding format standard has information defined as a variable length code. Applicable to all image encoding formats.

  In the present embodiment, the basic arithmetic processing unit length of the image composition means for rewriting information defined as a variable length code is described as 8 bits (that is, 1 byte which is normally called). The length may be any one or more.

  Further, in the present embodiment, horizontal position information is taken as an example of information to be rewritten, but any information can be used as long as it is defined as a variable-length code as a standard for an image encoding format. There may be a plurality of pieces of information to be changed at one time.

  Further, in this embodiment, a pair of extra_bit_slice and extra_information_slice is used as an extended code string to compensate for the missing bits. However, a code that does not affect the image in the image coding format or has little influence. A portion defined as a column may be output in the encoded image in advance.

(Sixth embodiment)
Next, a sixth embodiment of the present invention will be described. Note that the same numbers are assigned to those already described in the above-described embodiment, and the description thereof is omitted.

FIG. 40 shows a block diagram of the moving image composition apparatus in the present embodiment. Here, 4001 and 4002 are delay means for outputting after storing or holding input data or performing some processing temporarily or over a long period of time,
Reference numeral 4003 denotes a composition server for accumulating and managing encoded images;
4004 is a playback terminal for combining and playing back a multi-screen image stream;
It is.

The operation of the moving image composition apparatus in the present embodiment will be described below.
Delay means 4001 and 4002 perform processing such as delay output of the input data of the delay means and adjustment of delay time by frame thinning processing by storage means including a magnetic disk and a memory device.

  Here, the delay unit 4001 (hereinafter referred to as delay unit 1) outputs the code sequence of the encoded image from the image encoding unit 3701 after being delayed for a predetermined time. This delay time can be designated from the outside of the delay means. For example, it is possible to instruct the playback terminal 4004 to delay by 5 seconds. In addition, the delay time can also be changed. By gradually reducing the delay time from the initial state where the delay time was 10 seconds, so that there is no delay time (0 seconds) after 1 minute from the initial state, Performing chasing playback (Catch-Up) is also included in the processing of the delay means.

  On the other hand, the delay unit 4002 (hereinafter referred to as delay unit 2) outputs a code string every four frames for the encoded image extracted from the storage unit 3702 (that is, the frame numbers are 1, 5, 9, 13,...). Frame code string). That is, the encoded image output from the delay unit 2 is an image with a reproduction speed of 4 ×.

  As shown in FIG. 41, the image synthesizing unit in the present embodiment inputs a code string of an encoded image from four paths. In FIG. 41, the image encoding means inputs a real-time live video from the camera as an input image, performs MPEG encoding processing, and outputs an encoded image whose image ID is X.

  Further, the delay means 1 performs the above-described delay processing and changes the image ID, which is changed to X ′ here. Also, an encoded image whose image ID is B is extracted from the storage means. The delay means 2 performs the delay process described above and changes the image ID to B ′.

  Therefore, the image synthesizing unit 3703 performs a process of synthesizing four encoded images whose image IDs are X, X ′, B, and B ′.

  As shown in FIG. 40, as a configuration of the moving image synthesizing apparatus, it is possible to arrange an image synthesizing unit on the playback terminal side. In addition, the internal processing of the image synthesizing unit 3703 at that time is the same as that of the first embodiment, and the image decoding unit 3704 decodes and reproduces the multi-screen image stream synthesized by the image synthesizing unit.

  An example of a reproduction image of a multi-screen image according to this embodiment is shown in FIG. FIG. 42 is an example of a playback image when various videos such as live video, delayed video of live video, stored video, and shift video of stored video are combined as multi-screen images in each screen area.

  As described above, by configuring the moving image synthesizing apparatus of the present embodiment, multi-screen synthesis display such as real-time live video such as video from a camera and stored video can be displayed as an input image, and delay means It is possible to reproduce and display a multi-screen image that has been subjected to delayed playback of live video parts by means of, and adjustment of delay time by frame thinning processing.

(Seventh embodiment)
Next, a seventh embodiment of the present invention will be described. Note that the same numbers are assigned to those already described in the above-described embodiment, and the description thereof is omitted.

FIG. 43 shows a block diagram of the moving image synthesis apparatus according to the present embodiment. Here, 4301 is a layout table that holds the configuration of each area when the playback screen of the multi-screen image to be output is divided into at least one area as multi-screen layout information,
4302 is a layout table changing means for changing the value of the multi-screen layout information in the layout table,
It is.

The operation of the moving image composition apparatus in the present embodiment will be described below.
In the present embodiment, the description of the same parts as those described in the third embodiment will be omitted. That is, a description is given in which a thumbnail moving image is used as the input image and an encoded thumbnail moving image is used as the encoded image. The layout table 4301 is the same as the layout table in the third embodiment, and the same description applies to the configuration thereof. Further, the image synthesizing unit 3703 is the same as the moving image synthesizing unit 2503 in the third embodiment shown in FIG. 25, and the same description applies to the internal operation thereof.

  FIG. 44 shows the screen configuration of the multi-screen image before and after the layout table changing means 4302 changes the contents of the layout table. As shown in FIG. 44, according to the present embodiment, the image composition unit 3703 can generate a multi-screen image stream having a completely different screen configuration.

FIG. 45 shows a coded image composition process in the image composition means 3703. In FIG.
In step 4501, initialization processing is performed in the same manner as in step 2601 in the third embodiment shown in FIG. 26, and a layout table having the screen configuration as shown in FIG. 44 as an initial state is created.
In step 4502, the image composition means refers to the layout table. That is, the image synthesizing unit 3703 refers to the layout table at the time when processing at the frame level in the multi-screen image stream generation processing is started (step 4502), and synthesizes along the multi-screen layout information in the layout table. Processing (steps 803 to 809 in FIG. 45) is performed. Thereby, the screen configuration change of the multi-screen image by the layout table changing unit 4302 is performed at the frame level.

  As described above, by configuring the moving image synthesizing apparatus according to the present embodiment, it is possible to dynamically change the arrangement configuration of each area of the multi-screen image, the size of each area, the encoded image displayed in each area, and the like. It becomes.

(Eighth embodiment)
Next, an eighth embodiment of the present invention will be described. Note that the same numbers are assigned to those already described in the above-described embodiment, and the description thereof is omitted.

A block diagram of the moving image synthesizing apparatus according to the present embodiment is shown in FIG. Here, 4601 is an image encoding means for performing an encoding process based on an image encoding format for an input image and outputting an encoded image,
4602 is an image synthesizing unit that receives at least one encoded image and generates a multi-screen image stream;
It is.

The operation of the moving image composition apparatus in the present embodiment will be described below.
First, input image encoding processing by the image encoding means 4601 of the present embodiment will be described. The input image input to the image encoding unit by the image encoding unit 4601 of this embodiment is subjected to encoding processing as an MPEG format, and is stored in the storage unit 3702 as an encoded image. The MPEG encoding process flow of the image encoding unit 4601 is the same as that in FIG. 3 of the first embodiment. Note that the thumbnail video in the description of FIG. 3 and the input image in the present embodiment can be treated as the same.

  FIG. 47 shows an example of a code string when the image encoding unit 4601 encodes an input image in units of slices in steps 303, 305, 306, and 308 shown in FIG. 3 and outputs the encoded image as an encoded image. Is shown. 47 (a), (b), (c), and (d) apply the explanations of FIGS. 5 (a), (c), (d), and (e), respectively.

  As described above, the code string of the encoded image in the present embodiment is the same as that generated by a general MPEG encoding means or an MPEG encoding device (generally called an MPEG encoder). Therefore, a general MPEG encoding means can be used as the image encoding means 4601 in the present embodiment.

  The encoded image output by the image encoding unit 4601 is stored and managed by the storage unit 3702. FIG. 48 shows an example of the encoded image stored in the storage unit 3702. In the example of FIG. 48, the storage unit 3702 stores encoded images identified by image IDs A, B, C, D, and E, respectively. The encoded image is horizontally divided into two slices, and each slice is composed of three macro blocks.

  Next, the rewriting process of the information value defined by the variable length of the encoded image in the present invention will be described.

  In the present embodiment, a case where a multi-screen image as shown in FIG. 38 is synthesized will be described as an example. The multi-screen image synthesizing process is performed by the image synthesizing means 4602, and the process flow is the same as that of FIG. 8 of the first embodiment. Note that the encoded thumbnail moving image in the description of FIG. 8 and the encoded image in the present embodiment can be treated as the same thing.

  FIG. 49 shows the slice code string read processing performed by the image composition means 4602 in step 804 shown in FIG. The read slice code string is stored in a buffer whose head address is pointed by the variable Buf.

  The image synthesizing unit 4602 performs the slice display position information rewriting process performed in step 805 shown in FIG. 8 as follows.

  For the rewriting of the vertical position information, the description of FIG. 11 in the first embodiment is applied. The rewriting process of the horizontal position information defined by the variable length code in step 805 will be described by taking an example in which the value of the horizontal position is changed from 1 to 4. In the state stored in the storage means 3702, as shown in FIG. 47, the horizontal position information value of the encoded image is 1, and the code string is “1” having a 1-bit length. In order to rewrite this to the horizontal position 4, if the horizontal position 4 is represented by a variable length code, it becomes “0011” having a 4-bit length. Therefore, the code string for 3 bits is insufficient. Therefore, a method of compensating for the shortage code string and rewriting the horizontal position information value defined by the variable length code at high speed will be described with reference to FIG.

  FIG. 50A shows a state where the code string of slice 1A is read into the buffer in step 804, and FIG. 50B shows the vicinity of the slice header at that time in more detail.

The image synthesizing unit 4602 first rewrites the buf [4] portion. In this embodiment,
buf [4] = buf [4] | 0x07
And Thereafter, 5 bytes from buf [0] to buf [4] are output as a code string of the multi-screen image stream.

  Next, as an insertion process of 6 bytes, 0xff is output as 6 bytes, which is also a code string of a multi-screen image stream.

Finally, as rewriting processing of horizontal position information,
buf [10] = 0xc6 + (buf [4] & 0x01)
After that, all slice code sequences after buf [10] are output as code sequences of the multi-screen image stream.

  Strictly speaking, in order to refer to the value of buf [4] when rewriting the horizontal position information, the value of buf [4] is temporarily changed when rewriting buf [4] for the first time. Must be stored in a variable.

  Through the processing as described above, the rewriting of the information value defined by the variable length code can be completed only by performing a simple calculation several times. Therefore, it is possible to change the information value without requiring processing such as enormous bit shift calculation for the code length change of the information caused by changing the information value defined by the variable length code. Therefore, the speed of the multi-screen image generation process can be increased.

  Further, since the image encoding means for generating the encoded image to be input to the present image synthesizing means does not need to perform the special processing for realizing the above-described high speed, it is based on the image encoding format. It is possible to use general image encoding means for performing the encoding process. In addition, it is possible to increase the speed of editing processing between general encoded images based on the image encoding format.

  In the above description of the embodiment, the description has been made mainly on moving images. However, the same applies to all materials that handle variable-length data, such as moving images, still images, sounds, and texts. It can be handled. In that case, the term “synthetic” cannot fully describe the contents, so it is more appropriate to substitute the term “edit” instead of the term “synthesizing”.

Schematic diagram showing the configuration of the moving image composition apparatus of the present invention, 1 is a block diagram of a moving image synthesis apparatus according to the first embodiment of the present invention; FIG. 5 is a processing flowchart showing the operation of the moving image encoding means in the first embodiment of the present invention; An example diagram of block division of a thumbnail video in the first embodiment of the present invention, An exemplary diagram of a code string output by the moving image encoding means in the first embodiment of the present invention, FIG. 4 is an exemplary diagram of a list of encoded thumbnail videos stored in the storage unit according to the first embodiment of the present invention; FIG. 5 is a diagram illustrating an example of a multi-screen according to a request issued by the moving image list generating unit to the moving image combining unit according to the first embodiment of the present invention; FIG. 5 is a processing flowchart showing the operation of the moving image composition means in the first embodiment of the present invention; FIG. 4 is a diagram showing an example of loop processing inside the moving image composition unit in the first embodiment of the present invention; FIG. 3 is a diagram illustrating an example of slice code string read processing in the storage unit by the moving image synthesis unit according to the first embodiment of the present invention; FIG. 4 is an example diagram of rewriting processing of vertical position information of a slice according to the first embodiment of the present invention (vertical position = 3); FIG. 5 is an example diagram of horizontal slice information rewriting processing (horizontal position = 4) in the first embodiment of the present invention; FIG. 7 is an example diagram of horizontal slice information rewriting processing (horizontal position = 7) in the first embodiment of the present invention; FIG. 6 is an example diagram of horizontal slice information rewriting processing (horizontal position = 13) in the first embodiment of the present invention; FIG. 10 is a diagram illustrating an example of rewriting processing of horizontal position information of a slice according to the first embodiment of the present invention (horizontal position = 37); A block diagram of a moving image synthesizing apparatus according to a second embodiment of the present invention, FIG. 9 is a processing flowchart showing the operation of the moving image encoding means in the second embodiment of the present invention; An example diagram of a slice header code string in the second embodiment of the present invention, An example diagram of the operation of the work buffer in the second embodiment of the present invention, FIG. 5 is an example diagram of an operation of a work buffer at the time of MPEG encoding processing of moving image encoding means in the second embodiment of the present invention; FIG. 7 is a relationship diagram between the number of macroblock_stuffing insertions and the value of a variable b in the second embodiment of the present invention; FIG. 6 is a processing flowchart showing the operation of the moving image composition means in the second embodiment of the present invention; FIG. 4 is a diagram showing an example of loop processing inside moving image synthesis means in the second embodiment of the present invention; FIG. 10 is an example of an edit image processing order for one frame of a multi-screen in the second embodiment of the present invention; A block diagram of a moving image synthesizing device according to a third embodiment of the present invention, FIG. 10 is a processing flowchart showing the operation of the moving image encoding means in the third embodiment of the present invention; An example diagram of a multi-screen configuration in the third embodiment of the present invention, FIG. 8 is an example diagram of initialization of a two-dimensional array variable V in the third embodiment of the present invention; An example diagram of loop processing inside the moving image synthesizing means in the third embodiment of the present invention, FIG. 8 is an example diagram of a skip slice code string according to the third embodiment of the present invention; A block diagram of a moving image synthesizing device according to a fourth embodiment of the present invention, FIG. 10 is a processing flowchart showing the operation of the multi-screen moving image dividing means in the fourth embodiment of the present invention; FIG. 10 is a diagram showing an example of the format of a screen layout table in the fourth embodiment of the present invention. FIG. 7 is a diagram showing an example of a multi-screen according to a request issued by the moving image list generating unit to the moving image synthesizing unit in the fourth embodiment of the present invention; FIG. 9 is a diagram illustrating an example of processing progress of the multi-screen moving image dividing unit according to the fourth embodiment of the present invention; FIG. 10 is a diagram illustrating an example of slice code string conversion processing by the moving image re-encoding unit according to the fourth embodiment of the present invention. A block diagram of a moving image composition apparatus in a fifth embodiment of the present invention, An example diagram of a multi-screen in the fifth embodiment of the present invention, A multi-screen image code sequence in the fifth embodiment of the present invention; A block diagram of a moving image synthesizing device according to a sixth embodiment of the present invention, An input path of an encoded image to the image composition means in the sixth embodiment of the present invention; A reproduction image of a multi-screen image in the sixth embodiment of the present invention; A block diagram of a moving image synthesis apparatus in a seventh embodiment of the present invention, The screen configuration of the multi-screen image before and after the content change of the layout table in the seventh embodiment of the present invention, An encoded image editing process in the image synthesizing means in the seventh embodiment of the present invention; A block diagram of a moving image synthesizing device according to an eighth embodiment of the present invention, An exemplary diagram of a code string output by the image encoding means in the eighth embodiment of the present invention, FIG. 10 is a diagram illustrating an example of a list of encoded images stored in the storage unit according to the eighth embodiment of the present invention. An example diagram of slice code string read processing in the storage means by the image composition means in the eighth embodiment of the present invention, An example of rewriting processing of slice horizontal position information in the eighth embodiment of the present invention, FIG. Block configuration diagram in a conventional example, Example diagram of part of the syntax of a video stream in MPEG, Example diagram showing the phenomenon when rewriting slice horizontal position information, It is the example figure which showed the phenomenon at the time of the macroblock connection which is not byte-aligned.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Moving image taking-in means 102 Moving image reduction means 103 Moving image encoding means 104 Composition work memory 105 Accumulating means 106 Moving image synthesizing means 107 Moving image list generating means 108 Moving image decoding means 109 Moving image display means

Claims (3)

Image capturing means for capturing multi-screen image data; multi-screen image dividing means for extracting a slice code sequence that is a code sequence of each screen region portion from the code sequence of the multi-screen image data; and Screen layout table creating means for creating a screen layout table that holds correspondence between size and position information, image re-encoding means for converting the slice code string into a re-editable slice code string, Accumulating means for accumulating and storing encoded thumbnail images obtained by encoding thumbnail images included in the screen image data; re-editing the re-encoded slice code sequences; and a multi-screen image data code sequence having a different screen configuration. Image synthesizing means for generating and image list generating means for generating a combination / order list of image data constituting a multi-screen The provided image decoding means for decoding the multi-image image data code string, and image display means for screen display the multi-screen image data decoded,
The multi-screen image dividing means is
Deriving the identification ID of the encoded thumbnail image necessary for re-encoding the extracted slice code string, the position information in the multi-screen, the position information in each screen area from the screen layout table,
The image re-encoding means comprises:
A multimedia information editing apparatus that generates multi-screen image data in which the multi-screen image data has a different screen configuration by performing a conversion process from the derivation result to the re-editable code sequence. The image list generating means is
A list of image data is selectively generated from both the encoded thumbnail image stored in the storage means and the image data including the re-encoded slice code sequence,
The image synthesizing means;
The code sequence is acquired from both the encoded thumbnail image code sequence stored in the storage unit and the re-encoded slice code sequence based on the image data list generated by the image list generation unit. The multimedia information editing apparatus described. The image re-encoding means comprises:
Position information of a variable length code indicating a display position of the slice code string;
A supplementary code used when the position information is changed by the arrangement of the slice code string, and
The image synthesizing means;
3. The multimedia according to claim 2, wherein when the position information of the slice code sequence needs to be changed and the code length of the position information changes, the position information is changed using the position information and the compensation code. Information editing device.

Images