JP2002281463A - Device and method for encoding image, device and method for decoding image and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent mismatching of a reencoded and recorded video stream and other data when the video stream is reproduced. SOLUTION: A transport stream inputted to a recorder 1 is inputted to a demultiplexer 12 and an analyzing part 13. The analyzing part 13 outputs stream information other than video to an encoding controller 18. The encoding controller 18 controls encoding of an encoder 15 on the basis of the inputted information and outputs video encoding control information to an encoding part 20. The encoding part 20 prepares multimedia display subinformation (e.g. including information about a picture frame) in order not to cause mismatching between video and any data other than the video in a reproducing side on the basis of the video encoding control information and outputs the multimedia display subinformation to a writing part 21. The writing part 21 records a video stream, etc., on a recording medium 22 and also records the multimedia display subinformation prepared by the encoding part 20 on the recording medium 22.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image coding apparatus and method, an image decoding apparatus and method, and a recording medium. The present invention relates to a decoding device and method, an image decoding device and method, and a recording medium.

[0002]

2. Description of the Related Art European DVB (Digital Video Broadcast)
cast), US DTV (Digital Television) broadcast, Japanese BS (Broadcast Satellite) digital broadcast, and other digital television broadcasts.
Picture Expert Group) 2 transport streams are used. The transport stream is a stream in which transport packets are continuous, and the transport packets are, for example, packetized video data and audio data. The data length of one transport packet is 188 bytes.

[0003] Unlike digital television broadcasting, digital television broadcasting can provide a service to which multimedia encoded data is added. In this service, data such as video data, audio data, character / graphic data, and still image data is transmitted in association with multimedia encoded data. For example, in the case of BS digital broadcasting in Japan, XML (Extensibl
e Markup Language) based coding method is used. For details, for example, ARIB STD-B24 Data Coding a
nd transmission Specification for digital broadcas
ting (ARIB STD-B24 data broadcast coding and transmission schemes in digital broadcasting) and the like.

[0004] Data such as video data, audio data, character / graphic data, and still image data are each transmitted as a transport packet.

Referring to FIG. 1, an example of combining data and multimedia screens transmitted and received between a transmitting side and a receiving side will be described. As shown in FIG. 1A, the transmitting side transmits video data, character / graphic data for displaying buttons A to C, character data for displaying characters such as "XYZABC ...", and the like. The multimedia encoded data for association is transmitted to the receiving side. The transmitting side is a television broadcasting station or the like, but here, further, receives data transmitted from the broadcasting station,
A recording device (recording side) for recording is also included, and FIG. 1A includes data output from the recording device.

[0006] Multimedia encoded data includes video data, character / graphic data, and data that can be combined and displayed on the receiving side. That is, as shown in FIG. 1B, the multimedia encoded data has a size (plane_height, plane_height,
plane_width) and video display size (video_height, v
ideo_width), video data, character / graphic data, and data related to the display position of the video, character / graphic, and character displayed by the respective data of the character data.

On the receiving side, video data, character / graphic data, and character data are respectively processed based on the multimedia encoded data, and as shown in FIG.
An image based on each data is displayed.

When the user operates the button A, for example, from the screen displayed in this manner, information corresponding to the operation is displayed in a service such as displaying a video portion or in a lower portion of the screen. It is possible to receive a service such as obtaining information on a matter displayed in a video portion from character data that is present.

[0009]

If a television program of a transport stream broadcast by digital television broadcasting is recorded on a recording medium or the like as it is on the receiving side, it can be recorded without deteriorating image quality or audio quality at all. It is possible to When recording a program for as long a time as possible on a limited-capacity recording medium, assuming that the image quality will deteriorate, processing such as lowering the bit rate by decoding the received video stream once and re-encoding it Is required.

For example, when a video stream of a television program to which multimedia encoded data is added is re-encoded for recording at a reduced bit rate, it is conceivable to change the picture frame by sub-sampling the image. Can be However, at this time, there is a problem that a mismatch may occur between the video stream obtained by re-encoding and the multimedia encoded data. An example of the mismatch will be described with reference to FIG.

In the example shown in FIG. 2A, a conversion process to a picture frame smaller than the picture frame of the original video is performed at the time of re-encoding on the transmission side (recording side). As a result, as shown in FIG. 2B, the display size and the display position of the video are changed on the receiving side (reproducing side), and the display screen intended by the transmitting side (data before re-encoding is used). (Display screen to be displayed).

The present invention has been made in view of such a situation. On the recording side, information that does not cause inconsistency even when a video stream is re-encoded and recorded is generated, and the information is generated. And also record it. It is another object of the present invention to prevent the occurrence of inconsistency such as a reduced image frame by reproducing on the reproduction side with reference to the information.

[0013]

According to a first aspect of the present invention, there is provided an image coding apparatus comprising: input means for inputting a multiplexed stream containing multimedia coded data; and a video stream from the multiplexed stream input by the input means. Separation means for separating the video stream, a conversion means for performing a predetermined conversion on the video stream separated by the separation means, and indicating that a mismatch occurs when the converted video stream is displayed based on the multimedia encoded data. It is characterized by comprising generating means for generating additional information, and output means for outputting the converted video stream, multimedia encoded data, and additional information.

[0014] The information processing apparatus may further include encoding means for encoding the additional information generated by the generation means as data different from a multiplexed stream including the converted video stream.

[0015] The information processing apparatus may further include encoding means for multiplexing and encoding the additional information generated by the generating means into a multiplexed stream including the converted video stream.

[0016] The conversion means can convert the parameters of the picture frame of the video stream.

[0017] The conversion by the conversion means may include at least processing for decoding the video stream separated by the separation means and encoding the decoded video stream.

[0018] The additional information generated by the generating means may include at least one of information of an original image frame and an original screen aspect ratio.

The additional information generated by the generating means may include information on an original video format and information on a video format after conversion.

The additional information generated by the generating means may include information on the original screen aspect ratio and the converted screen aspect ratio.

The additional information generated by the generating means includes information indicating whether or not the image frame of the video stream has been converted by the converting means, information of the original image frame of the video stream separated by the separating means, and the original information. Of at least one screen aspect ratio.

According to a first image encoding method of the present invention, an input step of inputting a multiplexed stream including multimedia encoded data, and a separation for separating a video stream from the multiplexed stream input by the processing of the input step And a conversion step of performing a predetermined conversion on the video stream separated by the processing of the separation step, and additional information indicating that inconsistency occurs when the converted video stream is displayed based on the multimedia encoded data. , And an output step of outputting the converted video stream, the multimedia encoded data, and the additional information.

[0023] The method may further include an encoding step of encoding the additional information generated by the generation means as data different from a multiplexed stream including the converted video stream.

[0024] An encoding step of multiplexing and encoding the additional information generated by the processing of the generation step into a multiplexed stream including the converted video stream may be further included.

In the conversion step, the parameters of the picture frame of the video stream can be converted.

The conversion by the conversion step may include at least a process of decoding the video stream separated by the separation step and encoding the decoded video stream.

[0027] The additional information generated by the processing of the generation step may include at least one of information of an original image frame and an original screen aspect ratio.

[0028] The additional information generated by the processing of the generation step may include information on the original video format and the converted video format.

[0029] The additional information generated by the processing of the generation step may include information on the original screen aspect ratio and the converted screen aspect ratio.

The additional information generated by the processing of the generation step includes information indicating whether or not the image frame of the video stream has been converted by the processing of the conversion step, and the original image of the video stream separated by the processing of the separation step. At least one of the frame information and the original screen aspect ratio may be included.

The program of the first recording medium of the present invention comprises:
An input step of inputting a multiplexed stream including multimedia encoded data, a separating step of separating a video stream from the multiplexed stream input by the processing of the input step, and a video stream separated by the processing of the separating step. A conversion step of performing conversion of the following, a generation step of generating additional information indicating that inconsistency occurs when the converted video stream is displayed based on the multimedia encoded data, and a converted video stream, multimedia Outputting the encoded data and the additional information.

The first recording medium of the present invention is not suitable for displaying a video stream to which a predetermined conversion has been performed, multimedia encoded data, and displaying the converted video stream based on the multimedia encoded data. It is characterized in that additional information indicating that matching occurs is recorded.

The additional information may be encoded and recorded as data different from a multiplexed stream including the converted video stream.

The additional information is multiplexed and encoded on a multiplexed stream including the converted video stream,
It can be recorded.

In the video stream, the parameters of the picture frame may be converted.

The video stream may be once decoded and then encoded.

[0037] The additional information may include at least one of information of an original image frame and an original screen aspect ratio.

The additional information may include information on the original video format and the converted video format.

The additional information may include information on the original screen aspect ratio and the converted screen aspect ratio.

The additional information includes at least one of information indicating whether or not the picture frame of the video stream has been converted, information of the original picture frame of the video stream, and the original screen aspect ratio. can do.

The first image decoding apparatus according to the present invention comprises: input means for inputting a multiplexed stream containing multimedia encoded data; and separating means for separating a video stream from the multiplexed stream input by the input means. Decoding means for decoding the separated video stream, and decoding based on additional information indicating that inconsistency occurs when displaying the decoded video stream based on the multimedia encoded data. Processing means for performing a predetermined conversion on the video stream thus obtained.

[0042] An acquisition means for acquiring the additional information from data different from the multiplexed stream may be further provided.

[0043] The information processing apparatus may further include acquisition means for acquiring the additional information from a multiplexed stream in which the additional information is multiplexed.

The processing means can convert the parameters of the picture frame of the video stream.

The conversion by the processing means may include at least processing for decoding the video stream separated by the separation means and encoding the decoded video stream.

[0046] The additional information may include at least one of information of an original image frame and an original screen aspect ratio.

The additional information may include information on the original video format and the converted video format.

The additional information may include information on the original screen aspect ratio and the converted screen aspect ratio.

The additional information includes information indicating whether or not the picture frame of the video stream has been converted by the conversion means, information on the original picture frame of the video stream separated by the separation means, and the original screen aspect ratio. , At least one piece of information.

A first image decoding method according to the present invention comprises an inputting step of inputting a multiplexed stream including multimedia encoded data, and a separation for separating a video stream from the multiplexed stream input by the processing of the inputting step. A decoding step for decoding the separated video stream; anda decoding step for decoding the separated video stream based on the multimedia encoded data. Performing a predetermined conversion on the decoded video stream.

[0051] The method may further include an obtaining step of obtaining the additional information from data different from the multiplexed stream.

[0052] The method may further include an obtaining step of obtaining the additional information from a multiplexed stream in which the additional information is multiplexed.

The processing in the processing step can convert the parameters of the picture frame of the video stream.

The conversion by the processing of the processing step may include at least processing of decoding the video stream separated by the processing of the separation step and encoding the decoded video stream.

[0055] The additional information may include at least one of information of an original image frame and an original screen aspect ratio.

The additional information may include information on the original video format and the converted video format.

The additional information may include information on the original screen aspect ratio and the converted screen aspect ratio.

The additional information includes information indicating whether or not the picture frame of the video stream has been converted by the processing of the conversion step, information of the original picture frame of the video stream separated by the processing of the separation step, and the original screen. At least one of the aspect ratios may be included.

A second recording medium according to the present invention includes an input step of inputting a multiplexed stream including multimedia encoded data, and a separating step of separating a video stream from the multiplexed stream input by the processing of the input step. Decoding the separated video stream, and decoding the decoded video stream according to additional information indicating that a mismatch occurs when the decoded video stream is displayed based on the multimedia encoded data. And performing a predetermined conversion on the obtained video stream.

[0060] The second image encoding apparatus of the present invention comprises: input means for inputting a multiplexed stream; separating means for separating a video stream from the multiplexed stream input by the input means; Determining means for determining whether or not the multiplexed stream contains multimedia encoded data; and separating means for separating when the determining means determines that the multiplexed stream contains multimedia encoded data. Means for generating encoding control information for instructing not to change the display format of the generated video stream, and a video stream separated by the separating means based on the encoding control information generated by the generating means. Conversion means for performing conversion, and a multiplexed stream including a video stream converted by the conversion means Characterized in that it comprises a multiplexing means to generate.

[0061] The generation means can generate encoding control information for instructing not to change the picture frame, the video format, or the aspect ratio.

According to a second image encoding method of the present invention, an input step of inputting a multiplexed stream, a separation step of separating a video stream from the multiplexed stream input by the processing of the input step, and a processing of the input step And a determining step of determining whether or not the multiplexed stream input includes multimedia encoded data, and the processing of the determining step determines that the multiplexed stream includes multimedia encoded data. In the case, a generating step of generating encoding control information indicating that the display format of the video stream separated by the processing of the separating step is not changed, and based on the encoding control information generated by the processing of the generating step,
It is characterized by including a conversion step of performing a predetermined conversion on the video stream separated by the processing of the separation step, and a multiplexing step of generating a multiplexed stream including the video stream converted by the processing of the conversion step.

The processing of the generation step can generate encoding control information for instructing not to change the picture frame, video format, or aspect ratio.

The program of the third recording medium of the present invention comprises:
An input step of inputting the multiplexed stream, a separating step of separating the video stream from the multiplexed stream input by the processing of the input step, and the multiplexed stream input by the processing of the input step includes multimedia encoded data. A judging step of judging whether or not the multiplexed stream contains multimedia coded data by the processing of the judging step, and the display format of the video stream separated by the processing of the separating step Generating encoding control information instructing not to change the video stream, and performing a predetermined conversion on the video stream separated by the processing of the separating step based on the coding control information generated by the processing of the generating step Conversion step and conversion step processing Characterized in that it comprises a multiplexing step of generating a multiplexed stream including the video stream.

[0065] The recording medium of the present invention includes encoding control information instructing not to change the display format of a video stream, and a multiplexed stream including a video stream that has been subjected to a predetermined conversion based on the encoding control information. Is recorded.

In the first image encoding apparatus, image encoding method, and recording medium program according to the present invention, a video stream is separated from a multiplexed stream containing multimedia encoded data, and a predetermined The conversion is performed, and additional information is generated indicating that a mismatch occurs when the converted video stream is displayed based on the multimedia encoded data.

In the first recording medium of the present invention, a video stream subjected to predetermined conversion, multimedia encoded data, and the converted video stream are displayed based on the multimedia encoded data. At this time, additional information indicating that a mismatch occurs is recorded.

In the image decoding device and the image decoding method of the present invention, and the program of the second recording medium of the present invention, a video stream is separated from an input multiplexed stream, and the separated video stream is decoded. The decoded video stream is subjected to predetermined conversion according to additional information indicating that inconsistency occurs when the decoded video stream is displayed based on the multimedia encoded data. You.

According to the second image encoding apparatus and the second image encoding method of the present invention, and the program of the third recording medium of the present invention, the video stream is separated from the input multiplexed stream, and the input multiplexed stream is separated. It is determined whether or not the multiplexed stream contains multimedia encoded data, and if it is determined that the multiplexed stream contains multimedia encoded data, the display format of the separated video stream is not changed Is generated, and predetermined conversion is performed on the separated video stream based on the generated coding control information.

The second recording medium of the present invention includes encoding control information for instructing not to change the display format of a video stream, and a video stream that has been subjected to a predetermined conversion based on the encoding control information. A multiplexed stream is recorded.

[0071]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 3 is a diagram showing a configuration of an embodiment of the recording apparatus 1 to which the present invention is applied.
The transport stream received by an antenna or the like (not shown) is input to the selector 10. Selector 1
To 0, a program number (channel number) specified by the user is also input from the terminal 11. The selector 10 extracts a specified program from the input transport stream based on the program number, and outputs a partial transport stream. The partial transport stream is transmitted to the demultiplexer 12 and the analysis unit 13.
Is input to

The transport stream input to the demultiplexer 12 includes a video stream and other streams (audio, still picture, character / graphic data,
Stream composed of data such as multimedia encoded data). The separated video stream is output to the decoder 14. Other streams are output to the multiplexer 16. The demultiplexer 12 outputs, in addition to transport packets other than video, output timing information of the transport packets in an input transport stream.
Output to the multiplexer 16.

The decoder 14 performs a decoding process on the input video stream based on a predetermined decoding system, for example, the MPEG2 system, and outputs video data to the encoder 15. Further, the decoder 14 outputs stream information of the video stream obtained at the time of decoding to the encoding controller 18.

On the other hand, the analyzing unit 13 analyzes the input transport stream to obtain stream information of a stream other than video, for example, information such as a bit rate, and outputs the information to the encoding controller 18. The stream information of the stream other than the video output from the analysis unit 13, the video stream information output from the decoder 14, and the recording bit rate of the stream from the terminal 19 are input to the encoding controller 18. The encoding controller 18 sets conditions for encoding video data performed by the encoder 15 from these data, and outputs the set conditions (encoding control information) to the encoder 15 and the encoding unit 20.

The encoding controller 18 determines, for example, the bit rate of a stream other than video from the recording bit rate of the stream (data input via a terminal 19 from a control device (not shown) for controlling the operation of the recording apparatus 1). The value obtained by subtracting the sum of the data (data input from the analysis unit 13) is used as the bit rate assigned to the encoding of the video data, and the code such as the bit rate and picture frame is set so that the image quality is optimal at the bit rate. The encoding control information is set, and the set encoding control information is output to the encoder 15 and the encoding unit 20. Details of the encoding control information will be described later with reference to FIGS.

The recording bit rate of this stream is a fixed rate when recording on a recording medium at a fixed rate, and is an average bit rate per predetermined time when recording at a variable bit rate. However, the maximum value of the variable bit rate in this case needs to be equal to or less than the maximum recording bit rate guaranteed by the recording medium.

The encoder 15 encodes the video data output from the decoder 14 based on the encoding control information output from the encoding controller 18 (for example, MP
The encoding is performed based on the EG2 method) and output to the multiplexer 16. The multiplexer 16 inputs the video stream from the encoder 15, the transport stream packet other than video from the demultiplexer 12, and the information on the appearance timing of the transport stream packet other than the video. And multiplexes the video stream and the transport stream packet other than the video stream, and outputs the multiplexed transport stream to the arrival time stamp adding unit 17 as a transport stream.

FIG. 4 schematically shows the processing performed by the multiplexer 16 in this case. FIG. 4A shows the timing of the packets of the input transport stream. In this figure, a gray portion indicates a video packet, and a white portion indicates a packet of a stream other than the video. As shown in FIG. 4A, the input transport stream packets are continuous, but the video data is re-encoded by the encoder 15, so that the data amount of the video data is reduced. As a result, the number of video packets is reduced.

As shown in FIG. 4B, the multiplexer 16 does not change the timing of the packets of the stream other than the video, and only changes the timing of the video packets to the original state (the state shown in FIG. 4A). And different timing.

The arrival time stamp adding section 17
As shown in FIG. 5, a transport stream packet (FIG.
(A)), a header (TP_extra_header) including an arrival time stamp is added to the source packet (FIG. 5).
(B)), the source packets are arranged so as to be continuous (FIG. 5 (C)), and output to the writing unit 21. The arrival time stamp is information indicating the timing at which a transport stream packet appears in a transport stream. The writing unit 21 makes a file of a source packet stream including continuous source packets that are input, and records the file on a recording medium 22. The recording medium 22 may be any medium.

The information output from the encoding unit 20 is also input to the writing unit 21. The encoding unit 20 generates multimedia display sub information based on the video encoding information from the encoding controller 18 and outputs the multimedia display sub information to the writing unit 21. The multimedia display sub-information output to the writing unit 21 is such that the video stream is transcoded (decoded by the decoder 14 and then encoded by the encoder 15 again) to change the image frame size. Even in this case, it is information to make the display position and display size of the video on the multimedia plane the same as the screen intended by the transmission side (the screen that would be displayed if not re-encoded). , Information used in combination with multimedia encoded data.

Here, the multimedia display sub information will be specifically described. As shown in FIG. 6, the multimedia display sub information includes a mismatch flag (mismatch_MMi).
nfo_flag), Re-encoded flag (Re_encoded_fla)
g), and the frame size change flag (changd_frame_
size_flag), data relating to two sizes representing the original horizontal size (original_horizontal_size) and the original vertical size (original_vertical_size), and the original screen aspect ratio (original_display_aspect_rati).
o).

The mismatch flag is a flag indicating whether or not there is a mismatch between the video and the multimedia encoded data. The re-encoding flag is a flag indicating whether or not the video has been re-encoded at the time of recording. The frame size change flag is, for example, a flag indicating whether the picture frame of the video has been changed due to re-encoding. The original horizontal size is the size of the horizontal picture frame before re-encoding. The original vertical size is the size of the vertical picture frame before re-encoding. The original screen aspect ratio means the aspect ratio of the frame screen before being re-encoded.

Note that the above-described multimedia display sub-information is merely an example, and may include information other than that shown in FIG. 6, or conversely, a part of the information shown in FIG. It may be reduced if necessary.

Another example of the multimedia display sub information will be described. In this example, the multimedia display sub information is:
It is stored in ProgramInfo of the syntax shown in FIG. Fields related to the present invention in the syntax of ProgramInfo () will be described.

The length indicates the number of bytes from the byte immediately after the length field to the last byte of ProgramInfo ().

Num_of_program_sequences indicates the number of program-sequences in the AV stream file. AV
A source packet sequence in which a program content defined by this format in a stream file is constant is called a program-sequence.

SPN_program_sequence_start indicates the address where program-sequence starts on the AV stream file. SPN_program_sequence_start has a size in units of source packet numbers, and is counted from the first source packet of the AV stream file with zero as an initial value.

The program_map_PID is the program-sequenc
This is a PID value of a transport packet having a PMT (program map table) applicable to e.

Num_of_streams_in_ps is the program-se
Indicates the number of elementary streams defined in the quence.

The stream_PID is the pr of the program-sequence
Indicates the value of the PID for the elementary stream defined in the PMT referenced by ogram_map_PID.

[0092] StreamCodingInfo () is the stream_PID
Indicates the information of the elementary stream indicated by.

FIG. 8 shows the syntax of StreamCodingInfo (). length indicates the number of bytes from the byte immediately after this length field to the last byte of StreamCodingInfo ().

The stream_coding_type is the StreamCoding
Indicates the encoding type of the elementary stream indicated by stream_PID corresponding to Info (). The meaning of the values is shown in FIG.

When the value of stream_coding_type is 0x02,
Indicates that the elementary stream pointed to by stream_PID is a video stream.

Also, if the value of stream_coding_type is 0x0A,
0x0B, 0x0C or 0x0D indicates that the elementary stream indicated by stream_PID is multimedia encoded data.

When the value of stream_coding_type is 0x06, it indicates that the elementary stream indicated by stream_PID is a caption or a teletext.

Video_format is the StreamCodingInfo ()
Indicates the video format of the video stream pointed to by stream_PID corresponding to. The meaning of the values is shown in FIG.

480i is a video display of an NTSC standard TV (interlaced frame of 720 pixels × 480 lines). The 576i is a PAL standard TV (720 pixels x
576 lines of interlaced frames). 480p is a video display of a progressive frame of 720 pixels × 480 lines. Also, 1080i is 1920
This is a video display of an interlaced frame of pixels × 1080 lines. 720p is a video display of a progressive frame of 1280 pixels × 720 lines.

Frame_rate indicates the frame rate of the video stream specified by stream_PID corresponding to this StreamCodingInfo (). The meaning of the values is shown in FIG.

The display_aspect_ratio is the StreamCodi
Indicates the display aspect ratio of the video stream pointed to by stream_PID corresponding to ngInfo (). The meaning of the values is shown in FIG.

The original_video_format_flag is the Stre
original_video_format and originala in amCodingInfo ()
This flag indicates whether l_display_aspect_ratio exists.

Original_video_format is the StreamCod
This is the original video format before the video stream pointed to by stream_PID corresponding to ingInfo () is encoded. The meaning of the values is the same as in FIG.

Original_display_aspect_ratio is the S
This is the original display aspect ratio before the video stream pointed to by stream_PID corresponding to streamCodingInfo () is encoded. The meaning of the values is shown in FIG.
Is the same as

When transcoding a transport stream in which a multimedia data stream (BML stream, subtitle, etc.) is multiplexed together with the video stream, the video stream changes its video format by being re-encoded. (For example, change from 1080i to 480i) On the other hand, consider a case where the multimedia data stream keeps the content of the original stream. At this time, information mismatch may occur between the new video stream and the multimedia data stream. For example, the parameters related to the display of the multimedia data stream are determined by assuming the video format of the original video stream, but the video stream changes due to the re-encoding of the video stream.

The video format of the original video stream is video_format and display_aspect_ratio.
Is indicated by The video format of the video stream after re-encoding is original_video_format
And original_display_aspect_ratio.

The values of video_format and original_video_format are different, and / or display_aspect_ratio and or
If the value of iginal_display_aspect_ratio is different, it indicates that the video format has been changed by re-encoding the video during recording.

[0108] The stream_coding_type is multimedia encoded data and the stream_PID indicating subtitles is ProgramInfo.
Indicates that the multimedia data is multiplexed in the AV stream file (transport stream).

When ProgramInfo indicates that the video format has been changed by re-encoding the video at the time of recording and that the multimedia data is multiplexed in the AV stream file,
It is determined that there is a display mismatch between the video stream (re-encoded) of the AV stream file and the multimedia data (original).

In such a case, original_video_format
And the original_display_aspect_ratio and the information about the original video stream are valid. The playback device creates a display image from the new video stream and the multimedia data stream as follows. -The video stream is original_video_format and origi
Up-sampled to the video format indicated by nal_display_aspect_ratio. -The up-sampled image and the multimedia data stream are combined to create a correct display image.

The multimedia display sub-information created and output by the encoding unit 20 is recorded on the recording medium 22 by the writing unit 21. The source packet stream file output from the arrival time stamp adding unit 17 Recorded as a separate file. When the multimedia display sub-information is recorded on the recording medium 22 by the writing unit 21 as a file different from the source packet stream file, the encoding unit 20 outputs the filed multimedia display sub-information.

FIG. 13 shows a flowchart of an encoding process of an AV stream and multimedia display sub-information.

At step 50, a multiplexed stream including multimedia encoded data is input to the recording device 1.

In step 51, the demultiplexer 12 separates the video stream from the multiplexed stream.

At step 52, the encoder 15 re-encodes the video stream decoded by the decoder 14.

In step 53, the multiplexer 16 multiplexes the video stream and multimedia encoded data to create a multiplexed stream.

At step 54, the encoding unit 20 creates multimedia display sub information.

In the above description, the encoding controller 1
8 generates encoding control information including information such as a bit rate and an image frame based on the input data, but may generate the following information as other encoding control information. That is, when the analysis unit 13 analyzes the input transport stream and finds that the input transport stream contains multimedia encoded data, the encoding controller 18 performs encoding by the encoder 15. When the encoding is performed, encoding control information for instructing re-encoding with an image frame of the same size as the original video image frame (image frame before re-encoding) is generated and output to the encoder 15. May be.

In such a case, the encoder 15
The video data from the decoder 14 is re-encoded with the same value as the picture frame of the original video stream based on the input coding control information. When such encoding control information is generated and re-encoding is performed based on the encoding control information, the image frame is not changed by the re-encoding. The occurrence of inconsistency in the relationship with the encoded data can be prevented.

Alternatively, the following information may be generated as the coding control information generated by the coding controller 18. That is, as a result of analyzing the input transport stream by the analysis unit 13 and analyzing that the input transport stream contains multimedia encoded data, the encoding controller 18
When encoding is performed by the encoder 15, encoding control information for instructing re-encoding under the same conditions as the video format (shown in FIG. 10) and the screen aspect ratio (shown in FIG. 12) of the original video is Generate
You may make it output to the encoder 15.

In such a case, the encoder 15
Based on the input coding control information, the video data from the decoder 14 is re-encoded under the same conditions as the original video format (shown in FIG. 10) and the screen aspect ratio (shown in FIG. 12). When such encoding control information is generated and re-encoding is performed based on the encoding control information, a video stream obtained by re-encoding does not change the video format and the screen aspect ratio due to the re-encoding. And inconsistency in the relationship with the multimedia encoded data can be prevented.

FIG. 14 shows a flowchart of an encoding process in a case where the re-encoding of a video of a multiplexed stream including multimedia encoded data is restricted.

In step 70, the multiplexed stream is input to the recording device 1.

At step 70, the demultiplexer 12 separates the video stream from the multiplexed stream.

At step 70, the analysis unit 13 checks whether or not multimedia encoded data is included. When multimedia encoded data is included, the encoding control information is instructed to the encoder 15 so that the video stream is re-encoded without changing the video display format. The encoder 15 re-encodes the video stream based on the control information.

In step 70, the multiplexer 16 creates a multiplexed stream including the video stream.

Next, an example of control based on the coding control information will be described with reference to FIGS.

Now, for example, as shown in FIG.
Transport stream input to the selector 10 is assumed to have a constant bit rate R _I.
The video stream and the stream other than the video are each encoded at a variable bit rate. In the example of FIG. 15, in the unit time (for example, GOP) A, the bit rate of the video stream is R _VA, and the bit rate of the unit time other than video is R _OA . In the unit time B, the bit rate of the video stream is R _VB , the bit rate of the stream other than the video is R _OB, and in the unit time C, the bit rate of the video stream is R _VC , The bit rates of the other streams are set to R _OC .

Now, the transport stream as shown in FIG. 15 is re-encoded, and as shown in FIG. 16, the transport stream having a fixed bit rate S (S <R _I ) is output from the multiplexer 16. When outputting, the encoding controller 18
The processing shown in the flowchart of FIG. 17 is executed.

First, in step S 1, the encoding controller 18 outputs a signal from the multiplexer 16 based on a control signal input from a control device (not shown) from a terminal 19.
The bit rate (recording rate) of the transport stream to be output is set to S. Next, in step S2, the encoding controller 18 determines streams to be recorded other than the video, and calculates the maximum value D of the total bit rate of those streams.

The maximum value D is determined from the stream specification of the input transport stream. For example, when recording two audio streams in addition to video, the maximum bit rate of one audio stream is 384 Kbps according to the stream specifications of digital BS broadcasting in Japan.
Therefore, the maximum value D is 384 × 2 Kbps.

Next, in step S3, the encoding controller 18 subtracts the maximum value D calculated in step S2 from the bit rate S for recording set in step S1 to obtain a value C (= Let SD) be the assigned bit rate for re-encoding the video data.
Then, in step S4, the encoding controller 1
8 analyzes encoding information such as a bit rate and a picture frame of the video stream from the video stream information output from the decoder 14.

Next, proceeding to step S5, the encoding controller 18 sets the optimum image quality based on the value C calculated in step S3 and the encoding information of the video stream analyzed in step S4. A video coding parameter (video coding control information) is determined.

For example, in the example shown in FIG.
Is １ ／ of the value R _I. In this case, the bit rate of the stream other than the video is set to the maximum value D, and the maximum value D is directly used as the bit rate of the stream other than the video of the multiplexed stream after re-encoding.

The video coding parameters are determined so that the image quality is optimal within the range of (SD). When the image frame is controlled, for example, 720 × 48
The horizontal direction of the image frame of 0 pixels is sub-sampled to １ ／ to be 360 × 480 pixels. The determined encoding parameters (bit rate, angle of view, etc.) are supplied to the encoder 15 as video encoding control information.

Therefore, the encoder 15 determines in step S6
In step (1), based on the video encoding control information supplied from the encoding controller 18, the video data of the unit time (currently, unit time A in this case) to be processed is re-encoded. In the example of FIG. 16, in the unit time A, the bit rate R _OA is actually smaller than the maximum value D, but the maximum value D is a fixed value. ). Since the maximum value D is fixed, a useless portion _Rsa that cannot be used for video encoding occurs, but a stuffing bit is inserted therein.

Next, proceeding to step S7, the encoding controller 18 determines whether or not the stream to be re-encoded has been completed. If the stream has not been completed yet, the process returns to step S4, and the subsequent steps are performed. Repeat the process.

Then, if it is determined in step S7 that the stream to be encoded has been completed, the processing is completed.

As described above, in the example of FIG. 16, even in the unit time B, the bit rate of the stream other than the video is set to D, and the allocated bit rate of the video stream is the fixed bit rate. It is said. Then, the value R _sb (= S− (SD) −R _OB = D−
R _OB ) is inserted with a stuffing bit.

In the unit time C, the bit rate of the stream other than the video is D, and the assigned bit rate of the video stream is SD. In this unit time C, since D = R _OC , there is no stuffing bit.

As described above, in the example of FIG. 16, the video stream is encoded at a fixed bit rate.

FIG. 18 shows an example of processing when the assigned bit rate for video re-encoding is a variable bit rate. First, in step S21, the encoding controller 18 sets the recording rate S based on the input from the terminal 19. Next, in step S22, the encoding controller 18 analyzes the encoded information of the video stream based on the video stream information from the decoder 14. The processing in steps S21 and S22 described above is the same as the processing in steps S1 and S4 in FIG.

Next, in step S23, the encoding controller 18 calculates the total bit rate B for each time unit of a stream other than video from the output of the analysis unit 13.

Next, proceeding to step S24, the encoding controller 18 subtracts the value B calculated in the processing of step S23 from the value S set in step S1 to obtain the value C
Let (= S−B) be the video re-encoding allocation bit rate.

Further, in step S25, the encoding controller 18 determines the optimum image quality based on the value C obtained in step S24 and the analysis result of the encoding information of the video stream obtained in the processing in step S22. To determine the video coding parameters. The determined encoding parameters are output to the encoder 15.

[0146] Then, in step S26, the encoder 15 re-encodes the video data of the current unit time based on the encoding parameters determined in the processing of step S25. Thereby, for example, as shown in FIG. 19, after R _oa (= R _OA ) is secured as a bit rate per unit time other than video, the bit rate R _va defined by (S−R _oa ) Is set to the bit rate of the video stream.

[0147] In step S27, the encoding controller 18 determines whether or not the stream has ended. If the stream has not ended, the process returns to step S22, and the subsequent steps are repeatedly executed. In step S27,
If it is determined that the stream has ended, the process ends.

Thus, in the unit time B,
Bit rate R _ob of streams other than video (= SR
_{After OB} ) is secured, the remaining R _vb (= _SRob ) is set to the bit rate of the video stream. In the unit time C, the bit rate of the video stream is set to R _vc (= S−R _OC ) excluding the bit rate R _oc of the stream other than the video.

As described above, in this processing example, the bit rate of the video stream is changed to a variable bit rate.
The stuffing bits can be eliminated or reduced, and the video stream can be encoded more efficiently.

In the above, the case where the input transport stream has a fixed bit rate has been described as an example. For example, as shown in FIG. 20, even when the input transport stream has a variable bit rate, The present invention can be applied.

As described above, it is possible to record a transport stream corresponding to a longer-time content on the recording medium 22 at a lower bit rate if necessary.

It is possible to prevent the quality of data other than video data, such as audio data, still picture / character graphic data, and multimedia encoded data, from remarkably deteriorating. Since the amount of data other than video data is basically smaller than that of video data, if the bit rate is reduced at the same rate as video data, the effect on non-video data is relatively small. Although the effect is larger than the effect on the data, such an effect can be prevented.

Next, reproduction of the source packet stream file recorded on the recording medium 22 will be described. FIG. 21 is a diagram showing a configuration of an embodiment of a reproducing apparatus to which the present invention has been applied. The source packet stream file recorded on the recording medium 22 is read by the reading unit 31. The reading unit 31 also reads the multimedia display sub information recorded on the recording medium 22 as a file separate from the source packet stream.

The source packet stream read by the reading unit 31 is sent to the arrival time stamp separating unit 32, and the multimedia display sub information is sent to the combining unit 36.
Are respectively output. The arrival time stamp separation unit 32 has a built-in reference clock, compares the reference clock with the value of the arrival time stamp added to the source packet of the input source packet stream, and finds that both values are equal. The source packet with the equal arrival time stamp, remove the arrival time stamp,
Output to the demultiplexer 33 as a transport stream packet.

The demultiplexer 33 converts the input transport stream into a video audio stream,
It is separated into multimedia encoded data and a stream of data such as character figures / characters / still images. Among the separated data, the video / audio stream is sent to the AV decoder 34, the multimedia encoded data is sent to the synthesizing unit 36, and the data stream such as character / graphic / character / still image is sent to the character / graphic / still image decoder 35. Are respectively output.

[0156] The AV decoder 34 separates the input video / audio stream into video data and audio data, decodes them, and then sends the audio data to an audio playback device (not shown) and the video data to the synthesizing unit 36. Output each. The character / graphics / still image decoder 35 decodes an input data stream of character / graphics, characters, still images, and the like, and outputs the decoded character / graphics data, character data, still image data, and the like to the combining unit 36.

The synthesizing unit 36 includes video data from the AV decoder 34, multimedia encoded data output from the demultiplexer 33, and the character / graphic / still image decoder 3.
The data such as character figures / characters / still images output from 5 and the multimedia display sub information read by the reading unit 31 are input. The synthesizing unit 36 determines whether or not a mismatch has occurred in the relationship between the input video signal and the multimedia encoded data by checking the mismatch flag (FIG. 6) of the input multimedia display sub-information. .

Alternatively, the synthesizing unit 36 determines whether the video_form
at and original_video_format values are different, and / or display_aspect_ratio and original_display_aspect_
If the value of the ratio is different, it is determined that the video format has been changed by re-encoding the video at the time of recording, and that there is a mismatch between the input video signal and the multimedia encoded data. video_form
at and original_video_format have the same value, and d
isplay_aspect_ratio and original_display_aspect_ratio
Are the same, it is determined that there is no mismatch in the relationship between the input video signal and the multimedia encoded data.

In the case where it is indicated that the relationship between the input video signal and the multimedia encoded data is inconsistent, the synthesizing unit 36 further calculates the original horizontal size of the multimedia display sub-information and the original horizontal size. Refer to vertical size or original_video_for
Refer to mat and original_display_aspect_ratio. Then, the input video signal is scale-converted so as to be displayed in a frame of the reference size. And
A video signal obtained by combining a scaled video signal and other character / graphic data on a multimedia plane based on the multimedia encoded data is output to a television receiver (not shown) as a display device or the like. .

On the other hand, when it is indicated that there is no inconsistency in the relationship between the input video signal and the multimedia coded data, the synthesizing unit 36 does not change the scale of the input video signal, and The data is combined with the multimedia plane and output.

As described above, by recording the multimedia display sub-information at the time of recording and using it at the time of reproduction, the screen intended by the transmitting side is displayed on the receiving side. With reference to FIG. 22, even if the transmitting side (recording side) is changed to an image frame smaller than the original video image frame due to re-encoding, the multimedia display sub-information By recording the information and referring to the recorded multimedia display sub-information during reproduction, the receiving side (reproducing side) can reproduce the video and other data without inconsistency. The screen obtained when not encoding is obtained.

FIG. 24 is a flowchart showing the process of reproducing an AV stream using the multimedia display sub information.

At step 60, the multiplexed stream including the multimedia encoded data is read from the recording medium and input to the reproducing apparatus.

In step 61, multimedia display sub information is input. This information is read from the recording medium in the case of the reproducing apparatus of FIG. 21, and is separated from the multiplexed stream in the case of the reproducing apparatus of FIG.

In step 62, the video stream is separated from the multiplexed stream.

In step 63, the video stream is decoded.

In step 64, if there is a display mismatch between the video and the multimedia coded data, the synthesizing unit 36 scales the video based on the multimedia display sub-information.

At step 65, the synthesizing unit 36 synthesizes the processed image and the multimedia data to create a display image.

As described above, the multimedia display sub-information may be recorded on the recording medium 22 as a file different from the source packet stream file containing the character / graphic data and the video signal. And may be recorded in the recording medium 22. FIG. 23 shows a configuration of the recording apparatus 1 in a case where the multimedia display sub-information is embedded and recorded in the source packet stream file.

The configuration of the recording apparatus 1 shown in FIG.
23, the multimedia display sub-information output from the encoding unit 20 is output to the multiplexer 16 in the recording device 1 illustrated in FIG. The multiplexer 16 creates a transport packet from the input multimedia display sub information, embeds the transport packet in a source packet stream file, and outputs the transport packet to the arrival time stamp adding unit 17. Instead of embedding the multimedia display sub-information as a transport packet in the source packet stream file, it may be written in the user data area of the MPEG video stream.

In the present embodiment, the method of re-encoding the video data is not limited to such a method, and the input video stream is transformed on the DCT area to encode the image frame and other encoding parameters. You may make it convert.

FIG. 25 shows the configuration of the reproducing apparatus 30 when the multimedia display sub-information is embedded in the source packet stream file and recorded on the recording medium 22 in this manner. The configuration of the playback device 30 shown in FIG. 25 is compared with the configuration of the playback device 30 shown in FIG. 21. In the configuration of the playback device 30 shown in FIG. It's just a stream. Then, the source packet stream read by the reading unit 31 is input to the demultiplexer 33 via the arrival time stamp separation unit 32.

The demultiplexer 33 extracts the multimedia display sub information from the input source packet stream file, and outputs it to the synthesizing unit 36. Other configurations are the same as those in FIG.

As described above, even when the source packet stream file is stored with the multimedia display sub-information, the size and display position of the video frame intended by the transmission side can be obtained on the reception side. Become.

In the present embodiment, the case of a transport stream has been described. However, the present invention is not limited to this, and can be applied to a multiplexed stream such as a program stream.

The series of processes described above can be executed by hardware, but can also be executed by software. In this case, for example, the recording device 1
(Although not shown, the same applies to the playback device 30)
It is composed of a personal computer as shown in FIG.

In FIG. 26, the CPU (Central Process
g Unit) 101 is a program stored in a ROM (Read Only Memory) 102 or a RAM from a storage unit 108.
(Random Access Memory) 103 executes various processes according to the program loaded. The RAM 103 also stores data necessary for the CPU 101 to execute various types of processing, as appropriate.

CPU 101, ROM 102, and RAM 103
Are connected to each other via a bus 104. An input / output interface 105 is also connected to the bus 104.

The input / output interface 105 includes an input unit 106 such as a keyboard and a mouse, a display such as a CRT and an LCD, an output unit 107 such as a speaker, a storage unit 108 such as a hard disk, a modem, and a terminal. A communication unit 109 composed of an adapter or the like is connected. The communication unit 109 performs communication processing via a network.

A drive 110 is connected to the input / output interface 105 as necessary.
21, an optical disk 122, a magneto-optical disk 123, a semiconductor memory 124, and the like are appropriately mounted, and a computer program read therefrom is installed in the storage unit 108 as necessary.

When a series of processing is executed by software, a computer in which a program constituting the software is built in dedicated hardware,
Or, by installing various programs,
It is installed from a recording medium to, for example, a general-purpose personal computer that can execute various functions.

As shown in FIG. 26, this recording medium is a magnetic disk 121 (including a floppy disk) on which a program is recorded, which is distributed separately from a computer to provide the program to a user, and an optical disk 1
22 (CD-ROM (Compact Disk-Read Only Memory), DVD
(Including a Digital Versatile Disk), a magneto-optical disk 123 (including an MD (Mini-Disk)), or a packaged medium including a semiconductor memory 124, as well as a user installed in a computer in advance. Is stored in the program
The hard disk includes the ROM 102 and the storage unit 108, and the like.

In this specification, the steps of describing a program provided by a recording medium may be performed in a chronological order according to the described order. It also includes processes that are executed individually or individually.

[0184]

As described above, according to the first image encoding apparatus, image encoding method, and recording medium program of the present invention, a video stream is separated from a multiplexed stream containing multimedia encoded data. A predetermined conversion is performed on the video stream, and additional information indicating that a mismatch occurs when the converted video stream is displayed based on the multimedia encoded data is generated.

According to the first recording medium of the present invention, a video stream subjected to predetermined conversion, multimedia encoded data, and the converted video stream are converted based on the multimedia encoded data. Additional information indicating that inconsistency occurs at the time of display is recorded.

Therefore, in any case, it is possible to prevent a mismatch between the video stream and the multimedia encoded data on the reproducing side.

According to the picture decoding apparatus and picture decoding method of the present invention, and the program of the second recording medium of the present invention, a video stream is separated from an input multiplexed stream, and the separated video stream is separated. Decoding the stream, and applying a predetermined conversion to the decoded video stream according to additional information indicating that a mismatch occurs when displaying the decoded video stream based on the multimedia encoded data. Since this is performed, it is possible to prevent a mismatch from occurring between the video stream and the multimedia encoded data.

Further, according to the second image encoding device and the second image encoding method of the present invention, and the program of the third recording medium of the present invention, the video stream is separated from the input multiplexed stream, It is determined whether or not the multiplexed stream includes multimedia encoded data. If it is determined that the multiplexed stream includes multimedia encoded data, the display format of the separated video stream is determined. Is generated, and predetermined conversion is performed on the separated video stream based on the generated coding control information.

According to the second recording medium of the present invention,
Encoding control information indicating that the display format of the video stream is not changed, and a multiplexed stream including a video stream that has been subjected to a predetermined conversion based on the encoding control information are recorded.

Therefore, in any case, it is possible to prevent a mismatch between the video stream and the multimedia encoded data on the reproducing side.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a display screen based on multimedia encoded information.

FIG. 2 is a diagram illustrating a mismatch that occurs when re-encoding is performed.

FIG. 3 is a diagram illustrating a configuration of an embodiment of a recording apparatus to which the present invention has been applied.

FIG. 4 is a diagram illustrating the operation of the multiplexer of FIG.

FIG. 5 is a diagram illustrating processing of an arrival time stamp adding unit in FIG. 3;

FIG. 6 is a diagram illustrating multimedia display sub information.

FIG. 7 is a diagram illustrating another example of the syntax of ProgramInfo ().

FIG. 8 is a diagram illustrating the syntax of StreamCodingInfo ().

FIG. 9 is a diagram illustrating the meaning of stream_coding_type.

FIG. 10 is a diagram illustrating the meaning of video_format.

FIG. 11 is a diagram illustrating the meaning of frame_rate.

FIG. 12 is a diagram illustrating the meaning of display_aspect_ratio.

FIG. 13 is a flowchart illustrating an encoding process of an AV stream and multimedia display sub information.

FIG. 14 is a flowchart illustrating an encoding process when restricting re-encoding of video of a multiplexed stream including multimedia encoded data.

FIG. 15 is a diagram illustrating an input transport stream.

FIG. 16 is a diagram illustrating an example of a transport stream after the video stream in FIG. 15 is re-encoded.

FIG. 17 is a flowchart illustrating a recording rate control process in the recording apparatus of FIG. 3;

FIG. 18 is a flowchart illustrating another recording rate control process in the recording apparatus of FIG. 3;

FIG. 19 is a diagram illustrating another example of the transport stream after the video stream is re-encoded.

FIG. 20 is a diagram illustrating another example of the input transport stream.

FIG. 21 is a diagram illustrating a configuration of an embodiment of a playback device to which the present invention has been applied.

FIG. 22 is a diagram illustrating a display screen when multimedia display sub-information is added.

FIG. 23 is a diagram illustrating another configuration of a recording apparatus to which the present invention is applied.

FIG. 24-A using multimedia presentation sub-information
It is a flowchart explaining a reproduction process of a V stream.

FIG. 25 is a diagram illustrating another configuration of a playback device to which the present invention has been applied.

FIG. 26 is a diagram illustrating a medium.

[Explanation of symbols]

1 recording device, 11 terminals, 12 demultiplexers, 13 analysis units, 14 decoders, 15 encoders, 16 multiplexers, 17 arrival time stamp addition units, 18 encoding controllers,
19 terminals, 20 encoding unit, 21 writing unit,
Reference Signs List 22 recording medium, 30 playback device, 31 reading unit, 32 arrival time stamp separation unit, 33
Demultiplexer, 34 AV decoder, 35 character / still image decoder, 36 synthesis unit

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C059 KK41 LA06 PP04 RB02 RB09 RB10 RC04 RC19 RC32 RC33 RC34 SS11 SS20 UA02 UA05 5C063 AA02 AA03 AB03 AB05 AC01 AC02 AC05 AC10 BA01 BA14 CA11 CA12 CA25 DA02 DA03 DA05 DA07

Claims (53)

[Claims] An input unit for inputting a multiplexed stream including multimedia encoded data; a separating unit for separating a video stream from the multiplexed stream input by the input unit; Converting means for performing a predetermined conversion on the video stream; generating means for generating additional information indicating that inconsistency occurs when the converted video stream is displayed based on the multimedia encoded data; An image encoding apparatus comprising: an output unit that outputs the encoded video stream, the multimedia encoded data, and the additional information. 2. The image processing apparatus according to claim 1, further comprising an encoding unit that encodes the additional information generated by the generation unit as data different from a multiplexed stream including the converted video stream. 2. The image encoding device according to 1. 3. The apparatus according to claim 1, further comprising an encoding unit that multiplexes and encodes the additional information generated by the generation unit into a multiplexed stream including the converted video stream. The image encoding device according to claim 1. 4. The image encoding apparatus according to claim 1, wherein the conversion unit converts a parameter of an image frame of the video stream. 5. The method according to claim 1, wherein the conversion by the conversion unit includes at least a process of decoding the video stream separated by the separation unit and encoding the decoded video stream. Image coding device. 6. The apparatus according to claim 1, wherein the additional information generated by the generation unit includes at least one of information of an original image frame and an original screen aspect ratio. Image coding device. 7. The image encoding apparatus according to claim 1, wherein the additional information generated by the generation unit includes information on an original video format and the converted video format. 8. The image encoding apparatus according to claim 1, wherein the additional information generated by the generation unit includes information on an original screen aspect ratio and the converted screen aspect ratio. . 9. The additional information generated by the generating means includes information indicating whether or not an image frame of the video stream has been converted by the converting means, and an original image of the video stream separated by the separating means. 2. The image encoding apparatus according to claim 1, wherein the image encoding apparatus includes at least one of frame information and original screen aspect ratio. 10. An inputting step of inputting a multiplexed stream including multimedia encoded data, a separating step of separating a video stream from the multiplexed stream input by the processing of the inputting step, and a processing of the separating step A predetermined conversion is performed on the video stream separated according to the above, and a generation step of generating additional information indicating that a mismatch occurs when the converted video stream is displayed based on the multimedia encoded data. And an output step of outputting the converted video stream, the multimedia encoded data, and the additional information. 11. The method according to claim 11, further comprising an encoding step of encoding the additional information generated by the generation unit as data different from a multiplexed stream including the converted video stream. An image encoding method according to claim 10. 12. The method according to claim 1, further comprising an encoding step of multiplexing and encoding the additional information generated by the processing of the generation step into a multiplexed stream including the converted video stream. An image encoding method according to claim 10. 13. The image encoding method according to claim 10, wherein the processing in the conversion step includes converting a parameter of an image frame of a video stream. 14. The conversion according to the conversion step, wherein the conversion includes at least processing of decoding a video stream separated by the processing of the separation step and encoding the decoded video stream. An image encoding method according to claim 10. 15. The apparatus according to claim 10, wherein the additional information generated by the processing of the generating step includes at least one of information of an original image frame and an original screen aspect ratio. Image coding method as described in the above. 16. The image encoding method according to claim 10, wherein the additional information generated by the processing of the generating step includes information on an original video format and the converted video format. . 17. The image code according to claim 10, wherein the additional information generated by the processing of the generation step includes information of an original screen aspect ratio and the converted screen aspect ratio. Method. 18. The additional information generated by the processing of the generation step includes information indicating whether or not the image frame of the video stream has been converted by the processing of the conversion step;
The image encoding method according to claim 10, wherein the image encoding method includes at least one of information on an original image frame of the video stream separated by the processing in the separation step and information on an original screen aspect ratio. . 19. An inputting step of inputting a multiplexed stream including multimedia encoded data, a separating step of separating a video stream from the multiplexed stream input by the processing of the inputting step, and a processing of the separating step A predetermined conversion is performed on the video stream separated according to the above, and a generation step of generating additional information indicating that a mismatch occurs when the converted video stream is displayed based on the multimedia encoded data. And a step of outputting the converted video stream, the multimedia encoded data, and the additional information. 20. A predetermined converted video stream, multimedia encoded data, and an indication that a mismatch occurs when displaying the converted video stream based on the multimedia encoded data. A recording medium on which additional information is recorded. 21. The recording medium according to claim 20, wherein the additional information is encoded and recorded as data different from a multiplexed stream including the converted video stream. 22. The recording medium according to claim 20, wherein the additional information is multiplexed and encoded on a multiplexed stream including the converted video stream and recorded. 23. The video stream, wherein parameters of an image frame are converted.
0. The recording medium according to 0. 24. The recording medium according to claim 20, wherein said video stream is once decoded and further encoded. 25. The recording medium according to claim 20, wherein the additional information includes at least one of an original image frame information and an original screen aspect ratio. 26. The recording medium according to claim 20, wherein the additional information includes information on an original video format and the converted video format. 27. The recording medium according to claim 20, wherein the additional information includes information of an original screen aspect ratio and the converted screen aspect ratio. 28. The additional information includes at least one of information indicating whether an image frame of the video stream has been converted, information of an original image frame of the video stream, and an original screen aspect ratio. 21. The recording medium according to claim 20, comprising: 29. An input unit for inputting a multiplexed stream including multimedia encoded data; a separating unit for separating a video stream from the multiplexed stream input by the input unit; Decoding means for decoding, and in accordance with additional information indicating that inconsistency occurs when displaying the decoded video stream based on the multimedia encoded data, the decoded video stream An image decoding apparatus, comprising: processing means for performing a predetermined conversion. 30. The image decoding apparatus according to claim 29, further comprising an acquisition unit configured to acquire the additional information from data different from the multiplexed stream. 31. The image decoding apparatus according to claim 29, further comprising an acquisition unit configured to acquire the additional information from a multiplexed stream in which the additional information is multiplexed. 32. The apparatus according to claim 2, wherein the processing unit converts a parameter of an image frame of the video stream.
10. The image decoding device according to 9. 33. The conversion by the processing unit decodes the video stream separated by the separation unit,
30. The method according to claim 29, further comprising at least a process of encoding the decoded video stream.
An image decoding device according to claim 1. 34. The image decoding apparatus according to claim 29, wherein the additional information includes at least one of information of an original image frame and an original screen aspect ratio. 35. The image decoding apparatus according to claim 29, wherein the additional information includes information of an original video format and the converted video format. 36. The image decoding apparatus according to claim 29, wherein the additional information includes information of an original screen aspect ratio and the converted screen aspect ratio. 37. The additional information includes information indicating whether or not an image frame of the video stream has been converted by the conversion unit, information of an original image frame of the video stream separated by the separation unit, and an original image frame. 30. The image decoding apparatus according to claim 29, wherein the image decoding apparatus includes at least one piece of information among screen aspect ratios. 38. An inputting step of inputting a multiplexed stream including multimedia encoded data; a separating step of separating a video stream from the multiplexed stream input by the processing of the inputting step; A decoding step of decoding the stream; and displaying the decoded video stream in response to additional information indicating that a mismatch occurs when displaying the decoded video stream based on the multimedia encoded data. Performing a predetermined conversion on the stream. 39. The image decoding method according to claim 38, further comprising an obtaining step of obtaining the additional information from data different from the multiplexed stream. 40. The image decoding method according to claim 38, further comprising an obtaining step of obtaining the additional information from a multiplexed stream in which the additional information is multiplexed. 41. The image decoding method according to claim 38, wherein the processing in the processing step converts a parameter of an image frame of a video stream. 42. The conversion by the processing of the processing step includes at least a processing of decoding a video stream separated by the processing of the separation step, and encoding the decoded video stream. 38. The image decoding method according to claim 38. 43. The image decoding method according to claim 38, wherein the additional information includes at least one of an original image frame information and an original screen aspect ratio. 44. The image decoding method according to claim 38, wherein the additional information includes information on an original video format and the converted video format. 45. The image decoding method according to claim 38, wherein the additional information includes information of an original screen aspect ratio and the converted screen aspect ratio. 46. The additional information includes information indicating whether or not an image frame of the video stream has been converted by the processing of the converting step, and information of an original image frame of the video stream separated by the processing of the separating step. , And at least one of the original screen aspect ratios
39. The image decoding method according to claim 38, comprising two pieces of information. 47. An inputting step of inputting a multiplexed stream including multimedia encoded data; a separating step of separating a video stream from the multiplexed stream input by the processing of the inputting step; A decoding step of decoding the stream; and displaying the decoded video stream in response to additional information indicating that a mismatch occurs when displaying the decoded video stream based on the multimedia encoded data. And a processing step of performing a predetermined conversion on the stream. A recording medium storing a computer-readable program. 48. An input unit for inputting a multiplexed stream, a separating unit for separating a video stream from the multiplexed stream input by the input unit, and a multimedia output to the multiplexed stream input by the input unit Determining means for determining whether or not encoded data is included; and when the determining means determines that multimedia encoded data is included in the multiplexed stream, Generating means for generating encoding control information instructing not to change the display format of the video stream; and a video stream separated by the separating means based on the encoding control information generated by the generating means. Conversion means for converting the video stream; and the video stream converted by the conversion means The image encoding device characterized by comprising a multiplexing means for generating a multiplexed stream including the beam. 49. The apparatus according to claim 4, wherein said generating means generates encoding control information for instructing not to change an image frame, a video format, or an aspect ratio.
9. The image encoding device according to 8. 50. An input step of inputting a multiplexed stream, a separating step of separating a video stream from the multiplexed stream input by the processing of the input step, and the multiplexing input by the processing of the input step A judging step of judging whether or not the stream contains multimedia encoded data; and, if it is judged by the processing of the judging step that the multiplexed stream contains multimedia encoded data, the separation A generating step of generating encoding control information that instructs not to change a display format of the video stream separated by the processing of the step, based on the encoding control information generated by the processing of the generating step, Video streams separated by the processing of the separation step Image coding method, characterized in that a multiplexing step of generating a multiplexed stream including a conversion step for performing a predetermined conversion, the video stream which has been converted by the processing of said conversion step. 51. The image encoding method according to claim 50, wherein the processing of the generating step generates encoding control information for instructing not to change an image frame, a video format, or an aspect ratio. . 52. An input step of inputting a multiplexed stream, a separation step of separating a video stream from the multiplexed stream input by the processing of the input step, and the multiplexing input by the processing of the input step A judging step of judging whether or not the stream contains multimedia encoded data; and, if it is judged by the processing of the judging step that the multiplexed stream contains multimedia encoded data, the separation A generating step of generating encoding control information that instructs not to change a display format of the video stream separated by the processing of the step, based on the encoding control information generated by the processing of the generating step, Video streams separated by the processing of the separation step And a multiplexing step of generating a multiplexed stream including the video stream converted by the processing of the conversion step. Recording media. 53. Encoding control information instructing not to change the display format of a video stream, and a multiplexed stream including a video stream that has been subjected to a predetermined conversion based on the encoding control information. Recording medium characterized by the following.