JP2008054159A - Video-audio multiplexing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a multiplexing stream which does not make a video and an audio out of synch during a reproduction even if input timings of a video data and an audio data are asynchronous. <P>SOLUTION: A video-audio multiplexing generator is provided with video encoder 103 to generate an encoded video data by encoding the video data SIG101 in video frame unit, and an audio encoder 104 to generate an encoded audio data SIG104 by encoding the audio data SIG102 in audio frame unit. It is controlled by an operation controller 105 so that the video encoder 103 starts to encode at a timing of a video frame boundary of the video data SIG101, and it is controlled so that the audio encoder 104 starts to encode at a timing moved only by a time shown by an input time difference information SIG111 on the basis of an encoding starting timing of the video encoder 103. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video / audio multiplexing apparatus for encoding video data and audio data, and multiplexing encoded video data and encoded audio data.

  In recent years, information has been digitized, and signals such as video and audio have been digitized and used. Since there is a relationship between digital data such as video and audio, it is generally multiplexed as one stream data.

  In applications such as storage and transmission of digital data such as video and audio, there are multiplexing standards defined by international standards. For example, MPEG (Moving Pictures Experts Group) is defined by an international standard called MPEG (Moving Pictures Experts Group). There are ProgramStream (hereinafter abbreviated as PS), TransportStream (hereinafter abbreviated as TS), and the like.

  When there is a temporal correlation between video data and audio data to be multiplexed, PS and TS add reproduction time information called PTS to the video data and audio data to separate the multiplexed data. It has a mechanism that can restore temporal correlation.

  Also, digitized video data and audio data have a very large amount of information and are not suitable for use in storage or transmission as they are. In order to reduce the amount of information at the stage before multiplexing, processing for encoding video data and audio data is performed in accordance with an encoding standard such as MPEG2. Video data is generally a set of still image data (video frames) for each sample period. When video data is encoded, encoding is performed in units of video frames. On the other hand, audio data is generally a collection of sound intensity data for each sample period. However, since the sample period is very short, when audio data is encoded, data for a certain period larger than the sample period is used. The audio data is handled as a set (audio frame) of the above, and encoding is performed in units of audio frames.

  For example, a video / audio multiplexing apparatus that outputs MPEG-2 standard TS sequentially encodes input video data and audio data and transmits the generated TS at a predetermined output rate and output timing ( For example, see Patent Document 1).

Some video / audio multiplexing apparatuses are capable of temporarily stopping encoding. As an example of such a device, when receiving a pause instruction, the acquisition of the video signal and the audio signal is paused at the same time in synchronization with the timing of obtaining the video signal, and thereafter, each encoding, multiplexing, and recording are respectively performed. Is known that pauses at the timing (see, for example, Patent Document 2).
JP 2001-78195 A JP 2001-160968 A

  With the expansion of use of digitized video data and audio data, there is an inevitable demand for higher image quality of video data and higher sound quality of audio data. Along with this, image quality improvement processing and sound quality improvement processing may be performed on the encoded video data and audio data as a pre-stage of the encoding processing.

  In most cases, the processing time differs between the image quality enhancement processing for video data and the sound quality enhancement processing for audio data. Also, the processing time varies depending on the processing content. For this reason, there is a possibility that a time difference occurs between the input timing of the video data and the audio data to the video / audio multiplexing device due to the high image quality of the video data and the high quality of the audio data.

  However, since the conventional video / audio multiplexing apparatus performs processing such as encoding on the assumption that video data and audio data are input simultaneously, there is a time difference in the input timing of the video data and audio data. If so, the correct playback time information cannot be added, and the video and audio are shifted during playback, and as a result, the viewer may feel uncomfortable.

  The present invention has been made paying attention to the above-mentioned problem. Even if there is a difference in the input timing between the video data and the audio data, the video / audio multiplexing capable of generating a multiplexed stream in which the video and the audio do not shift at the time of reproduction. An object is to provide a device.

  In order to solve the above problems, in the present invention, the encoding start timing of video data and the encoding start timing of audio data are adjusted according to the amount of deviation between the input timing of video data and audio data. .

One embodiment of the present invention provides:
Video data and audio data having temporal correlation are inputted, and encoded video data obtained by encoding the video data and encoded audio data encoded by the audio data are generated, and the encoded video data And an audio / video multiplexing apparatus that multiplexes the encoded audio data and generates one multiplexed stream,
A video encoding unit that encodes the video data in units of video frames and generates the encoded video data;
An audio encoding unit that encodes the audio data in units of audio frames and generates the encoded audio data;
Input time difference information indicating a time difference between input timings of the video data and the audio data is input, and the video encoding unit starts encoding at a detection timing of a video frame boundary of the video data. And an operation control unit that controls the audio encoding unit to start encoding at a timing shifted by a time indicated by the input time difference information with reference to an encoding start timing of the video encoding unit. ,
It is provided with.

One embodiment of the present invention includes
Video data and audio data having temporal correlation are inputted, and encoded video data obtained by encoding the video data and encoded audio data encoded by the audio data are generated, and the encoded video data And an audio / video multiplexing apparatus that multiplexes the encoded audio data and generates one multiplexed stream,
A video encoding unit that encodes the video data in units of video frames and generates the encoded video data;
An audio encoding unit that encodes the audio data in units of audio frames and generates the encoded audio data;
Input time difference information indicating a time difference between input timings of the video data and the audio data is input, and the video encoding unit starts encoding at a detection timing of a video frame boundary of the video data. And controlling the audio encoding unit to start encoding at a timing shifted by a time of a video frame period with reference to the encoding start timing of the video encoding unit, and further, An operation control unit for outputting time difference information indicating a difference between a coding start timing shift time of the encoding unit and the audio encoding unit and the input time difference information;
A multiplexing unit for adding reproduction time information corresponding to the time difference indicated by the time difference information and multiplexing the encoded video data and the encoded audio data;
It is provided with.

  According to the present invention, the encoding start timing of the video data and the encoding start timing of the audio data are adjusted according to the shift amount of the input timing of the video data and the audio data. Even if there is a difference in input timing, it is possible to generate a multiplexed stream in which video and audio do not shift during playback.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description of each embodiment, components having the same functions as those described once are given the same reference numerals and description thereof is omitted.

Embodiment 1 of the Invention
FIG. 1 is a block diagram showing a configuration of a video / audio multiplexing apparatus 100 according to Embodiment 1 of the present invention. The video / audio multiplexing apparatus 100 encodes the input video data (video data SIG101) and audio data (audio data SIG102), and encodes the encoded video data (encoded video data SIG105). This is a device that multiplexes data (encoded audio data SIG106) into one stream (multiplexed stream SIG107).

[Configuration of Video / Audio Multiplexer 100]
As shown in FIG. 1, the video / audio multiplexing apparatus 100 includes a detection unit 101, a time measurement unit 102, a video encoding unit 103, an audio encoding unit 104, an operation control unit 105, a video data buffer 106, and audio data. A buffer 107 and a multiplexing unit 108 are provided.

(Detector 101)
The detection unit 101 detects a video frame boundary from the video data SIG101, and notifies the operation control unit 105 of the detection of the video frame boundary based on the video frame boundary information SIG112.

(Time measuring unit 102)
The time measurement unit 102 notifies the operation control unit 105 of time information (time information SIG116).

(Video encoding unit 103)
The video encoding unit 103 is input with video encoding control information SIG113 for controlling the start and pause of encoding, and encodes video data SIG101 according to the video encoding control information SIG113 to generate encoded video data. The SIG 103 is generated and output to the video data buffer 106. Further, every time encoding of one video frame is completed, the video encoding unit 103 notifies the multiplexing unit 108 of video frame information (video frame information SIG108) of the encoded video data SIG103. The video frame information SIG108 includes information on the size of the encoded video data SIG103, and, for example, if the encoded video data SIG103 is data in the MPEG format, the order information at the time of multiplexing is included.

(Audio encoding unit 104)
The audio encoding unit 104 receives the audio encoding control information SIG114 for controlling the start and pause of encoding, encodes the audio data SIG102 according to the audio encoding control information SIG114, and encodes the encoded audio data SIG104. Is generated and output to the audio data buffer 107. In addition, every time encoding of one audio frame is completed, the audio encoding unit 104 notifies the multiplexing unit 108 of audio frame information (audio frame information SIG109) of the encoded audio data SIG104. The audio frame information SIG109 includes information on the size of the encoded audio data SIG104.

(Operation control unit 105)
The operation control unit 105 receives control information SIG110 including information for controlling start and pause of encoding, and the video encoding unit 103, the audio encoding unit 104, and the multiplexing are controlled according to the control information SIG110. The operation of the conversion unit 108 is controlled. This control differs depending on which input of the audio data SIG102 and the video data SIG101 is delayed.

  When the audio data SIG102 is input later than the video data SIG101, the operation control unit 105 controls the start and pause of encoding and multiplexing as follows.

  When the control information SIG110 instructing the start of encoding is input, the operation control unit 105 outputs the multiplexing control information SIG115 instructing the start of multiplexing to the multiplexing unit 108 and starts the operation to the multiplexing unit 108 Let

  Then, the operation control unit 105 outputs video encoding control information SIG113 instructing the start of encoding at the timing (video data encoding start time) when the video frame boundary information SIG112 of the first video frame to be encoded is input. Then, the video encoding unit 103 starts the encoding operation.

  In addition, the operation control unit 105 receives the input time difference information SIG111 indicating the input time difference between the video data SIG101 and the audio data SIG102, and is delayed from the video data encoding start time by the input time difference indicated by the input time difference information SIG111. The timing is detected based on the time information SIG116, and the audio encoding control information SIG114 is output at the detected timing to cause the audio encoding unit 104 to start the encoding operation.

  When the control information SIG110 instructing the temporary stop of the encoding is input, the operation control unit 105 receives the input timing (video data encoding) of the first video frame boundary information SIG112 input after receiving the control information SIG110. Video encoding control information SIG113 for instructing to stop encoding is output at a pause time), and the encoding operation of the video encoding unit 103 is temporarily stopped. Furthermore, the operation control unit 105 detects a timing delayed by the time difference indicated by the input time difference information SIG111 based on the video data encoding pause time based on the time information SIG116, and pauses the encoding at the detected timing. Audio encoding control information SIG114 is output, and the encoding operation of the audio encoding unit 104 is temporarily stopped.

  On the other hand, when the video data SIG101 is input later than the audio data SIG102, the operation control unit 105 controls the start and pause of encoding and multiplexing as follows.

  When the control information SIG110 instructing the start of encoding is input, the operation control unit 105 receives the time difference Δt indicated by the input time difference information SIG111 (when the video data SIG101 is delayed) from the video frame boundary of the first video frame to be encoded. The speech encoding control information SIG114 is output at a timing earlier by (negative value).

  Specifically, the operation control unit 105 sets t + (video frame period) − (| Δt |% (video frame period)) as a timing (audio data encoding start time) for outputting the audio encoding control information SIG114. calculate. However, t is the detection time of the video frame boundary detected first after the control information SIG110 is input. Also, | Δt |% (video frame period) is the remainder when | Δt | is divided by (video frame period).

  Further, the operation control unit 105 notifies the video encoding unit 103 of video encoding control information SIG113 instructing the encoding start at a time represented by (audio data encoding start time) + | Δt |. To do.

  When the control information SIG110 instructing the temporary stop of the encoding is input, the operation control unit 105 determines a time (audio data encoding temporary) determined by t + (video frame period) − (| Δt |% (video frame period)). At the stop time), the audio encoding unit 104 is notified of the audio encoding control information SIG114 instructing the encoding pause. However, t is the detection time of the first video frame boundary detected after the control information SIG110 is input. Further, the operation control unit 105 notifies the video encoding unit 103 of video encoding control information SIG113 instructing encoding pause at a time represented by (audio data encoding pause time) + | Δt |. .

(Video data buffer 106 and audio data buffer 107)
The video data buffer 106 holds the encoded video data SIG 103 output from the video encoding unit 103.

  The capacity of the video data buffer 106 is a capacity required when there is no deviation in the input timing of the video data SIG101 and the audio data SIG102 (that is, for multiplexing of the encoded video data SIG105 and the encoded audio data SIG106). In addition to the capacity of the encoded video data SIG 105 that needs to be buffered), the video encoding unit 103 encoded the difference in the encoding start time difference between the video encoding unit 103 and the audio encoding unit 104. A capacity that can hold data is required.

  In the present embodiment, the timing at which the encoded video data SIG103 is multiplexed is delayed by the time difference (Δt) indicated by the input time difference information SIG111 when the audio data is delayed from the video data, as compared to when there is no delay. . That is, the retention period of the encoded video data SIG103 in the video data buffer 106 is increased by Δt. Therefore, the video data buffer 106 is encoded with ([Δt / (video frame period)] + 1) × (encoded in addition to the capacity required when there is no deviation in the input timing of the video data SIG101 and the audio data SIG102. It is necessary to have a capacity of a maximum size of one video frame of the video data. Note that the video data buffer 106 may have a configuration in which a necessary capacity is secured in advance, or may have a configuration in which a necessary capacity is secured in response to a start instruction from the outside.

  The audio data buffer 107 holds the encoded audio data SIG 104 output from the audio encoding unit 104.

  The capacity of the audio data buffer 107 is a capacity required when there is no difference in input timing between the video data SIG101 and the audio data SIG102 (that is, for multiplexing of the encoded video data SIG105 and the encoded audio data SIG106). In addition to the capacity of the encoded audio data SIG 106 that needs to be buffered), the audio encoding unit 104 encoded the time difference of the encoding start time between the video encoding unit 103 and the audio encoding unit 104. A capacity that can hold data is required.

  In the present embodiment, the timing at which the audio data SIG102 is multiplexed is delayed by the time difference (Δt) indicated by the input time difference information SIG111 when the video data is delayed from the audio data, compared to when there is no delay. That is, the holding period of the encoded audio data SIG104 in the audio data buffer 107 is increased by Δt. Therefore, the audio data buffer 107 adds ([| Δt | / (audio frame period)] + 1) × (sign in addition to the capacity required when there is no deviation in the input timing of the video data SIG101 and the audio data SIG102. It is necessary to have a capacity of (maximum size for one audio frame of the converted audio data). The audio data buffer 107 may have a configuration in which a necessary capacity is secured in advance, or may have a configuration in which a necessary capacity is secured in response to an external start instruction.

(Multiplexer 108)
Multiplexer 108 starts operating when multiplexing control information SIG 115 is input, and stores data in encoded data buffer 106 and audio data buffer 107 (encoded video data SIG 105 and encoded audio data). A stream (multiplexed stream SIG107) obtained by multiplexing data SIG106) is output. Specifically, the multiplexing unit 108 calculates the reproduction time (reproduction time information) of the encoded video data SIG105 and the encoded audio data SIG106 based on the video frame information SIG108 and the audio frame information SIG109, and the calculated reproduction time information. The multiplexing order is determined based on the above. Then, according to the determined multiplexing order, the encoded video data SIG 105 acquired from the video data buffer 106 and the encoded audio data SIG 106 acquired from the audio data buffer 107 are multiplexed with reproduction time information added. .

[Operation of Video / Audio Multiplexer 100]
(Operation when audio data is input at a timing delayed from video data)
The encoding start, pause, and encoding restart operations when audio data is input at a timing delayed from the video data will be described with reference to FIG. FIG. 2 is a diagram illustrating control timings of the operation control unit 105, the video encoding unit 103, and the audio encoding unit 104.

  In order to cause the video / audio multiplexing apparatus 100 to perform multiplexing, for example, control information SIG110 and input time difference information SIG111 are input to the operation control unit 105 from the outside. Here, it is assumed that operation control unit 105 receives control information SIG110 and input time difference information SIG111 at start instruction time t200 shown in FIG. Also, the time difference indicated by the input time difference information SIG111 is Δt2 (a positive value when the audio data is delayed).

  When the control information SIG110 and the input time difference information SIG111 are input, the operation control unit 105 transmits the video frame boundary to the video encoding unit 103 for the first time after the start instruction time t200 is detected. Video encoding control information SIG113 is output. Accordingly, the video encoding unit 103 outputs the encoded video data SIG103 obtained by encoding the video data SIG101 to the video data buffer 106.

  Furthermore, the operation control unit 105 detects the timing at the time t202 that is delayed by Δt2 from the time t201 (video data encoding start time) based on the time information SIG116, and detects the timing at the detected timing with respect to the audio encoding unit 104. The speech encoding control information SIG114 is output. As a result, the audio encoding unit 104 outputs the encoded audio data SIG104 obtained by encoding the audio data SIG102 to the audio data buffer 107.

  Further, when the control information SIG 110 is input, the operation control unit 105 outputs the multiplexing control information SIG 115 to the multiplexing unit 108. Thereby, the multiplexing unit 108 starts the multiplexing operation. Specifically, the multiplexing unit 108 calculates the reproduction time information of the encoded video data SIG105 and the encoded audio data SIG106 based on the video frame information SIG108 and the audio frame information SIG109, and determines the multiplexing order. Then, in accordance with the determined multiplexing order, the multiplexing unit 108 converts the encoded video data SIG 105 acquired from the video data buffer 106 and the encoded audio data SIG 106 acquired from the audio data buffer 107 into reproduction time information. Add and multiplex.

  FIG. 3 shows reproduction time information added to each video frame and audio frame when audio data is delayed. As described above, the multiplexing unit 108 calculates the playback time information based on the video frame information SIG108 and the audio frame information SIG109, so that the playback time information of the video frame v201 (first video frame) and the audio frame are calculated. The playback time information of a201 (first audio frame) indicates the same playback time (playback time information t300) as shown in FIG.

(Pause and resume encoding)
Further, for example, when the operation control unit 105 receives a pause instruction at the pause instruction time t203 shown in FIG. 2, the time when the video frame boundary is first detected after the pause instruction time t203. At t <b> 204, the operation control unit 105 notifies the video encoding unit 103 of video encoding control information SIG <b> 113 that instructs to pause. Further, the operation control unit 105 detects the timing of the time t205 delayed by Δt2 from the time t204 by the time information SIG116, and the audio encoding control information SIG114 instructing the suspension at the detected timing. Output to.

  Also, for example, when the operation control unit 105 receives an instruction to start (resume) encoding at the restart instruction time t206, at the time t207 when the video frame boundary is first detected after the restart instruction time t206. The operation control unit 105 notifies the video encoding unit 103 of video encoding control information SIG113 instructing the start of encoding. Further, the operation control unit 105 detects the timing at time t208, which is delayed by Δt2 from the time t207, from the time information SIG116, and at the detected timing, the audio control information SIG114 instructing the start (resumption) of encoding is audio. The data is output to the encoding unit 104. Thereby, the video frame v204 and the video frame v205 before and after the pause and restart are seamlessly connected in the reproduction time information t301 as shown in FIG. Also, the audio frame a207 extending over the timing of pause and restart is seamlessly connected in the playback time information t301 as shown in FIG.

(Operation when video data is input at a timing delayed from audio data)
Next, operations of encoding start, pause, and encoding restart when video data is input at a timing delayed from audio data will be described with reference to FIG. FIG. 4 is a diagram illustrating control timings of the operation control unit 105, the video encoding unit 103, and the audio encoding unit 104.

  In order to cause the video / audio multiplexing apparatus 100 to perform multiplexing, first, for example, control information SIG110 and input time difference information SIG111 are input to the operation control unit 105 from the outside. Here, it is assumed that operation control unit 105 receives control information SIG110 and input time difference information SIG111 at start instruction time t400 shown in FIG. Further, the time difference indicated by the input time difference information SIG111 is assumed to be Δt4 (a negative value when the video data is delayed). Also, the detection time of the video frame boundary detected first after the start instruction time t400 is assumed to be time t401.

  When the control information SIG110 is input, the operation control unit 105 obtains a time t402 (audio data encoding start time) represented by a time t401 + (video frame period) − (| Δt4 |% (video frame period)). At time t402, the audio encoding unit 104 is notified of the audio encoding control information SIG114 that instructs to start encoding.

  In addition, the operation control unit 105 notifies the video encoding unit 103 of video encoding control information SIG113 instructing to start encoding at time t403 represented by time t402 + | Δt4 |.

  Further, when the control information SIG110 is input, the operation control unit 105 outputs the multiplexing control information SIG115 to the multiplexing unit 108, and starts the multiplexing operation of the multiplexing unit 108.

  As described above, the video data SIG101 and the audio data SIG102 are encoded and then multiplexed and output.

  FIG. 5 shows reproduction time information added to each video frame and audio frame when video data is delayed. The multiplexing unit 108 calculates the reproduction time information based on the video frame information SIG108 and the audio frame information SIG109, so that the reproduction time information of the video frame v401 (first video frame) and the audio frame a401 (first audio frame) are calculated. As shown in FIG. 5, the reproduction time information of (frame) indicates the same reproduction time (reproduction time information t500).

(Pause and resume encoding)
Also, for example, when the operation control unit 105 receives a pause instruction at the pause instruction time t404 shown in FIG. 4, the first video frame boundary after the pause instruction time t404 is time t405. Then, at time t406 represented by time t405 + (video frame period) − (| Δt4 |% (video frame period)), audio encoding that instructs the audio encoding unit 104 to pause encoding is performed. Control information SIG114 is notified.

  Also, at time t407 represented by time t406 + | Δt4 |, video encoding control information SIG113 instructing the temporary stop of encoding is notified to the video encoding unit 103.

  Further, when the operation control unit 105 receives an instruction to start (resume) encoding at the restart instruction time t408, if the first video frame boundary after the restart instruction time t408 is the time t409, At time t410 expressed by t409 + (video frame period) − (| Δt4 |% (video frame period)), audio encoding control information SIG114 instructing the start of encoding is notified to the audio encoding unit 104. . Also, at time t411 represented by time t410 + | Δt4 |, video encoding control information SIG113 instructing to start (restart) encoding is notified to the video encoding unit 103.

  As described above, the video frame v405 and the video frame v406 before and after the pause and restart are seamlessly connected with the reproduction time information t501 as shown in FIG. Also, the audio frame a408 extending over the pause and restart timings is seamlessly connected with the playback time information t501 as shown in FIG.

  As described above, according to the present embodiment, the encoding start timing in the video encoding unit and the encoding start timing in the audio encoding unit are adjusted according to the shift amount of the input timing of the video data and the audio data. Therefore, even if the input timing of the video data and the audio data is deviated, it is possible to prevent the video and audio from deviating when the multiplexed stream is reproduced.

  In addition, after encoding video data and audio data, buffering is performed to absorb the difference in input timing, so that, for example, when the video data is improved in quality or the audio data is improved in quality. However, the capacity of the video data buffer and the audio data buffer can be kept within the feasible range.

<< Embodiment 2 of the Invention >>
FIG. 6 is a block diagram showing a configuration of a video / audio multiplexing apparatus 200 according to Embodiment 2 of the present invention. As shown in the figure, the video / audio multiplexing apparatus 200 includes a detection unit 101, a video encoding unit 103, an audio encoding unit 104, a video data buffer 106, an audio data buffer 107, an operation control unit 201, and a multiplexing unit. The conversion unit 202 is provided.

(Operation control unit 201)
The operation control unit 201 receives control information SIG110 including information for controlling the start and pause of encoding from the outside of the video / audio multiplexing apparatus 200, and the video encoding unit 103 according to the control information SIG110. In addition, the control of the encoding operation of the audio encoding unit 104 and the multiplexing operation of the multiplexing unit 202 are controlled. This control differs depending on which input of the audio data SIG102 and the video data SIG101 is delayed.

  When the audio data SIG102 is input later than the video data SIG101, the operation control unit 105 controls the start and pause of encoding and multiplexing as follows.

  When the control information SIG110 instructing the start of encoding is input, the operation control unit 201 performs encoding at the timing (video data encoding start time) when the video frame boundary information SIG112 of the first video frame to be encoded is input. Video encoding control information SIG113 instructing the start of encoding is output to cause the video encoding unit 103 to start the encoding operation.

  Further, the input time difference information SIG111 is input to the operation control unit 201, and when the time difference indicated by the input time difference information SIG111 is Δt (a positive value when audio data is delayed), the operation control unit 201 (Data encoding start time) After the detection time of the video frame boundary at the earliest timing after (+ t) (this time is detected by the video frame boundary information SIG112), the audio encoding unit 104 starts encoding. Instructed voice encoding control information SIG114 is notified.

  At the same time, the operation control unit 201 notifies the multiplexing unit 202 of the value of (video frame period) −Δt% (video frame period) as time difference information SIG201. In addition, the multiplexing unit 202 is notified of multiplexing control information SIG 115 that instructs to start multiplexing.

  When the control information SIG110 instructing the temporary stop of the encoding is input, the operation control unit 201 receives the input timing (video data encoding) of the first video frame boundary information SIG112 input after receiving the control information SIG110. Video encoding control information SIG113 instructing to pause encoding is output at a pause time), and the encoding operation of the video encoding unit 103 is temporarily stopped. The operation control unit 201 detects the audio frame boundary detection time (this time is detected by the video frame boundary information SIG112) after (video data encoding pause time) + Δt and at the earliest timing. The encoding unit 104 is notified of the audio encoding control information SIG114 for instructing a pause.

  On the other hand, when the video data SIG101 is input later than the audio data SIG102, the operation control unit 201 controls the start and pause of encoding and multiplexing as follows.

  The operation control unit 201 sets the audio encoding control information SIG114 instructing the start of encoding at the time when the video frame boundary is first detected after the time when the control information SIG110 is input (audio data encoding start time). This is notified to the audio encoding unit 104. If the time difference indicated by the input time difference information SIG111 is Δt (a negative value when the video data is delayed), the operation control unit 201 is smaller than (audio data encoding start time) + | Δt | and the latest timing At the detection time of the video frame boundary, video encoding control information SIG113 instructing the start of encoding is notified to the video encoding unit 103. At the same time, the operation control unit 201 notifies the multiplexing unit 202 of the value of | Δt |% (video frame period) as time difference information SIG201. Further, the multiplexing unit 202 is notified of multiplexing control information SIG 115 that instructs to start multiplexing.

  When control information SIG110 instructing to pause encoding is input, operation control unit 201 first detects the time and time when the video frame boundary is detected after receiving control information SIG110 (audio data encoding temporary). At the stop time), the audio encoding unit 104 is notified of the audio encoding control information SIG114 instructing the temporary stop of encoding. Further, the operation control unit 201 provides video encoding control information for instructing a pause at the detection time of the video frame boundary at the latest timing before (audio data encoding pause time) + | Δt |. SIG 113 is notified to the video encoding unit 103.

(Multiplexer 202)
The multiplexing unit 202 starts operating when the multiplexing control information SIG115 is input, and based on the video frame information SIG108, the audio frame information SIG109, and the time difference information SIG201, the multiplexed video data SIG105 and the encoded audio data SIG106. Is calculated. Specifically, the multiplexing unit 202 uses the value calculated from the notified number of times of the video frame information SIG108 with the time interval of one video frame as the reproduction time information of the video frame, and the audio with the time interval of one audio frame. A value indicated by the time difference information SIG201 is added to a value calculated from the notified number of times of the frame information SIG109 to obtain reproduction time information of the audio frame.

  Then, the operation control unit 201 determines the multiplexing order based on the calculated reproduction time information, and in accordance with the determined multiplexing order, the encoded video data SIG 105 acquired from the video data buffer 106 and the audio data buffer The encoded audio data SIG 106 acquired from 107 is multiplexed with reproduction time information.

(Operation of Video / Audio Multiplexer 200)
(Operation when audio data is input at a timing delayed from video data)
The encoding start, pause, and encoding restart operations when audio data is input at a timing delayed from the video data will be described with reference to FIG. FIG. 7 is a diagram illustrating control timings of the operation control unit 201, the video encoding unit 103, and the audio encoding unit 104.

  In order to cause the video / audio multiplexing apparatus 200 to perform multiplexing, first, for example, control information SIG110 and input time difference information SIG111 are input to the operation control unit 201 from the outside. Here, it is assumed that the operation control unit 201 receives the control information SIG110 and the input time difference information SIG111 at the start instruction time t700 illustrated in FIG. Also, the time difference indicated by the input time difference information SIG111 is Δt7 (a positive value when the audio data is delayed).

  When the control information SIG110 and the input time difference information SIG111 are input, the operation control unit 201 detects the video at time t701 (video data encoding start time) when the video frame boundary is first detected after the start instruction time t700. Video encoding control information SIG 113 is output to encoding section 103. Accordingly, the video encoding unit 103 outputs the encoded video data SIG103 to the video data buffer 106.

  Further, the motion control unit 201 instructs the audio encoding unit 104 to start encoding at the earliest detection time of the video frame boundary (time t702) after the time t701 + Δt7. SIG 114 is notified. As a result, the audio encoding unit 104 outputs the encoded audio data SIG 104 to the audio data buffer 107.

  In addition, the operation control unit 201 outputs the multiplexing control information SIG 115 and the time difference information SIG 201 to the multiplexing unit 202. The value of the time difference information SIG201 is (video frame period) −Δt7% (video frame period). At this time, the time lag between the video data and the audio data captured at the start of encoding by the video encoding unit 103 and the audio encoding unit 104 is the same as the time difference information SIG201.

  Thereby, the multiplexing unit 202 calculates the reproduction time information of the encoded video data SIG105 based on the video frame information SIG108. Further, based on the values of the audio frame information SIG109 and the time difference information SIG201, the reproduction time information of the encoded audio / video data SIG106 is calculated. Then, the multiplexing unit 202 determines the multiplexing order of the encoded video data SIG105 and the encoded audio data SIG106, and the encoded video data SIG105 acquired from the video data buffer 106 according to the determined multiplexing order, and the audio The encoded audio data SIG 106 acquired from the data buffer 107 is multiplexed with reproduction time information added.

  FIG. 8 shows reproduction time information added to each video frame and audio frame when audio data is delayed. As shown in FIG. 8, assuming that the playback time information of the video frame v701 (first video frame) is the playback time information t800, the playback time information t801 of the audio frame a701 (first audio frame) is the playback time information t800 + (video Frame period) −Δt 7% (video frame period). That is, the difference in input timing between the video frame v701 and the audio frame a701 is equal to the difference in reproduction time information to be added. During the period from t801 to t800 at the time of reproduction, there is a silent state.

  When the audio data is input at a timing delayed from the video data, the encoded video data holding period in the video data buffer 106 is (time t702-time t701) as described above. Only gets longer. Therefore, in the present embodiment, the video data buffer 106 has the capacity (([(time t702−time t701) / (video) in addition to the capacity required when the input timing of the video data SIG101 and the audio data SIG102 is not shifted). It is necessary to have a capacity of (frame period)] + 1) × (maximum size for one video frame of encoded video data).

(Pause and resume encoding)
For example, when the operation control unit 201 receives a pause instruction at the pause instruction time t703 shown in FIG. 7, at the time t704 when the video frame boundary is first detected after the pause instruction time t703. The operation control unit 201 notifies the video encoding unit 103 of video encoding control information SIG113 instructing a pause. Furthermore, the operation control unit 201 outputs the audio encoding control information SIG114 instructing the pause to the audio encoding unit 104 after the time t704 + Δt7 and at the earliest timing video frame boundary (time t705). .

  When the operation control unit 201 receives an instruction to start (resume) encoding at the restart instruction time t706, the operation control unit 201 operates at the time t707 when the video frame boundary is first detected after the restart instruction time t706. The control unit 201 notifies the video encoding unit 103 of video encoding control information SIG113 that instructs the start of encoding. Further, audio encoding control information SIG 114 instructing to start (restart) encoding is output to the audio encoding unit 104 after the time t707 + Δt7 and at the earliest timing video frame boundary (time t708).

  Therefore, as shown in FIG. 8, the video frame v704 and the video frame v705 before and after the pause and restart are seamlessly connected with the playback time information t802. The audio frame a708 extending over the pause and restart timings is seamlessly connected at time t803 when the playback time information t802 + (video frame period) −Δt7% (video frame period). Also in this case, the difference between the reproduction time information t802 and the reproduction time information t803 is one video frame period or less, and does not affect the viewing.

(Operation when video data is input at a timing delayed from audio data)
Next, encoding start, pause, and encoding restart operations when video data is input at a timing delayed from audio data will be described with reference to FIG. FIG. 9 is a diagram illustrating control timings of the operation control unit 201, the video encoding unit 103, and the audio encoding unit 104.

  Here, it is assumed that the operation control unit 201 receives the control information SIG110 and the input time difference information SIG111 at the start instruction time t900 shown in FIG. Further, the time difference indicated by the input time difference information SIG111 is assumed to be Δt9 (a negative value when the video data is delayed). Also, the detection time of the video frame boundary detected first after the start instruction time t900 is assumed to be time t901.

  When the control information SIG110 is input, the operation control unit 201 notifies the audio encoding unit 104 of the audio encoding control information SIG114 instructing the start of encoding at time t901.

  Further, the operation control unit 201 instructs the video encoding unit 103 to start encoding at the detection time (time t902) of the video frame boundary at the latest timing before the time t901 + | Δt9 |. Video encoding control information SIG113 is notified.

  Further, | Δt9 |% (video frame period) is notified to the multiplexing unit 202 as time difference information SIG201 at the same time as the multiplexing control information SIG115. At this time, the time lag between the video data and the audio data captured at the start of encoding by the video encoding unit 103 and the audio encoding unit 104 is the same as the time difference information SIG201.

  Thereby, the multiplexing unit 202 calculates the reproduction time information of the encoded video data SIG105 based on the video frame information SIG108. Further, based on the values of the audio frame information SIG109 and the time difference information SIG201, the reproduction time information of the encoded audio / video data SIG106 is calculated. Then, the multiplexing unit 202 determines the multiplexing order of the encoded video data SIG105 and the encoded audio data SIG106, and the encoded video data SIG105 acquired from the video data buffer 106 according to the determined multiplexing order, and the audio The encoded audio data SIG 106 acquired from the data buffer 107 is multiplexed with reproduction time information added.

  FIG. 10 shows reproduction time information added to each video frame and audio frame when video data is delayed. As shown in FIG. 10, when the playback time information of the video frame v901 (first video frame) is the playback time information t1000, the playback time information t1001 of the audio frame a901 (first audio frame) is the playback time information t1000 + | Δt9. |% (Video frame period). That is, the difference in input timing between the video frame v901 and the audio frame a901 is equal to the difference in reproduction time information to be added. During the period from t1001 to t1000 at the time of reproduction, there is no sound, but since it is less than one video frame period, there is no influence on viewing.

  When the audio / video data is input at a timing delayed from the audio data, the encoded audio data holding period in the audio data buffer 107 is (time t902−time t901) as described above. ). Therefore, in the present embodiment, the audio data buffer 107 has ([| time t902−time t901 | / (audio) in addition to the capacity required when there is no deviation in the input timing of the video data SIG101 and the audio data SIG102. It is necessary to have a capacity of (frame period)] + 1) × (maximum size for one audio frame of encoded audio data).

(Pause and resume encoding)
Also, for example, when the operation control unit 201 receives a pause instruction at the pause instruction time t904 illustrated in FIG. 9, the operation control unit 201 starts the video frame first after the pause instruction time t904. At time t905 when the boundary is detected, the audio encoding control information SIG114 that instructs the audio encoding unit 104 to pause is notified. Further, the video encoding control information SIG113 instructing the suspension is notified to the video encoding unit 103 at time t905 before the time t905 + | Δt9 | and at the latest video frame boundary.

  For example, when the operation control unit 201 receives an instruction to start (resume) encoding at the restart instruction time t908, the operation control unit 201 sets the video frame boundary first after the restart instruction time t908. At the detected time t909, the audio encoding unit 104 is notified of the audio encoding control information SIG114 instructing to start (restart) encoding. Also, the video encoding control information SIG113 instructing the video encoding unit 103 to start (resume) encoding at time t910 before the time t909 + | Δt9 | and at the latest video frame boundary. To be notified.

  Therefore, as shown in FIG. 10, the video frame v905 and the video frame v906 before and after the pause and restart are seamlessly connected with the playback time information t1002. Also, the audio frame a907 extending over the pause and restart timings is seamlessly connected with playback time information t1003 that is playback time information t1002 + | Δt9 |% (video frame period). Also in this case, the difference between the reproduction time information t1002 and the reproduction time information t1003 is not more than one video frame period and does not affect the viewing.

  As described above, according to the present embodiment, the encoding start timing in the video encoding unit and the encoding start timing in the audio encoding unit are set according to the shift amount of the input timing of the video data and the audio data. Adjust by video frame period. In addition, a deviation that cannot be adjusted in units of video frame periods is adjusted by changing reproduction time information added to the multiplexed stream. Therefore, even if there is a difference between the input timings of the video data and the audio data, it is possible to prevent the video and the audio from shifting when the multiplexed stream is played back.

  In addition, in the present embodiment, unlike the video / audio multiplexing apparatus 100 of the first embodiment, the time measuring unit 102 is not required, so that it can be configured on a smaller scale.

  In addition, each component of Embodiment 1 and Embodiment 2 may be implement | achieved by hardware, and may be implement | achieved by operating a central processing unit (CPU) by software.

  In the video / audio multiplexing apparatus according to the present invention, the video data encoding start timing and the audio data encoding start timing are adjusted in accordance with the shift amount of the video data and audio data input timing. Even if there is a discrepancy between the input timing of data and audio data, it has the effect of being able to generate a multiplexed stream in which video and audio are not misaligned during playback, and video data and audio data are encoded and encoded. The present invention is useful as a video / audio multiplexing apparatus that multiplexes video data and encoded audio data.

1 is a block diagram showing a configuration of a video / audio multiplexing apparatus 100 according to Embodiment 1 of the present invention. FIG. 10 is a diagram illustrating control timings of the operation control unit 105, the video encoding unit 103, and the audio encoding unit 104 when audio data is delayed in the first embodiment. In Embodiment 1, it is a figure which shows the reproduction time information added to each video frame and audio frame, when audio | voice data are overdue. FIG. 6 is a diagram illustrating control timings of the operation control unit 105, the video encoding unit 103, and the audio encoding unit 104 when video data is delayed in the first embodiment. In Embodiment 1, it is a figure which shows the reproduction time information added to each video frame and audio frame when image data is delayed. It is a block diagram which shows the structure of the video / audio multiplexing apparatus 200 which concerns on Embodiment 2 of this invention. In Embodiment 2, it is a figure which shows the control timing of the operation control part 201, the video encoding part 103, and the audio encoding part 104 when audio | voice data are overdue. In Embodiment 2, it is a figure which shows the reproduction time information added to each video frame and audio frame, when audio | voice data are overdue. In Embodiment 2, it is a figure which shows the control timing of the operation control part 201, the video encoding part 103, and the audio encoding part 104 when video data is overdue. FIG. 10 is a diagram illustrating reproduction time information added to each video frame and audio frame when video data is delayed in the second embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Video / Audio Multiplexer 101 Detection Unit 102 Time Measurement Unit 103 Video Coding Unit 104 Audio Coding Unit 105 Operation Control Unit 106 Video Data Buffer 107 Audio Data Buffer 108 Multiplexing Unit 200 Video / Audio Multiplexing Device 201 Operation Control Unit 202 Multiplexing unit

Claims (8)

Video data and audio data having temporal correlation are inputted, and encoded video data obtained by encoding the video data and encoded audio data encoded by the audio data are generated, and the encoded video data And an audio / video multiplexing apparatus that multiplexes the encoded audio data and generates one multiplexed stream,
A video encoding unit that encodes the video data in units of video frames and generates the encoded video data;
An audio encoding unit that encodes the audio data in units of audio frames and generates the encoded audio data;
Input time difference information indicating a time difference between input timings of the video data and the audio data is input, and the video encoding unit starts encoding at a detection timing of a video frame boundary of the video data. And an operation control unit that controls the audio encoding unit to start encoding at a timing shifted by a time indicated by the input time difference information with reference to an encoding start timing of the video encoding unit. ,
A video / audio multiplexing apparatus comprising: Video data and audio data having temporal correlation are inputted, and encoded video data obtained by encoding the video data and encoded audio data encoded by the audio data are generated, and the encoded video data And an audio / video multiplexing apparatus that multiplexes the encoded audio data and generates one multiplexed stream,
A video encoding unit that encodes the video data in units of video frames and generates the encoded video data;
An audio encoding unit that encodes the audio data in units of audio frames and generates the encoded audio data;
Input time difference information indicating a time difference between input timings of the video data and the audio data is input, and the video encoding unit starts encoding at a detection timing of a video frame boundary of the video data. And controlling the audio encoding unit to start encoding at a timing shifted by a time of a video frame period with reference to the encoding start timing of the video encoding unit, and further, An operation control unit for outputting time difference information indicating a difference between a coding start timing shift time of the encoding unit and the audio encoding unit and the input time difference information;
A multiplexing unit for adding reproduction time information corresponding to the time difference indicated by the time difference information and multiplexing the encoded video data and the encoded audio data;
A video / audio multiplexing apparatus comprising: The video / audio multiplexing apparatus according to any one of claims 1 and 2,
When the operation control unit temporarily stops the encoding of the video encoding unit and the audio encoding unit, the operation control unit has the same time as the time difference between the encoding start timings of the video encoding unit and the audio encoding unit. A video / audio multiplexing apparatus characterized by temporarily stopping the encoding of the video encoding unit and the audio encoding unit at a timing shifted by a certain amount. The video / audio multiplexing apparatus according to any one of claims 1 and 2, further comprising:
A video data buffer for holding the output of the video encoder;
An audio data buffer that holds the output of the audio encoding unit;
The video data buffer includes, in addition to the capacity of the encoded video data that needs to be buffered for the multiplexing, a time difference between encoding start timings of the video encoding unit and the audio encoding unit Having a capacity capable of holding the encoded video data generated by the video encoding unit during
The audio data buffer has a difference in encoding start timing between the video encoding unit and the audio encoding unit in addition to the capacity of the encoded audio data that needs to be buffered for the multiplexing. A video / audio multiplexing apparatus having a capacity capable of holding the encoded audio data generated by the audio encoding unit during The video / audio multiplexing apparatus according to claim 1, wherein
When the input timing of the audio data is later than the input timing of the video data, the operation control unit is delayed by the time indicated by the input time difference information with reference to the encoding start timing of the video encoding unit. A video / audio multiplexing apparatus, wherein the audio encoding unit controls to start encoding at a timing. The video / audio multiplexing apparatus according to claim 1, wherein
When the input timing of the video data is later than the input timing of the audio data, the operation control unit is a timing earlier by the time indicated by the input time difference information with reference to the encoding start timing of the video encoding unit. And a video / audio multiplexing apparatus in which the audio encoding unit is controlled to start encoding. The video / audio multiplexing apparatus according to claim 2, wherein
When the input timing of the audio data is later than the input timing of the video data, the operation control unit is delayed by a time of a video frame period with reference to the encoding start timing of the video encoding unit. A video / audio multiplexing apparatus, wherein the audio encoding unit controls to start encoding at a timing. The video / audio multiplexing apparatus according to claim 2, wherein
When the input timing of the video data is later than the input timing of the audio data, the operation control unit is a timing earlier by a time in units of a video frame period based on the encoding start timing of the video encoding unit. In the video / audio multiplexing apparatus, the audio encoding unit is controlled to start encoding.

Images