JP2012023576A - Image/voice conversion apparatus and image/voice conversion method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten processing time by highly-efficient and parallel operation of a separation unit, an image decoding unit, a voice decoding unit, an image coding unit, a voice coding unit, and a multiplexing unit in rate conversion copy.SOLUTION: A separation unit 101, an image decoding unit 102, a voice decoding unit 103, an image coding unit 104, a voice coding unit 105, and a multiplexing unit 106 control the stop and restart of processing while monitoring the processing states of the preceding and following processing units. This enables each processing unit to operate at the best processing speed, which shortens the processing time of the rate conversion copy.

Description

  The present invention relates to a video / audio conversion apparatus and method for receiving rate-converted dubbing at high speed by receiving, as input, first multiplexed data in which first encoded video data and first encoded audio data are multiplexed. is there.

  In a recording / reproducing apparatus that performs general recording / reproduction such as a conventional HDD (Hard Disk Drive) / DVD (Digital Versatile Disc) recorder, video / audio data having a size larger than the recording capacity of the DVD recorded in the HDD is recorded on a DVD. There is one having a rate conversion dubbing function for performing rate conversion for converting to a size that fits in a DVD and recording on a DVD.

  In general, rate conversion dubbing is performed by decoding video / audio data compressed by MPEG (Moving Picture Experts Group), etc., recorded on an HDD, etc., and performing display control of the decoded video / audio data. The video / audio data is encoded and recorded on a DVD or the like.

  However, in the above control, since decoding and encoding involve rate conversion dubbing in synchronization with the output of the television, dubbing time as much as the display time is required.

  As a method for solving this problem and realizing high-speed rate conversion dubbing, for example, there is a rate conversion dubbing method related to video / audio data described in Patent Document 1.

  In the rate conversion dubbing described in Patent Document 1, first, the video / audio decoding unit starts decoding video / audio data of a predetermined unit. When the decoding is completed, a decoding completion notification is sent from the video / audio decoding unit to the video / audio encoding unit. The video / audio encoding unit that has received the completion notification immediately starts encoding the decoded video / audio data (in the decoding process, the video / audio data is re-encoded at a compression rate higher than the compression rate before conversion). Compression). When the encoding is completed, an encoding completion notification is sent from the video / audio encoding unit to the video / audio decoding unit. The video / audio decoding unit that has received the completion notice immediately starts decoding the next video / audio data. By performing such control, it is not necessary to synchronize the processing of the video / audio decoding unit and the video / audio encoding unit with the output of the television, so that the rate conversion dubbing processing time can be shortened.

International Publication No. 08/023763

  However, in the rate conversion technique described in Patent Document 1 described above, rate conversion dubbing is performed while switching between the video / audio decoding unit and the video / audio encoding unit by sequential processing. As a result, the video / audio decoding unit and the video / audio encoding unit each have a stop period, so that the processing efficiency is low and the processing time for rate conversion dubbing is long.

  In order to solve the above-described problems, the present invention provides a video / audio that can shorten the processing time by the video / audio decoding unit and the video / audio encoding unit operating in parallel with high efficiency during rate conversion dubbing for video / audio data. It is an object to provide a conversion device and method.

  In order to achieve the above object, a video / audio conversion device according to an aspect of the present invention provides a second code from first multiplexed data in which first encoded video data and first encoded audio data are multiplexed. A video / audio conversion device for generating second multiplexed data in which encoded video data and second encoded audio data are multiplexed, wherein the first encoded video data and the first encoding are generated from the first multiplexed data A demultiplexer for demultiplexing audio data, a video decoder for generating video data by decoding the first encoded video data demultiplexed by the demultiplexer, and a demultiplexer by the demultiplexer By decoding the first encoded audio data, an audio decoding unit that generates audio data and a second encoded video data are generated by encoding the video data generated by the video decoding unit. And a video encoding unit An audio encoding unit that generates second encoded audio data by encoding audio data generated by the audio decoding unit, a second encoded video data generated by the video encoding unit, and the A multiplexing unit that generates second multiplexed data by multiplexing the second encoded audio data generated by the audio encoding unit, and the separation unit includes the video decoding unit and the audio Based on the progress of decoding in the decoding unit, the demultiplexing unit is controlled to stop and restart the demultiplexing, and the video decoding unit performs the demultiplexing progress in the demultiplexing unit and the video code The audio decoding unit controls the stop and restart of decoding in the video decoding unit based on the progress of encoding in the encoding unit, and the audio decoding unit determines the progress of demultiplexing in the demultiplexing unit and the audio Based on the progress of encoding in the encoder Controlling the stop and restart of decoding in the audio decoding unit, the video encoding unit in accordance with the progress of decoding in the video decoding unit and the progress of multiplexing in the multiplexing unit Based on this, the stop and restart of encoding in the video encoding unit is controlled, and the audio encoding unit is configured to perform decoding progress in the audio decoding unit and multiplexing progress in the multiplexing unit. Based on the progress of encoding in the video encoding unit and the audio encoding unit, the multiplexing unit controls the stop and restart of encoding in the audio encoding unit Controls the stop and restart of multiplexing in the conversion unit.

  According to this configuration, in order to absorb the difference in processing speed between the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit, each processing unit Based on the progress of the process, the stop and restart of its own process is controlled. Accordingly, high-speed rate conversion dubbing can be performed by the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit operating at the optimum processing speed.

  In addition, the demultiplexing unit further notifies the multiplexing unit of time information added to the first multiplexed data, and the multiplexing unit uses the time information notified from the demultiplexing unit. The second encoded video data and the second encoded audio data may be multiplexed to generate the second multiplexed data.

  The separation unit monitors a data accumulation state of a first intermediate buffer that temporarily stores the first encoded video data demultiplexed by the separation unit when the separation unit transfers the first encoded video data from the separation unit to the video decoding unit. By determining the progress of decoding by the video decoding unit, the first encoded audio data demultiplexed by the demultiplexing unit is temporarily transferred from the demultiplexing unit to the audio decoding unit. The progress of decoding in the audio decoding unit is determined by monitoring the data storage state of the second intermediate buffer to be stored, and the video decoding unit monitors the data storage state of the first intermediate buffer Thus, the progress of demultiplexing in the demultiplexing unit is determined, and the video data generated by the video decoding unit is temporarily stored when it is transferred from the video decoding unit to the video encoding unit. Intermediate buffer The progress of encoding in the video encoding unit is determined by monitoring the data storage status, and the audio decoding unit monitors the data storage status of the second intermediate buffer, Determining the progress of demultiplexing, the data storage status of the fourth intermediate buffer temporarily stored when transferring the voice data generated by the voice decoding unit from the voice decoding unit to the voice encoding unit The status of encoding in the audio encoding unit is determined by monitoring, and the video encoding unit decodes in the video decoding unit by monitoring the data storage status in the third intermediate buffer The fifth intermediate buffer stores data temporarily when the second encoded video data generated by the video encoding unit is transferred from the video encoding unit to the multiplexing unit. And determining the progress of multiplexing in the multiplexing unit, and the speech encoding unit monitors the data storage status of the fourth intermediate buffer to determine the progress of decoding in the speech decoding unit. Judging the situation, monitoring the data accumulation status of the sixth intermediate buffer that temporarily stores the second encoded audio data generated by the audio encoding unit when it is transferred from the audio encoding unit to the multiplexing unit Thus, the progress of multiplexing in the multiplexing unit is determined, and the multiplexing unit determines the progress of encoding in the video encoding unit by monitoring the data storage state of the fifth intermediate buffer. The progress of encoding in the speech encoding unit may be determined by monitoring the data storage state of the sixth intermediate buffer.

  The video / audio conversion device described above further transfers the video data generated by the video decoding unit to the video encoding unit and transfers the audio data generated by the audio decoding unit to the audio encoding unit. A synchronization control unit that adjusts a transfer timing; and the separation unit further notifies the synchronization control unit of time information added to the first multiplexed data, and the time information is stored in video data. Video output time information related to output and audio output time information related to output of audio data, and the synchronization control unit has reached that the reference time for counting up in proportion to the system reference time has reached the video output time information At the same time, the video data generated by the video decoding unit is transferred, and the reference time for counting up in proportion to the system reference time reaches the audio output time information. Thus, the audio data generated by the audio decoding unit is transferred to perform synchronization control of the video data and the audio data, and the video decoding unit performs the encoding progress in the video encoding unit. , Based on the progress status of the synchronization control in the synchronization control unit, control the stop and restart of the decoding in the video decoding unit, the audio decoding unit, the encoding of the audio encoding unit Based on the progress status of the synchronization control in the synchronization control unit, the stop and restart of decoding in the audio decoding unit is controlled as the progress status, and the video encoding unit is generated by the video decoding unit As the encoded video data, the synchronized video data is encoded by the synchronization control unit, and the progress of the decoding by the video decoding unit is set as the progress of the synchronization control by the synchronization control unit. Stop and restart encoding The speech encoding unit encodes speech data synchronized by the synchronization control unit as encoding of speech data generated by the speech decoding unit, and performs decoding by the speech decoding unit. As the progress of encoding, stop and restart of encoding may be controlled based on the progress of synchronization control in the synchronization control unit.

  According to this configuration, the synchronization control unit performs synchronization control between video data and audio data. Accordingly, high-speed rate conversion dubbing can be performed by the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit operating at the optimum processing speed.

  In addition, the separation unit monitors a data accumulation state of a seventh intermediate buffer that temporarily stores the first encoded video data demultiplexed by the separation unit when the first encoded video data is transferred from the separation unit to the video decoding unit. By determining the progress of decoding by the video decoding unit, the first encoded audio data demultiplexed by the demultiplexing unit is temporarily transferred from the demultiplexing unit to the audio decoding unit. The progress of decoding in the audio decoding unit is determined by monitoring the data storage state of the eighth intermediate buffer to be stored, and the video decoding unit monitors the data storage state of the seventh intermediate buffer Determining the progress of demultiplexing in the demultiplexing unit, and temporarily storing the video data generated by the video decoding unit when the video data is transferred from the video decoding unit to the synchronization control unit Buffer The progress of synchronization control in the synchronization control unit is determined by monitoring the data storage status, and the speech decoding unit monitors the data storage status of the eighth intermediate buffer. Determining the demultiplexing progress, and monitoring the data accumulation status of the tenth intermediate buffer temporarily stored when the speech data generated by the speech decoding unit is transferred from the speech decoding unit to the synchronization control unit By determining the progress of the synchronization control in the synchronization control unit, the video encoding unit transfers the video data synchronously controlled by the synchronization control unit from the synchronization control unit to the video encoding unit. The progress of synchronization control in the synchronization control unit is determined by monitoring the data accumulation status of the eleventh intermediate buffer temporarily stored in the second encoded video data generated by the video encoding unit. Monitoring the data accumulation status of the thirteenth intermediate buffer that is temporarily stored when the video encoding unit is transferred from the video encoding unit to the multiplexing unit, thereby determining the progress of multiplexing in the multiplexing unit, and The encoding unit monitors the data accumulation state of the twelfth intermediate buffer that temporarily stores the audio data synchronously controlled by the synchronization control unit when the audio data is transferred from the synchronization control unit to the audio encoding unit. The synchronization control unit determines a progress status of the synchronization control, and temporarily stores the second encoded audio data generated by the audio encoding unit when transferring from the audio encoding unit to the multiplexing unit. Determining the progress of multiplexing in the multiplexing unit by monitoring the data storage status of the fourteenth intermediate buffer, the multiplexing unit monitoring the data storage status of the thirteenth intermediate buffer, The progress of encoding in the speech encoding unit may be determined by determining the progress of encoding in the encoding unit and monitoring the data storage state of the fourteenth intermediate buffer.

  Further, the synchronization control unit performs synchronization control at a speed of N times, where N is the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and The minimum value of the double speed value of the conversion speed of the rate conversion dubbing capable of operating the multiplexing unit may be used.

  The video / audio conversion apparatus further includes a delay control unit that notifies the synchronization control unit of delay control, and a decoding delay in the video decoding unit with respect to the synchronization control in the synchronization control unit. A first delay detection unit that notifies the delay control unit of the delay status, and monitors the delay of decoding in the speech decoding unit with respect to the synchronization control in the synchronization control unit, and determines the delay status A second delay detection unit that notifies the delay control unit and a synchronization delay in the synchronization control unit are monitored for an encoding delay in the video encoding unit, and a delay state is notified to the delay control unit. A third delay detection unit; and a fourth delay detection unit that monitors a delay in encoding in the speech encoding unit with respect to the synchronization control in the synchronization control unit and notifies the delay control unit of a delay state. The first delay detector, the second delay detector, the third delay detector And each of the fourth delay detection units performs a delay notification to the delay control unit if a delay is detected, and a restart notification if the delay is detected, and the delay control unit includes the first delay detection unit. When a delay notification is received from any of the delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, a synchronization control stop notification is sent to the synchronization control unit as a delay control notification. When the restart notification is received from any one of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, the synchronization control unit performs synchronization control as a delay control notification. Notification of resumption may be performed.

  According to this configuration, each delay detection unit monitors the processing delay in each processing unit, and controls the processing in the synchronization control unit according to the monitoring result. Accordingly, high-speed rate conversion dubbing can be performed by the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit operating at the optimum processing speed.

  Further, the first delay detection unit is configured to store data in a ninth intermediate buffer that temporarily stores the video data generated by the video decoding unit when the video data is transferred from the video decoding unit to the synchronization control unit. And detecting a delay in decoding by the video decoding unit, and the second delay detecting unit transfers the audio data generated by the audio decoding unit from the audio decoding unit to the synchronization control unit. The third delay detection unit detects video data that is synchronously controlled by the synchronization control unit based on the data accumulation state of the tenth intermediate buffer temporarily stored in the audio decoding unit. Detecting a delay in encoding in the video encoding unit based on the data accumulation state of the eleventh intermediate buffer temporarily stored when transferring from the synchronization control unit to the video encoding unit, The four delay detection unit The speech encoding unit performs encoding based on the data accumulation state of the twelfth intermediate buffer temporarily stored when the speech data controlled by the unit is transferred from the synchronization control unit to the speech encoding unit. A delay may be detected.

  The first delay detector may be configured to monitor the number of frames of the video data generated by the video decoder and the number of frames of the video data subjected to synchronization control by the synchronization controller. The second delay detection unit monitors the number of frames of audio data generated by the audio decoding unit and the number of frames of audio data on which the synchronization control unit has performed synchronization control. The third delay detection unit detects the number of frames of video data that the synchronization control unit performs synchronization control and the second generated by the video encoding unit. By detecting the number of frames of the encoded video data, the delay of the encoding in the video encoding unit is detected, and the fourth delay detection unit is the number of frames of the audio data on which the synchronization control unit performs synchronization control And the voice It may detect the delay of coding in the audio coding unit by-coding unit monitors the number of frames of the second encoded audio data generated.

  Further, the delay control unit, when receiving a delay notification from any of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, performs synchronization control to the synchronization control unit. When a stop signal is notified and a restart notification is received from any of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, a synchronization control restart signal is sent to the synchronization control unit When the stop signal is received from the delay control unit, the synchronization control unit holds the state of the synchronization control and stops the synchronization control. When the stop signal is received from the delay control unit, the synchronization control unit holds Synchronization control may be resumed in this state.

  Further, the synchronization control unit performs synchronization control at a speed of N times, where N is the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and The minimum value of the double speed conversion rate of rate conversion dubbing operable by the multiplexing unit, and the delay control unit includes the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth When a delay notification is received from any one of the delay detection units, the N switching signal used in the synchronization control is notified to the synchronization control unit, and the first delay detection unit, the second delay detection unit, and the third delay detection When a restart notification is received from any one of the first delay detection unit and the fourth delay detection unit, a synchronization control restart signal is notified to the synchronization control unit, and the synchronization control unit receives a switching signal from the delay control unit and receives a predetermined value. The subtracted N is the double speed value of synchronous control And using, may be used the N and receives a restart signal from the delay controller as the speed value of the synchronous control.

  In addition, the video / audio conversion device described above may further include a double speed notification unit that notifies the synchronization control unit of the N.

  Further, the double speed notification unit may select the N to be notified according to an instruction from the outside.

  In addition, the video / audio conversion device described above further includes demultiplexing in the demultiplexing unit, decoding in the video decoding unit, decoding in the audio decoding unit, encoding in the video encoding unit, An operation mode determination unit that notifies the N-speed notification unit of the N according to an operation setting that affects encoding in the speech encoding unit or multiplexing in the multiplexing unit, the double-speed notification unit, The N notified from the operation mode determination unit may be notified to the synchronization control unit.

  With this configuration, optimum rate conversion dubbing is performed by selecting an N value at which the separation unit, video decoding unit, audio decoding unit, video encoding unit, audio encoding unit, and multiplexing unit operate optimally. Can do. For this reason, the video decoding unit, the audio decoding unit, the video encoding unit, and the audio encoding unit can take time for high image quality and high sound quality processing, and the second multiplexed data with high quality. Can be generated.

  In addition, the video / audio conversion device described above further includes a double speed measurement unit that performs the measurement of N before starting rate conversion dubbing and notifies the double speed notification unit of the measured N, wherein the double speed notification unit includes: The N notified from the double speed measuring unit may be notified to the synchronization control unit.

  With this configuration, the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit operate optimally without determining an optimal N value for each operation mode in advance. Rate conversion dubbing can be performed with N values.

  In addition, the video / audio conversion device described above further includes demultiplexing in the demultiplexing unit, decoding in the video decoding unit, decoding in the audio decoding unit, encoding in the video encoding unit, An operation mode determination unit that determines an operation mode according to an operation setting that affects the encoding in the speech encoding unit or the multiplexing in the multiplexing unit, and the double speed that measures the N before the start of rate conversion dubbing A measurement unit and a double speed recording memory for recording the measured N, and the operation mode determination unit records the N according to the operation mode when the N is recorded in the double speed recording memory. N is notified to the double speed notification unit, and when the N corresponding to the operation mode is not recorded in the double speed recording memory, the double speed measurement unit is notified of activation, and the double speed measurement unit Measure the optimal N , And records the N of each operating mode on the speed recording memory, it may notify the N to the speed notification unit.

  With this configuration, the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit operate optimally without determining an optimal N value for each operation mode in advance. Rate conversion dubbing can be performed with N values. Further, when operating in the same operation mode, it is possible to shorten the time required to start rate conversion dubbing.

  The video / audio conversion device described above may further include a double speed notification unit that switches N, which is a magnification of the speed for performing the synchronization control, during rate conversion dubbing and notifies the synchronization control unit.

  Further, the video / audio conversion device described above further calculates a value obtained by subtracting a predetermined value from N when the number of times of the timing adjustment of the synchronization control by the delay control unit is larger than the predetermined number in a predetermined period. The double speed measurement unit that notifies the double speed notification unit as N, and when the synchronization control timing adjustment does not occur, a value obtained by adding a predetermined value to N is used as the N to notify the double speed notification unit The delay control unit notifies the double speed measurement unit that synchronization control timing adjustment has occurred, and the double speed notification unit notifies the synchronous control unit of the N notified from the double speed measurement unit May be.

  With this configuration, even when the operation mode is switched, rate conversion dubbing is performed with N values at which the separation unit, video decoding unit, audio decoding unit, video encoding unit, audio encoding unit, and multiplexing unit operate optimally. be able to. Further, in accordance with the first multiplexed data input to the demultiplexing unit, the N value at which the demultiplexing unit, video decoding unit, audio decoding unit, video coding unit, audio coding unit, and multiplexing unit operate optimally Can perform rate conversion dubbing.

  Further, the video / audio conversion device described above further includes, according to the N, demultiplexing in the demultiplexing unit, decoding in the video decoding unit, decoding in the audio decoding unit, and in the video encoding unit. And a double speed determination unit that performs operation settings that affect the encoding in the speech encoding unit or the multiplexing in the multiplexing unit.

  Note that the present invention can be realized not only as a video / audio conversion device including such a characteristic processing unit, but also as a stepping process performed by the characteristic processing unit included in the video / audio conversion device. It can be realized as a voice conversion method. Also, it can be realized as a program for causing a computer to execute the characteristic steps included in the video / audio conversion method. Needless to say, such a program can be distributed via a computer-readable non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory) or a communication network such as the Internet.

  According to the present invention, adjustment control for absorbing a difference in processing speed between the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit is performed. By performing the demultiplexing unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit at high processing speed, high-speed rate conversion dubbing is performed. It can be performed.

1 is a block diagram showing a functional configuration of a video / audio conversion device according to Embodiment 1; FIG. 3 is a block diagram showing a functional configuration of a video / audio conversion device according to Embodiment 2. Timing chart of synchronous control in embodiment 2 Relationship diagram between reference time STC 'and VPTS and APTS at synchronization timing in the second embodiment FIG. 9 is a block diagram showing a functional configuration of a video / audio conversion device according to Embodiment 3. Timing diagram of delay control by stop and restart in Embodiment 3 Timing diagram of delay control by N value adjustment in embodiment 3 The figure which shows the pattern in which the audio | voice data in Embodiment 3 are not supplied to the said 10th intermediate | middle buffer FIG. 9 is a block diagram showing a functional configuration of a video / audio conversion device according to Embodiment 4; The figure which shows the double speed information table in Embodiment 4. FIG. 7 is a block diagram showing a functional configuration of a video / audio conversion device according to a fifth embodiment. Flowchart showing operation of double speed measurement control in the fifth embodiment FIG. 7 is a block diagram showing a functional configuration of a video / audio conversion device according to Embodiment 6; Flowchart showing the operation of double speed determination control in the sixth embodiment FIG. 9 is a block diagram showing a functional configuration of a video / audio conversion device according to Embodiment 7; Explanatory drawing of double speed switching control in the seventh embodiment Flowchart showing the operation of double speed switching control in the seventh embodiment FIG. 9 is a block diagram illustrating a functional configuration of a video / audio conversion device according to an eighth embodiment. The figure which shows the double speed information table in Embodiment 8.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a functional configuration of a video / audio conversion apparatus according to Embodiment 1 of the present invention.

  In the first embodiment, the HDD 001 and the DVD 002 are recording means, and the first multiplexed data compressed by, for example, MPEG of a television broadcast is recorded on the HDD 001. DVD002 is used for recording the second multiplexed data after rate conversion dubbing. The recording means may be a recording medium such as an HDD, a BD, or an SD card.

  The video / audio conversion apparatus includes a separation unit 101, a video decoding unit 102, an audio decoding unit 103, a video encoding unit 104, an audio encoding unit 105, and a multiplexing unit 106. Further, the separation unit 101 has a time information notification function (SIG 201) to the multiplexing unit 106, and the multiplexing unit 106 has a function of performing synchronization control of video data and audio data using the time information. Hereinafter, each block and the flow of signals will be described.

  The demultiplexing unit 101 demultiplexes the first multiplexed data recorded in the HDD 001, and encodes the first encoded video data encoded by MPEG or the like and AC-3 (Audio Code number 3) or the like. The first encoded audio data is stored in the first intermediate buffer 301, and the first encoded audio data is stored in the second intermediate buffer 302. The separation unit 101 also outputs output time information (PTS) and decoding time information (DTS), system time reference information (SCR), and program time reference for video data and audio data added to the first multiplexed data. Time information such as information (PCR) is notified to the multiplexing unit 106 (SIG201).

  The video decoding unit 102 performs a decoding process on the first encoded video data stored in the first intermediate buffer 301 and stores the decoded video data in the third intermediate buffer 303.

  The audio decoding unit 103 performs a decoding process on the first encoded audio data stored in the second intermediate buffer 302 and stores the decoded audio data in the fourth intermediate buffer 304.

  The video encoding unit 104 performs encoding processing of the video data stored in the third intermediate buffer 303 into compressed data such as MPEG, and stores the encoded second encoded video data in the fifth intermediate buffer 305. . At this time, the video encoding unit 104 may perform the encoding process after the video data decoded by the video decoding unit 102 is separately subjected to video processing such as high image quality and resolution conversion.

  The audio encoding unit 105 performs encoding processing of audio data stored in the fourth intermediate buffer 304 into predetermined compressed data, and stores the encoded second encoded audio data in the sixth intermediate buffer 306. At this time, the speech encoding unit 105 may perform the encoding process after the speech data decoded by the speech decoding unit 103 is subjected to speech processing such as improving the sound quality separately.

  In the encoding process in the video encoding unit 104 and the audio encoding unit 105, data is recompressed (rate conversion) at a compression rate different from the compression rate used when recording in the HDD001. As a recompression method, for example, there is a method described in the conventional example, and this method can also be applied.

  The multiplexing unit 106 uses the time information notified from the separation unit 101 to convert the second encoded video data stored in the fifth intermediate buffer 305 and the second encoded audio data stored in the sixth intermediate buffer 306. The second multiplexed data is generated by arranging the first encoded video data and the first encoded audio data in the first multiplexed data so that the relationship on the time axis is the same. After multiplexing while performing synchronization control of the second encoded video data and the second encoded audio data by this control, the second multiplexed data is recorded on the DVD002.

  The video / audio conversion device may recompress only one of the video data and the audio data. A case where only video data is recompressed will be described. The demultiplexing unit 101 demultiplexes the first multiplexed data, and separates the first encoded video data encoded by MPEG or the like and the first encoded audio data encoded by AC-3 or the like. The first encoded video data is transferred to the first intermediate buffer 301. Thereafter, the video decoding unit 102 performs decoding, the video encoding unit 104 performs encoding, and the fifth encoded buffer data 305 is transferred to the fifth intermediate buffer 305. . On the other hand, the first encoded audio data is directly transferred to the sixth intermediate buffer 306 without being transferred to the second intermediate buffer 302. The multiplexing unit 106 multiplexes the second encoded video data stored in the fifth intermediate buffer 305 and the first encoded audio data stored in the sixth intermediate buffer 306, and stores the second multiplexed data in the DVD002. To record. Here, the case of recompression of video data has been described, but the same processing is performed also in the case of recompression of only audio data.

  Next, the overflow and underflow of each intermediate buffer due to the difference in processing speed among the separation unit 101, the video decoding unit 102, the audio decoding unit 103, the video encoding unit 104, the audio encoding unit 105, and the multiplexing unit 106 The avoidance control will be described. The separation unit 101, the video decoding unit 102, the audio decoding unit 103, the video encoding unit 104, the audio encoding unit 105, and the multiplexing unit 106 are each an intermediate buffer existing in the previous stage (the separation unit 101 has a buffer in the previous stage). Is not applicable because it does not exist) and the intermediate buffer existing in the subsequent stage (not applicable since there is no intermediate buffer in the subsequent stage in the multiplexing unit 106) is monitored. The separation unit 101, the video decoding unit 102, the audio decoding unit 103, the video encoding unit 104, the audio encoding unit 105, and the multiplexing unit 106 each have a remaining data amount in the intermediate buffer existing in the preceding stage If the amount of data is less than, the processing is stopped so that the underflow of the intermediate buffer does not occur. If the remaining data amount exceeds the predetermined threshold, the processing is resumed. In addition, when the remaining free space in the intermediate buffer existing in the subsequent stage falls below a predetermined threshold, the processing is stopped so that the overflow of the intermediate buffer does not occur, and when the remaining free space exceeds the predetermined threshold, Resume processing.

  With this configuration, the separation unit 101, the video decoding unit 102, the audio decoding unit 103, the video encoding unit 104, the audio encoding unit 105, and the multiplexing unit 106 can realize rate conversion dubbing at the optimum processing speed. . Further, by using the time information of the first multiplexed data, it is possible to generate second multiplexed data that enables accurate reproduction in which video and audio are synchronized.

  The essential components of the video / audio conversion device according to the present invention are the separation unit 101, the video decoding unit 102, the audio decoding unit 103, the video encoding unit 104, the audio encoding unit 105, The multiplexing unit 106 and other components may be provided outside the video / audio conversion device.

(Embodiment 2)
FIG. 2 is a block diagram showing a functional configuration of the video / audio conversion apparatus according to Embodiment 2 of the present invention.

  Similar to the first embodiment, the HDD 001 and the DVD 002 are recording means, and the HDD 001 records first multiplexed data compressed by, for example, MPEG of a television broadcast. DVD002 is used for recording the second multiplexed data after rate conversion dubbing. The recording means may be an SD card or the like.

  The video / audio conversion apparatus according to the second embodiment includes a separation unit 107, a video decoding unit 108, an audio decoding unit 109, a video encoding unit 110, an audio encoding unit 111, a multiplexing unit 112, and a synchronization unit. The control unit 401, the seventh intermediate buffer 501, the eighth intermediate buffer 502, the ninth intermediate buffer 503, the tenth intermediate buffer 504, the eleventh intermediate buffer 505, the twelfth intermediate buffer 506, the first A thirteenth intermediate buffer 507 and a fourteenth intermediate buffer 508 are configured.

  The demultiplexing unit 107 demultiplexes the first multiplexed data recorded in the HDD 001, and the first encoded video data encoded by MPEG or the like and the first encoding encoded by AC-3 or the like. Separate into audio data.

  The video decoding unit 108 decodes the first encoded video data demultiplexed by the separation unit 107.

  The audio decoding unit 109 decodes the first encoded audio data demultiplexed by the demultiplexing unit 107.

  The video encoding unit 110 encodes the video data decoded by the video decoding unit 108.

  The voice encoding unit 111 encodes the voice data decoded by the voice decoding unit 109.

  The multiplexing unit 112 multiplexes the second encoded video data encoded by the video encoding unit 110 and the second encoded audio data encoded by the audio encoding unit 111, and the second multiplexed data Is generated.

  The synchronization control unit 401 determines the transfer timing of the video data decoded by the video decoding unit 108 to the video encoding unit 110 and the transfer of the audio data decoded by the audio decoding unit 109 to the audio encoding unit 111. adjust.

  The seventh intermediate buffer 501 temporarily stores the first encoded video data demultiplexed by the separation unit 107 when transferring it from the separation unit 107 to the video decoding unit 108.

  The eighth intermediate buffer 502 temporarily stores the first encoded speech data demultiplexed by the separation unit 107 when transferring it from the separation unit 107 to the speech decoding unit 109.

  The ninth intermediate buffer 503 temporarily stores the video data decoded by the video decoding unit 108 when the video data is transferred from the video decoding unit 108 to the synchronization control unit 401.

  The tenth intermediate buffer 504 temporarily stores the audio data decoded by the audio decoding unit 109 when the audio data is transferred from the audio decoding unit 109 to the synchronization control unit 401.

  The eleventh intermediate buffer 505 temporarily stores the video data synchronously controlled by the synchronization control unit 401 when the video data is transferred from the synchronization control unit 401 to the video encoding unit 110.

  The twelfth intermediate buffer 506 temporarily stores the audio data synchronously controlled by the synchronization control unit 401 when it is transferred from the synchronization control unit 401 to the audio encoding unit 111.

  The thirteenth intermediate buffer 507 temporarily stores the second encoded video data encoded by the video encoding unit 110 when the second encoded video data is transferred from the video encoding unit 110 to the multiplexing unit 112.

  The fourteenth intermediate buffer 508 temporarily stores the second encoded audio data encoded by the audio encoding unit 111 when transferring it from the audio encoding unit 111 to the multiplexing unit 112.

  Further, the separation unit 107 has a function of notifying time information (SIG 402) to the synchronization control unit 401. Hereinafter, each block and the flow of signals will be described.

  FIG. 3 shows the synchronization control timing in the synchronization control unit 401. The horizontal axis in FIG. 3 indicates the elapsed time t from the start of rate conversion dubbing.

  When the double speed value of the rate conversion dubbing conversion speed is N, the synchronization control unit 401 is a period obtained by multiplying the video frame period (for example, 1 / 29.97 Hz for 1080i of HDTV (High Definition Television)) by 1 / N times. Then, transfer control of the video data decoded by the video decoding unit 108 is performed. In addition, the synchronization control unit 401 performs transfer control of the audio data decoded by the audio decoding unit 109 at a cycle that is 1 / N times the audio frame cycle (for example, 1024/48 kHz for Advanced Audio Coding (AAC)). . In other words, when N = 1 (1 × speed), synchronization control is performed in the frame period of video and audio, and when N = 2 (2 × speed), synchronization is performed at a cycle that is 1/2 of the frame period of video and audio. Take control.

  FIG. 4 is a diagram for explaining the synchronization control in the synchronization control unit 401. The horizontal axis in FIG. 4 indicates elapsed time t from the start of rate conversion dubbing, and the vertical axis indicates time information values.

  The synchronization control unit 401 obtains a value obtained by multiplying a system reference time STC (System Time Clock) by an N value (N value) at the start of rate conversion dubbing, and first time information (time stamp or By subtracting from SCR (System Clock Reference), PCR (Program Clock Reference, etc.), stc_base is obtained. Next, the synchronization control unit 401 adds the STC ′ obtained by adding the value obtained by multiplying the STC by the N value to stc_base at the synchronization control timing of the video data transfer, and the time information notified from the separation unit 107. The video output time information (VPTS) relating to the output of the included video data is compared. If the value obtained by subtracting STC ′ from VPTS is smaller than the predetermined threshold value, or the value obtained by subtracting VPTS from STC ′ is smaller than the predetermined threshold value, the synchronization control unit 401 is decoded by the video decoding unit 108. Performs video data transfer control. If the value obtained by subtracting STC ′ from VPTS is greater than a predetermined threshold, the synchronization control unit 401 determines that the output time of the corresponding video data has not been reached, and the black data or the video data transferred at the previous synchronization control timing Perform transfer control. When the value obtained by subtracting VPTS from STC ′ is larger than a predetermined threshold, the synchronization control unit 401 determines that the output time of the corresponding video data has passed, and discards the corresponding video data without performing transfer. The next VPTS and STC 'are compared. The same control is performed when the VPTS is an invalid value due to missing video data or an error. If no new VPTS is notified from the separation unit 107 when there is no video data or video data is missing, the synchronization control unit 401 transfers the black data or the video transferred at the previous synchronization control timing. Perform data transfer control. Expressions for obtaining stc_base and STC ′ are shown below.

stc_base = time information−STC × N Equation 1
STC ′ = stc_base + STC × N Equation 2

  In addition, the synchronization control unit 401 compares the STC ′ with the audio data output synchronization control timing and the audio output time information (APTS) related to the output of the audio data included in the time information notified from the separation unit 107. If the value obtained by subtracting STC ′ from APTS is smaller than the predetermined threshold value, or the value obtained by subtracting APTS from STC ′ is smaller than the predetermined threshold value, the synchronization control unit 401 is decoded by the speech decoding unit 109. Performs audio data transfer control. When the value obtained by subtracting STC ′ from APTS is greater than a predetermined threshold, the synchronization control unit 401 determines that the output time of the corresponding audio data has not been reached, and performs transfer control of silent data. When the value obtained by subtracting APTS from STC ′ is greater than a predetermined threshold, the synchronization control unit 401 determines that the output time of the corresponding audio data has passed, and discards the corresponding audio data without performing transfer. The next APTS and STC 'are compared. The same control is performed when the APTS is an invalid value due to missing audio data or an error. In addition, when there is no audio data or when audio data is missing, if the new APTS is not notified from the separation unit 107, the synchronization control unit 401 performs transfer control of silent data.

  Accordingly, the video encoding unit 110 and the audio encoding unit 111 can perform encoding in a state where the video data and the audio data are synchronized. In addition, the multiplexing unit 112 encodes the second time encoded by the video encoding unit 110 and the audio encoding unit 111 with respect to the multiplexing time determined by the frame period of the video data and the audio data and the number of encoded frames. Since the encoded video data and the second encoded audio data may be multiplexed in a predetermined order, the synchronization control shown in the first embodiment is not necessary.

  Note that the difference between the time information and STC ′ due to the timing at which the separation unit 107 notifies the time information to the multiplexing unit 112 is not constant because it is not equal rate conversion dubbing that requires real-time performance. There is no need.

  N of the double speed value of the conversion speed of rate conversion dubbing will be described. The N value is determined by using the video / audio conversion apparatus shown in the second embodiment, with respect to synchronous control, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, and the audio encoding unit 111. In addition, it is a value that can prevent the rate conversion dubbing from being performed without causing a delay in the multiplexing unit 112. The formula for calculating the N value is shown below.

      N = min (S1, S2, S3, S4, S5, S6) Equation 3

S1 (S1min ≦ S1 ≦ S1max): the throughput factor of the separation unit 107 S2 (S2min ≦ S2 ≦ S2max): the magnification factor of the video decoding unit 108 S3 (S3min ≦ S3 ≦ S3max): the throughput of the audio decoding unit 109 S4 (S4min.ltoreq.S4.ltoreq.S4max): Throughput rate of the video encoding unit 110. S5 (S5min.ltoreq.S5.ltoreq.S5max): Throughput rate of the audio encoding unit 111. S6 (S6min.ltoreq.S6.ltoreq.S6max): Multiplexing unit 112. Throughput rate

  The function min () for calculating N is a function for obtaining the minimum value from the arguments. Each argument indicates the throughput magnification of the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, or the multiplexing unit 112. To Simax (i is the number of each block).

  For example, when the throughput factor S of each block that is an argument of the function min () takes the following values, N is 2.4.

S1 (3.1 ≦ S1 ≦ 4.9)
S2 (2.7 ≦ S2 ≦ 4.0)
S3 (2.4 ≦ S3 ≦ 4.1)
S4 (3.4 ≦ S4 ≦ 4.4)
S5 (2.9 ≦ S5 ≦ 4.7)
S6 (2.8 ≦ S6 ≦ 3.9)

  By selecting the minimum N, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 are not delayed with respect to the synchronization control. Rate conversion dubbing can be performed.

  In addition, the video decoding unit 108 stops the decoding in the video decoding unit 108 based on the progress of the synchronization control in the synchronization control unit 401 as the progress of the encoding in the video encoding unit 110. The restart may be controlled. The speech decoding unit 109 stops and restarts the decoding in the speech decoding unit 109 based on the progress of synchronization control in the synchronization control unit 401 as the progress of encoding in the speech encoding unit 111. You may control. The video encoding unit 110 encodes the video data synchronized by the synchronization control unit 401 as the encoding of the video data generated by the video decoding unit 108, and the decoding progress of the video decoding unit 108 As an alternative, the stop and restart of encoding may be controlled based on the progress of synchronization control in the synchronization control unit 401. The speech encoding unit 111 encodes the speech data synchronized by the synchronization control unit 401 as the speech data generated by the speech decoding unit 109, and the progress of decoding by the speech decoding unit 109 As an alternative, the stop and restart of encoding may be controlled based on the progress of synchronization control in the synchronization control unit 401.

  Further, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 are each an intermediate buffer existing in the preceding stage (the separating unit 107 includes Therefore, the data storage state of the intermediate buffer (not applicable since the buffer does not exist in the multiplexing unit 112 is not applicable) is monitored. The separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 each have a data remaining amount in the intermediate buffer existing in the preceding stage as a predetermined threshold value. If the value is less than, the process is stopped so that the intermediate buffer does not underflow. If the remaining data exceeds a predetermined threshold value, the process is resumed. Also, if the remaining free space in the intermediate buffer in the subsequent stage falls below a predetermined threshold, the processing is stopped so that the intermediate buffer does not overflow, and if the remaining free space exceeds the predetermined threshold, the processing is stopped. Resume. Accordingly, each intermediate buffer according to the difference between the processing speed of the synchronization control and the processing speed of the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112. Overflow and underflow can be avoided.

  With this configuration, it is possible to realize N-times rate conversion dubbing corresponding to abnormal systems such as abnormal time information of the first multiplexed data and missing video data and audio data. Further, by using the time information of the first multiplexed data, it is possible to generate second multiplexed data that enables accurate reproduction in which video and audio are synchronized.

(Embodiment 3)
FIG. 5 is a block diagram showing a functional configuration of the video / audio conversion apparatus according to the third embodiment.

  In addition to the configuration of the video / audio conversion device in the second embodiment shown in FIG. 2, the video / audio conversion device in the third embodiment includes a double speed notification unit 1201, a delay control unit 701, a first delay detection unit 801, A second delay detection unit 802, a third delay detection unit 803, and a fourth delay detection unit 804 are provided.

  The double speed notification unit 1201 notifies the synchronization control unit 401 of the double speed value of the conversion speed of rate conversion dubbing.

  The delay control unit 701 performs a delay control notification (SIG 702) to the synchronization control unit 401.

  The first delay detection unit 801 detects a decoding delay in the video decoding unit 108.

  The second delay detection unit 802 detects the decoding delay in the audio decoding unit 109.

  The third delay detection unit 803 detects the encoding delay in the video encoding unit 110.

  The fourth delay detection unit 804 detects the encoding delay in the speech encoding unit 111.

  Further, the separation unit 107 has a function of stopping the first delay detection unit 801, the second delay detection unit 802, the third delay detection unit 803, and the fourth delay detection unit 804 when the first multiplexed data is abnormal. . Hereinafter, each block and the flow of signals will be described.

  The double speed notification unit 1201 notifies the synchronization control unit 401 of a value (for example, N + 1) exceeding the N value obtained by Equation 3 of Embodiment 2 as the N value at the start of rate conversion dubbing (SIG 1202). The synchronization control unit 401 uses the N value from the double speed notification unit 1201 as the synchronization control timing and the calculated value of the STC ′.

  The first delay detection unit 801 detects a decoding delay in the video decoding unit 108 with respect to the synchronization control. The second delay detection unit 802 detects a decoding delay in the audio decoding unit 109 with respect to the synchronization control. The third delay detection unit 803 detects a coding delay in the video coding unit 110 with respect to the synchronization control. The fourth delay detection unit 804 detects the encoding delay in the audio encoding unit 111 with respect to the synchronization control. The first delay detector 801, the second delay detector 802, the third delay detector 803, and the fourth delay detector 804 notify the delay controller 701 of a delay when a delay is detected.

  When the delay control unit 701 receives a delay notification from any of the first delay detection unit 801, the second delay detection unit 802, the third delay detection unit 803, and the fourth delay detection unit 804, the delay control unit 701 performs synchronous control as a delay control notification. The unit 401 is notified of the stop of synchronization control. When the delay is eliminated by stopping the synchronization control, the block that made the delay notification among the first delay detection unit 801, the second delay detection unit 802, the third delay detection unit 803, and the fourth delay detection unit 804 A restart notification is sent to the control unit 701. When the delay control unit 701 receives the restart notification, the delay control unit 701 sends a synchronization control restart notification to the synchronization control unit 401 as a delay control notification (the above control is referred to as delay control). By the delay control, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video for the synchronous control by performing the synchronous control at a speed exceeding N times that the video / audio conversion device of the second embodiment can not guarantee Processing delays of the encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 can be avoided.

  Details of the delay control method will be described with reference to FIG. Numerical values are simplified for simplicity. The horizontal axis in FIG. 6 indicates the elapsed time t from the start of rate conversion dubbing.

  Upon receiving the stop notification from the delay control unit 701, the synchronization control unit 401 stops the synchronization control and stops each synchronization control stop position (for the synchronization control period of the video data and the synchronization control period of the audio data). The advanced position) and the STC 'at the time of stop (pastSTC' = 40) are recorded. During the period when the synchronization control is stopped, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 continue to operate. As a result, when the bottleneck portion is eliminated, the block that made the delay notification out of the first delay detection unit 801, the second delay detection unit 802, the third delay detection unit 803, and the fourth delay detection unit 804 When the delay control unit 701 issues a restart notification to the delay control unit 701 and receives the delay restart notification, the delay control unit 701 issues a restart notification to the synchronization control unit 401. Upon receiving the restart notification, the synchronization control unit 401 restarts the synchronization control while maintaining the video data transfer timing and the audio data transfer timing before stopping. At this time, since the STC is also advanced during the stop period, STC ′ (= 52) obtained by the equation 2 is also advanced, and correction is necessary. Therefore, the difference between STC ′ at restart (preSTC ′ = 52) and STC ′ at stop (pastSTC = 40) is subtracted from STC ′ as a correction value STC′_diff so that synchronization does not shift even at restart. . In addition, STC′_diff is cleared to 0 when stc_base is calculated in Expression 1. The calculation formula of STC 'including correction is shown below.

STC'_diff = STC'_diff + (preSTC-pastSTC)
... Formula 4
STC ′ = stc_base + STC × N−STC′_diff Equation 5

STC'_diff: Correction value at restart preSTC ': STC' value at restart pastSTC ': STC' value at stop

  By thus returning the state of the synchronization control timing and correcting the STC ', the synchronization control can be resumed without disturbing the transfer timing of the video data and the transfer timing of the audio data even after the delay control.

  In addition, the delay control unit 701 is configured such that when the delay notification and the restart notification from the first delay detection unit 801, the second delay detection unit 802, the third delay detection unit 803, and the fourth delay detection unit 804 overlap, This can be dealt with by providing a flag corresponding to the delay detection unit. The delay control unit 701 sets a flag by a delay notification from each delay detection unit, drops a flag by a restart notification, and performs control to resume synchronous control when all the flags are dropped. Alternatively, the first delay detection unit 801, the second delay detection unit 802, the third delay detection unit 803, and the like are performed by performing control to restart the synchronization control when the same number of restart notifications are received for a plurality of delay notifications. Even when the delay notification and the restart notification from the fourth delay detection unit 804 overlap, control with consistency can be performed.

  Note that the delay control method may be performed by adjusting the N value, which is a double speed value of rate conversion dubbing, instead of stop / restart control of synchronous control.

  FIG. 7 is a diagram showing a delay control method by adjusting the N value. The horizontal axis in FIG. 7 indicates the elapsed time t from the start of rate conversion dubbing.

  When the delay control unit 701 receives a delay notification from one of the delay detection units, the delay control unit 701 notifies the synchronization control unit 401 of the N value (2 → 1). Upon receiving the N value from the delay control unit 701, the synchronization control unit 401 obtains a value obtained by multiplying the system reference time STC by the N value, and subtracts it from STC ′ to correct stc_base. Also, the synchronization control unit 401 performs synchronization control by switching between the timing of synchronization control (frame period × 1/2 → frame period × 1/1) and the STC ′ count-up value (2 → 1). Accordingly, when the bottleneck portion is eliminated, the delay detection unit that has issued the delay notification issues a restart notification to the delay control unit 701. Upon receiving the restart notification, the delay control unit 701 notifies the synchronization control unit 401 of the N value (1 → 2). Upon receiving the N value from the delay control unit 701, the synchronization control unit 401 obtains a value obtained by multiplying the system reference time STC by the N value, and subtracts it from STC ′ to correct stc_base. The synchronization control unit 401 performs synchronization control by switching between synchronization control timing (frame period × 1/1 → frame period × 1/2) and STC ′ count-up value (1 → 2). The calculation formula of stc_base at the time of correction is shown below.

      stc_base = STC′−STC × N Equation 6

  With the above control, delay control by adjusting the N value can be realized.

  Next, detailed operations of the first delay detection unit 801, the second delay detection unit 802, the third delay detection unit 803, and the fourth delay detection unit 804 will be described.

  The first delay detection unit 801 monitors the data amount in the ninth intermediate buffer 503 in order to detect the decoding delay in the video decoding unit 108. When the amount of data in the ninth intermediate buffer 503 falls below the delay determination threshold due to the delay in decoding by the video decoding unit 108, the first delay detection unit 801 temporarily stops synchronization control. The delay notification to the delay control unit 701 is performed. Since the synchronization control is stopped by the delay control of the delay control unit 701, the data in the first intermediate buffer 301 is not consumed, so that the data amount is increased by the decoding by the video decoding unit. When the amount of data increases and exceeds the threshold value for restart determination, the first delay detection unit 801 notifies the delay control unit 701 to restart in order to restart the synchronization control. Accordingly, it is possible to prevent occurrence of underflow in the ninth intermediate buffer 503 due to a delay in processing of the video decoding unit 108.

  The third delay detection unit 803 monitors the amount of data in the eleventh intermediate buffer 505 in order to detect the encoding delay in the video encoding unit 110. If the encoding in the video encoding unit 110 is delayed and the amount of data in the eleventh intermediate buffer 505 exceeds the delay determination threshold, the third delay detection unit 803 temporarily stops the synchronization control. Therefore, a delay notification is sent to the delay control unit 701. Since the synchronization control is stopped by the delay control of the delay control unit 701, the video data is not supplied to the eleventh intermediate buffer 505. Therefore, the data amount is reduced by the encoding by the video encoding unit 110. When the data amount decreases and falls below the restart determination threshold, the third delay detection unit 803 notifies the delay control unit 701 to restart in order to restart the synchronization control. Accordingly, it is possible to prevent the eleventh intermediate buffer 505 from overflowing due to the delay in encoding in the video encoding unit 110.

  For example, when a threshold is provided for the data amount of the intermediate buffer, the size for one frame may be obtained from the resolution of the video data and used as the threshold. That is, when the resolution of the decoded video data is 1920 × 1080 and one pixel is 24 bits, the threshold value is calculated and used in units of 1920 × 1080 × 24 = 497766400 bits (6220800 bytes).

  The delay detection method may monitor the number of frames stored in the intermediate buffer instead of monitoring the data amount of the intermediate buffer.

  Here, the decoding delay in the video decoding unit 108 and the encoding delay in the video encoding unit 110 have been described. However, the decoding in the audio decoding unit 109 and the encoding delay in the audio encoding unit 111 are described. As for the second delay detection unit 802 and the fourth delay detection unit 804, the same processing is performed.

  The first delay detection unit 801 monitors the number of frames of video data generated by the video decoding unit 108 and the number of frames of video data subjected to synchronization control by the synchronization control unit 401 so that the video decoding unit 108 The decoding delay may be detected. The second delay detection unit 802 monitors the number of frames of the voice data generated by the voice decoding unit 109 and the number of frames of the voice data subjected to synchronization control by the synchronization control unit 401, thereby decoding in the voice decoding unit 109. A delay in the conversion may be detected. The third delay detection unit 803 monitors the number of frames of video data for which synchronization control is performed by the synchronization control unit 401 and the number of frames of second encoded video data generated by the video encoding unit 110, thereby monitoring the video encoding unit. A coding delay at 110 may be detected. The fourth delay detection unit 804 monitors the number of frames of audio data for which the synchronization control unit 401 has performed synchronization control and the number of frames of second encoded audio data generated by the audio encoding unit 111 to thereby detect the audio encoding unit. A coding delay at 111 may be detected.

  Next, abnormal delay control will be described. FIG. 8 is a diagram showing a pattern when audio data is not supplied to the tenth intermediate buffer 504. The delay control in each pattern will be described.

  A case will be described in which audio data is not supplied to the tenth intermediate buffer 504 due to first multiplexed data in which no audio exists. The separation unit 107 determines that the first encoded audio data does not exist by analyzing information such as an MPEG system header, for example. If it is determined by this determination that the first encoded audio data does not exist in the first multiplexed data, the separation unit 107 stops the second delay detection unit 802 and the fourth delay detection unit 804 (SIG 703). By this abnormal control, the synchronization control is not stopped even when data is not supplied to the tenth intermediate buffer 504 due to the first multiplexed data in which no sound exists.

  A case will be described in which audio data is not supplied to the tenth intermediate buffer 504 due to missing first encoded audio data or absence of the first encoded audio data for an indefinite period. Separating section 107 determines a period in which no first encoded audio data exists for STC ′ from time information such as SCR and PCR and audio output time information. If it is determined by this determination that the first encoded audio data does not exist in the first multiplexed data, the separation unit 107 stops the second delay detection unit 802 and the fourth delay detection unit 804. Due to this abnormal control, the synchronization control stops even when data is not supplied to the tenth intermediate buffer 504 because the first encoded audio data is missing or the first encoded audio data does not exist for an indefinite period. Will disappear.

  When audio data is not supplied to the tenth intermediate buffer 504 due to delay of demultiplexing by the demultiplexing unit 107 with respect to the synchronization control, abnormal system control is not necessary, and normal system delay control may be performed.

  Although the case of audio data has been described here, the same processing is performed for video data.

  With this configuration, while having an abnormal system mechanism equivalent to that of the second embodiment, synchronization control is performed with an N value larger than the N value in the second embodiment, and the separation unit 107 and the video decoding unit 108 for the synchronization control are performed. In the embodiment, the delay of the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 is controlled so as not to disturb the synchronization of the video data and the audio data. It is possible to realize rate conversion dubbing faster than 2. Also, when the processing performance of each block is increased, the rate conversion dubbing performance can be improved without changing this configuration by simply changing the N value.

(Embodiment 4)
FIG. 9 is a block diagram illustrating a functional configuration of the video / audio conversion device according to the fourth embodiment.

  The video / audio conversion apparatus according to the fourth embodiment includes an operation mode determination unit 1401 in addition to the configuration of the video / audio conversion apparatus according to the third embodiment shown in FIG. Hereinafter, each block and signal flow will be described.

  The operation mode determination unit 1401 performs demultiplexing by the separation unit 107 during rate conversion dubbing, decoding by the video decoding unit 108, decoding by the audio decoding unit 109, encoding by the video encoding unit 110, and audio Each element that affects the encoding in the encoding unit 111 or the multiplexing in the multiplexing unit 112 (for example, the resolution of the video data in the first multiplexed data, the resolution of the video data in the second multiplexed data, the first Bit rate of multiplexed data, bit rate of second multiplexed data, frame rate of video data in first multiplexed data, frame rate of video data in second multiplexed data, in first multiplexed data Audio data sampling frequency, audio data sampling frequency in the second multiplexed data, video data codec in the first multiplexed data, Codec of video data in multiplexed data, codec of audio data in first multiplexed data, codec of audio data in second multiplexed data, number of channels of audio data in first multiplexed data, second multiplexing All the elements such as the number of channels of audio data in the data, the multiplexed format of the first multiplexed data, or the multiplexed format of the second multiplexed data are indicated), and the operation mode is selected. decide. The operation modes are defined as operation modes 1, 2, 3,... According to combinations of elements such as resolution and bit rate. The operation mode determination unit 1401 selects an optimum N value for each operation mode from the double speed information table shown in FIG. 10, and notifies the double speed notification unit 1201 (SIG1402). The double speed notification unit 1201 that has received the N value notifies the synchronization control unit 401 of the N value.

  With this configuration, the N value at which the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 operate optimally is selected for each operation mode. Thus, optimum rate conversion dubbing can be performed. Further, since the N value operates at a double speed less than 1, many processes are performed in the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112. Since time can be allocated, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, and the audio encoding unit 111 can devote time to high image quality and high quality audio processing, High-quality second multiplexed data can be generated.

  In this embodiment, an example in which the N value is determined according to the operation mode has been described. However, regardless of the operation mode, the double speed notification unit 1201 may select N to be notified by an external instruction. Good.

(Embodiment 5)
FIG. 11 is a block diagram illustrating a functional configuration of the video / audio conversion device according to the fifth embodiment.

  The video / audio conversion apparatus according to the fifth embodiment includes a double speed measuring unit 1501 in addition to the configuration of the video / audio conversion apparatus according to the third embodiment shown in FIG. In addition, the delay control unit 701 has a function of notifying the double speed measurement unit 1501 of delay occurrence information (SIG 1503). Hereinafter, each block and the flow of signals will be described.

  In the fifth embodiment, a method for selecting an N value by measuring the N value before the start of rate conversion dubbing will be described as a method for selecting an N value for rate conversion dubbing different from that in the fourth embodiment.

  FIG. 12 is a flowchart for explaining a double speed measurement method for measuring the N value before starting rate conversion dubbing.

  In S101, the double speed measurement unit 1501 obtains the N value obtained by the equation 3.

  In S102, the double speed measurement unit 1501 saves the N value used when the N value for rate conversion dubbing is determined as N 'in a buffer (not shown).

  In S103, the double speed measurement unit 1501 obtains an N value to be used for rate conversion dubbing by adding a predetermined value α to the N value.

  In S104, rate conversion dubbing for a predetermined period is executed, and double speed measurement unit 1501 receives delay generation information (SIG 1502) from delay control unit 701, and counts the number of occurrences of delay control.

  In S105, the double speed measurement unit 1501 determines the number of occurrences of delay control and a predetermined threshold. Here, if the number of occurrences of delay control exceeds the threshold value, the process proceeds to S106, and if it is lower, the process returns to S102, and the N value is saved as N 'in a buffer (not shown) and rate conversion dubbing is performed again.

  In S106, the double speed measurement unit 1501 determines the N value to be used in the actual rate conversion dubbing notified to the double speed notification unit 1201 by substituting the value of N ′ saved in the buffer in S102 into the N value. To do.

  With this configuration, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit can be performed without determining an optimal N value for each operation mode in advance. Rate conversion dubbing can be performed with an N value at which 112 operates optimally.

  If the value of α is given a small value, the accuracy of the optimum N value is improved, but it takes time to reach the optimum value. If a large value is given, the time to reach the optimum value is shortened, but the accuracy of the optimum N value is lowered. The value of α may be a value unique to the device or may be given from the outside.

(Embodiment 6)
FIG. 13 is a block diagram illustrating a functional configuration of the video / audio conversion device according to the sixth embodiment.

  The video / audio conversion apparatus according to the sixth embodiment includes a second operation mode determination unit 1601, a second double speed measurement unit 1602, and a double speed recording memory 1603 in addition to the configuration of the video / audio conversion apparatus according to the third embodiment shown in FIG. Prepare. Hereinafter, each block and the flow of signals will be described.

  The second operation mode determination unit 1601 performs demultiplexing in the demultiplexing unit 107, decoding in the video decoding unit 108, decoding in the audio decoding unit 109, encoding in the video encoding unit 110, and audio encoding. The operation mode is determined according to the operation setting that affects the encoding in the unit 111 or the multiplexing in the multiplexing unit 112. As described above, the operation modes are defined as operation modes 1, 2, 3,... According to combinations of elements such as resolution and bit rate.

  Second double speed measuring section 1602 measures N before starting rate conversion dubbing.

  The double speed recording memory 1603 records the measured N.

  When the N corresponding to the operation mode is recorded in the double-speed recording memory 1603, the second operation mode determination unit 1601 notifies the recorded N to the double-speed notification unit 1201 (SIG1604), and enters the operation mode. When the corresponding N is not recorded in the double speed recording memory 1603, the second double speed measurement unit 1602 is notified of activation (SIG 1605).

  The second double speed measurement unit 1602 measures the optimum N for each operation mode, records the N for each operation mode on the double speed recording memory 1603 (SIG1607), and notifies the double speed notification unit 1201 of the N ( SIG1606).

  In the sixth embodiment, an N value selection method for rate conversion dubbing different from that of the fourth and fifth embodiments will be described.

  FIG. 14 is a flowchart for explaining a double speed determination method for measuring the N value before starting the rate conversion dubbing.

  In step S201, the second operation mode determination unit 1601 determines each element (for example, the resolution of the video data in the first multiplexed data, the resolution of the video data in the second multiplexed data, Bit rate of one multiplexed data, bit rate of second multiplexed data, frame rate of video data in first multiplexed data, frame rate of video data in second multiplexed data, first multiplexing Audio data sampling frequency in data, audio data sampling frequency in second multiplexed data, video data codec in first multiplexed data, video data codec in second multiplexed data, first multiplexing Audio data codec in data, audio data codec in second multiplexed data, first multiplexing All the elements such as the number of audio data channels in the data, the number of audio data channels in the second multiplexed data, the multiplexed format of the first multiplexed data, or the multiplexed format of the second multiplexed data Pointed without omission) and determine the operation mode.

  In S202, the second operation mode determination unit 1601 determines whether the N value of the operation mode is recorded in the double speed recording memory 1603. If the N value is recorded, the process proceeds to S203, and if not recorded, the process proceeds to S204.

  In S203, the second operation mode determination unit 1601 selects an optimum N value for each operation mode from the double speed recording memory 1603, and notifies the N value to the double speed notification unit 1201 (SIG1604), thereby using the rate conversion dubbing. N value to be determined is determined.

  In S204, the second operation mode determination unit 1601 notifies the second double speed measurement unit 1602 of activation (SIG1605).

  In S205, the second double speed measurement unit 1602 acquires the N value obtained by Equation 3 above.

  In S206, the second double speed measurement unit 1602 saves the N value used when the N value for rate conversion dubbing is determined as N 'in a buffer (not shown).

  In S207, the second double speed measurement unit 1602 obtains an N value to be used for rate conversion dubbing by adding a predetermined value α to the N value.

  In S208, rate conversion dubbing for a predetermined period is executed, and the second double speed measurement unit 1602 receives delay generation information (SIG1502) from the delay control unit 701 and counts the number of occurrences of delay control.

  In step S209, the second double speed measurement unit 1602 determines the number of occurrences of delay control and a predetermined threshold value. Here, if the number of occurrences of delay control exceeds the threshold value, the process proceeds to S210, and if it is lower, the process returns to S206. The N value is saved as N 'in a buffer (not shown) and the rate conversion dubbing is performed again.

  In S210, the second double speed measurement unit 1602 substitutes the value of N ′ saved in the buffer in S206 for the N value.

  In S211, the second double speed measurement unit 1602 records the N value in the double speed recording memory 1603 (SIG1607).

  In S212, the second double speed measurement unit 1602 notifies the N value to the double speed notification unit 1201 (SIG1606), thereby determining the N value used in the rate conversion dubbing.

  With this configuration, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit can be performed without determining an optimal N value for each operation mode in advance. Rate conversion dubbing can be performed with an N value at which 112 operates optimally. Further, when operating in the same operation mode, it is possible to shorten the time until rate conversion dubbing is started as compared with the fifth embodiment.

  As described above, when the value of α is given a small value, the accuracy of the optimum N value is improved, but it takes time to reach the optimum value. If a large value is given, the time to reach the optimum value is shortened, but the accuracy of the optimum N value is lowered. The value of α may be a value unique to the device or may be given from the outside.

(Embodiment 7)
FIG. 15 is a block diagram illustrating a functional configuration of the video / audio conversion device according to the seventh embodiment.

  The video / audio conversion device according to the seventh embodiment includes a third operation mode determination unit 1801 and a third double speed measurement unit 1802 in addition to the configuration of the video / audio conversion device according to the third embodiment shown in FIG. Hereinafter, each block and the flow of signals will be described.

  The double speed notification unit 1201 switches the N to be notified to the synchronization control unit 401 during rate conversion dubbing.

  When the operation mode is switched during the rate conversion dubbing, the N value used in the synchronization control is not an optimal value, and therefore the rate conversion dubbing speed may be reduced. In the seventh embodiment, an operation of switching to an optimal N value when the operation mode is switched during rate conversion dubbing will be described.

  FIG. 16 is a diagram showing N value switching. The horizontal axis in FIG. 16 indicates the elapsed time t from the start of rate conversion dubbing.

  The third operation mode determination unit 1801 monitors operation mode switching during rate conversion dubbing. When the operation mode is switched, the third operation mode determination unit 1801 selects an optimum N value for each operation mode and notifies the double speed notification unit 1201 (SIG1803). Receiving the N value, the double speed notification unit 1201 notifies the synchronization control unit 401 of the N value (2 → 1). The synchronization control unit 401 that has received the N value from the double speed notification unit 1201 obtains a new stc_base according to Equation 6 above. In addition, the synchronization control unit 401 performs synchronization control by switching the timing (frame period × 1/2 → frame period × 1/1) of the synchronization control unit 401 and the STC ′ count-up value (2 → 1).

  The above control makes it possible to switch the N value during rate conversion dubbing.

  FIG. 17 is a flowchart for explaining control for measuring an optimum N value during rate conversion dubbing.

  In S301, the third double speed measurement unit 1802 counts the number of delay occurrences during a predetermined period during rate conversion dubbing.

  In S302, the third double speed measurement unit 1802 determines the number of occurrences of delay control and a predetermined threshold value. Here, if the number of occurrences of delay control exceeds the threshold value, the process proceeds to S303, and if it is less than the threshold value, the process proceeds to S304.

  In S303, the third double speed measurement unit 1802 obtains an N value to be used for rate conversion dubbing by subtracting a predetermined value β from the N value.

  In step S304, the third double speed measurement unit 1802 determines whether delay control has occurred. Here, if delay control has not occurred, the process proceeds to S305, and if it has occurred, the process returns to S301 to again count the number of occurrences of delay in a predetermined period.

  In S305, the third double speed measurement unit 1802 obtains an N value to be used for rate conversion dubbing by adding a predetermined value γ to the N value.

  In S306, the third double speed measurement unit 1802 notifies the N value to the double speed notification unit 1201 (SIG1804).

  In S307, the double speed notification unit 1201 controls switching to the N value.

  With this configuration, even when the operation mode is switched, the N value at which the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 operate optimally. Can perform rate conversion dubbing. Further, in accordance with the first multiplexed data input to the separation unit 107, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112. The rate conversion dubbing can be performed with the N value that operates optimally.

  As for the values of β and γ, as with the value of α, giving a small value improves the accuracy of the optimum N value, but it takes time to reach the optimum value. If a large value is given, the time to reach the optimum value is shortened, but the accuracy of the optimum N value is lowered. Note that the values of β and γ may be values inherent to the device or may be given from the outside.

(Embodiment 8)
FIG. 18 is a block diagram illustrating a functional configuration of the video / audio conversion device according to the eighth embodiment.

  The video / audio conversion apparatus according to the eighth embodiment includes a double speed determination unit 1901 and a double speed notification unit 1905 in addition to the configuration of the video / audio conversion apparatus according to the second embodiment shown in FIG. Hereinafter, each block and the flow of signals will be described.

  The double speed determination unit 1901 receives an N value from the double speed notification unit 1905 at the start of rate conversion dubbing (SIG 1902), and sets an operation mode so that rate conversion dubbing with the N value can be realized (SIG 1903). For example, as in the double speed information table shown in FIG. 19, an operation mode is selected for each N value, and each element that affects the operation during rate conversion dubbing (for example, the resolution of video data in the second multiplexed data, the second Bit rate of multiplexed data, frame rate of video data in second multiplexed data, sampling frequency of audio data in second multiplexed data, codec of video data in second multiplexed data, second multiplexing All factors such as codec of audio data in the multiplexed data, number of channels of audio data in the first multiplexed data, number of channels of audio data in the second multiplexed data, or multiplexing format of the second multiplexed data Point to without omission). If the N value is outside the range defined in the table, the separation unit 107, the video decoding unit 108, the audio decoding unit 109, and the video encoding are performed in the operation mode corresponding to the closest N value on the table. Unit 110, speech encoding unit 111, and multiplexing unit 112 are set, and N value is notified to double speed notification unit 1905 (SIG1904). The double speed notification unit 1905 notifies the synchronization control unit 401 of the N value received from the double speed determination unit 1901 (SIG 1906).

  In addition, selection of the priority of the separation unit 107, the video decoding unit 108, the audio decoding unit 109, the video encoding unit 110, the audio encoding unit 111, and the multiplexing unit 112 and whether or not to change the mode is added to the selection of the operation mode. May be.

  With this configuration, high-quality second multiplexed data can be generated for rate conversion dubbing at a predetermined magnification.

  Each of the above devices may be realized by hardware.

  Further, each of the above devices may be specifically configured as a computer system including a microprocessor, ROM, RAM, hard disk drive, display unit, keyboard, mouse, and the like. A computer program is stored in the RAM or hard disk drive. Each device achieves its functions by the microprocessor operating according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.

  Furthermore, some or all of the constituent elements constituting each of the above-described devices may be configured by a single system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip, and specifically, a computer system including a microprocessor, ROM, RAM, and the like. . A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

  Furthermore, some or all of the constituent elements constituting each of the above-described devices may be configured from an IC card that can be attached to and detached from each device or a single module. The IC card or module is a computer system that includes a microprocessor, ROM, RAM, and the like. The IC card or the module may include the super multifunctional LSI described above. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may have tamper resistance.

  Further, the present invention may be the method described above. Further, the present invention may be a computer program that realizes these methods by a computer, or may be a digital signal composed of the computer program.

  Furthermore, the present invention relates to a non-volatile recording medium that can read the computer program or the digital signal, such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray). -ray Disc (registered trademark)), recorded on a semiconductor memory, or the like. The digital signal may be recorded on these non-volatile recording media.

  In the present invention, the computer program or the digital signal may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like.

  The present invention may be a computer system including a microprocessor and a memory, wherein the memory stores the computer program, and the microprocessor operates according to the computer program.

  Further, another computer system independent by recording the program or the digital signal on the non-volatile recording medium and transferring the program or transferring the program or the digital signal via the network or the like. May be carried out.

  Furthermore, the above embodiment and the above modification examples may be combined.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  The video / audio conversion apparatus and method according to the present invention are useful for a video / audio conversion apparatus and method for performing high-speed rate conversion dubbing.

001 HDD
002 DVD
101, 107 Separator 102, 108 Video decoder 103, 109 Audio decoder 104, 110 Video encoder 105, 111 Audio encoder 106, 112 Multiplexer 301 First intermediate buffer 302 Second intermediate buffer 303 Third intermediate buffer 304 Fourth intermediate buffer 305 Fifth intermediate buffer 306 Sixth intermediate buffer 401 Synchronization controller 501 Seventh intermediate buffer 502 Eighth intermediate buffer 503 Nine intermediate buffer 504 Tenth intermediate buffer 505 Eleventh intermediate buffer 506 Twelve intermediate buffer 507 Thirteenth intermediate buffer 508 Fourteenth intermediate buffer 701 Delay control unit 801 First delay detection unit 802 Second delay detection unit 803 Third delay detection unit 804 Fourth delay detection units 1201, 1905 Double speed notification unit 1401 Operation Mode Determination Unit 1501 Double Speed Measurement Unit 16 1 second operation mode determination unit 1602 second speed measurement unit 1603 speed recording memory 1801 third operation mode determination unit 1802 third speed measurement unit 1901 speed determination unit

Claims (38)

Second multiplexed data in which second encoded video data and second encoded audio data are multiplexed from first multiplexed data in which first encoded video data and first encoded audio data are multiplexed A video / audio conversion device for generating
A separator for demultiplexing the first encoded video data and the first encoded audio data from the first multiplexed data;
A video decoding unit that generates video data by decoding the first encoded video data demultiplexed by the demultiplexing unit;
A speech decoding unit that generates speech data by decoding the first encoded speech data demultiplexed by the separation unit;
A video encoding unit that generates second encoded video data by encoding the video data generated by the video decoding unit;
A speech encoding unit that generates second encoded speech data by encoding speech data generated by the speech decoding unit;
Multiplexer for generating second multiplexed data by multiplexing the second encoded video data generated by the video encoder and the second encoded audio data generated by the audio encoder And
The demultiplexing unit controls stop and restart of demultiplexing in the demultiplexing unit based on the progress of decoding in the video decoding unit and the audio decoding unit,
The video decoding unit controls stop and restart of decoding in the video decoding unit based on a progress of demultiplexing in the demultiplexing unit and a progress of encoding in the video encoding unit. ,
The speech decoding unit controls stop and restart of decoding in the speech decoding unit based on the progress of demultiplexing in the separating unit and the progress of encoding in the speech encoding unit. ,
The video encoding unit controls stop and restart of encoding in the video encoding unit based on the progress of decoding in the video decoding unit and the progress of multiplexing in the multiplexing unit. ,
The speech encoding unit controls the stop and restart of encoding in the speech encoding unit based on the progress of decoding in the speech decoding unit and the progress of multiplexing in the multiplexing unit. ,
The video / audio conversion device, wherein the multiplexing unit controls stop and restart of multiplexing in the multiplexing unit based on a progress of encoding in the video encoding unit and the audio encoding unit. The separation unit further notifies the multiplexing unit of time information added to the first multiplexed data,
The multiplexing unit multiplexes second encoded video data and second encoded audio data using the time information notified from the demultiplexing unit to generate the second multiplexed data. 2. The audio / video conversion apparatus according to 1. The separation unit monitors a data storage state of a first intermediate buffer that temporarily stores the first encoded video data demultiplexed by the separation unit when the first encoded video data is transferred from the separation unit to the video decoding unit. To determine the progress of decoding in the video decoding unit, and temporarily store the first encoded audio data demultiplexed by the demultiplexing unit from the demultiplexing unit to the audio decoding unit Determining the progress of decoding in the speech decoding unit by monitoring the data storage state of the second intermediate buffer;
The video decoding unit determines the progress of demultiplexing in the demultiplexing unit by monitoring the data accumulation status of the first intermediate buffer, and decodes the video data generated by the video decoding unit. Determining the progress of encoding in the video encoding unit by monitoring the data accumulation status of the third intermediate buffer temporarily stored when transferring from the video encoding unit to the video encoding unit,
The speech decoding unit determines the progress of demultiplexing in the separation unit by monitoring the data storage status of the second intermediate buffer, and the speech decoding unit generates speech data generated by the speech decoding unit. Determining the progress of encoding in the speech encoding unit by monitoring the data accumulation status of the fourth intermediate buffer temporarily stored when transferring from the unit to the speech encoding unit,
The video encoding unit judges the progress of decoding in the video decoding unit by monitoring the data storage status of the third intermediate buffer, and the second encoded video generated by the video encoding unit Determining the progress of multiplexing in the multiplexing unit by monitoring the data accumulation status of the fifth intermediate buffer temporarily stored when transferring data from the video encoding unit to the multiplexing unit;
The speech encoding unit determines the progress of decoding in the speech decoding unit by monitoring the data accumulation state of the fourth intermediate buffer, and the second encoded speech generated by the speech encoding unit Determining the progress of multiplexing in the multiplexing unit by monitoring the data accumulation status of the sixth intermediate buffer temporarily stored when transferring data from the speech encoding unit to the multiplexing unit;
The multiplexing unit determines the progress of encoding in the video encoding unit by monitoring the data storage state of the fifth intermediate buffer, and monitors the data storage state of the sixth intermediate buffer by monitoring the data storage state of the sixth intermediate buffer. The video / audio conversion apparatus according to claim 2, wherein the progress of encoding in the audio encoding unit is determined. A synchronization control unit for adjusting a timing of transferring the video data generated by the video decoding unit to the video encoding unit and transferring the audio data generated by the audio decoding unit to the audio encoding unit; Prepared,
The separation unit further notifies the synchronization control unit of time information added to the first multiplexed data,
The time information includes video output time information related to output of video data and audio output time information related to output of audio data,
The synchronization control unit transfers the video data generated by the video decoding unit in accordance with the arrival of the video output time information in accordance with a reference time that is counted up in proportion to the system reference time. In accordance with the fact that the reference time to be counted up in proportion to the audio output time information has been reached, the audio data generated by the audio decoding unit is transferred to perform synchronization control of the video data and audio data,
The video decoding unit stops and restarts decoding in the video decoding unit based on the progress of synchronization control in the synchronization control unit as the progress of encoding in the video encoding unit. Control
The speech decoding unit stops and restarts the decoding in the speech decoding unit based on the progress of synchronization control in the synchronization control unit as the progress of encoding in the speech encoding unit. Control
The video encoding unit encodes the video data synchronized by the synchronization control unit as encoding of the video data generated by the video decoding unit, and progress of decoding by the video decoding unit Based on the progress status of the synchronization control in the synchronization control unit, to control the stop and restart of the encoding,
The speech encoding unit encodes speech data synchronized by the synchronization control unit as encoding of speech data generated by the speech decoding unit, and progress of decoding in the speech decoding unit The video / audio conversion device according to claim 1, wherein the stop and restart of encoding are controlled based on a progress of synchronization control in the synchronization control unit. The separation unit monitors a data accumulation state of a seventh intermediate buffer that temporarily stores the first encoded video data demultiplexed by the separation unit when the first encoded video data is transferred from the separation unit to the video decoding unit. To determine the progress of decoding in the video decoding unit, and temporarily store the first encoded audio data demultiplexed by the demultiplexing unit from the demultiplexing unit to the audio decoding unit Determining the progress of decoding in the speech decoding unit by monitoring the data storage state of the eighth intermediate buffer;
The video decoding unit determines the progress of demultiplexing in the demultiplexing unit by monitoring the data storage status of the seventh intermediate buffer, and decodes the video data generated by the video decoding unit. Determining the progress of synchronization control in the synchronization control unit by monitoring the data storage status of the ninth intermediate buffer temporarily stored when transferring from the unit to the synchronization control unit,
The speech decoding unit judges the progress of demultiplexing in the separation unit by monitoring the data storage status of the eighth intermediate buffer, and the speech decoding unit generates speech data generated by the speech decoding unit. Determining the progress of synchronization control in the synchronization control unit by monitoring the data accumulation status of the tenth intermediate buffer temporarily stored when transferring from the unit to the synchronization control unit,
The video encoding unit monitors the data accumulation state of the eleventh intermediate buffer that temporarily stores the video data controlled by the synchronization control unit when the video data is transferred from the synchronization control unit to the video encoding unit. Thus, the progress status of the synchronization control in the synchronization control unit is determined, and the second encoded video data generated by the video encoding unit is temporarily stored when it is transferred from the video encoding unit to the multiplexing unit. Determining the progress of multiplexing in the multiplexing unit by monitoring the data accumulation status of the thirteenth intermediate buffer,
The speech encoding unit monitors a data accumulation state of a twelfth intermediate buffer that temporarily stores speech data controlled by the synchronization control unit from the synchronization control unit to the speech encoding unit. Thus, the progress status of the synchronization control in the synchronization control unit is determined, and the second encoded speech data generated by the speech encoding unit is temporarily stored when transferred from the speech encoding unit to the multiplexing unit. Determining the progress of multiplexing in the multiplexing unit by monitoring the data storage status of the fourteenth intermediate buffer,
The multiplexing unit determines the progress of encoding in the video encoding unit by monitoring the data storage state of the thirteenth intermediate buffer, and monitors the data storage state of the fourteenth intermediate buffer. The video / audio conversion apparatus according to claim 4, wherein the progress of encoding in the audio encoding unit is determined by the step (a). The synchronization control unit performs synchronization control at N times speed,
N is the minimum double speed value of the rate conversion dubbing conversion rate at which the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit can operate. The video / audio conversion device according to claim 4, which is a value. further,
A delay control unit for performing a delay control notification to the synchronization control unit;
A first delay detection unit that monitors a delay in decoding in the video decoding unit with respect to synchronization control in the synchronization control unit, and notifies the delay control unit of a delay state;
A second delay detection unit that monitors a delay in decoding in the speech decoding unit with respect to synchronization control in the synchronization control unit, and notifies the delay control unit of a delay state;
A third delay detection unit that monitors a delay in encoding in the video encoding unit with respect to the synchronization control in the synchronization control unit, and notifies the delay control unit of a delay state;
A fourth delay detection unit for monitoring a delay in encoding in the speech encoding unit with respect to the synchronization control in the synchronization control unit, and notifying the delay control unit of a delay state;
Each of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit makes a delay notification to the delay control unit if a delay is detected. If it is detected that there is a resolution,
When the delay control unit receives a delay notification from any of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, the synchronization control is performed as a delay control notification. When a restart notification is received from any one of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, a delay control notification is made. The audio / video conversion apparatus according to claim 4, wherein the synchronization control unit is notified of restart of synchronization control. The first delay detection unit is based on a data accumulation state of a ninth intermediate buffer that temporarily stores the video data generated by the video decoding unit when the video data is transferred from the video decoding unit to the synchronization control unit. Detecting a decoding delay in the video decoding unit,
The second delay detection unit is based on a data accumulation state of a tenth intermediate buffer that temporarily stores the audio data generated by the audio decoding unit when the audio data is transferred from the audio decoding unit to the synchronization control unit. Detecting a decoding delay in the voice decoding unit,
The third delay detection unit is based on a data accumulation state of an eleventh intermediate buffer that temporarily stores the video data synchronously controlled by the synchronization control unit when the video data is transferred from the synchronization control unit to the video encoding unit. Detecting the encoding delay in the video encoding unit,
The fourth delay detection unit is based on a data accumulation state of a twelfth intermediate buffer that temporarily stores the audio data synchronously controlled by the synchronization control unit when the audio data is transferred from the synchronization control unit to the audio encoding unit. The video / audio conversion device according to claim 7, wherein an encoding delay in the audio encoding unit is detected. The first delay detection unit monitors the number of frames of video data generated by the video decoding unit and the number of frames of video data subjected to synchronization control by the synchronization control unit, thereby decoding in the video decoding unit. Detect delays in
The second delay detecting unit monitors the number of frames of the voice data generated by the voice decoding unit and the number of frames of the voice data subjected to synchronization control by the synchronization control unit, thereby decoding in the voice decoding unit. Detect delays in
The third delay detection unit monitors the number of frames of video data for which the synchronization control unit has performed synchronization control and the number of frames of second encoded video data generated by the video encoding unit, thereby monitoring the video encoding. Detection of encoding delay in
The fourth delay detection unit monitors the number of frames of audio data that the synchronization control unit has performed synchronization control and the number of frames of second encoded audio data generated by the audio encoding unit, thereby monitoring the audio encoding. The video / audio conversion apparatus according to claim 7, wherein an encoding delay in the unit is detected. When the delay control unit receives a delay notification from any of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, a synchronization control stop signal is sent to the synchronization control unit. When a restart notification is received from any one of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, a synchronization control restart signal is notified to the synchronization control unit. And
When the synchronization control unit receives a stop signal from the delay control unit, the synchronization control unit holds the synchronization control state and stops the synchronization control. When the synchronization control unit receives a restart signal from the delay control unit, the synchronization control unit holds the synchronization control state. The video / audio conversion device according to claim 7, wherein the control is resumed. The synchronization control unit performs synchronization control at N times speed,
N is the minimum double speed value of the rate conversion dubbing conversion rate at which the separation unit, the video decoding unit, the audio decoding unit, the video encoding unit, the audio encoding unit, and the multiplexing unit can operate. Value,
When the delay control unit receives a delay notification from any one of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, the delay control unit uses the synchronization control unit for synchronization control. When the N switching signal is notified and a restart notification is received from any of the first delay detection unit, the second delay detection unit, the third delay detection unit, and the fourth delay detection unit, the synchronization control unit Notify the restart signal of synchronous control,
When the synchronization control unit receives the switching signal from the delay control unit, the synchronization control unit uses the N subtracted by a predetermined value as a double speed value of the synchronization control, and receives the restart signal from the delay control unit, The video / audio conversion device according to claim 7, wherein the video / audio conversion device is used as a double speed value. The video / audio conversion device according to claim 6, further comprising a double-speed notification unit that notifies the synchronization control unit of the N. The video / audio conversion apparatus according to claim 12, wherein the double speed notification unit selects N to be notified in accordance with an instruction from the outside. Further, demultiplexing in the demultiplexing unit, decoding in the video decoding unit, decoding in the audio decoding unit, encoding in the video encoding unit, encoding in the audio encoding unit, Or an operation mode determination unit for notifying the N-speed notification unit of the N according to the operation setting affecting the multiplexing in the multiplexing unit,
The video / audio conversion device according to claim 12, wherein the double speed notification unit notifies the synchronization control unit of the N notified from the operation mode determination unit. Furthermore, the N speed measurement is performed before starting the rate conversion dubbing, and the double speed measurement unit that notifies the N of the measured N to the double speed notification unit,
The video / audio conversion apparatus according to claim 12, wherein the double speed notification unit notifies the synchronization control unit of the N notified from the double speed measurement unit. further,
Demultiplexing in the demultiplexing unit, decoding in the video decoding unit, decoding in the audio decoding unit, encoding in the video encoding unit, encoding in the audio encoding unit, or An operation mode determination unit that determines an operation mode according to an operation setting that affects multiplexing in the multiplexing unit;
A double speed measurement unit for measuring N before starting rate conversion dubbing;
A double speed recording memory for recording the measured N,
When the N corresponding to the operation mode is recorded in the double-speed recording memory, the operation mode determination unit notifies the recorded N to the double-speed notification unit, and the N corresponding to the operation mode is If it is not recorded in the double speed recording memory, perform a start notification to the double speed measurement unit,
The double speed measurement unit measures the optimum N for each operation mode, records the N for each operation mode on the double speed recording memory, and notifies the double speed notification unit of the N. Video / audio converter. further,
The video / audio conversion apparatus according to claim 7, further comprising: a double speed notification unit that switches N during the rate conversion dubbing and notifies the synchronous control unit of N that is a magnification of a speed for performing the synchronous control. further,
When the number of occurrences of the timing adjustment of the synchronization control by the delay control unit in a predetermined period is larger than the predetermined number, the value obtained by subtracting a predetermined value from N is notified to the double speed notification unit as N, When the timing adjustment of the synchronous control does not occur, a double speed measurement unit that notifies the double speed notification unit of the value obtained by adding a predetermined value to N as the N,
The delay control unit notifies the double speed measurement unit that the timing adjustment of the synchronization control has occurred,
The video / audio conversion device according to claim 17, wherein the double-speed notification unit notifies the synchronization control unit of the N notified from the double-speed measurement unit. Further, according to the N, the demultiplexing in the demultiplexing unit, the decoding in the video decoding unit, the decoding in the audio decoding unit, the encoding in the video encoding unit, and the audio encoding unit The video / audio conversion device according to claim 6, further comprising: a double speed determination unit configured to perform an operation setting that affects the encoding of the video signal or the multiplexing in the multiplexing unit. Second multiplexed data in which second encoded video data and second encoded audio data are multiplexed from first multiplexed data in which first encoded video data and first encoded audio data are multiplexed A video / audio conversion method for generating
A separation step of demultiplexing video data and audio data from the first multiplexed data;
A video decoding step of generating video data by decoding the first encoded video data demultiplexed in the separation step;
A speech decoding step of generating speech data by decoding the first encoded speech data demultiplexed in the separation step;
A video encoding step of generating second encoded video data by encoding the video data generated in the video decoding step;
A voice encoding step of generating second encoded voice data by encoding the voice data generated in the voice decoding step;
Multiplexing step of generating second multiplexed data by multiplexing the second encoded video data generated in the video encoding step and the second encoded audio data generated in the audio encoding step Including
In the separation step, based on the progress of the decoding in the video decoding step and the audio decoding step, the stop and restart of demultiplexing is controlled,
In the video decoding step, based on the progress of demultiplexing in the separation step and the progress of encoding in the video encoding step, control the stop and restart of decoding,
In the speech decoding step, the stop and restart of decoding are controlled based on the progress of demultiplexing in the separation step and the progress of encoding in the speech encoding step,
In the video encoding step, based on the progress of decoding in the video decoding step and the progress of multiplexing in the multiplexing step, control the stop and restart of encoding,
In the speech encoding step, based on the progress of decoding in the speech decoding step and the progress of multiplexing in the multiplexing step, the stop and restart of encoding are controlled,
In the multiplexing step, a video / audio conversion method of controlling the stop and restart of multiplexing based on the progress of encoding in the video encoding step and the audio encoding step. In the separation step, the time information added to the first multiplexed data is further output,
The video / audio conversion method according to claim 20, wherein, in the multiplexing step, second multiplexed data is generated using the time information output in the separation step. In the separation step, by monitoring a data accumulation state of a first intermediate buffer that temporarily stores the first encoded video data demultiplexed in the separation step for decoding in the video decoding step. A second state of determining the progress of decoding in the video decoding step and temporarily storing the first encoded audio data demultiplexed in the separation step for decoding in the audio decoding step Determining the progress of decoding in the speech decoding step by monitoring the data storage state of the intermediate buffer;
In the video decoding step, the progress of demultiplexing in the separation step is determined by monitoring the data accumulation status of the first intermediate buffer, and the video data generated in the video decoding step is converted into the video code Determining the progress of encoding in the video encoding step by monitoring the data storage status of the third intermediate buffer temporarily stored for encoding in the encoding step;
In the speech decoding step, the progress of demultiplexing in the separation step is determined by monitoring the data storage status of the second intermediate buffer, and the speech data generated in the speech decoding step is determined as the speech code Determining the progress of encoding in the speech encoding step by monitoring the data storage status of the fourth intermediate buffer temporarily stored for encoding in the encoding step;
In the video encoding step, the progress of decoding in the video decoding step is determined by monitoring the data storage status of the third intermediate buffer, and the video data generated in the video encoding step is Determining the progress of multiplexing in the multiplexing step by monitoring the data storage status of the fifth intermediate buffer temporarily stored for multiplexing in the multiplexing step;
In the speech encoding step, the progress of decoding in the speech decoding step is determined by monitoring the data accumulation status of the fourth intermediate buffer, and the speech data generated in the speech encoding step is Determining the progress of multiplexing in the multiplexing step by monitoring the data storage status of the sixth intermediate buffer temporarily stored for multiplexing in the multiplexing step;
In the multiplexing step, the progress of encoding in the video encoding step is determined by monitoring the data storage state of the fifth intermediate buffer, and the data storage state of the sixth intermediate buffer is monitored by monitoring the data storage state of the sixth intermediate buffer. The video / audio conversion method according to claim 21, wherein the progress of encoding in the audio encoding step is determined. Furthermore, transfer of the video data generated in the video decoding step for encoding in the video encoding step and encoding of the audio data generated in the audio decoding step in the audio encoding step A synchronization control step for adjusting the timing of transfer for
In the separation step, the time information added to the first multiplexed data is further output,
The time information includes video output time information related to output of video data and audio output time information related to output of audio data,
In the synchronization control step, the video data generated in the video decoding step is transferred in accordance with the reference time for counting up in proportion to the system reference time reaching the video output time information. Synchronous control of video data and audio data is performed by transferring the audio data generated in the audio decoding step in accordance with the arrival of the audio output time information in accordance with a reference time that counts up in proportion to time. Done
In the video decoding step, as the progress of the encoding in the video encoding step, the stop and restart of the decoding are controlled based on the progress of the synchronization control in the synchronization control step,
In the speech decoding step, as the progress of the encoding in the speech encoding step, the stop and restart of the decoding are controlled based on the progress of the synchronization control in the synchronization control step,
In the video encoding step, the video data synchronized in the synchronization control step is encoded as the video data generated in the video decoding step, and the decoding progress in the video decoding step As a situation, based on the progress of synchronization control in the synchronization control step, control the stop and restart of encoding,
In the speech encoding step, the speech data synchronized in the synchronization control step is encoded as the speech data generated in the speech decoding step, and the decoding progress in the speech decoding step 21. The video / audio conversion method according to claim 20, wherein, as a situation, encoding stop and restart are controlled based on a progress of synchronization control in the synchronization control step. In the separation step, by monitoring a data accumulation state of a seventh intermediate buffer that temporarily stores the first encoded video data demultiplexed in the separation step for decoding in the video decoding step. Determining the progress of decoding in the video decoding step, and temporarily storing the first encoded audio data demultiplexed in the separation step for decoding in the audio decoding step; Determining the progress of decoding in the speech decoding step by monitoring the data storage state of the intermediate buffer;
In the video decoding step, the progress of demultiplexing in the separation step is determined by monitoring the data storage status of the seventh intermediate buffer, and the video data generated in the video decoding step is controlled by the synchronization control. Determining the progress of the synchronization control in the synchronization control step by monitoring the data accumulation status of the ninth intermediate buffer temporarily stored for synchronization control in the step;
In the speech decoding step, the progress of demultiplexing in the separation step is determined by monitoring the data storage status of the eighth intermediate buffer, and the synchronization control is performed on the speech data generated in the speech decoding step. Determining the progress of synchronization control in the synchronization control step by monitoring the data storage status of the tenth intermediate buffer temporarily stored for synchronization control in the step;
In the video encoding step, by monitoring the data accumulation status of the eleventh intermediate buffer that temporarily stores the video data synchronously controlled in the synchronization control step for encoding in the video encoding step. Data accumulation in a thirteenth intermediate buffer that determines the progress of synchronization control in the synchronization control step and temporarily stores the video data generated in the video encoding step for multiplexing in the multiplexing step Determining the progress of multiplexing in the multiplex step by monitoring the situation;
In the speech encoding step, by monitoring the data accumulation status of the twelfth intermediate buffer that temporarily stores the speech data synchronously controlled in the synchronization control step for encoding in the speech encoding step. Data storage in a fourteenth intermediate buffer that determines the progress of synchronization control in the synchronization control step and temporarily stores the audio data generated in the audio encoding step for multiplexing in the multiplexing step Determining the progress of multiplexing in the multiplex step by monitoring the situation,
In the multiplexing step, the progress of the encoding in the video encoding step is determined by monitoring the data storage state of the thirteenth intermediate buffer, and the data storage state of the fourteenth intermediate buffer is monitored. The video / audio conversion method according to claim 23, wherein the progress of encoding in the audio encoding step is determined by: In the synchronous control step, synchronous control is performed at a speed of N times,
24. The N is a minimum value of double speed values operable in the separation step, the video decoding step, the audio decoding step, the video encoding step, the audio encoding step, and the multiplexing step. The video / audio conversion method described. further,
A delay control step for performing a delay control notification for synchronization control in the synchronization control step;
A first delay detection step of monitoring a delay in decoding in the video decoding step with respect to the synchronization control in the synchronization control step, and notifying a delay situation for delay control in the delay control step;
A second delay detection step for monitoring a delay in decoding in the speech decoding step with respect to the synchronization control in the synchronization control step, and notifying a delay situation for delay control in the delay control step;
A third delay detection step of monitoring a delay in encoding in the video encoding step with respect to the synchronization control in the synchronization control step, and notifying a delay situation for delay control in the delay control step;
A fourth delay detection step of monitoring a delay in encoding in the speech encoding step with respect to the synchronization control in the synchronization control step and notifying a delay state for delay control in the delay control step. ,
In each of the first delay detection step, the second delay detection step, the third delay detection step, and the fourth delay detection step, a delay notification is performed if a delay is detected, and a restart notification is performed if a delay is detected. And
In the delay control step, when a delay notification is performed in any one of the first delay detection step, the second delay detection step, the third delay detection step, and the fourth delay detection step, the synchronization control is performed as a delay control notification. A synchronization control stop notification is made for the synchronization control in the control step, and a restart notification is made in any of the first delay detection step, the second delay detection step, the third delay detection step, and the fourth delay detection step. 24. The video / audio conversion method according to claim 23, wherein when the operation is performed, a restart notification of the synchronization control is performed as a delay control notification for the synchronization control in the synchronization control step. In the first delay detection step, the video decoding is performed based on a data accumulation state of a ninth intermediate buffer that temporarily stores the video data generated in the video decoding step for synchronization control in the synchronization control step. Detecting the decoding delay in the conversion step,
In the second delay detection step, the speech decoding is performed based on a data accumulation state of a tenth intermediate buffer that temporarily stores the speech data generated in the speech decoding step for synchronization control in the synchronization control step. Detecting the decoding delay in the conversion step,
In the third delay detection step, the video data synchronously controlled in the synchronization control step is temporarily stored based on the data accumulation state of the eleventh intermediate buffer that temporarily stores the video data for encoding in the video encoding step. Detect the encoding delay in the video encoding step,
In the fourth delay detection step, the audio data synchronously controlled in the synchronization control step is temporarily stored based on the data accumulation state of the twelfth intermediate buffer for temporarily storing the audio data for encoding in the audio encoding step. 27. The video / audio conversion method according to claim 26, wherein an encoding delay in the audio encoding step is detected. In the first delay detection step, in the video decoding step, the number of frames of the video data generated in the video decoding step and the number of frames of the video data subjected to synchronization control in the synchronization control step are monitored. Detect the decryption delay of
In the second delay detection step, the number of frames of the voice data generated in the voice decoding step and the number of frames of the voice data subjected to synchronization control in the synchronization control step are monitored, thereby Detect the decryption delay of
In the third delay detection step, the video encoding step includes monitoring the number of frames of the video data subjected to the synchronization control in the synchronization control step and the number of frames of the video data generated in the video encoding step. Detect the encoding delay of
In the fourth delay detection step, by monitoring the number of frames of the audio data subjected to the synchronization control in the synchronization control step and the number of frames of the audio data generated in the audio encoding step, 27. The video / audio conversion method according to claim 26, wherein an encoding delay is detected. In the delay control step, when the delay notification is performed in any of the first delay detection step, the second delay detection step, the third delay detection step, and the fourth delay detection step, the synchronization control in the synchronization control step For example, a synchronization control stop signal is notified, and when the restart notification is performed in any of the first delay detection step, the second delay detection step, the third delay detection step, and the fourth delay detection step, the synchronization Notify the restart signal of synchronous control for synchronous control in the control step,
In the synchronization control step, when the stop signal is notified in the delay control step, the state of the synchronization control is maintained and the synchronization control is stopped, and when the restart signal is notified in the delay control step, the stop time is maintained. 27. The video / audio conversion method according to claim 26, wherein synchronization control is resumed in a state of. In the delay control step, when the delay notification is performed in any of the first delay detection step, the second delay detection step, the third delay detection step, and the fourth delay detection step, the synchronization control in the synchronization control step When the N switching signal used in the notification is notified and the restart notification is performed in any of the first delay detection step, the second delay detection step, the third delay detection step, and the fourth delay detection step, the synchronization is performed. Notify the restart signal of the synchronous control in the control step,
In the synchronization control step, when the switching signal is notified in the delay control step, the N subtracted by a predetermined value is used as the double speed value of the synchronization control, and when the restart signal is notified in the delay control step, the N 27. The video / audio conversion method according to claim 26, wherein: is used as a double speed value of synchronous control. 26. The video / audio conversion method according to claim 25, further comprising a double speed notification step of performing the notification of N for synchronization control in the synchronization control step. 32. The video / audio conversion method according to claim 31, wherein in the double speed notification step, the N to be notified is selected by an instruction from the outside. Further, demultiplexing in the separation step, decoding in the video decoding step, decoding in the audio decoding step, encoding in the video encoding step, encoding in the audio encoding step, Or an operation mode determination step of notifying the N according to the operation setting affecting the multiplexing in the multiplexing step for notification in the double speed notification step,
32. The video / audio conversion method according to claim 31, wherein, in the double speed notification step, the N notified in the operation mode determination step is notified for synchronization control in the synchronization control step. Further, it includes a double speed measurement step of measuring the N before the start of rate conversion dubbing and notifying the measured N for notification in the double speed notification step,
32. The video / audio conversion method according to claim 31, wherein in the double speed notification step, the N notified in the double speed measurement step is notified for synchronization control in the synchronization control step. further,
Demultiplexing in the separation step, decoding in the video decoding step, decoding in the audio decoding step, encoding in the video encoding step, encoding in the audio encoding step, or An operation mode determination step for determining an operation mode according to an operation setting that affects multiplexing in the multiplex step;
A double speed measurement step of measuring N before the start of rate conversion dubbing;
Recording the measured N in a double speed recording memory,
In the operation mode determination step, when the N corresponding to the operation mode is recorded in the double speed recording memory, the recorded N is notified for notification in the double speed notification step, and the operation mode is set. If the corresponding N is not recorded in the double speed recording memory, a start notification is made for measurement in the double speed measurement step,
In the double speed measurement step, the optimum N for each operation mode is measured, the N for each operation mode is recorded on the double speed recording memory, and the N is notified for notification in the double speed notification step. The video / audio conversion method according to claim 31. further,
27. The video according to claim 26, further comprising a double speed notification step of switching N during rate conversion dubbing and performing notification for synchronization control in the synchronization control step by switching N which is a magnification of the speed for performing the synchronization control in the synchronization control step. Voice conversion method. further,
When the number of occurrences of the timing adjustment of the synchronization control by the delay control step is larger than the predetermined number in a predetermined period, the value obtained by subtracting the predetermined value from N is set as N to be notified in the double speed notification step When the timing adjustment of the synchronous control has not occurred, a double speed measurement step of notifying for the notification in the double speed notification step, where N is a value obtained by adding a predetermined value to N,
In the delay control step, it is notified for the measurement of the number of occurrences of the timing adjustment in the double speed measurement step that the synchronization control timing adjustment has occurred,
37. The video / audio conversion method according to claim 36, wherein, in the double speed notification step, the N notified in the double speed measurement step is notified for synchronization control in the synchronization control step. Further, according to the N, the demultiplexing in the separation step, the decoding in the video decoding step, the decoding in the audio decoding step, the encoding in the video encoding step, and the audio encoding step 26. The video / audio conversion method according to claim 25, further comprising: a double speed determination step of performing an operation setting that affects the encoding of the video or the multiplexing in the multiplexing step.

Images