JP2012204876A - Reproduction device, reproduction method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a reproducing device, a reproducing method and a program capable of establishing synchronization between video and voice without uncomfortable feeling.SOLUTION: In a reproducing device of video and voice, a deviation determining part determines the deviation amount between video data and voice data reproduced by a decoding reproduction part at the time of reproduction. A video importance degree calculation part calculates the degree of importance of video showing the degree of complexity of video reproduced by the video data. A voice importance degree calculation part calculates the degree of importance of voice showing characteristics of voice volume reproduced by the voice data. An automatic correction part controls reproduction of the video data and the voice data so that deviation between the video data and the voice data during reproduction is corrected based on the degree of importance of video, the degree of importance of voice and the deviation amount.

Description

  The present invention relates to a playback device, a playback method, and a program.

  In video content, since video and audio are separate pieces of information, it is necessary to synchronize video and audio. However, for example, for content with a frame rate of 30 fps and video that is determined to be 30 frames per second, the audio is a continuous stream, and there is no audio divided for each video. Also, there is a difference in the amount of data between video and audio. In particular, the reason for the large difference between the amount of video data and the amount of audio data is that the amount of video data increases dramatically as the image quality becomes higher. The amount of data per fixed time may change greatly. On the other hand, the amount of audio data does not change greatly.

  In this way, while video data is separated by frames, etc., audio data is not separated, and there is a difference in the amount of data between audio data and video data. Cause. However, since synchronization between video and audio is largely manual and not necessarily accurate, it is a problem to execute synchronization with little deviation.

  As an example corresponding to such a problem, for example, there is a reproducing apparatus that reproduces video and audio with different clocks. In this reproducing apparatus, audio data is silently reproduced prior to reproduction of moving image data, an error between the audio data clock and the video data clock is calculated from the reproduction time, and the timing at the time of reproduction is adjusted. There is also a method of synchronizing the reproduction position deviation between the video data and the audio data by changing the reproduction speed of the audio data in the section where the volume level is lower than a specific value. Further, there is an example in which the playback speed is changed by dividing the audio data at a constant time period and connecting the data. The reproduction speed is changed by determining the start and end points of cutout and connecting the waveforms according to the waveform style of the audio data near the break position.

Japanese Patent Laid-Open No. 2004-7140 JP 2006-050362 A Japanese Patent Application Laid-Open No. 06-259093

  However, as described above, in the method of simply synchronizing video and audio by changing the playback speed, the audio and / or video may be interrupted at the moment of synchronization, causing the viewer to feel uncomfortable or uncomfortable. Bring. In addition, when there is an interruption of other processing and the state where real-time processing cannot be continued, and the video and audio are out of synchronization, it is difficult to correct the shift. That is, there is a problem that the viewer feels uncomfortable or uncomfortable, for example, the playback video drops, the sound is interrupted, or the deviation is not corrected at a certain interval due to insufficient processing capability.

  Accordingly, an object of the present invention is to provide a playback apparatus and a playback method that can synchronize video and audio without a sense of incongruity.

  A playback apparatus according to one aspect is a playback apparatus that acquires a stream including encoded video and audio and plays back the video and audio. In this playback apparatus, the signal acquisition unit acquires the stream. The decoding / playback unit generates video data and audio data by separating and decoding the acquired stream, and plays back the first video and the first audio at the first shift determination time. The deviation determination unit determines an amount of deviation at the time of reproduction between the first video and the first audio during the first deviation determination time. The video importance level calculation unit calculates a video importance level indicating a degree of complexity of the second video to be reproduced in a second shift determination time next to the first shift determination time. The voice importance level calculation unit calculates a voice importance level indicating a volume characteristic of the second voice reproduced at a second shift determination time next to the first shift determination time. The automatic correction unit reproduces the second video and the second audio so as to correct a deviation in reproduction between the video and the audio based on the video importance, the audio importance, and the shift amount. It is characterized by control.

  The reproduction method which is another aspect is a reproduction method for acquiring a stream including encoded video and audio and reproducing the video and audio. In this reproduction method, the stream is acquired, the first video and the first audio are reproduced based on the first deviation determination time and the stream, and the first video and the first audio at the first deviation judgment time are reproduced. A deviation amount at the time of reproduction with the sound of 1 is determined. In addition, a video importance level indicating a degree of complexity of the second video to be played back at the second shift determination time next to the first shift determination time is calculated, and played back at the second shift determination time. The voice importance indicating the volume characteristic of the second voice is calculated. In this reproduction method, the second video and the second video at the second deviation determination time so as to correct a deviation at the time of reproduction between the video and the audio based on the video importance, the audio importance, and the deviation amount. The reproduction of the second sound is controlled.

  In addition, even if it is a program for causing a computer to perform the method according to the present invention described above, since the program is executed by the computer, the same operations and effects as the method according to the present invention described above are achieved. The aforementioned problems are solved.

  According to the above-described aspect, a playback device, a playback method, and a program that can synchronize video and audio without a sense of incongruity are provided.

It is a block diagram which shows the structure of the reproducing | regenerating apparatus by one Embodiment. It is a block diagram which shows the function of the reproducing | regenerating apparatus by one Embodiment. It is a block diagram which shows the function of the reproducing | regenerating apparatus by one Embodiment. It is a flowchart which shows operation | movement of the reproducing | regenerating apparatus by one Embodiment. It is a flowchart which shows operation | movement of the reproducing | regenerating apparatus by one Embodiment. It is a flowchart which shows operation | movement of the reproducing | regenerating apparatus by one Embodiment. It is a flowchart which shows operation | movement of the reproducing | regenerating apparatus by one Embodiment. It is a figure explaining the calculation method of the sound volume for every frequency domain by one embodiment. It is a block diagram which shows an example of the hardware constitutions of a standard computer.

  Hereinafter, embodiments will be described with reference to the drawings. First, the configuration of the playback apparatus 1 according to an embodiment will be described with reference to FIGS. 1, 2A, and 2B. FIG. 1 is a block diagram illustrating a configuration of a playback device according to the present embodiment, and FIGS. 2A and 2B are block diagrams illustrating functions of the playback device 1 according to the present embodiment. The playback device 1 according to the present embodiment has a function of synchronizing audio and video when decoding and reproducing video and audio content data in a local environment. The playback device 1 is realized as, for example, a personal information terminal (PDA), a personal computer (PC), a mobile phone, or the like.

  As shown in FIG. 1, the playback device 1 includes an input playback unit 5, a shift determination unit 7, an audio importance level determination unit 9, a video importance level determination unit 11, a comparative evaluation unit 13, an automatic correction unit 15, a storage unit 35, A timer 37 is included, connected to each other via the system bus 17 and controlled by the main control unit 3. The input reproduction unit 5 includes a signal acquisition unit 19, a separation unit 21, an audio decoding unit 23, an audio reproduction unit 25, a video decoding unit 27, and a video reproduction unit 29. The automatic correction unit 15 includes an audio operation unit 31 and a video operation unit 33.

  As illustrated in FIGS. 1, 2A, and 2B, the signal acquisition unit 19 acquires a stream 53 including encoded video data and encoded audio data. The separation unit 21 separates the stream 53 acquired by the signal acquisition unit 19 into a video stream 57 of encoded video data and an audio stream 55 of encoded audio data.

  The audio decoding unit 23 decodes the audio stream 55 and performs an operation according to the output from the automatic correction unit 15 to generate reproducible audio data. The audio reproduction unit 25 reproduces audio based on the information regarding the reproduction start location from the automatic correction unit 15 and the audio data from the audio decoding unit 23. The video decoding unit 27 decodes the video stream 57 and performs an operation according to the output from the automatic correction unit 15 to generate reproducible video data. The video playback unit 29 plays back the video based on the information regarding the playback start location from the automatic correction unit 15 and the video data from the video decoding unit 27.

  The deviation determination unit 7 determines the amount of deviation between the audio reproduced by the audio reproduction unit 25 and the video reproduced by the video reproduction unit 29. The voice importance level determination unit 9 calculates the voice importance level SL based on the voice data decoded by the voice decoding unit 23. The video importance determination unit 11 calculates the video importance IL based on the video stream 57 separated by the separation unit 21. The comparative evaluation unit 13 compares the calculated audio importance level SL with the video importance level IL.

  The automatic correction unit 15 is a device that automatically corrects the reproduction state of the content data based on the comparison result in the comparative evaluation unit 13. The audio operation unit 31 operates audio data, and the video operation unit 33 displays video. Manipulate data. In the audio operation unit 31, at least one of the silent thinning operation 61, the silent interpolation operation 63, and the reproduction speed changing operation 65 is performed based on the deviation amount and the audio data. In the video operation unit 33, a frame thinning operation 67 or a frame interpolation operation 69 is performed based on the shift amount and the video data.

  The storage unit 35 is a random access memory (RAM), a read only memory (ROM) or the like, and a program for performing the above operation, a video stream 57, decoded video data, an audio stream 55, a decoded Stores audio data. The main control unit 3 is an arithmetic processing device for controlling the operation of the playback device 1.

  Hereinafter, the operation of the playback apparatus 1 according to the present embodiment will be described with reference to FIGS. 3 to 5. 3 and 4A to 4C are flowcharts showing the operation of the playback apparatus 1. As shown in FIG. 3, the signal acquisition unit 19 acquires a stream including encoded audio data and video data such as moving image content, and the stream is separated into a video stream and an audio stream by the separation unit 21 (S100). ). The main control unit 3 sets the threshold value A by the deviation determination unit 7 (S101). The audio decoding unit 23 decodes the audio stream 55 by a certain amount of buffer. This buffer length is set as the threshold A, and can be set to 1 second, for example.

  Subsequently, the main control unit 3 calculates the threshold value B as the number of frames of the moving image corresponding to the threshold value A by the deviation determination unit 7 (S101). That is, the deviation determination unit 7 calculates the number of frames corresponding to the time of the threshold A from the stream 53 and sets this as the threshold B. For example, when the threshold A = 1 second, the threshold B = 30 frames is calculated from the frame rate of the moving image.

  The input reproduction unit 5 starts reproduction of moving images and sounds, and activates an unillustrated audio timer and moving image counter (S102). The audio timer has a time measuring function, and measures the time of reproduced audio by counting the number of reproduced samples of the audio. That is, the value of the audio timer is the number of reproduced audio samples divided by the audio sampling rate. The moving picture counter has a counting function and counts the number of frames of the displayed video.

  The input playback unit 5 plays back one frame of the moving image (S103). That is, the audio decoding unit 23 decodes the audio stream 55 to create audio data, and the audio reproduction unit 25 reproduces the audio data. The video decoding unit 27 decodes the video stream 57 to create video data, and the video playback unit 29 plays back a moving image.

  The deviation determination unit 7 compares the value of the moving image counter (moving image counter value ICt) indicating the number of frames of the image reproduced so far with the threshold value B when one frame is reproduced (S104). When the deviation determination unit 7 determines that the moving image counter value ICt does not satisfy the threshold value B (S104: Yes), whether or not the frame currently being reproduced by the video reproduction unit 29 is the end of the Group of Pictures (GOP) of the video stream. (S107). If the currently playing frame is not at the GOP end (S107: No), the process returns to S103, and one frame is played back additionally.

  When the currently reproduced portion is the GOP end (S107: Yes), the difference between the audio and the video when the number of frames included in the GOP length is reproduced is determined. Here, the section in which the shift is determined is referred to as a shift determination section JA. Here, the deviation determination section JA = GOP length.

  If the moving image counter value ICt is equal to or greater than the threshold value B (S104: No) and the moving image counter value ICt is the threshold value B (S105: Yes), the main control unit 3 advances the process to S108. At this time, deviation determination section JA = threshold value B. If the moving image counter value ICt is larger than the threshold value B (S105: No), the main control unit 3 outputs an error (S106) and ends the process.

  Subsequently, the deviation determination unit 7 calculates a difference between the value of the audio timer (audio timer value ST) and the value obtained by multiplying the reproduced video frame by the threshold A and dividing by the threshold B (time corresponding to the reproduced video frame). To do. That is, the deviation determination unit 7 calculates a value of (audio timer value ST−threshold value A × (moving image counter value ICt / threshold value B)) as the deviation amount L1. The deviation determination unit 7 determines whether or not the calculated value is less than the predetermined deviation allowable time AT (S108). The predetermined deviation allowable time AT can be set to 1/30 seconds, for example.

  The deviation determination unit 7 corrects the deviation because synchronization is established when (audio timer value ST−threshold A × (movie counter value ICt / threshold B)) <(deviation allowable time AT) (S108: Yes). Determine that it is not necessary. At this time, the deviation determination unit 7 resets the audio timer value ST = 0 and the moving image counter value ICt = 0 after setting the threshold A = the audio timer value ST and the threshold B = the moving image counter value ICt (S109). Return.

  The deviation determination unit 7 determines that synchronization is not established when (voice timer value ST−threshold A × (moving image counter value ICt / threshold B) <(deviation allowable time AT)) (S108: No). The main control unit 3 advances the process to the process of the flowchart of FIG. 4A.

  As shown in FIG. 4A, the main control unit 3 sets (shift time L1) = (audio timer value ST−threshold A × (moving image counter value ICt / threshold B)) by the automatic correction unit 15 (S131). . The deviation time L1 represents the deviation time between video and audio, and is the time obtained by subtracting the reproduced video time from the reproduced audio time. When the audio is delayed, a negative value is obtained. When the video is delayed, the deviation time L1 is positive. Takes a value.

  The main control unit 3 uses the video decoding unit 23 to determine the GOP structure of the portion to be reproduced next for the video stream 57, and determines whether or not it includes a Bi-directional Predicted Frame (B frame) (S131). For example, it is determined whether or not the GOP structure is “Inter Video Bitrate Balance Profile: IBBP”.

  When the B frame is not included (S132: No), the automatic correction unit 15 advances the process to S135 in FIG. If a B frame is included (S132: Yes), it is determined based on the audio importance level SL and the video importance level IL whether the priority is given to the video or audio operation.

   In S133, the video importance IL indicating the importance of the video and the audio importance SL indicating the importance of the audio are calculated. Hereinafter, the process of S133 will be described. The video importance IL is calculated as a value indicating the complexity of the video. The voice importance level SL is calculated as a value indicating the characteristics of the sound volume.

  First, a method for calculating the video importance IL will be described. The video importance level determination unit 11 performs, for example, H.P. H.264 parameter information is analyzed, and (-1) × quantization coefficient (Quantization Parameter: QP), data amount per frame before decoding, and motion vector total amount are normalized and added to each other, thereby adding video importance. IL is calculated.

  Since it is difficult to analyze the video while reproducing the video itself, the video importance IL is based on the quantization coefficient added when the video is encoded, the frame (picture) size, and the total amount of motion vectors. Ask. Here, it is assumed that the video importance based on the quantization coefficient is the quantization importance IL1, the video importance based on the frame size is the size importance IL2, and the video importance based on the total motion vector is the vector importance IL3.

Hereinafter, a method for calculating the quantization importance IL1 will be described. The quantization importance IL1 is calculated based on a quantization coefficient QP that is set for each macroblock (Macroblock: MB) during encoding. The quantization coefficient QP is a parameter that is set according to the complexity of the video and the difficulty of understanding the degradation so that the target data size is obtained when the video data is compressed. The quantization coefficient QP is set in the form of a difference from the quantization coefficient QP of the previous MB in each MB header, and the quantization value QPp per picture can be calculated by the following equation 1.
QPp = 26 + PIQM + Σ (SQD + (ΣMQD / Mb)) / (NSGM + 1)
... (Formula 1)

  Here, the first “Σ” indicates the sum of all slices included in each picture, and the second “Σ” indicates the sum of all macroblocks included in each slice. Each variable is as follows.

  In Equation 1, PIQM indicates pic_init_qp_minus 26, which is a value for setting an initial value of QP defined in the Picture Parameter Set (PPS), and a value obtained by subtracting 26 from the actual initial value is set. SQD indicates slice_qp_delta, is a value defined in the slice header, and is a value for setting an initial value of QP for each slice. MQD indicates mb_qp_delta and is a value defined for each macroblock, and is a difference value between the quantization parameter QP of the macroblock and the immediately preceding macroblock. Mb indicates Macroblocks, is a value defined in the slice header, and is the number of macroblocks included in the slice. NSGM indicates num_slice_groups_minus1, is a value defined in the PPS, and represents a value obtained by subtracting 1 from the number of slices included in the picture.

Further, if the quantized value QPp per picture is added to the number of pictures corresponding to the shift determination section JA, and the average is taken as the picture average QPa,
Picture average QPa = ΣQPp / deviation determination section JA (Expression 2)
It is expressed. Here, “Σ” indicates the sum of the number of pictures (that is, GOP length or threshold value B) included in the shift determination section JA.

The video importance level determination unit 11 calculates the quantization importance level IL1 as shown in Expression 3. That is,
Quantization importance IL1 = 102− (2 × picture average QPa) (Expression 3)
By so doing, the quantization importance IL1 is normalized to a value in the range of 1 to 100.

Hereinafter, a method of calculating the size importance IL2 will be described. The size importance IL2 is calculated based on the data size of the picture as one of the parameters for calculating the importance of the video. The data size of the picture is measured according to nal_unit_type in the network abstraction layer (NAL). The data size can be measured as a binary size of nal_unit_type of “1”, that is, an Instantaneous Decoding Refresh (IDR) picture, or “5”, that is, a picture other than IDR. In order to normalize this with respect to the size of the video, the picture data size is divided by the picture size (the number of vertical pixels of one picture × the number of horizontal pixels). Further, the video importance level determination unit 11 calculates the total picture data size included in the shift determination section JA and sets it as the video data amount DV. That is,
Video data amount DV = Σ (picture data size / picture size) (Formula 4)

Here, “Σ” represents the sum of pictures included in the shift determination section JA. Further, the video importance level determination unit 11 calculates the average of the video data amount DV of the portion reproduced so far, and if this is (average size), the size importance level IL2 is expressed as in the following Expression 5. Is done.
Size importance IL2
= Max (((video data amount DV) / (average size)) × 50, 100) (Equation 5)
According to Equation 5, the size importance IL2 is normalized to a value in the range of 1 to 100.

Hereinafter, a method of calculating the vector importance IL3 based on the total motion vector amount of the picture will be described. In calculating the vector importance level IL3, the video importance level determination unit 11 first adds the total amount of motion vectors for all macroblocks for each frame and takes an average per macroblock. For the vector importance IL3, the total amount of average motion vectors per macroblock is multiplied by 10, for example, and the upper limit is suppressed to 100. As an example of the vector importance IL3, the following Expression 6 is given.
Vector importance IL3 = max ((Σ (motion vector length of each macroblock)
/ Number of macroblocks of picture × 10), 100) (Equation 6)

  Here, “Σ” represents the sum of the number of macroblocks in one picture. According to Equation 6, the vector importance IL3 is normalized to a value in the range of 1-100.

  The video importance determination unit 11 calculates the video importance IL based on the quantization importance IL1, the size importance IL2, and the vector importance IL3 calculated as described above. The video importance IL is calculated as, for example, an arithmetic average of the quantization importance IL1, the size importance IL2, and the vector importance IL3, that is, the video importance IL = (IL1 + IL2 + IL3) / 3.

  Next, calculation of the voice importance level SL will be described. The voice importance level determination unit 9 transfers, for example, one frame worth of the voice data decoded by the voice decoding unit 23 to the frequency domain. The voice importance determination unit 9 calculates the integral of the amplitude for each predetermined frequency range, and sets the volume for each frequency in the time of one frame.

式７により、周波数毎の音量ＳＶａ（ｊ）は、０〜１００までの範囲に正規化される。 FIG. 5 is a diagram for explaining a method of calculating a volume for each frequency domain. In FIG. 5, the vertical axis indicates the amplitude x (i) corresponding to the volume, and the horizontal axis indicates the frequency i on the logarithmic axis. FIG. 5 shows, for example, the result of converting audio data for one frame into amplitude in the frequency domain. The horizontal axis is divided into frequency ranges for each frequency 10 ^k to 10 ^{k + 1} (k is an integer). At this time, the integration of the amplitude for each frequency range corresponds to the volume for each frequency range (hereinafter referred to as the volume for each frequency). A frame number j indicating a temporal order is attached to the frame, and a volume SVa (j) for each frequency in the frame number j is expressed by the following Expression 7.

According to Equation 7, the volume SVa (j) for each frequency is normalized to a range from 0 to 100.

Hereinafter, a method of calculating the variation importance SL1 using the volume SVa (j) will be described. The variation importance SL1 is an average of the sound level change amount for each frequency band. That is, the degree of importance SL1 takes the difference from the previous sample (here, the previous frame) for each volume SVa (j) for each frequency of Equation 7, and the difference corresponds to the audio timer value ST in S108. The determination time and the frequency region corresponding to all frequencies are added together, and the average is calculated. When the number of frames included in the shift determination time is the number of frames FN and the entire frequency region is 10 ⁰ to 10 ^N (N is a natural number), the variation importance SL1 is expressed by the following Expression 8.
Fluctuation importance SL1 = 1 / NΣ (Σ | SVa (j) −SV (j−1) |) / FN (Expression 8)

  Here, the first “Σ” represents the sum of all frequency regions, and the second “Σ” represents the sum of the number of frames in the shift determination time (j = 1 to FN). Note that the voice importance SL1 is calculated as a value from 0 to 100 according to Equation 8.

Next, a method for calculating the volume importance SL2 using the volume SVa (j) will be described. The volume importance level SL2 is the total sum of the volume levels. For each volume SVa for each frequency, the frame included in the deviation determination time corresponding to the audio timer value ST in S108 and the frequency region corresponding to all frequencies are added. Calculate the average. That is, it is expressed by the following formula 9.
Volume importance level SL2 = 1 / NΣΣSVa (j) / FN (Expression 9)

Here, the first “Σ” represents the sum of all frequency regions (10 ⁰ to 10 ^N ), and the second “Σ” represents the sum of the number of frames in the shift determination time (j = 1 to 1). FN). The voice importance SL2 is calculated as a value from 0 to 100.

  The voice importance level determination unit 9 sets the voice importance level SL based on the fluctuation importance level SL1 and the volume importance level SL2 calculated as described above. For example, voice importance SL = (variation importance SL1 + volume importance SL2) / 2 is calculated.

  In S134 of FIG. 4A, the comparative evaluation unit 13 determines the magnitude of the video importance IL calculated as described above and the audio importance SL. If the video importance level IL is greater than the audio importance level SL (S134: Yes), the comparative evaluation unit 13 advances the process to S135 in FIG. 4B in order to preferentially operate the audio. When the audio importance level SL is greater than the video importance level IL (S134: No), the comparative evaluation unit 13 advances the processing to S145 in FIG. 4C in order to preferentially operate the video. Hereinafter, according to the determination result of the comparative evaluation unit 13, the automatic correction unit 15 synchronizes video and audio.

  First, a case where the voice is preferentially operated will be described. As shown in FIG. 4B, in S135, the voice operation unit 31 of the automatic correction unit 15 can determine that the voice part of the operation target (for the deviation determination time to be played back) is near silence or that the voice change amount is small. Determine if there is a part. It can be determined that the sound volume is close to silence, for example, when the volume importance level SL2 is equal to or lower than the voice importance level SL calculated in advance for white noise. For example, the change importance SL1 may be determined to be equal to or less than the change importance SL1 calculated in advance for white noise.

  In S135, there is a portion that is close to the operation target sound portion, or a portion that can be determined that the sound change amount is small (hereinafter, a portion that is close to silence and a portion that can be determined that the sound change amount is small is collectively referred to as a silence-corresponding portion). When it is determined that there is (S135: Yes), the voice operation unit 31 determines whether or not the deviation time L1 is a positive number (S136). When the deviation time L1 is a positive number (S136: Yes), since the voice is advanced, the voice operation unit 31 can determine that the deviation time L1 minutes is close to silence or the amount of change in voice is small. Is reproduced and expanded many times to complete the synchronization process (S137).

  In S136, when it is determined that the shift time L1 is a negative number and the video is delayed (S136: No), the voice operation unit 31 determines that the absolute value of the shift time L1 is the time corresponding to the silence ( It is determined whether it is longer than the silent equivalent time (S138). When the absolute value of the deviation time L1 is longer than the silence equivalent time (S138: Yes), the voice operation unit 31 deletes the silence equivalent part without reproducing it (S139), and the deviation time L1 = the deviation time L1 + silence. The corresponding time is set (S140), and the process proceeds to S142. If the absolute value of the deviation time L1 is equal to or less than the silence equivalent time (S138: No), the voice operation unit 31 deletes the voice corresponding to the silence corresponding to the deviation time L1 and completes the synchronization process (S141). The process proceeds to S153.

  In S135, when there is no silent equivalent (S135: No), the voice operation unit 31 determines whether the speed change rate VC = (voice timer value ST / (voice timer value ST + | deviation time L1 |) is equal to or greater than a certain value). (S142) The fixed value used for the determination can be set to 0.8, for example.

  When the speed change rate VC is less than a certain value (S142: No), the voice operation unit 31 advances the process to S145. This is because, when the audio playback speed is changed more than a certain level, it is determined that the uncomfortable feeling due to changing the sound speed is greater than the uncomfortable feeling caused by manipulating the video. Here, the voice operation unit 31 may change the voice playback speed by a ratio corresponding to the fixed value, and then proceed to S145.

  When the speed change rate VC is equal to or greater than a certain value (S142: No), the voice operation unit 31 sets voice playback speed change value = (voice timer value ST + shift time L1) / voice timer value ST (S143), The synchronization process is completed by changing the reproduction speed according to the voice reproduction speed change value (S144).

  Here, the audio reproduction speed changing method will be described. The audio reproduction speed changing method employed here is a method of converting the reproduction speed of digital audio data without changing the pitch by omitting or inserting a part of the digital audio data.

  First, the voice operation unit 31 divides the voice data corresponding to the operation target, that is, the voice judgment section JA to be reproduced from now, into sections of a certain period, and determines the rate of omission or expansion. For example, if 10% is omitted, the voice operation unit 31 may omit one section every 10 sections as a guide in the divided voice data, and if 10% is expanded, every 10 sections. Insert one section as a guide. Next, the voice operation unit 31 operates the voice data according to the rate of omission or expansion. Various conventional methods such as the method described in Patent Document 3 can be applied to such a change in the audio reproduction speed.

  In the case of processing a video, as shown in FIG. 4C, first, the video operation unit 33 determines whether the shift time L1 is positive or negative (S145). When the shift time L1 is positive (S145: Yes), the video operation unit 33 performs frame thinning processing because the video is delayed with respect to the audio, and when negative (S145: NO), On the other hand, since the video is progressing, frame interpolation processing is performed.

  In the frame decimation process, the video operation unit 33 first calculates the number of frame decimation (S146). That is, the video operation unit 33 calculates the number of frames to be thinned = the shift time L1 / frame rate (Frame per Second: FPS). Round off to the nearest decimal point.

  When there is a non-reference picture in the operation target image (for the shift determination section JA to be played back), the video operation unit 33 thins the video data so that it is not included in the video data to be played back within a range that does not exceed the frame skipping number ( S147). If the number of non-reference pictures does not reach the number of frames to be thinned out, the video operation unit 33 subsequently thins out the B picture after the P picture from the one with the higher quantization coefficient QP (S148). If the number of frames thinned out does not reach the processing up to S148, the video operation unit 33 thins out reference pictures such as I frames and P frames. When the reference picture is thinned out in this way, the picture is decoded for the next frame (S149).

  As described above, when thinning out a frame, the video operation unit 33 thins out a non-reference frame first and then thins out a B frame in IBBP in order to reduce the influence on the subsequent frame. Further, frames are thinned out in the order of frames with a large quantization coefficient QP in IPPP. When the frame is thinned, the video operation unit 33 performs decoding for subsequent frames except for the non-reference frame. Thus, the synchronization process is completed.

  If the shift time L1 is a negative number (S145: No), frame interpolation processing is performed. The video operation unit 33 calculates frame interpolation number = −shift time L1 / FPS (S150). Round off to the nearest decimal point.

  The video manipulating unit 33 creates an average frame of the frame and the next frame for the interpolated frame having as high a quantization coefficient QP as possible (S151). For example, when frame interpolation is performed, the video operation unit 33 obtains an average value of the pixel values of each pixel for the previous and subsequent frames of the target portion, and inserts this between the previous and next frames as an interpolation frame.

  At the time of frame interpolation, the video operation unit 33 performs the interpolation processing for the number of interpolations so that the frames to be interpolated are arranged as evenly as possible with respect to the threshold value B (S152). After completing the thinning for the number of frames to be interpolated, the process proceeds to the initialization process of S153.

  As an initialization process of S153, the main control unit 3 substitutes the reproduced audio time (audio timer value ST) for the threshold A and substitutes the number of displayed video frames (moving picture counter value ICt) for the threshold B. Thereafter, the main control unit 3 initializes the audio timer value ST = 0 and the moving image counter value ICt = 0, and returns to S103 to repeat the processing.

  Note that the separation unit 21, the audio decoding unit 23, the audio reproduction unit 25, the video decoding unit 27, and the video reproduction unit 29 of the present embodiment are examples of the decoding and reproduction unit of the present invention. The audio importance level determination unit 9 is an example of an audio importance level calculation unit, and the video importance level determination unit 11 is an example of a video importance level calculation unit.

  As described above, in the playback apparatus 1 according to the present embodiment, the video importance level IL and the audio importance level SL are calculated. The video importance IL indicates the degree of video complexity, and is calculated based on the video stream before decoding. The voice importance level SL indicates the characteristics of the volume of the voice and is calculated based on the decoded voice data. Further, the playback device 1 performs synchronization by determining which one of video and audio is to be preferentially operated according to the video importance IL, the audio importance SL, and the shift time L1 in the shift determination section JA. .

  Therefore, in the playback apparatus 1 according to the present embodiment, the calculation of the video importance level IL and the audio importance level SL is not performed during the playback of the video itself, so that there is an effect that the playback is not affected. Further, according to the playback apparatus 1, it is possible to perform the synchronization process using a method that is considered to have little influence on the viewer by the synchronization process. Therefore, due to lack of processing capacity, it is less likely to give viewers a sense of discomfort and discomfort, such as the playback video dropping frames, sound being interrupted, or deviation not being corrected at fixed intervals. effective.

  The present invention is not limited to the embodiments described above, and various configurations or embodiments can be adopted without departing from the gist of the present invention. For example, in the initialization process of S153, the audio timer value ST is set as the threshold A and the moving image counter value ICt is set as the threshold B. However, the present invention is not limited to this. For example, every time S153 is performed, the values of the audio timer value S and the moving image counter value ICt are stored in the storage unit 35, and the average values thereof are calculated and substituted as initial values of the threshold A and the threshold B. It may be.

  The threshold A may be decreased when the variation importance SL1 is large, and may be increased when the variation importance SL1 is small. For example, the calculated fluctuation importance SL1 is stored in the storage unit 35, the statistical distribution of the fluctuation importance SL1 is calculated, and whether or not the current fluctuation importance SL1 falls within the upper 20% range in the probability distribution. Thus, it is determined whether or not the change amount of the voice is large. When it is determined that the amount of change in the sound is large, the threshold A may be updated to 0.5 seconds, and otherwise, the threshold A may be sequentially updated to 1 second. This is because the part where the amount of change is large is considered that there are many opportunities for the viewer to feel a lip sync shift.

  The video importance IL can be calculated based on at least one of the quantization importance IL1, the size importance IL2, and the vector importance IL3. It should be noted that the size importance IL2 may be reflected after 30 seconds from the start of reproduction, where (average size) will stabilize. The voice importance SL can be calculated based on at least one of the fluctuation importance SL1 and the volume importance SL2.

The video importance level IL and the audio importance level SL may be calculated using another calculation method. For example, if the movement is small and the quantization coefficient is large, the pattern is fine but the movement is small (for example, when the music is the main and the video is used as the background). Adjustments such as removing clogs may be performed. Further, since the size of small speech tends to be noticed by human beings, the integrand x (i) in Equation 7 in the calculation of speech importance SL is replaced with 10 × (x (i)) ^1/2. It may be. As a result, there is an effect that the operation for synchronization is made closer to the degree of concern when viewed by a human.

The reproduction apparatus according to the present embodiment is an H.264 encoded with data compression. The present invention can be applied as a digital moving image reproducing apparatus based on H.264, MPEG2, or the like.
Here, an example of a computer that is commonly applied to cause the computer to perform the operation of the video / audio reproduction method according to the above embodiment will be described. FIG. 6 is a block diagram illustrating an example of a hardware configuration of a standard computer. As shown in FIG. 6, a computer 300 includes a central processing unit (CPU) 302, a memory 304, an input device 306, an output device 308, an external storage device 312, a medium driving device 314, a network connection device, and the like via a bus 310. It is connected.

  The CPU 302 is an arithmetic processing unit that controls the operation of the entire computer 300. The memory 304 is a storage unit for storing in advance a program for controlling the operation of the computer 300 or using it as a work area when necessary when executing the program. The memory 304 is, for example, a random access memory (RAM), a read only memory (ROM), or the like. The input device 306 is a device that, when operated by a computer user, acquires various information input from the user associated with the operation content and sends the acquired input information to the CPU 302. Keyboard device, mouse device, etc. The output device 308 is a device that outputs a processing result by the computer 300, and includes a display device and the like. For example, the display device displays text and images according to display data sent by the CPU 302.

  The external storage device 312 is a storage device such as a hard disk, and stores various control programs executed by the CPU 302, acquired data, and the like. The medium driving device 314 is a device for writing to and reading from the portable recording medium 316. The CPU 302 can read out and execute a predetermined control program recorded on the portable recording medium 316 via the recording medium driving device 314 to perform various control processes. The CPU 302 can read out and reproduce the moving image content recorded on the portable recording medium 316. The storage portable recording medium 316 is, for example, a Compact Disc (CD) -ROM, a Digital Versatile Disc (DVD), a Universal Serial Bus (USB) memory, or the like.

  The network connection device 318 is an interface device that manages transmission / reception of various data performed between the outside by wired or wireless. The CPU 302 can also acquire and reproduce external video content via the network connection device 318. A bus 310 is a communication path for connecting the above devices and the like to exchange data.

A program that causes a computer to execute the video / audio reproduction method according to the above-described embodiment is stored in, for example, the external storage device 312. The CPU 302 reads the program from the external storage device 312 and causes the computer 300 to perform an audio / video reproduction operation. At this time, first, a control program for causing the CPU 302 to perform video / audio reproduction processing is created and stored in the external storage device 312. Then, a predetermined instruction is given from the input device 306 to the CPU 302 so that the control program is read from the external storage device 312 and executed. The program may be stored in the portable recording medium 316.
The CPU 302 can read out and reproduce the moving image content recorded on the portable recording medium 316.

Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
A playback device that acquires a stream including encoded video and audio and reproduces the video and audio,
A signal acquisition unit for acquiring the stream;
A decoding reproduction unit that generates video data and audio data by separating and decoding the acquired stream, and reproduces the first video and the first audio at the first shift determination time;
A shift determination unit that determines a shift amount during reproduction of the first video and the first audio during the first shift determination time;
A video importance level calculating unit that calculates a video importance level indicating a degree of complexity of a second video that is reproduced in a second shift determination time next to the first shift determination time;
A voice importance level calculation unit for calculating a voice importance level indicating a volume characteristic of the second voice reproduced at a second shift determination time next to the first shift determination time;
An automatic correction unit that controls reproduction of the second video and the second audio so as to correct a deviation during reproduction between the video and the audio based on the video importance, the audio importance, and the shift amount. When,
A playback apparatus comprising:
(Appendix 2)
The supplementary note 1 is characterized in that the video importance is calculated based on at least one of a quantization coefficient added when the video is encoded, a data amount, and a total amount of motion vector lengths. The reproducing apparatus as described.
(Appendix 3)
The voice importance is calculated based on at least one of an average value of temporal changes in volume for each predetermined frequency range of the audio data and an average value of volume for each predetermined frequency range of the audio data. The reproducing apparatus according to appendix 1 or appendix 2, wherein:
(Appendix 4)
The automatic correction unit further corrects a deviation in reproduction between the second video and the second audio based on whether or not there is a non-reference picture in the second video before decoding. 4. The playback device according to any one of appendix 1 to appendix 3, which is characterized.
(Appendix 5)
The automatic correction unit is
A voice operation unit for operating the voice data;
A video operation unit for operating the video data;
Have
Based on whether there is a non-reference picture in the video before decoding and the magnitude relationship between the video importance level and the audio importance level, an audio operation unit and a video operation unit are used when correcting the shift at the time of reproduction. The playback apparatus according to appendix 4, wherein which operation is to be prioritized is determined.
(Appendix 6)
The automatic correction unit is
When the video importance is greater than the audio importance, priority is given to the operation of the audio operation unit,
6. The playback apparatus according to appendix 5, wherein when the audio importance level is greater than the video importance level, a shift between the video and the audio is corrected by performing an operation using the video operation unit.
(Appendix 7)
The voice operation unit
In a section corresponding to the second deviation determination time, there is a silent equivalent section in which the voice importance is equal to or less than a predetermined value,
When the reproduction of the first audio is ahead of the reproduction of the first video, the silent equivalent section is reproduced repeatedly,
When the reproduction of the first video is ahead of the reproduction of the first audio, an operation for deleting the silent equivalent section according to the deviation amount, an operation for changing the audio reproduction speed, Perform at least one of the operations to perform operations on the video operation unit,
In a section where the voice importance is calculated, there is no silent equivalent section where the voice importance is a predetermined value or less,
When the deviation amount is less than a predetermined ratio with respect to the second deviation determination time, the audio playback speed is changed,
The reproduction apparatus according to appendix 5 or appendix 6, wherein when the shift amount is equal to or greater than a predetermined ratio with respect to the second shift determination time, the video operation unit is operated.
(Appendix 8)
The video operation unit
If the reproduction of the first audio is ahead of the reproduction of the first video, at least a process of not reproducing the non-reference picture is performed,
The supplementary note 5 or 5, wherein when the reproduction of the first video is more advanced than the reproduction of the first audio, a process of interpolating a frame having an average of pixel values of preceding and succeeding frames is performed. The reproducing apparatus according to appendix 6.
(Appendix 9)
A reproduction method for obtaining a stream including encoded video and audio and reproducing the video and audio,
Get the stream,
Playing the first video and the first audio based on the first deviation determination time and the stream;
Determining a shift amount during reproduction of the first video and the first audio in the first shift determination time;
Calculating a video importance level indicating a degree of complexity of a second video to be reproduced in a second shift determination time next to the first shift determination time;
Calculating a voice importance level indicating a volume characteristic of the second voice reproduced in the second deviation determination time;
Based on the video importance level, the audio importance level, and the shift amount, the second video and the second audio at the second shift determination time so as to correct a shift during playback between the video and the audio. Control the playback of the
A reproduction method characterized by the above.
(Appendix 10)
Playing back the first video and the first audio, determining the shift amount, calculating the video importance, calculating the audio importance, and controlling the playback are repeatedly performed. The reproduction method according to appendix 9.
(Appendix 11)
A program for obtaining a stream including encoded video and audio and causing a computer to execute a process of reproducing the video and audio,
Get the stream,
Playing the first video and the first audio based on the first deviation determination time and the stream;
Determining a shift amount during reproduction of the first video and the first sound in the first shift determination time;
Calculating a video importance level indicating a degree of complexity of a second video to be reproduced in a second shift determination time next to the first shift determination time;
Calculating a voice importance level indicating a volume characteristic of the second voice reproduced in the second deviation determination time;
Based on the video importance level, the audio importance level, and the shift amount, the second video and the second audio at the second shift determination time so as to correct a shift during playback between the video and the audio. A program for causing the computer to execute a process for controlling the reproduction of an image.
(Appendix 12)
Processing for reproducing the first video and the first audio, processing for determining the shift amount, processing for calculating the video importance, processing for calculating the audio importance, and processing for controlling the reproduction The program according to appendix 11, which is repeatedly executed by the computer.

DESCRIPTION OF SYMBOLS 1 Video / audio reproduction device 3 Main control part 5 Input reproduction part 7 Deviation determination part 9 Audio importance degree determination part 11 Video importance degree determination part 13 Comparison evaluation part 15 Automatic correction part 17 System bus 19 Signal acquisition part 21 Separation part 23 Voice decoding Unit 25 Audio playback unit 27 Video decoding unit 29 Video playback unit 31 Audio operation unit 33 Video operation unit

Claims (8)

A playback device that acquires a stream including encoded video and audio and reproduces the video and audio,
A signal acquisition unit for acquiring the stream;
A decoding reproduction unit that generates video data and audio data by separating and decoding the acquired stream, and reproduces the first video and the first audio at the first shift determination time;
A shift determination unit that determines a shift amount during reproduction of the first video and the first audio during the first shift determination time;
A video importance level calculating unit that calculates a video importance level indicating a degree of complexity of a second video that is reproduced in a second shift determination time next to the first shift determination time;
A voice importance level calculation unit for calculating a voice importance level indicating a volume characteristic of the second voice reproduced at a second shift determination time next to the first shift determination time;
An automatic correction unit that controls reproduction of the second video and the second audio so as to correct a deviation during reproduction between the video and the audio based on the video importance, the audio importance, and the shift amount. When,
A playback apparatus comprising:   2. The video importance level is calculated based on at least one of a quantization coefficient, a data amount, and a total motion vector length added when the video is encoded. The playback device described in 1.   The voice importance is calculated based on at least one of an average value of temporal changes in volume for each predetermined frequency range of the audio data and an average value of volume for each predetermined frequency range of the audio data. The reproducing apparatus according to claim 1 or 2, characterized in that:   The automatic correction unit further corrects a deviation in reproduction between the second video and the second audio based on whether or not there is a non-reference picture in the second video before decoding. The reproducing apparatus according to any one of claims 1 to 3, wherein the reproducing apparatus is characterized. The automatic correction unit is
A voice operation unit for operating the voice data;
A video operation unit for operating the video data;
Have
Based on whether there is a non-reference picture in the video before decoding and the magnitude relationship between the video importance level and the audio importance level, an audio operation unit and a video operation unit are used when correcting the shift at the time of reproduction. The playback apparatus according to claim 4, wherein which operation is to be prioritized is determined. The automatic correction unit is
When the video importance is greater than the audio importance, priority is given to the operation of the audio operation unit,
6. The playback apparatus according to claim 5, wherein when the audio importance level is greater than the video importance level, a deviation between the video and the audio is corrected by performing an operation using the video operation unit. A reproduction method for obtaining a stream including encoded video and audio and reproducing the video and audio,
Get the stream,
Playing the first video and the first audio based on the first deviation determination time and the stream;
Determining a shift amount during reproduction of the first video and the first audio in the first shift determination time;
Calculating a video importance level indicating a degree of complexity of a second video to be reproduced in a second shift determination time next to the first shift determination time;
Calculating a voice importance level indicating a volume characteristic of the second voice reproduced in the second deviation determination time;
Based on the video importance level, the audio importance level, and the shift amount, the second video and the second audio at the second shift determination time so as to correct a shift during playback between the video and the audio. A reproduction method characterized by controlling reproduction of the video. A program for obtaining a stream including encoded video and audio and causing a computer to execute a process of reproducing the video and audio,
Get the stream,
Playing the first video and the first audio based on the first deviation determination time and the stream;
Determining a shift amount during reproduction of the first video and the first sound in the first shift determination time;
Calculating a video importance level indicating a degree of complexity of a second video to be reproduced in a second shift determination time next to the first shift determination time;
Calculating a voice importance level indicating a volume characteristic of the second voice reproduced in the second deviation determination time;
Based on the video importance level, the audio importance level, and the shift amount, the second video and the second audio at the second shift determination time so as to correct a shift during playback between the video and the audio. A program for causing the computer to execute a process for controlling the reproduction of an image.

Images