JP2012124586A - Video reproducing method, video reproducing device, video reproducing program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video reproducing method capable of concealing a transmission error in a video reproducing device to which video encoded data with a hierarchical structure is input.SOLUTION: A vide reproducing method for concealing a transmission error and reproducing a video when a desired hierarchy to be reproduced and hierarchies lower than the desired hierarchy are decoded at a predetermined time comprises: a decoding signal formation step of extracting encoding information from a received encoding packet of a predetermined time stored in a storage part and forming a decoding signal; a signal reading step of reading received encoding information of other than the predetermined time stored in the storage part, a generated decoding signal, and a reproduction signal; a pseudo signal generation step of generating signals in the desired hierarchy to be reproduced at the predetermined time in a pseudo manner using encoding information and decoding signals of the predetermined time and encoding information, decoding signals, and reproduction signals of other than the predetermined time; and a signal output step of outputting a signal generated in the pseudo manner as a reproduction signal of the predetermined time.

Description

  The present invention relates to a video playback device, a video playback method, a video playback program, and a recording medium for concealing transmission errors in a video playback device to which video encoded data having a hierarchical structure is input.

  When streaming playback of video data using a network line with transmission errors such as packet delay and packet loss, video quality degradation such as generation of block noise, playback delay, and frame skipping occurs. In order to avoid this quality degradation, a transmission error recovery technique and a transmission error concealment technique are usually used. The transmission error recovery technique is a technique in which redundant data is given to original video data in advance, and information of packets lost due to a transmission error is recovered using the redundant data. A typical example is a forward error correction (FEC) technique. On the other hand, with transmission error concealment technology, packets are lost due to transmission errors and cannot be recovered even using forward error correction technology, etc., or video is not completed at the timing to be reproduced due to transmission delay (not reproducible) In some cases, it is a technique for constructing a high-quality playback video as much as possible by using only received information. For example, when an error occurs in decoded image data, an image display control method is known that conceals a transmission error by displaying a currently displayed image repeatedly (see, for example, Patent Document 1).

JP 2001-119893 A

  However, since the image display control method described in Patent Document 1 is a method of repeatedly displaying the currently displayed image, there is a problem that it is difficult to reproduce the movement of the subject in the error occurrence section. This is because the deterioration of the subjective quality becomes more remarkable as the movement of the subject increases. In addition, although the technique described in Patent Document 1 can be applied to hierarchical coding such as scalable video coding, there is a similar problem that it is difficult to reproduce the motion of a subject.

  The present invention has been made in view of such circumstances, and a video playback apparatus, a video playback method, and a video capable of concealing transmission errors in a video playback apparatus to which video encoded data having a hierarchical structure is input. An object is to provide a reproduction program and a recording medium.

  The present invention can decode up to a lower layer than a desired layer to be reproduced at a predetermined time in order to conceal transmission errors that occur when decoding and reproducing encoded data layered in a plurality of received layers. A decoded signal forming step for extracting encoded information from a received encoded packet at the predetermined time stored in a storage unit and forming a decoded signal, A signal reading step of reading received encoded information other than the predetermined time stored in the storage unit, a generated decoded signal and a reproduction signal, the encoded information and decoded signal at the predetermined time, and the predetermined time Pseudo signal generation that artificially generates a signal of the desired layer to be reproduced at the predetermined time using encoded information other than the above, a decoded signal, and a reproduction signal And steps, and having a signal output step of outputting the pseudo-generated signal as a reproduction signal of the predetermined time.

  The present invention is characterized in that in the pseudo signal generation step, there are two or more methods for generating the pseudo signal, and an optimal method is selected for each predetermined hierarchy and predetermined image area.

  In the present invention, in the pseudo signal generation step, there are two methods for generating the pseudo signal, and the pseudo signals generated by the two methods are weighted and averaged using a predetermined weighting factor. A pseudo signal is generated.

  According to the present invention, in the case of spatial scalable coding, the method of generating the pseudo signal expands the decoded signal of the lower layer at the predetermined time according to the resolution ratio of the desired layer and the lower layer according to a predetermined procedure. A method of using the processed signal as the pseudo signal, and using the motion information obtained by scaling the motion information of the lower layer at the predetermined time according to the resolution ratio of the desired layer and the lower layer according to a predetermined procedure. The lower layer prediction residual signal at the predetermined time is enlarged according to the resolution ratio of the desired layer and the lower layer according to a predetermined procedure, and the motion compensation signal and the prediction after enlargement are performed. There are two methods that use the pseudo signal as a sum of residual signals.

  In the case of SNR scalable coding, the present invention relates to a method for generating the pseudo signal in which the decoded signal of the lower layer at the predetermined time is the pseudo signal, and the motion of the lower layer at the predetermined time. The method includes two methods of performing motion compensation in the desired layer using information and using the motion compensation signal and the prediction residual signal in the lower layer at the predetermined time as the pseudo signal. To do.

  According to the present invention, the method for generating the pseudo signal may be selected by calculating a sum of absolute values of prediction residual signals held by the lower-level image areas existing in the same spatial position as the target image area as a predetermined value. The selection is made based on the determination result of whether or not it is larger than the threshold value.

  The present invention is characterized in that the method of generating the pseudo signal is selected based on a time accumulated value of transmission errors continuously generated before the predetermined time.

  According to the present invention, the selection of the method for generating the pseudo signal has two or more candidates to be performed in the image region based on information on the method of generating the pseudo signal already selected in the predetermined image region. It is characterized in that an optimum method is selected from among error concealment signal generation method candidates after the number of error concealment signal generation method candidates is reduced.

  The present invention can decode up to a lower layer than a desired layer to be reproduced at a predetermined time in order to conceal transmission errors that occur when decoding and reproducing encoded data layered in a plurality of received layers. A decoded signal forming unit for extracting encoded information from the received encoded packet at the predetermined time stored in the storage unit and forming a decoded signal, A signal reading unit that reads received encoded information other than the predetermined time stored in the storage unit, a generated decoded signal and a reproduction signal, the encoded information and decoded signal at the predetermined time, and the predetermined time A pseudo signal generation unit that artificially generates a signal of the desired layer to be reproduced at the predetermined time using encoded information other than the above, a decoded signal, and a reproduction signal; Characterized in that it comprises a signal output section for outputting the generated signal as a reproduction signal of the predetermined time.

  The present invention can decode up to a lower layer than a desired layer to be reproduced at a predetermined time in order to conceal transmission errors that occur when decoding and reproducing encoded data layered in a plurality of received layers. A video reproduction program for causing a computer on a video reproduction apparatus to conceal a transmission error to perform video reproduction when encoding information is received from an encoded packet received at the predetermined time stored in a storage unit. A decoded signal forming step of extracting and forming a decoded signal; a signal reading step of reading received encoded information other than the predetermined time stored in the storage unit; a generated decoded signal and a reproduction signal; and the predetermined time Is reproduced at the predetermined time using the encoded information and the decoded signal, and the encoded information, the decoded signal and the reproduction signal other than the predetermined time. A pseudo signal generating step for generating a signal of the desired layer in a pseudo manner, and a signal output step for outputting the pseudo generated signal as a reproduction signal at the predetermined time. To do.

  The present invention is characterized in that the video reproduction program according to claim 10 is recorded.

  According to the present invention, when the desired layer of the desired frame is not decoded at the timing to be reproduced due to packet loss or transmission delay, the image quality of the finally reproduced video can be improved as compared with the conventional technique. can get. In particular, when the frequency characteristics are greatly different between hierarchies or when the movement of the subject in the frame is large, the effect of improving the image quality from the prior art becomes significant.

It is a block diagram which shows the structure of the video reproduction apparatus by one Embodiment of this invention. 3 is a flowchart showing an operation of the video reproduction apparatus shown in FIG. It is a block diagram which shows the structure of the error concealment process part 6 shown in FIG. It is a flowchart which shows operation | movement of the error concealment process part 6 shown in FIG. It is a block diagram which shows the structure of the error concealment signal generation part 12 shown in FIG. 6 is a flowchart illustrating an operation of the error concealment signal generation unit 12 illustrated in FIG. 5. It is a figure which shows an experimental result. It is a figure which shows an experimental result.

  A video playback apparatus according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. Here, as an example, it is assumed that hierarchically encoded data composed of three layers of the basic layer, the extended layer # 1, and the extended layer # 2 is input. In addition, as an example of hierarchically encoded data, data that has been subjected to spatial scalable encoding is input. As the spatial scalable coding, SVC (scalable video coding) or the like can be applied.

  In FIG. 1, reference numeral 1 denotes a packet receiving unit that receives a video packet in which encoded data is stored and stores the received video packet in the storage unit 8. In the following description, the storage unit 8 will be described as being configured with a register, a video memory, a physical memory, a cache, and the like. Reference numeral 2 denotes a packet loss detection unit that reads a video packet from the storage unit 8, checks whether a packet of each layer at the time is normally received, and derives a flag indicating whether or not the packet can be received for each layer. . The packet reception enable / disable flag indicates that the packet can be normally received when the flag is true, and the packet cannot be normally received when the flag is false.

  Note that the case where a packet cannot be received normally refers to the case where part or all of the packet is missing, or the case where a packet waiting for reception has not arrived within a predetermined time. This includes a case where data arrives without loss but data of another packet necessary for decoding the data in the packet is lost. For example, although packet #A storing information such as a quantization coefficient and a motion vector can be normally received, packet #B including information such as SPS (Sequence Parameter Set) and PPS (Picture Parameter Set) can be received. If there is no packet #A, it is impossible to generate a decoded signal of the image area included in the packet #A, and therefore it is considered that the packet #A cannot be received normally.

  The packet loss detection unit 2 is connected to the branch of the path A when the packet reception availability flag of all layers is true (true), and the packet reception availability flag of the extension layer # 2 is false (false), and the extension layer # 1 If the packet reception availability flag of the base layer is true, the packet is connected to the branch of the route B, the packet reception availability flag of the extension layer # 2 and the extension layer # 1 is false, and the packet reception availability flag of the base layer is true. For example, if the packet reception enable / disable flag of all layers is false, it is connected to the branch of route D.

  Reference numeral 3 is a basic that reads a video packet having base layer encoded data from the storage unit 8, extracts encoded data from the packet payload, performs decoding processing, and writes the decoded motion information and decoded signal to the storage unit 8. A hierarchical data decoding unit. Reference numeral 4 reads a video packet having encoded data of the enhancement layer # 1 from the storage unit 8, extracts the encoded data from the packet payload, performs a decoding process, and stores the decoded motion information and decoded signal in the storage unit 8. This is an extended hierarchy # 1 data decoding unit to be written.

  Reference numeral 5 reads a video packet having encoded data of enhancement layer # 2 from the storage unit 8, extracts the encoded data from the packet payload, performs a decoding process, and stores the decoded motion information and decoded signal in the storage unit 8. It is an enhancement layer # 2 data decoding unit that writes out and outputs the decoded signal as a playback video. Reference numeral 6 denotes an error concealment processing unit that performs error concealment processing using a lower layer. In the case of the branch of the path B, the error concealment processing unit 6 reads the motion information and decoded signal of the base layer and the extended layer # 1, the encoded data and the decoded signal at times other than the time from the storage unit 8, and A reproduction signal is generated and written to the storage unit 8. Further, in the case of the branch of the path C, the base layer motion information and the decoded signal, and the encoded data and the decoded signal at a time other than the time are read, and the reproduction signal at the time is generated. Write out.

  Code 7 is an error concealment that reads encoded data or a decoded signal at a time other than the time, generates a reproduction signal at the time, and writes it to the storage unit 8 without using a lower layer. It is a processing unit. The error concealment processing unit 7 performs error concealment processing by a known technique described in Patent Document 1, for example.

  Next, the operation of the video playback apparatus shown in FIG. 1 will be described with reference to FIG. First, the packet receiving unit 1 receives a video packet in which encoded data is stored, and stores the received video packet in the storage unit 8 (step S1). Subsequently, the packet loss detection unit 2 reads the video packet from the storage unit 8, checks whether the packet of each layer at the time is normally received, and returns a flag indicating whether or not the packet can be received for each layer ( Step S2).

  Next, the packet loss detection unit 2 determines whether or not the base layer packet is lost based on whether or not the base layer packet reception enable / disable flag is true (step S3). As a result of this determination, if the packet of the base layer is lost, it is connected to the route D, and the error concealment processing unit 7 performs an error concealment process (step S4). The error concealment process applied here is, for example, the error concealment process described in Patent Document 1. Then, the error concealment processing unit 7 reads encoded data and a decoded signal at times other than the time, generates a reproduction signal at the time, and writes it to the storage unit 8.

  On the other hand, if the packet of the base layer is not lost, the packet loss detection unit 2 determines whether the packet of the extended layer # 1 is lost based on whether the packet reception enable / disable flag of the extended layer # 1 is true. It is determined whether or not (step S5). As a result of the determination, if the packet of the enhancement layer # 1 is lost, the base layer data decoding unit 3 connected to the path C reads the video packet having the base layer encoded data from the storage unit 8. Then, the encoded data is extracted from the packet payload, the decoding process is performed, and the decoded motion information and the decoded signal are written in the storage unit 8 (step S6).

  Next, when the packet of the enhancement layer # 1 is not lost, the packet loss detection unit 2 loses the packet of the enhancement layer # 2 based on whether the packet reception enable / disable flag of the enhancement layer # 2 is true. It is determined whether or not (step S7). As a result of the determination, if the packet of the enhancement layer # 2 is lost, the base layer data decoding unit 3 connected to the path B reads the video packet having the base layer encoded data from the storage unit 8 Then, the encoded data is extracted from the packet payload, the decoding process is performed, the decoded motion information and the decoded signal are written to the storage unit 8, and then the extended layer # 1 data decoding unit 4 reads the extended layer # 1 from the storage unit 8. The video packet having the encoded data is read, the encoded data is extracted from the packet payload, the decoding process is performed, and the decoded motion information and the decoded signal are written in the storage unit 8 (step S8).

  Next, the error concealment processing unit 6 performs an error concealment process (step S9). When the process moves from step S6, the decoded motion information and decoded signal written in step S6 and the encoded data and decoded signal at times other than the time are read to generate a reproduction signal at the time, and the storage unit Write to 8. In addition, when the process proceeds from step S8, the decoded motion information and decoded signal written in step S8 and the encoded data and decoded signal at times other than the time are read to generate a reproduction signal at the time, Write to the storage unit 8.

  Next, when the packet of the extension layer # 2 is not lost, the base layer data decoding unit 3 connected to the path A reads the video packet having the base layer encoded data from the storage unit 8, and The encoded data is extracted from the payload, decoded, and the decoded motion information and decoded signal are written in the storage unit 8. Then, the enhancement layer # 1 data decoding unit 4 reads the video packet having the encoded data of the enhancement layer # 1 from the storage unit 8, extracts the encoded data from the packet payload, performs the decoding process, and decodes the motion information Or the decoded signal is written to the storage unit 8. Further, the enhancement layer # 2 data decoding unit 5 reads the video packet having the encoded data of the enhancement layer # 2, extracts the encoded data from the packet payload, performs a decoding process, and receives the decoded motion information and the decoded signal. By writing to the storage unit 8, the entire hierarchy is decoded (step S10). The decoded signal output here becomes a video for reproduction at the time.

  Next, the detailed configuration and operation of the error concealment processing unit 6 shown in FIG. 1 will be described with reference to FIGS. Here, it is assumed that an error concealment process is performed in an image area unit (16 × 16 pixel size or more) called a slice. In the following description, it is assumed that the packets of the base layer and the extension layer # 1 can be received and the packet of the extension layer # 2 cannot be received. As shown in FIG. 3, the error concealment processing unit 6 includes an all macroblock processing completion determination unit 11, an error concealment signal generation unit 12, and a post filter unit 13.

  The all macroblock processing completion determination unit 11 determines whether or not the error concealment processing is completed for all the macroblocks in the slice. If all the macroblock processing is completed, the processing proceeds to the post filter unit 13. If not completed, the error concealment signal generation process by the error concealment signal generation unit 12 is continued. The error concealment signal generation unit 12 reads the motion information and decoded signals of the base layer and the enhancement layer # 1 from the storage unit 8, performs error concealment processing, generates an error concealment signal for this macroblock, and stores it in the storage unit 8. Write out. The post filter unit 13 reads an error concealment signal in each macro block of the slice, applies a post filter, and outputs the filtered signal as a reproduction signal for the slice at the time. An example of a post filter here is H.264. A deblocking filter in H.264 / AVC or SVC can be applied.

  Next, the operation of the error concealment processing unit 6 shown in FIG. 3 will be described with reference to FIG. First, when a macro block unit loop is started, the error concealment signal generation unit 12 reads the motion information and decoded signals of the base layer and the enhancement layer # 1 from the storage unit 8, performs error concealment processing, and An error concealment signal is generated and written to the storage unit 8 (step S11). The all macroblock processing completion determination unit 11 determines whether the processing has been completed for all macroblocks, and repeats until the processing is completed for all macroblocks. The unit 13 reads the error concealment signal in each macroblock of the slice, applies a post filter, and outputs the filtered signal as a reproduced video signal of the slice at the time (step S12).

  Next, the detailed configuration and operation of the error concealment signal generator 12 shown in FIG. 3 will be described with reference to FIGS. In FIG. 5, reference numeral 21 denotes a lower layer corresponding block prediction mode determination unit that reads the prediction mode possessed by the lower layer corresponding block from the storage unit 8 and branches the process based on the prediction mode. The code | symbol 22 reads the prediction residual signal which a lower hierarchy corresponding | compatible block holds from the memory | storage part 8, calculates the sum total (SAD: Sum of Absolute Distance) of the prediction residual signal of a lower hierarchy corresponding | compatible block, and the memory | storage part 8 is a lower layer corresponding block residual signal SAD calculation processing unit to be output to FIG.

  Reference numeral 23 denotes a SAD value threshold value comparison unit that reads the calculated SAD value and a threshold value for the SAD value and branches the process based on whether the SAD value is larger than the threshold value. Here, the threshold value for the SAD value is given in advance from an external module, and an empirical value is set and determined as an external setting value from the application side, or calculated in another module. It is determined by the method etc. An individual threshold value may be set for each GOP (Group of Picture) in which slices, frames, and a plurality of frames are bundled, or may be determined from the relationship between a desired layer and a higher layer that can be decoded.

  Reference numeral 24 reads the decoded signal of the lower layer corresponding block from the storage unit 8, expands the decoded signal according to the resolution ratio of the extended layer # 1 and the extended layer # 2 by a predetermined decoded signal expansion method, This is a lower layer corresponding block decoded signal expanding unit that performs clipping processing on the expanded signal to an effective pixel value as necessary and outputs it to the storage unit 8 as an error concealment signal. As an example of the decoded signal expansion method, an expansion method using a linear filter such as an upsample filter used for inter-layer prediction in SVC, a super-resolution method, or the like can be applied.

  Reference numeral 25 reads the motion information held by the lower layer corresponding block, and changes the motion information from the resolution of the enhancement layer # 1 to the resolution of the enhancement layer # 2 by a predetermined motion information scaling method. The lower layer corresponding block motion information scaling unit which outputs the scaled motion information to the storage unit 8. The motion information here is information used for motion compensation, such as a motion vector, a reference frame index for motion prediction, a prediction mode, and a block division shape. In addition, as a scaling method, for example, when handling spatial scalable coding data, if the vertical / horizontal pixel number ratio between the extended layer # 2 and the extended layer # 1 is 2: 1, the motion vector is doubled in each of the xy directions. Processing to be performed is applicable. The prediction mode and the reference frame index inherit the information held by the lower layer corresponding block as it is.

  A motion compensation unit 26 reads scaled motion information from the storage unit 8, performs motion compensation in the macroblock using the scaled motion information, and outputs a signal obtained by motion compensation to the storage unit 8. It is. The code | symbol 27 reads the prediction residual signal which a lower hierarchy corresponding | compatible block hold | maintains, and according to the resolution ratio of the extended hierarchy # 1 and the extended hierarchy # 2 according to the resolution ratio of the extended hierarchy # 1 by the prediction residual signal expansion method defined beforehand It is a lower layer corresponding block residual signal expansion unit that expands and outputs the expanded prediction residual signal to the storage unit 8. As a prediction residual signal expansion method, an expansion method using a linear filter such as an upsample filter used for inter-layer prediction in SVC, a super-resolution method, or the like can be applied.

  Reference numeral 28 adds the output signal obtained by motion compensation and the enlarged prediction residual signal, performs clipping processing to the effective pixel value range as necessary, and stores it in the storage unit 8 as an error concealment signal. It is the signal addition part to output.

  Next, the detailed operation of the error concealment signal generator 12 shown in FIG. 5 will be described with reference to FIG. First, the lower layer corresponding block prediction mode determining unit 21 reads the prediction mode possessed by the lower layer corresponding block from the storage unit 8, and determines whether or not the prediction mode is the intra prediction mode (step S21). As a result of the determination, if the mode is an intra prediction mode, the process proceeds to the lower layer corresponding block decoded signal expanding unit 24. If the prediction mode is other than the intra prediction mode, the lower layer corresponding block residual signal SAD calculation processing unit 22 is performed. Transfer processing to.

  Here, the lower layer refers to a layer that is lower than the desired layer and that can be decoded, and the lower layer corresponding block is spatially the same as the macroblock in the lower layer that can be decoded. Refers to the block that exists at the location. For example, when handling spatial scalable coding data, assuming that the vertical pixel number ratio and horizontal pixel number ratio of the extension layer # 2 and the extension layer # 1 are both 2: 1, the upper left corner of the image is the origin ( The block corresponding to the lower layer of the macro block of the extension layer # 2 whose center is located at 2X, 2Y) is an 8 × 8 pixel size block centered at the (X, Y) position in the extension layer # 1. If the extension layer # 1 cannot be decoded, the base layer is the lower layer here.

  Next, the lower layer corresponding block residual signal SAD calculation processing unit 22 reads the prediction residual signal held by the lower layer corresponding block from the storage unit 8 and sums the absolute values of the prediction residual signals of the lower layer corresponding block ( SAD: Sum of Absolute Distance is calculated and output to the storage unit 8 (step S22). Subsequently, the SAD value threshold value comparison unit 23 reads the calculated SAD value and the threshold value for the SAD value from the storage unit 8, and determines whether or not the calculated SAD value is larger than the threshold value (step S31). S23). As a result of this determination, if the calculated SAD value is greater than the threshold value, the process proceeds to the lower layer corresponding block decoded signal expanding unit 24, and if the calculated SAD value is less than the threshold value, the process to the lower layer corresponding block motion information scaling unit 25 is performed. Transition.

  Next, when the calculated SAD value is larger than the threshold value, the lower layer corresponding block decoded signal expanding unit 24 reads the decoded signal of the lower layer corresponding block from the storage unit 8, and by a predetermined decoded signal expanding method, The decoded signal is enlarged according to the resolution ratio between the enhancement layer # 1 and the enhancement layer # 2, and the enlarged signal is subjected to clipping processing to an effective pixel value as necessary, and stored in the storage unit 8 as an error concealment signal. Output (step S24).

  On the other hand, if it is equal to or lower than the threshold value, the lower layer-corresponding block motion information scaling unit 25 reads the motion information held by the lower layer-corresponding block, and converts the motion information into the extension layer The resolution is changed from 1 to match the resolution of the extension layer # 2, and the scaled motion information is output to the storage unit 8 (step S25).

  Next, the motion compensation unit 26 reads the scaled motion information from the storage unit 8, performs motion compensation in the macroblock using the scaled motion information, and stores the signal obtained by the motion compensation in the storage unit 8. Output (step S26). Subsequently, the lower layer corresponding block residual signal expanding unit 27 reads the prediction residual signal held by the lower layer corresponding block from the storage unit 8, and uses the predetermined prediction residual signal expanding method to generate the prediction residual signal. Enlargement is performed according to the resolution ratio between the extension hierarchy # 1 and the extension hierarchy # 2, and the enlarged prediction residual signal is output to the storage unit 8 (step S27).

  Next, the signal adding unit 28 adds the signal obtained by motion compensation and the expanded prediction residual signal, performs clipping processing to the effective pixel value range as necessary, and stores it as an error concealment signal 8 (step S28).

  Note that the lower layer correspondence block motion information scaling unit 25, the motion compensation unit 26, and the lower layer correspondence block residual signal expansion unit 27 shown in FIG. The corresponding block motion information scaling unit 25 and the motion compensation unit 26 may be performed in this order, or the lower layer correspondence block motion information scaling unit 25, the series processing unit of the motion compensation unit 26, and the lower layer correspondence block residual signal expansion unit 27. These processes may be performed in parallel, and signal addition may be performed by the signal adder 28 after completion of both processes.

The series of processes in steps S22 and S23 described above is an example of how to switch between the error concealment signal generation process in step S24 and the error concealment signal generation process in steps S25 to S28. The switching method is not limited as long as it is switched with reference to the received encoded information and decoded signal. For example, as means other than the series processing unit in steps S22 and S23, the following can be applied.
(1) Whether the information amount of the encoded information held by the lower hierarchy block is larger than a threshold value.
(2) Whether the value of the quantization parameter / quantization step possessed by the lower hierarchy block is larger than the threshold value.
(3) Whether the prediction mode possessed by the lower hierarchy block is a prediction mode designated in advance to shift to S24.
(4) Whether or not the prediction mode possessed by the lower hierarchy block is a prediction mode designated in advance to move to steps S25 to S28.
(5) Whether the number of slices in which packet loss has occurred in the past or the number of slices in which packet loss has occurred in time is greater than a threshold value (6) Number of slices in which packet loss has occurred around that time Or whether the time interval is larger than the threshold value (7) whether the norm value of the motion vector held by the lower layer corresponding block is larger than the threshold value (8) generating an error concealment signal by motion compensation Whether the reference signal itself is a signal generated by error concealment.
One of these means may be applied, or switching may be performed based on a combination of these in multiple stages including the series of processes of steps S22 and S23 described above.

  In addition, an error concealment signal generation process to be applied to the macroblock is defined in advance by an external module such as a setting file, and switching may be performed according to the information. A predetermined rule may be added to the switching. For example, error concealment processing is performed in the block of the immediately preceding frame at the same position as the macroblock or the block of the frame that the macroblock intends to refer to motion, and at that time, the error concealment signal generation processing in step S24 If is selected, the error concealment signal generation processing of the macroblock may be forcibly determined as S24. Further, the above description is a processing flow in units of macroblocks, but the unit of processing does not depend on this. For example, a slice unit, a frame unit, or a GOP (Group Of Picture) unit obtained by bundling a plurality of frames may be used.

  Further, the error concealment signal generation processing once determined by any of the switching means described above may be changed again with reference to information on the surrounding area. For example, in the macroblock, the error concealment signal generation process in step S24 is selected, but in many of the surrounding blocks, the error concealment signal generation process in steps S25 to S28 is selected. The error concealment signal generation process is changed to that in steps S25 to S28.

Also, the pseudo signals generated by the two methods described above may be weighted average using a predetermined weighting factor. For example, in the macroblock, the signal generated in step S24 is A, the signal generated in steps S25 to S28 is B, and the weighting coefficient applied to signal A is applied to wA and signal B, respectively. When the weighting factor is wB, the error concealment signal Z of the macroblock uses a predetermined function f defined in advance,
Z = f (A, B, wA, wB)
Is expressed as
As an example of this function, a function defined by the following formula is applicable.
Z = wA × A + wB × B (however, wA + wB = 1)

Various determination criteria for wA and wB can be applied. For example, the following elements may be determined as parameters. Basically, the same parameters as those used when switching between the two methods can be applied.
(1) Prediction residual signal value (absolute value sum or square value sum) held by the lower hierarchy block
(2) Information amount of encoded information held by lower layer corresponding block (3) Quantization parameter / quantization step value held by lower layer corresponding block (4) Norm of motion vector held by lower layer corresponding block Value (5) Type of prediction mode possessed by block corresponding to lower layer (6) Number of slices in which packet loss has occurred in the past or number of slices in which packet loss has occurred in time (7) Packet loss near the time The time interval that occurred

  Even when an error concealment signal is generated by weighted averaging, the same processing procedure and processing structure as in FIGS. In FIG. 5, reference numeral 23 is not the SAD threshold value comparison unit but the weighting factor calculation unit 23, and there is no conditional branch. In accordance with this, step 23 in FIG. 6 is a process of inputting a SAD value and outputting a weighting coefficient. Then, a weighted average unit may be newly added after the lower layer corresponding block decoded signal expanding unit 24 and the signal adding unit 28. Here, the signals output from the lower layer corresponding block decoded signal expanding unit 24 and the signal adding unit 28 are input, and the weighted averages of these signals are output by the weighting coefficient calculating unit, and the error concealment signal is output. It becomes processing to do. Here, the weighting coefficient is calculated by applying only the example (1) mentioned above, and only one of these parameters may be used, or a plurality of these parameters may be used according to a predetermined function or rule. You may follow the procedure of calculating a weighting coefficient.

For example, when the SAD of the prediction residual signal held by the lower layer corresponding block is x and the packet loss has occurred in the past, and the number of frames that are continuous in time until the time is y, the weight coefficient wA and wB
wA = 1-wB
wB = (threshold Y−y) × y if (x <threshold X && y <threshold Y)
wB = 0 otherwise
You may decide like this. Here, the threshold value X is a threshold value for the SAD of the prediction residual signal possessed by the lower layer corresponding block, and the threshold value Y is a packet loss that has occurred in the past until the corresponding time. This is a threshold for the number of frames.

  In the above description, in the case of spatial scalable coding, as a method of generating a pseudo signal, a lower layer decoded signal at a predetermined time is expanded according to a resolution ratio between a desired layer and a lower layer by a predetermined procedure. A method of using the processed signal as a pseudo signal, and motion compensation in the desired layer using motion information obtained by scaling the lower layer motion information at a predetermined time according to the resolution ratio between the desired layer and the lower layer according to a predetermined procedure. The pseudo residual signal is obtained by enlarging the prediction residual signal of the lower layer at a predetermined time according to the resolution ratio between the desired layer and the lower layer according to a predetermined procedure, and adding the motion compensation signal and the expanded prediction residual signal. However, the present invention is also applicable to SNR scalable coding. In the case of SNR scalable coding, a method in which a decoded signal in a lower layer at a predetermined time is used as a pseudo signal, and motion compensation is performed in a desired layer using motion information in a lower layer at a predetermined time. A method of using a signal obtained by adding the prediction residual signal of the lower hierarchy of time as a pseudo signal may be used.

  In this way, in order to generate a video signal for playback at the time, a flag clearly indicating that the desired layer cannot be decoded and error concealment is performed at the time is received and stored in advance. Read the previously received encoded information, decoded signal and reproduction signal, and encoded information and decoded signal in a layer lower than the desired layer received at the time, and generate a video by a predetermined video generation procedure I made it. Here, a different video generation procedure can be specified for each specific image area for each layer. Then, the video signal generated by the designated video generation procedure is output as a reproduction signal at the time. This is because video data encoded by hierarchical encoding such as scalable encoding is input, and not only errors on the network such as packet errors and transmission delays, but also decoders such as decoding processing delays. This is applicable when the desired hierarchy is not obtained at the time at which playback is desired at that time due to an error).

  With this configuration, when the desired layer of the desired frame is not decoded at the timing to be reproduced due to packet loss or transmission delay, the image quality of the video to be finally reproduced is compared with the prior art described in Patent Document 1. It can be improved. In particular, when the movement of the subject is large, the effect of improving the image quality is remarkable as compared with the prior art.

Experimental results are shown in FIGS. This experiment was performed under the following conditions.
(1) Video: Vidyo1 (MPEG test video)
(2) Resolution / frame rate: 1280 x 720/60 fps
(3) Number of layers: 2 layers of space (4) QP: 28 (both basic and extended)
(5) Picture structure: IPPP
(6) Intra interval: top only (7) Number of encoded frames: 150 (8) Comparison method: PictCopy (previous frame repeat display), BLSK / BLUP (TH192) (present invention)
(9) SAD threshold: 192
(10) Slice: 1 frame 1 slice (11) Frame loss interval: 10 frames, 150 frames For example, if the loss interval is 10 frames, there is no loss at the beginning, 9 frames are lost, 10 frames are lost, and 9 frames are lost. The sheet is a loss.

  As shown in FIG. 7 and FIG. 8, it can be seen that the present invention can constitute a high-quality playback video at any frame loss interval as compared to PictCopy.

As described above, when video encoded data having a hierarchical structure is input, if the desired hierarchy at a predetermined time is not decoded at the timing to be reproduced, the data stored in the memory area on the receiving side is stored. Using the encoded information, the decoded signal and the reproduction signal received before the time, and the encoded information and decoded signal of the lower layer at the received time, the method for each layer and each specific image region While switching, generate a pseudo signal for playback at that time,
The generated signal can be output as a final reproduction signal at the time.

  In particular, in the present invention, in order to artificially create a signal of a desired layer lost at the time, a signal value close to the missing signal is obtained by using encoded information and a decoded signal of a lower layer at the time. Can be generated. Furthermore, the accuracy can be improved by switching a plurality of methods having different utilization methods of encoded information and decoded signals in a lower layer depending on a specific image region. For example, when the movement of the subject is large, it is better to generate a decoded signal of the lower layer and use an enlarged signal. When the movement of the subject is small, only the information of the movement of the lower layer is scaled, Since it is better to perform motion compensation in the desired hierarchy using the information after scaling, the quality of the entire image can be improved by automatically switching between these two methods.

  A program for realizing the functions of the processing units shown in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by a computer system and executed to reproduce video. Processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  The present invention can be applied to applications in which it is essential to conceal transmission errors in a video reproduction apparatus to which video encoded data having a hierarchical structure is input.

  DESCRIPTION OF SYMBOLS 1 ... Packet receiving part, 2 ... Packet loss detection part, 3 ... Base layer data decoding part, 4 ... Extension layer # 1 data decoding part, 5 ... Extension layer # 2 data decoding part , 6 ... error concealment processing unit, 7 ... error concealment processing unit, 8 ... storage unit

Claims (11)

In order to conceal transmission errors that occur when decoding and playing back encoded data that has been hierarchized into multiple hierarchies, it is transmitted when a layer lower than the desired layer to be reproduced can be decoded at a predetermined time. A video playback method that conceals and plays back errors,
A decoded signal forming step of extracting encoded information from the encoded packet received at the predetermined time stored in the storage unit and forming a decoded signal;
A signal reading step of reading received encoded information other than the predetermined time stored in the storage unit, a generated decoded signal and a reproduction signal;
Using the encoded information and decoded signal at the predetermined time and the encoded information other than the predetermined time, the decoded signal, and the reproduction signal, a pseudo signal for generating a desired layer signal to be reproduced at the predetermined time. A signal generation step;
And a signal output step of outputting the pseudo-generated signal as a reproduction signal at the predetermined time. The pseudo signal generation step includes:
2. The video reproduction method according to claim 1, wherein there are two or more methods for generating the pseudo signal, and an optimal method is selected for each predetermined hierarchy and predetermined image area. The pseudo signal generation step includes:
There are two methods for generating the pseudo signal, and the pseudo signal generated by the two methods is weighted using a predetermined weighting factor to generate the pseudo signal. Item 2. The video playback method according to Item 1. In the case of spatial scalable coding, a method for generating the pseudo signal is:
A method in which a signal obtained by enlarging the decoded signal of the lower layer at the predetermined time according to a resolution ratio of the desired layer and the lower layer according to a predetermined procedure is used as the pseudo signal;
The motion information of the lower layer at the predetermined time is compensated for motion in the desired layer using the motion information scaled according to the resolution ratio of the desired layer and the lower layer according to a predetermined procedure, and the lower layer at the predetermined time A predicted residual signal of a layer is expanded according to a resolution ratio between the desired layer and the lower layer according to a predetermined procedure, and the sum of the motion compensation signal and the expanded predicted residual signal is the pseudo signal. The video reproducing method according to claim 2, wherein the video reproducing method is one of two methods. In the case of SNR scalable coding, a method for generating the pseudo signal is:
A method of using the decoded signal of the lower layer at the predetermined time as the pseudo signal;
Method of performing motion compensation in the desired layer using the motion information of the lower layer at the predetermined time, and adding the motion compensation signal and the prediction residual signal of the lower layer at the predetermined time as the pseudo signal The video reproduction method according to claim 2, wherein the number of the video reproduction methods is two. The selection of the method of generating the pseudo signal is as follows:
Selection based on the determination result whether or not the total value of the absolute values of the prediction residual signals held by the lower-level image areas existing in the same spatial position as the target image area is larger than a predetermined threshold value The video reproduction method according to claim 2, wherein: The selection of the method of generating the pseudo signal is as follows:
3. The video reproduction method according to claim 2, wherein the selection is made based on a time accumulated value of transmission errors continuously occurring before the predetermined time. The selection of the method of generating the pseudo signal is as follows:
An error concealment signal after reducing the number of error concealment signal generation methods having two or more candidates to be implemented in the image region based on information on a method of generating the pseudo signal already selected in a predetermined image region 3. The video reproduction method according to claim 2, wherein an optimum method is selected from the generation method candidates. In order to conceal transmission errors that occur when decoding and playing back encoded data that has been hierarchized into multiple hierarchies, it is transmitted when a layer lower than the desired layer to be reproduced can be decoded at a predetermined time. A video playback device that conceals and plays back errors,
A decoded signal forming unit that extracts encoded information from the encoded packet received at the predetermined time stored in the storage unit and forms a decoded signal;
A signal reading unit that reads received encoded information other than the predetermined time stored in the storage unit, a generated decoded signal, and a reproduction signal;
Using the encoded information and decoded signal at the predetermined time and the encoded information other than the predetermined time, the decoded signal, and the reproduction signal, a pseudo signal for generating a desired layer signal to be reproduced at the predetermined time. A signal generator;
And a signal output unit that outputs the pseudo-generated signal as a reproduction signal at the predetermined time. In order to conceal transmission errors that occur when decoding and playing back encoded data that has been hierarchized into multiple hierarchies, it is transmitted when a layer lower than the desired layer to be reproduced can be decoded at a predetermined time. A video playback program for causing a computer on a video playback device that conceals an error to perform video playback,
A decoded signal forming step of extracting encoded information from the encoded packet received at the predetermined time stored in the storage unit and forming a decoded signal;
A signal reading step of reading received encoded information other than the predetermined time stored in the storage unit, a generated decoded signal and a reproduction signal;
Using the encoded information and decoded signal at the predetermined time and the encoded information other than the predetermined time, the decoded signal, and the reproduction signal, a pseudo signal for generating a desired layer signal to be reproduced at the predetermined time. A signal generation step;
A video reproduction program that causes the computer to perform a signal output step of outputting the pseudo-generated signal as a reproduction signal at the predetermined time.   A computer-readable recording medium on which the video reproduction program according to claim 10 is recorded.

Images