JP2011166486A - Device and method for decoding - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress degradation of reproduction quality. <P>SOLUTION: A reference track determination part 32 determines a track which is to be used as a reference (i.e., a track having a high priority) in multi-track edition based on analysis information from a reproduction information analyzing part 31, and determines the order in which frames in a GOP are to be decoded based on the reference track. A decoding control part 33 sequentially supplies information indicating whether frame dropping is to be performed to decoding parts 23-1 and 23-2 based on the order information determined by the reference track determination part 32 and load information from a load monitoring part 22. The invention is applicable to, for example, an image decoding device which decodes an encoded bit stream. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a decoding apparatus and method, and more particularly, to a decoding apparatus and method capable of suppressing deterioration in reproduction quality.

  As a technique for realizing real-time decoding processing when playing back a moving image, for example, in Patent Document 1, when full-frame real-time decompression processing cannot be performed, delay determination with audio is performed and frame drop processing is performed. ing. A frame drop target picture is a “B picture” that does not become a reference frame.

JP 2001-309303 A

  However, there may be situations where it is difficult to synchronize audio and video when multiple moving images are decoded at the same time, such as moving image editing. In addition, in a situation where a plurality of moving image frames such as crossfades are taken out simultaneously, the pictures of the plurality of frames may not all become B frames, and the technique described in Patent Document 1 causes image quality degradation and the like. Optimal frame drop was difficult.

  As described above, many of the prior arts often refer to a frame drop method when a single moving image is reproduced in real time. Few things were mentioned about frame drops.

  The present invention has been made in view of such a situation, and can suppress a decrease in reproduction quality.

  According to one aspect of the present invention, there is provided a decoding device that performs bitstream decoding processing in units of GOPs, load monitoring means for monitoring the load of the decoding processing, and multitrack editing according to the contents of special effects. Based on reference track determination means for determining a reference track, information on the load monitored by the load monitoring means, and information on the reference track determined by the reference track determination means, the decoding process Decoding control means for controlling execution or interruption of the bit stream, and decoding means for performing the decoding processing of the bitstream under the control of the decoding control means.

  The decoding control means controls execution or interruption of the decoding process so as to satisfy real-time characteristics.

  The decoding control unit can individually control execution or interruption of the decoding process with respect to the plurality of decoding units.

  The special effect is a wipe or an alpha blend.

  The image processing apparatus may further include special effect processing means for performing the special effect processing using the image decoded by the decoding means.

  A decoding method according to one aspect of the present invention is a decoding method of a decoding device that performs a decoding process of a bitstream in GOP units, and includes a load monitoring unit, a reference track determination unit, a decoding control unit, and a decoding unit. The load monitoring means monitors the load of the decoding process, the reference track determining means determines a reference track according to the content of the special effect in multitrack editing, and the decoding control means is monitored Based on the load information and the determined reference track information, execution or interruption of the decoding process is controlled, and the decoding unit performs the decoding process of the bitstream under the control. I do.

  In one aspect of the present invention, the load of the decoding process is monitored, and in multitrack editing, a reference track is determined according to the content of the special effect, the monitored load information, and the determined reference Based on the track information, the execution or interruption of the decoding process is controlled. Under the control, the decoding process of the bit stream is performed.

  Note that the above-described decoding device may be an independent device, or may be an internal block constituting one decoding device.

  According to one aspect of the present invention, the number of frame drops can be reduced. Thus, according to one aspect of the present invention, it is possible to suppress a decrease in reproduction quality.

It is a block diagram which shows the structure of one Embodiment of the image decoding apparatus to which this invention is applied. It is a figure explaining reproduction | regeneration (edit) information. It is a figure which shows the example of the arrangement order of a picture. It is a figure explaining the track | truck used as the reference | standard when a special effect is a wipe. It is a figure explaining the track | truck used as a reference | standard when a special effect is (alpha) blend. It is a figure which shows the example of the flame | frame which can be displayed by determination of a reference | standard track. It is a figure which shows the example of the flame | frame which can be displayed by determination of a reference | standard track. It is a figure which shows the example of the flame | frame which can display the conventional frame drop. It is a flowchart explaining the reproduction | regeneration processing of an image decoding apparatus. It is a block diagram which shows the structural example of the hardware of a computer.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Configuration Example of Image Decoding Device]
FIG. 1 shows a configuration of an embodiment of an image decoding apparatus to which the present invention is applied.

  The image decoding device 11 decodes the encoded bit stream in GOP units.

  In the example of FIG. 1, the image decoding device 11 includes a control unit 21, a load monitoring unit 22, decoding units 23-1 and 23-2, a special effect processing unit 24, and a display unit 25.

  Reproduction (editing) information at the time of moving image editing input by a user from an operation input unit (not shown) is stored in a storage unit (not shown). The control unit 21 acquires the reproduction (editing) information, analyzes the acquired reproduction (editing) information, determines the decoding order of the frames in the GOP, and determines the decoding units 23-1 and 23-2. , Information on whether or not to perform frame drop is sequentially supplied.

  Specifically, the control unit 21 includes a reproduction information analysis unit 31, a reference track determination unit 32, and a decoding control unit 33. The reproduction information analysis unit 31 acquires and analyzes reproduction (editing) information, and supplies the analyzed information to the reference track determination unit 32.

  Based on the analysis information from the reproduction information analysis unit 31, the reference track determination unit 32 determines a track with a high priority as a reference track in multitrack editing. Then, the reference track determination unit 32 determines the decoding order of the frames in the GOP based on the determined reference track, and supplies information on the determined order to the decoding control unit 33.

  Whether the decoding control unit 33 performs frame drop on the decoding units 23-1 and 23-2 based on the order information determined by the reference track determination unit 32 and the load information from the load monitoring unit 22. Information on whether or not is supplied sequentially. That is, the decoding control unit 33 controls the decoding units 23-1 and 23-2 to execute or interrupt the decoding process.

  The load monitoring unit 22 analyzes the system information, and monitors the decoding processing load, that is, the system load based on the analyzed system information during reproduction of the editing result. Then, the load monitoring unit 22 supplies load information to the decoding control unit 33 when the real time reproduction is not in time or when the decoding buffer is insufficient.

  Specifically, the load monitoring unit 22 includes a system information analysis unit 41 and a load monitoring processing unit 42.

  The system information analysis unit 41 acquires and analyzes system information and supplies the analysis information to the load monitoring processing unit 42. The system information analysis unit 41 monitors the load of the decoding process, that is, the system load during reproduction of the edited result, and acquires the load information. As the system load, for example, information such as the performance of the processor, the remaining memory capacity, and the read speed of the hard disk is monitored. The processor includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like.

  The load monitoring processing unit 42 analyzes the acquired load information based on the analysis information from the system information analysis unit 41. For example, when the real-time reproduction is not in time or the decoding buffer is insufficient, Load information is supplied to the decoding control unit 33.

  The encoded bit streams are input to the decoding unit 23-1 and the decoding unit 23-2, respectively. Under the control of the decoding control unit 33, the decoding unit 23-1 and the decoding unit 23-2 perform a decoding process on the input bitstream, and output the decoded result images to the special effect processing unit 24, respectively. .

  The special effect processing unit 24 performs special effect processing such as effects and transitions on a plurality of images, and outputs the resulting image to the display unit 25. The display unit 25 displays the image from the special effect processing unit 24.

  In the example of FIG. 1, an example in which two decoding units 23-1 and 23-2 are shown is shown, but the number of decoding units is not limited to two. For example, a plurality of decoding units corresponding to the number of bit streams may be configured.

[Description of how to determine the reference track]
The reference track determination unit 32 calculates the order in which frames within a GOP are decoded by determining a track that prioritizes decoding. As a method for determining the track, reproduction (editing) information shown in FIG. 2 is used.

  In the example of FIG. 2, two tracks, Track A and Track B, as materials used for reproduction are shown with the reproduction direction from left to right. The hatched portions in these tracks represent portions used for reproduction, and the boundaries indicate the start point and the end point. Further, the right side of the overlapping portion of the hatched portion of Track A and the hatched portion of Track B is shown as an effect transition section E1, and the left side is shown as an effect transition section E2.

  As described above, the reproduction (editing) information holds information (period and content) of a material used for reproduction, its start point and end point, and special effects such as effects and transitions.

  In the example of FIG. 3, the arrangement order of pictures in the effect transition section E1 of FIG. 2 is shown. That is, in the effect transition section E1, TrackA is configured by sequentially arranging BPBPBPBP pictures, and TrackB is configured by sequentially arranging BBIBPBPB pictures.

  When the special effect performed in the effect transition section E1 is a wipe as shown in FIG. 4, the reference track determination unit 32 determines the larger of the decoded image writing area as the higher priority track, that is, Determine as the reference track.

  In the example of FIG. 4, the ratio of the TrackA and TrackB image areas written by wipe in the effect transition section E1 is shown.

  In the section from the first picture (picture with circled number 1) to the fourth picture (picture with rounded numeral 4) in the effect transition period E1, the image writing area of TrackB is larger than TrackA. Therefore, the reference track determination unit 32 determines that the section from the first picture to the fourth picture has priority to TrackB, that is, is a reference section.

  On the other hand, in the section from the fifth picture (picture with the circled number 5) to the eighth picture (picture with the circled number 8) in the effect transition period E1, the area for writing the image of TrackA is larger than that of TrackB. Therefore, the reference track determination unit 32 determines that the section from the fifth picture to the eighth picture has priority on Track A, that is, is the reference section.

  Further, when the special effect performed in the effect transition section E1 is α blend as shown in FIG. 5, the reference track determination unit 32 determines the one with the higher blend value applied to the decoded image as the priority. It is determined as a high track, that is, a reference track.

  In the example of FIG. 5, the intensity ratio of the α blend of the TrackA and TrackB images to be written in the effect transition section E1 is shown. In the example of FIG. 5, the image written by TrackA is white, and the image written by TrackB is dark hatched. That is, the first picture in the effect transition section E1 (the picture with the circled number 1) has the highest ratio of the α blend of the TrackB image, and the ratio of each picture gradually decreases in the playback direction. In the last picture (the picture with the circled number 8), the α blend intensity of the TrackA image has the highest ratio.

  Therefore, the intensity of the α blend value of the TrackB image is higher than that of TrackA in the section from the first picture (picture with circled number 1) to the fourth picture (picture with circled number 4) in the effect transition period E1. Therefore, the reference track determination unit 32 determines that the section from the first picture to the fourth picture has priority to TrackB, that is, is a reference section.

  On the other hand, the intensity of the α blend value of Track A is higher than that of Track B in the section from the fifth picture (picture with circled number 5) to the eighth picture (picture with round numeral 8) in the effect transition period E1. Therefore, the reference track determination unit 32 determines that the section from the fifth picture to the eighth picture has priority on Track A, that is, is the reference section.

  In the above description, the example in which the content of the special effect is used for the reference track determination method has been described. However, the reference track may be determined according to the weight of the decoding process. For example, if one track is H.264. 264 and MPEG-4 Part10 (Advanced Video Coding, hereinafter referred to as H.264 / AVC) is a bitstream encoded, and the other track is encoded with MPEG (Moving Picture Experts Group) 2 Suppose it is a bitstream. In this case, H. Processing is heavier for tracks encoded with H.264 / AVC. As described above, when the weight of the decoding process greatly varies depending on the track, this track can be determined as the reference track in order to give priority to the decoding of the track having the light processing material.

  Furthermore, depending on the track, the GOP may end earlier or continue longer in the time direction. It is also possible to determine the reference track according to the end timing of the GOP in the time direction.

[Effects of determining the reference track]
When the reference track determination method as described above is used, for example, when Track A is determined as the reference track in FIG. 3, the order of decoding processing is as follows.
(1) Decode only I picture and P picture of TrackA
(2) Decode only necessary pictures among I and P pictures in TrackB
(3) Decode the picture displayed in the same timing as the I picture and P picture of Track A in Track B
(4) Decoding undecoded pictures of TrackB I and P pictures
(5) Decode pictures in Track A that are displayed at the same timing as the I and P pictures of Track B
(6) Decode other B pictures that have not been decoded yet

  Note that this decoding processing order can be said to be the order of frame drop.

  In the above decoding processing order, when decoding in the GOP is interrupted after decoding (1), the displayed frame is the frame indicated by the dotted line in FIG.

  In the example of FIG. 6, it is shown that the hatched frame has been decoded. In the effect transition section E1, both Track A and Track B have been decoded, and the displayable frame is surrounded by a dotted line. In addition, TrackA or TrackB has been decoded outside the effect transition section E1, and displayable frames are appropriately surrounded by dotted lines. The same applies to FIGS. 7 and 8 below.

   Since the decoding within the GOP is interrupted after the decoding of (1), only the I picture and the P picture of TrackA are decoded in the example of FIG.

  That is, before the effect transition period E1, it is not an effect transition period, so that even if the track A frame is decoded, it can be displayed. Therefore, the frames of the I picture and the P picture that are temporally before the effect transition section E1 can be displayed.

  However, no frames can be displayed in the effect transition section E1.

  Next, in the decoding processing order described above, when decoding within the GOP is interrupted after decoding (1), (2), (3), the displayed frame is the frame indicated by the dotted line in FIG. Become.

  Since decoding in the GOP is interrupted after decoding of (1), (2), and (3), in the case of the example in FIG. 7, the I picture and P picture of Track A are decoded, and the necessary I picture in Track B And the P picture is decoded. Next, in Track B, a picture to be displayed at the same timing as the I picture and P picture of Track A is decoded.

  That is, before and after the effect transition section E1, it is not an effect transition section, so it can be displayed as long as the track A or track B frame is decoded. Therefore, the frame of the Track A I picture and P picture before the effect transition section E1 can be displayed. In addition, the frame of the I picture and P picture of TrackB that is temporally after the effect transition section E1 can be displayed.

  Further, in the effect transition section E1, since the B picture to be displayed is decoded at the same timing as the P picture of Track A, these four frames can be displayed.

  On the other hand, in the prior art, for example, as shown in FIG. 8, when a B picture is simply dropped without determining a reference track, the frames before and after the effect transition section E1 are determined. I and P picture frames can be displayed.

  However, there is no displayable frame in the effect transition section E1.

  As described above, in the image decoding device 11, by determining the reference track, it is possible to increase the number of frames that can be displayed compared to the conventional one.

[Reproduction processing of image decoding apparatus]
Next, the reproduction processing of the image decoding device 11 will be described with reference to the flowchart of FIG.

  When operation information by a user is input from an operation input unit (not shown), in step S11, the system information analysis unit 41 acquires system information and analyzes the acquired system information. The system information analysis unit 41 supplies the analysis information to the load monitoring processing unit 42.

  In step S12, the load monitoring processing unit 42 starts monitoring the performance of the processor and the memory information as monitoring the load of the decoding process.

  In step S13, the reproduction information analysis unit 31 and the reference track determination unit 32 analyze the reproduction information and determine the frame drop order. That is, the reproduction information analysis unit 31 acquires and analyzes reproduction (editing) information, and supplies the analyzed information to the reference track determination unit 32. Based on the analysis information from the reproduction information analysis unit 31, the reference track determination unit 32 determines a track with a high priority as a reference track in multitrack editing. Then, the reference track determination unit 32 determines the decoding order of the frames in the GOP based on the determined reference track, and supplies information on the determined order to the decoding control unit 33.

  In step S14, the decoding units 23-1 and 23-2 perform a decoding process on the input bitstream under the control of the decoding control unit 33. The decoding units 23-1 and 23-2 output the decoding result images to the special effect processing unit 24, respectively.

  In step S <b> 15, the load monitoring processor 42 acquires the load information by monitoring the CPU performance and memory information, and analyzes the acquired load information based on the analysis information from the system information analysis unit 41.

  The load monitoring processing unit 42 refers to the analyzed load information and determines in step S16 whether or not real-time reproduction is in time. If it is determined in step S16 that the real-time reproduction is not in time, the load monitoring processing unit 42 supplies the analyzed load information to the decoding control unit 33, and the process proceeds to step S17.

  In step S17, the decoding control unit 33 instructs the corresponding decoding unit among the decoding units 23-1 and 23-2 to drop the frame. That is, the decoding control unit 33 selects the corresponding decoding unit among the decoding units 23-1 and 23-2 based on the order information determined by the reference track determination unit 32 and the load information from the load monitoring unit 22. In response to this, a frame drop (decoding process stop) instruction is issued.

  For example, in the case of the above-described decoding processing order, the other B picture that has not been decoded in (6) is indicated as a frame drop target.

  After the process of step S17, the process returns to step S16. That is, if it is determined in step S16 that real-time playback is not yet in time, in step S17, in the case of the decoding process described above, out of Track A in (5), the I picture and P picture of Track B A picture to be displayed at the same timing is designated as a frame drop target.

  If it is determined in step S16 that real-time playback is in time, the process proceeds to step S18. In step S18, the special effect processing unit 24 determines whether or not all the pictures necessary for the special effect processing have been acquired. If it is determined in step S18 that all the pictures necessary for the special effect processing have been acquired, the special effect processing unit 24 performs the special effect processing using the necessary pictures in step S19, and obtains the resulting image. And output to the display unit 25. Thereafter, the process proceeds to step S20.

  On the other hand, if it is determined in step S18 that all the pictures required for the special effect processing have not been acquired, the processing proceeds to step S20.

  In step S20, the decoding units 23-1 and 23-2 determine whether there is a next frame to be decoded. If it is determined in step S20 that there is a next frame, the process proceeds to step S14, and the subsequent processes are repeated.

  If it is determined in step S20 that there is no next frame, the reproduction process ends.

  Note that the decoding control unit 33 controls the decoding units 23-1 and 23-2 individually.

  As described above, in the image decoding device 11, in multitrack editing, a reference track is determined based on reproduction information, and the decoding processing order (that is, frame drop order) is controlled.

  This reduces the number of frame drops when performing real-time playback in devices that perform decoding and special effects processing on multiple streams such as when previewing video editing, and suppresses degradation in playback quality can do.

  Also, when the effect application range and its start and end points are updated as needed, such as when editing a movie, it is possible to handle the frame drop appropriately according to the weight of the decoding process and the effect application range. It becomes possible.

  In the above, the encoding method is H.264. The H.264 / AVC system and MPEG2 are used, but the present invention is not limited to this, and other encoding / decoding systems can be applied.

  It should be noted that the present invention includes, for example, MPEG, H.264, and the like. When receiving image information (bitstream) compressed by orthogonal transformation such as discrete cosine transformation and motion compensation, such as 26x, via network media such as satellite broadcasting, cable television, the Internet, or mobile phones. The present invention can be applied to an image encoding device and an image decoding device used in the above. Further, the present invention can be applied to an image encoding device and an image decoding device used when processing on a storage medium such as an optical, magnetic disk, and flash memory. Furthermore, the present invention can also be applied to motion prediction / compensation devices included in such image encoding devices and image decoding devices.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes a computer incorporated in dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like.

[Configuration example of personal computer]
FIG. 10 is a block diagram illustrating an example of a hardware configuration of a computer that executes the above-described series of processing by a program.

  In a computer, a central processing unit (CPU) 301, a read only memory (ROM) 302, and a random access memory (RAM) 303 are connected to each other via a bus 304.

  An input / output interface 305 is further connected to the bus 304. An input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected to the input / output interface 305.

  The input unit 306 includes a keyboard, a mouse, a microphone, and the like. The output unit 307 includes a display, a speaker, and the like. The storage unit 308 includes a hard disk, a nonvolatile memory, and the like. The communication unit 309 includes a network interface and the like. The drive 310 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer configured as described above, for example, the CPU 301 loads the program stored in the storage unit 308 to the RAM 303 via the input / output interface 305 and the bus 304 and executes the program, thereby performing the series of processes described above. Is done.

  The program executed by the computer (CPU 301) can be provided by being recorded on a removable medium 311 as a package medium or the like, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

  In the computer, the program can be installed in the storage unit 308 via the input / output interface 305 by attaching the removable medium 311 to the drive 310. Further, the program can be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. In addition, the program can be installed in advance in the ROM 302 or the storage unit 308.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  DESCRIPTION OF SYMBOLS 11 Image decoding apparatus, 21 Control part, 22 Load monitoring part, 23-1, 23-2 Decoding part, 24 Special effect processing part, 25 Display part, 31 Playback information analysis part, 32 Reference track determination part, 33 Decoding control part , 41 System information analysis unit, 42 Load monitoring processing unit

Claims (6)

In a decoding device that performs bitstream decoding processing in GOP units,
Load monitoring means for monitoring the load of the decoding process;
In multitrack editing, reference track determination means for determining a reference track according to the content of special effects;
Decoding control means for controlling execution or interruption of the decoding process based on information on the load monitored by the load monitoring means and information on the reference track determined by the reference track determination means; A decoding device comprising: decoding means for performing the decoding processing of the bitstream under the control of a decoding control means. The decoding apparatus according to claim 1, wherein the decoding control unit controls execution or interruption of the decoding process so as to satisfy real-time characteristics. The decoding device according to claim 1, wherein the decoding control unit individually controls execution or interruption of the decoding process with respect to the plurality of decoding units. The decoding device according to claim 1, wherein the special effect is a wipe or an α blend. The decoding device according to claim 1, further comprising: special effect processing means for performing the special effect processing using the image decoded by the decoding means. In a decoding method of a decoding device that performs a decoding process of a bitstream in GOP units and includes a load monitoring unit, a reference track determination unit, a decoding control unit, and a decoding unit,
The load monitoring means monitors the load of the decoding process;
The reference track determining means determines a reference track according to the contents of special effects in multi-track editing,
The decoding control means performs execution or interruption control of the decoding process based on the monitored information on the load and the determined information on the reference track,
A decoding method in which the decoding means performs the decoding processing of the bitstream under the control.

Images