EP2335419B1 - Methods and apparatus for video imaging pruning - Google Patents

Methods and apparatus for video imaging pruning Download PDF

Info

Publication number
EP2335419B1
EP2335419B1 EP09789245.9A EP09789245A EP2335419B1 EP 2335419 B1 EP2335419 B1 EP 2335419B1 EP 09789245 A EP09789245 A EP 09789245A EP 2335419 B1 EP2335419 B1 EP 2335419B1
Authority
EP
European Patent Office
Prior art keywords
picture
portions
data
decoder
columns
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP09789245.9A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP2335419A1 (en
Inventor
Dung Trung Vo
Joel Sole
Peng Yin
Yunfei Zheng
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
InterDigital Madison Patent Holdings SAS
Original Assignee
InterDigital Madison Patent Holdings SAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by InterDigital Madison Patent Holdings SAS filed Critical InterDigital Madison Patent Holdings SAS
Publication of EP2335419A1 publication Critical patent/EP2335419A1/en
Application granted granted Critical
Publication of EP2335419B1 publication Critical patent/EP2335419B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/587—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression

Definitions

  • the present principles relate generally to video encoding and decoding and, more particularly, to methods and apparatus for video image pruning.
  • the first approach involves improving compression technology, and the second approach involves performing some sort of preprocessing prior to compression.
  • the goal of such preprocessing is to remove the information that is least important in terms of visual perception, or the information that is able to be recovered after the decoding process without significantly altering the content.
  • This bit rate reduction is also commonly referred to as data pruning.
  • Some common techniques to perform this bit rate reduction through data pruning are the use of low-pass filters and down-sampling (which can be seen as a filtering process), followed by an up-sampling at the decoder.
  • One effect of these schemes is that the decoded and reconstructed video looks a bit blurry since these techniques are designed to eliminate high frequency information in the video in order to reduce the bit-rate.
  • interpolation which, for example, can be used for the previously described upsampling
  • a wide range of interpolation methods and schemes have been discussed and developed, beginning with conventional bilinear and bi-cubic interpolation and continuing to sophisticated iterative interpolation methods such as projection onto convex sets (POCS) and non-convex nonlinear partial differential equations.
  • POCS projection onto convex sets
  • edge-oriented interpolation methods using a Markov random field and covariance of the low resolution image have been proposed.
  • One prior art approach employs a combination of directional filtering and data fusion to estimate the missing high resolution (HR) pixels by a linear minimum mean square error (LMMSE).
  • LMMSE linear minimum mean square error
  • Another group of interpolation algorithms predict the fine structure of the HR image from its corresponding low resolution (LR) image using different kinds of transforms such as the wavelet transform or the countourlet transform.
  • Each of the above methods are suitable for up-sampling the same ratio in both horizontal and vertical directions; that is, in a fixed and regular data grid (i.e., all data points are found in a rectangular grid).
  • a fixed and regular data grid i.e., all data points are found in a rectangular grid.
  • interpolation is used along with data pruning, flexibility is desired in order to adapt to the discarded data and to adjust to the varying surroundings of each pixel to achieve the best performance.
  • the present principles are directed to methods and apparatus for video image data pruning.
  • the invention concerns pre-processing (215) a picture prior to, and in preparation for, compression by encoding, wherein said pre-processing step comprises selectively removing, based on location of pixels within the spatial domain of the picture, and not in the frequency domain, portions of said picture and leaving the remaining portions of said picture to be encoded using a data pruner having a processor (1200), namely in transmitting side information indicating a location of the removed portions within the picture to a decoder or other device and in selectively removing diagonal line portions of said picture, wherein the diagonal line portions are neither rows nor columns and encoding the pre-processed picture.
  • processor or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (“DSP”) hardware, read-only memory (“ROM”) for storing software, random access memory (“RAM”), and non-volatile storage.
  • DSP digital signal processor
  • ROM read-only memory
  • RAM random access memory
  • any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
  • any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function.
  • the present principles as defined by such claims reside in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.
  • such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C).
  • This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.
  • image and picture are used interchangeably, and may refer to any of video pictures that are part of a video sequence and/or still images.
  • the present principles are directed to methods and apparatus for video image data pruning.
  • the system 100 includes a data pruner 110, an encoder (or compressor) 120, a decoder (or decompressor) 130, and a data restorer 140.
  • the data pruner 110 performs pre-processing to remove, in the spatial domain, portions (e.g., a line, curve, region, etc.) from a picture prior to compression by encoding.
  • the encoder 120 then codes the picture with the portions removed.
  • the decoder 130 decodes the coded picture with the portions removed.
  • the data restorer 140 performs post-processing to restore, in the spatial domain, the portions to the decoded picture.
  • any of the data pruner 110, the encoder 120, the decoder 130, and the data restorer 140 may include, for example, one or more processors, memory, and so forth, in order to perform their corresponding functions.
  • at least one or more processors in the data pruner 110 may perform the pre-processing described herein, while one or more processors in the data restorer 140 may perform the post-processing described herein.
  • the data pruner 110 may include other elements which, in some embodiments, may be separate from the data pruner, while in other embodiments may be integrated with the data pruner.
  • the data pruner 110 (or a data pruning phase) may include a decimator and an interpolator (see, e.g., FIG.
  • the data pruner 110 may include a data pruner (as a sub-element that performs the actual removal of the portions), a store or compressor, and an interpolator (see, e.g., FIG. 10 ).
  • a data pruner as a sub-element that performs the actual removal of the portions
  • a store or compressor as a store or compressor
  • an interpolator see, e.g., FIG. 10 .
  • the method 200 includes a start block 205 that passes control to a function block 210.
  • the function block 210 receives a picture, and passes control to a function block 215.
  • the function block 215 performs pre-processing of the picture to remove, in the spatial domain, portions (e.g., a line, curve, region, etc.) from the picture, and passes control to a function block 220.
  • the function block 220 codes the picture with the portions removed, and passes control to a function block 225.
  • the function block 225 provides (e.g., transmits) the coded picture to a decoder (or a device having a decoder), and passes control to a function block 230.
  • the function block 230 decodes the coded picture with the portions removed, and passes control to a function block 225.
  • the function block 235 receives the decoded picture with the portions removed, performs post-processing to restore, in the spatial domain, the portions to the decoded picture, and passes control to an end block 299.
  • an exemplary video encoder to which the present principles may be applied is indicated generally by the reference numeral 300.
  • the video encoder 300 includes a frame ordering buffer 310 having an output in signal communication with a non-inverting input of a combiner 385.
  • An output of the combiner 385 is connected in signal communication with a first input of a transformer and quantizer 325.
  • An output of the transformer and quantizer 325 is connected in signal communication with a first input of an entropy coder 345 and a first input of an inverse transformer and inverse quantizer 350.
  • An output of the entropy coder 345 is connected in signal communication with a first non-inverting input of a combiner 390.
  • An output of the combiner 390 is connected in signal communication with a first input of an output buffer 335.
  • a first output of an encoder controller 305 is connected in signal communication with a second input of the frame ordering buffer 310, a second input of the inverse transformer and inverse quantizer 350, an input of a picture-type decision module 315, an input of a macroblock-type (MB-type) decision module 320, a second input of an intra prediction module 360, a second input of a deblocking filter 365, a first input of a motion compensator 370, a first input of a motion estimator 375, and a second input of a reference picture buffer 380.
  • MB-type macroblock-type
  • a second output of the encoder controller 305 is connected in signal communication with a first input of a Supplemental Enhancement Information (SEI) inserter 330, a second input of the transformer and quantizer 325, a second input of the entropy coder 345, a second input of the output buffer 335, and an input of the Sequence Parameter Set (SPS) and Picture Parameter Set (PPS) inserter 340.
  • SEI Supplemental Enhancement Information
  • a first output of the picture-type decision module 315 is connected in signal communication with a third input of a frame ordering buffer 310.
  • a second output of the picture-type decision module 315 is connected in signal communication with a second input of a macroblock-type decision module 320.
  • SPS Sequence Parameter Set
  • PPS Picture Parameter Set
  • An output of the inverse quantizer and inverse transformer 350 is connected in signal communication with a first non-inverting input of a combiner 319.
  • An output of the combiner 319 is connected in signal communication with a first input of the intra prediction module 360 and a first input of the deblocking filter 365.
  • An output of the deblocking filter 365 is connected in signal communication with a first input of a reference picture buffer 380.
  • An output of the reference picture buffer 380 is connected in signal communication with a second input of the motion estimator 375.
  • a first output of the motion estimator 375 is connected in signal communication with a second input of the motion compensator 370.
  • a second output of the motion estimator 375 is connected in signal communication with a third input of the entropy coder 345.
  • An output of the motion compensator 370 is connected in signal communication with a first input of a switch 397.
  • An output of the intra prediction module 360 is connected in signal communication with a second input of the switch 397.
  • An output of the macroblock-type decision module 320 is connected in signal communication with a third input of the switch 397.
  • the third input of the switch 397 determines whether or not the "data" input of the switch (as compared to the control input, i.e., the third input) is to be provided by the motion compensator 370 or the intra prediction module 360.
  • the output of the switch 397 is connected in signal communication with a second non-inverting input of the combiner 319 and with an inverting input of the combiner 385.
  • Inputs of the frame ordering buffer 310 and the encoder controller 305 are available as input of the encoder 300, for receiving an input picture 301.
  • an input of the Supplemental Enhancement Information (SEI) inserter 330 is available as an input of the encoder 300, for receiving metadata.
  • An output of the output buffer 335 is available as an output of the encoder 300, for outputting a bitstream.
  • SEI Supplemental Enhancement Information
  • FIG. 4 an exemplary video decoder to which the present principles may be applied is indicated generally by the reference numeral 400.
  • the video decoder 400 includes an input buffer 410 having an output connected in signal communication with a first input of the entropy decoder 445.
  • a first output of the entropy decoder 445 is connected in signal communication with a first input of an inverse transformer and inverse quantizer 450.
  • An output of the inverse transformer and inverse quantizer 450 is connected in signal communication with a second non-inverting input of a combiner 425.
  • An output of the combiner 425 is connected in signal communication with a second input of a deblocking filter 465 and a first input of an intra prediction module 460.
  • a second output of the deblocking filter 465 is connected in signal communication with a first input of a reference picture buffer 480.
  • An output of the reference picture buffer 480 is connected in signal communication with a second input of a motion compensator 470.
  • a second output of the entropy decoder 445 is connected in signal communication with a third input of the motion compensator 470 and a first input of the deblocking filter 465.
  • a third output of the entropy decoder 445 is connected in signal communication with an input of a decoder controller 405.
  • a first output of the decoder controller 405 is connected in signal communication with a second input of the entropy decoder 445.
  • a second output of the decoder controller 405 is connected in signal communication with a second input of the inverse transformer and inverse quantizer 450.
  • a third output of the decoder controller 405 is connected in signal communication with a third input of the deblocking filter 465.
  • a fourth output of the decoder controller 405 is connected in signal communication with a second input of the intra prediction module 460, with a first input of the motion compensator 470, and with a second input of the reference picture buffer 480.
  • An output of the motion compensator 470 is connected in signal communication with a first input of a switch 497.
  • An output of the intra prediction module 460 is connected in signal communication with a second input of the switch 497.
  • An output of the switch 497 is connected in signal communication with a first non-inverting input of the combiner 425.
  • An input of the input buffer 410 is available as an input of the decoder 400, for receiving an input bitstream.
  • a first output of the deblocking filter 465 is available as an output of the decoder 400, for outputting an output picture.
  • the original images/pictures are pruned to a smaller size prior to compression and then interpolated to their original size after decoding.
  • This interpolation uses information about which lines, segments, or regions were pruned during the data pruning step to return the video image/pictures to their original size. Pruning is performed prior to encoding by dropping lines, segments, and/or regions in the spatial domain, according to one of several possible evaluation schemes and then restoring these lines, segments, and/or regions after decoding the bitstream back into the spatial domain.
  • edge-directed interpolation is utilized to reconstruct the decoded picture to its original size. Since we avoid filtering the remaining pixels in the image/video pictures, the reconstructed pictures can achieve higher quality from a lower bit-rate.
  • data pruning is performed in a manner in which only rows or columns are dropped, so only up-sampling in one direction is needed in these areas.
  • the lines dropped may be rows or columns, but are not limited to only the preceding.
  • the lines dropped may be diagonal or may be non-linear curves instead of lines and/or in addition to lines.
  • the same principles would apply and are within the scope of the present principles in that the requisite unit to be pruned is not a spatial domain line or curve, but rather a section or region.
  • model parameters of high-order edge-directed interpolation relating to NEDI-4 and NEDI-8 are indicated generally by the reference numeral 500.
  • two passes are performed using the quincunx sub-lattice.
  • the high resolution pixels at the cross of any four nearest low resolution pixels are interpolated from their four nearest pixels by using an adaptive fourth-order interpolator.
  • the remaining high resolution pixels are interpolated with the same algorithm in the second pass, using the low resolution pixels and the pixels interpolated in the first pass.
  • the high resolution model parameters of the interpolator are estimated from the lower resolution model parameters.
  • Data pruning is implemented by dropping lines in the pictures.
  • a particular case would be that the line is a straight line.
  • a line can be either a row or a column, or any straight line with direction (as seen in FIG. 6 ).
  • FIG. 6 an example of data pruning by dropping arbitrary lines is indicated generally by the reference numeral 600.
  • the data pruning 600 involves a frame 610, a pruned line 620, and a pruned curve 630.
  • the data pruning 700 involves a frame 710, a dropped column 720, and a dropped row 730.
  • FIG. 8 an example of data pruning by dropping the same columns and rows for a whole group of pictures (GOP) and/or scene is indicated generally by the reference numeral 800.
  • the data pruning 800 involves a first frame 810, middle frames 820, and a last frame 830, dropped rows 840, and dropped columns 850.
  • FIG. 9A lines for indicating the dropped columns and rows in the first frame of the well-known Akiyo sequence are indicated generally by the reference numeral 900.
  • FIG. 9B the pruned frame corresponding to the example of FIG. 9A is indicated generally by the reference numeral 950.
  • all the pictures in one GOP/scene have the same dropping patterns in order to further reduce the overhead.
  • the target can be distortion, percentage of dropping pixels and/or any complexity constraint.
  • the target is not limited to the preceding and, thus, other targets may also be used in accordance with the present principles, while maintaining the present principles.
  • presuming one line is dropped we interpolate the dropped line without considering compression. Then, we compare the MSE (Mean Square Error) of the interpolated line to the same line in the original image. The line that has the minimum MSE is selected to be dropped first.
  • the line bit-rate can be approximated using its variance.
  • the system 1000 includes a data pruner 1010, a data store or compressor 1020, and an interpolator 1030. While shown as separate elements, one or more of the data pruner 1010, data store or compressor 1020, or interpolator 1030 may be combined into an integrated element. For example, in an embodiment, the data pruner 1010 may be included within the data store or compressor 1020. The data store or compressor 1020 may be considered to be an encoder for the purposes of the present principles.
  • the original frame I of size M ⁇ N is pruned to frame P of smaller size ( M - M p ) ⁇ ( N - N p ), where M p and N p are the number of dropped rows and columns, respectively.
  • the purpose of data pruning is to reduce the number of bits to represent the stored or compressed frame P'.
  • P' is then interpolated to I ' to reconstruct to the original frame size.
  • the compression stage considered here is the MPEG-4 AVC Standard codec, with the encoder as shown in FIG. 3 and the decoder as shown in FIG. 4 .
  • the present principles are not limited to solely the MPEG-4 AVC Standard and, thus, may be applied to other video coding standards, recommendations, and extensions thereof, while maintaining the present principles.
  • FIG. 11 an exemplary system for a data pruning phase is indicated generally by the reference numeral 1100.
  • the system 1100 includes a decimator 1110 and an interpolator 1120.
  • the original frame I is decimated by the decimator 1110 to the low resolution (LR) frame I l by a ratio of 1 ⁇ 2 , 2 ⁇ 1 and 2 ⁇ 2 for the respective cases of dropping only columns, dropping only rows, and dropping both columns and rows.
  • LR low resolution
  • frame I l is interpolated by the interpolator 1120 to the high resolution (HR) frame I h .
  • HR high resolution
  • the mean squared error ( MSE ) is defined as the mean square error between I h and I .
  • the data pruning is optimized to drop the most pixels while still keeping the overall MSE less than MSE max .
  • the location of the dropped rows and columns is respectively indicated by ⁇ m and ⁇ n .
  • the i th even line (row or column) is dropped if the value of ⁇ i is 1; otherwise it will be kept. These indicators are stored as side information in the coded bit-stream.
  • the mean square line error ( MSLE ) for one dropped line is defined as the mean square error between I h and I for pixels on that line only. Lines with a smaller MSLE have a higher priority to be dropped than lines with a higher MSLE .
  • MSE max the maximum MSLE of these lines.
  • the overall MSE becomes the averaged MSE of all dropped pixels.
  • An upper limit of MSE is achieved based on, for example, the number of dropped rows M p , dropped columns N p , and MSLE max . This upper limit should be less than MSE max as the condition to find the optimal data pruning scheme.
  • the method 1200 includes a start block 1201 that passes control to a function block 1203.
  • the function block 1203 inputs a frame or GOP, and passes control to a function block 1206, a function block 1209, and a function block 1212.
  • the function block 1206 removes the even rows, and passes control to a function block 1015.
  • the function block 1215 performs interpolation based on, for example, NEDI-6, and passes control to a function block 1224.
  • the function block 1224 finds the error between the original and reconstructed rows (1), and passes control to a function block 1233.
  • the function block 1209 removes the even rows, and passes control to a function block 1218.
  • the function block 1218 performs interpolation based on, for example, NEDI-6, and passes control to a function block 1227.
  • the function block 1227 finds the error between the original and reconstructed columns (2), and passes control to the function block 1233.
  • the function block 1212 removes even rows and columns, and passes control to a function block 1221.
  • the function block 1221 performs interpolation based on, for example, NEDI-8, and passes control to a function block 1230.
  • the function block 1230 finds the error between the original and reconstructed rows/columns (3), and passes control to the function block 1233.
  • the function block 1242 chooses to drop the N_d columns with the least error in (1), i.e., the error calculated per function block 1224, and passes control to a function block 1246.
  • the function block 1246 calculates the error MSE and number of dropped pixels Pel, and passes control to a decision block 1252.
  • the decision block 1252 determines whether or not MSE ⁇ MSE_max and Pel>Pel_max. If so, then control is passed to a function block 1255. Otherwise, control is passed to a function block 1270.
  • the function block 1258 determines whether or not N_d ⁇ N. If so, then control is passed to a function block 1264. Otherwise, control is passed to a function block 1261.
  • the function block 1261 determines whether or not M_d ⁇ M. If so, then control is passed to a function block 1267. Otherwise, control is passed to a function block 1270.
  • the function block 1245 chooses to drop the M_d columns with the least error in (2), i.e., the error calculated per function block 1227, and passes control to the function block 1246.
  • the function block 1249 chooses to drop the M_d columns and N_d rows with the least error in (3), i.e., the error calculated per function block 1230, and passes control to the function block 1046.
  • the function block 1270 outputs M_d_opt and N_d_opt, and passes control to an end block 1299.
  • the method 1300 includes a start block 1305 that passes control to a function block 1310.
  • the function block 1310 inputs a pruned frame and/or group of pictures (GOP), and a list of dropped frames, and passes control to a function block 1315.
  • the function block 1315 resizes the frames to their original size, and passes control to a loop limit block 1320.
  • the loop limit block 1320 loops over the pruned data, and passes control to a decision block 1325.
  • the decision block 1325 determines whether or not an even row and column has been removed. If so, then control is passed to a function block 1330. Otherwise, control is passed to a decision block 1340.
  • the function block 1330 interpolates the row and column with NEDI-8, and passes control to a loop limit block 1335.
  • the loop limit block 1335 ends the loop, and passes control to a function block 1360.
  • the function block 1360 outputs a reconstructed frame/GOP, and passes control to an end block 1399.
  • the decision block 1340 determines whether or not an even row has been removed. If so, then control is passed to a function block 1345. Otherwise, control is passed to a decision block 1350.
  • the function block 1345 interpolates the row with NEDI-6, and passes control to the loop limit block 1335.
  • the decision block 1350 determines whether or not an even column has been removed. If so, then control is passed to a function block 1355. Otherwise, control is passed to the loop limit block 1335.
  • the function block 1355 interpolates the column with NEDI-6, and passes control to the loop limit block 1335.
  • interpolation we mean any method that is able to estimate/fill missing pixels from the surrounding pixels of the same or temporally related frames.
  • Such methods can be classical interpolation, like the bi-cubic method, newer methods like NEDI, or other kind of methods like inpainting, which fills holes of missing pixels from their surroundings.
  • interpolation in accordance with the present principles is not limited to only the preceding described interpolation techniques and, thus, other interpolation techniques may be also be used, while maintaining the present principles.
  • the algorithm for dropping only columns is discussed because the same algorithm can be applied for the cases of dropping only columns and dropping only rows.
  • the columns of P ' corresponding to the odd columns in the original frame I are extracted to form the low resolution frame P ' l of size M ⁇ N 2 .
  • Columns of P' l are mapped to the odd columns of the HR frame P' h of size M ⁇ N.
  • the even columns of P' h are interpolated from the odd columns by a sixth-order interpolation as explained herein.
  • the interpolated pixel is the summation of the weighted 6-neighbor surrounding pixels as shown in FIG. 5B .
  • model parameters of high-order edge-directed interpolation relating to NEDI-6 are indicated generally by the reference numeral 500. These weights are the model parameters of the high resolution frame.
  • the optimal model parameters are obtained by minimizing the MSE between the interpolated pixels and the original pixels with the assumption that the model parameters are constant in a local window. Applying the geometric duality assumption, the model parameters are considered constant for different scales and can be estimated from the low resolution pixels.
  • the model parameters of the low resolution frame with the availability of the original pixels and interpolated pixels, can be obtained by minimizing the error between them.
  • the column indicator is used to choose the pixels from the compressed pruned frame or interpolated frame to form the final reconstructed frame. If the column indicator value is 1, then the reconstructed column is taken from the interpolated frame. Otherwise, if the column indicator value is 0, then the reconstructed column is taken from the compressed pruned frame.
  • the pixels in P ' corresponding to the low resolution pixels of the down-sampling ratio 2 ⁇ 2 in the original frame I are extracted to form the LR frame P' l of size M 2 ⁇ N 2 .
  • the interpolation is performed using NEDI-4 for the first round and NEDI-8 for the second round.
  • NEDI-8 is implemented to get extra information from 4 directions.
  • the advantage of this round is that the model parameters can be directly estimated from its high resolution pixels, so the over-fitting problem of NEDI-4 is lessened.
  • NEDI-8 is applied to all the missing high resolution pixels in order to be consistent.
  • the fourth-order model parameters h 4 and eighth-order model parameters h 8 for HR scale are shown in FIG. 5A .
  • the optimal h 8 is calculated by a similar algorithm to the one employed for NEDI-6.
  • one advantage/feature is an apparatus having a data pruner for pre-processing a picture prior to, and in preparation for, compression by encoding.
  • the data pruner selectively removes, in the spatial domain, portions within the picture.
  • Another advantage/feature is the apparatus having the data pruner as described above, wherein the data pruner is included in an encoder.
  • Still another advantage/feature is the apparatus having the data pruner as described above, further comprising an encoder coupled to the data pruner for encoding the picture with the portions removed.
  • Another advantage/feature is the apparatus having the data pruner as described above, wherein side information indicating a location of the portions within the picture is one of explicitly or implicitly transmitted to a decoder or other device.
  • Yet another advantage/feature is the apparatus having the data pruner as described above, wherein the portions includes at least one of a straight line, a non-linear curve, and a region.
  • Another advantage/feature is the apparatus having the data pruner as described above, wherein only even or odd lines are removed by the data pruner.
  • Another advantage/feature is the apparatus having the data pruner as described above, wherein the picture is included in a group of pictures, and all pictures within the group of pictures are assigned a same removal pattern.
  • another advantage/feature is the apparatus having the data pruner as described above, wherein the portions selectively removed is determined responsive to an evaluation of at least one of an image distortion metric, a rate distortion metric, a dropped pixel metric, and a complexity metric.
  • Another advantage/feature is the apparatus having the data pruner as described above, wherein 6-order interpolation is used for removing the portions in one dimension, and 8-order interpolation is used for removing the portions in two dimensions.
  • the teachings of the present principles are implemented as a combination of hardware and software.
  • the software may be implemented as an application program tangibly embodied on a program storage unit.
  • the application program may be uploaded to, and executed by, a machine comprising any suitable architecture.
  • the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPU"), a random access memory (“RAM”), and input/output ("I/O") interfaces.
  • CPU central processing units
  • RAM random access memory
  • I/O input/output
  • the computer platform may also include an operating system and microinstruction code.
  • the various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU.
  • various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
EP09789245.9A 2008-09-18 2009-09-01 Methods and apparatus for video imaging pruning Active EP2335419B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US9812508P 2008-09-18 2008-09-18
PCT/US2009/004950 WO2010033151A1 (en) 2008-09-18 2009-09-01 Methods and apparatus for video imaging pruning

Publications (2)

Publication Number Publication Date
EP2335419A1 EP2335419A1 (en) 2011-06-22
EP2335419B1 true EP2335419B1 (en) 2020-11-04

Family

ID=41582015

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09789245.9A Active EP2335419B1 (en) 2008-09-18 2009-09-01 Methods and apparatus for video imaging pruning

Country Status (8)

Country Link
US (1) US9571857B2 (pt)
EP (1) EP2335419B1 (pt)
JP (2) JP2012503416A (pt)
KR (3) KR20160114186A (pt)
CN (1) CN102217314B (pt)
BR (1) BRPI0918044A2 (pt)
TW (1) TWI499310B (pt)
WO (1) WO2010033151A1 (pt)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20110011361A (ko) * 2009-07-28 2011-02-08 삼성전자주식회사 샘플링을 이용한 영상 데이터 인코딩／디코딩 장치 및 인코딩／디코딩 방법
US9602814B2 (en) 2010-01-22 2017-03-21 Thomson Licensing Methods and apparatus for sampling-based super resolution video encoding and decoding
CN102726044B (zh) 2010-01-22 2016-08-10 汤姆逊许可证公司 使用基于示例的超分辨率的用于视频压缩的数据剪切
US9338477B2 (en) 2010-09-10 2016-05-10 Thomson Licensing Recovering a pruned version of a picture in a video sequence for example-based data pruning using intra-frame patch similarity
CN103210648B (zh) * 2010-09-10 2017-06-09 汤姆逊许可公司 使用基于块的混合分辨率数据修剪的视频解码
CN103202017B (zh) * 2010-09-10 2017-02-22 汤姆逊许可公司 使用基于样本的数据修剪的视频解码的方法和装置
US9544598B2 (en) 2010-09-10 2017-01-10 Thomson Licensing Methods and apparatus for pruning decision optimization in example-based data pruning compression
US20120275511A1 (en) * 2011-04-29 2012-11-01 Google Inc. System and method for providing content aware video adaptation
US9300980B2 (en) * 2011-11-10 2016-03-29 Luca Rossato Upsampling and downsampling of motion maps and other auxiliary maps in a tiered signal quality hierarchy
US9197888B2 (en) 2012-03-13 2015-11-24 Dolby Laboratories Licensing Corporation Overlapped rate control for video splicing applications
CN103379348B (zh) * 2012-04-20 2016-11-16 乐金电子(中国)研究开发中心有限公司 一种编码深度信息时的视点合成方法、装置及编码器
US20140072048A1 (en) * 2012-09-13 2014-03-13 Samsung Electronics Co., Ltd Method and apparatus for a switchable de-ringing filter for image/video coding
WO2017199149A1 (en) * 2016-05-16 2017-11-23 Numeri Ltd. A new pyramid algorithm for video compression and video analysis
KR102010479B1 (ko) * 2018-01-15 2019-08-13 옵티시스 주식회사 비디오 월 시스템
JP6985609B2 (ja) * 2018-05-21 2021-12-22 日本電信電話株式会社 符号化装置、画像補間システム及び符号化プログラム
WO2020037566A1 (zh) * 2018-08-22 2020-02-27 深圳配天智能技术研究院有限公司 一种图像处理、匹配方法、装置及存储介质
US11432009B2 (en) * 2019-07-02 2022-08-30 Intel Corporation Techniques for encoding and decoding immersive video
US11477429B2 (en) 2019-07-05 2022-10-18 Electronics And Telecommunications Research Institute Method for processing immersive video and method for producing immersive video
EP3968635A1 (en) 2020-09-11 2022-03-16 Axis AB A method for providing prunable video
EP3968636A1 (en) * 2020-09-11 2022-03-16 Axis AB A method for providing prunable video

Family Cites Families (57)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2537269B2 (ja) 1988-07-28 1996-09-25 株式会社竹中工務店 危険薬品使用工場等用クリ―ンル―ム
JP2856219B2 (ja) 1990-12-28 1999-02-10 ソニー株式会社 画像データの符号化装置及び符号化方法
US5293434A (en) * 1991-04-10 1994-03-08 International Business Machines Corporation Technique for use in a transform coder for imparting robustness to compressed image data through use of global block transformations
JPH07177507A (ja) * 1993-12-17 1995-07-14 Victor Co Of Japan Ltd 画像データ圧縮装置
US5881176A (en) * 1994-09-21 1999-03-09 Ricoh Corporation Compression and decompression with wavelet style and binary style including quantization by device-dependent parser
US5966465A (en) * 1994-09-21 1999-10-12 Ricoh Corporation Compression/decompression using reversible embedded wavelets
US6195465B1 (en) * 1994-09-21 2001-02-27 Ricoh Company, Ltd. Method and apparatus for compression using reversible wavelet transforms and an embedded codestream
US5867602A (en) * 1994-09-21 1999-02-02 Ricoh Corporation Reversible wavelet transform and embedded codestream manipulation
US6873734B1 (en) * 1994-09-21 2005-03-29 Ricoh Company Ltd Method and apparatus for compression using reversible wavelet transforms and an embedded codestream
US6181825B1 (en) * 1994-12-02 2001-01-30 Comsat Corporation Methods for performing 2-dimensional maximum differences coding and decoding during real-time facsimile image compression and apparatus therefor
JPH08331498A (ja) * 1995-06-01 1996-12-13 Asahi Optical Co Ltd 画像データ再圧縮装置
JP3540447B2 (ja) * 1995-07-20 2004-07-07 三洋電機株式会社 動画像符号化装置及び復号装置
KR100192270B1 (ko) * 1996-02-03 1999-06-15 구자홍 에이치디티브이 비데오 디코더 회로
TW364107B (en) * 1996-03-25 1999-07-11 Trw Inc Method and system for three-dimensional compression of digital video signals
JP3747970B2 (ja) 1996-12-24 2006-02-22 ソニー株式会社 画像符号化装置および画像符号化方法、画像復号化装置および画像復号化方法
US5928313A (en) * 1997-05-05 1999-07-27 Apple Computer, Inc. Method and apparatus for sample rate conversion
US5995923A (en) * 1997-06-26 1999-11-30 Nortel Networks Corporation Method and apparatus for improving the voice quality of tandemed vocoders
GB2333656B (en) * 1998-01-22 2002-08-14 British Broadcasting Corp Compressed signals
US6195394B1 (en) * 1998-11-30 2001-02-27 North Shore Laboratories, Inc. Processing apparatus for use in reducing visible artifacts in the display of statistically compressed and then decompressed digital motion pictures
US6259741B1 (en) * 1999-02-18 2001-07-10 General Instrument Corporation Method of architecture for converting MPEG-2 4:2:2-profile bitstreams into main-profile bitstreams
US6625322B1 (en) 1999-06-08 2003-09-23 Matsushita Electric Industrial Co., Ltd. Image coding apparatus
JP2001359096A (ja) 1999-06-08 2001-12-26 Matsushita Electric Ind Co Ltd 画像符号化装置
KR20010094694A (ko) * 2000-04-06 2001-11-01 윤종용 티시엠 복호장치 및 방법
DE10048735A1 (de) * 2000-09-29 2002-04-11 Bosch Gmbh Robert Verfahren zur Codierung und Decodierung von Bildsequenzen sowie Einrichtungen hierzu
AU2002255237A1 (en) * 2001-03-30 2003-07-24 Yissum Research Development Company Of The Hebrew University Of Jerusalem Discriminative feature selection for data sequences
US7046854B2 (en) * 2001-05-07 2006-05-16 Hrl Laboratories, Llc Signal processing subband coder architecture
EP1452039B1 (en) * 2001-11-29 2008-12-31 Panasonic Corporation Coding distortion removal method and video encoding and decoding methods
JP3797209B2 (ja) * 2001-11-30 2006-07-12 ソニー株式会社 画像情報符号化方法及び装置、画像情報復号方法及び装置、並びにプログラム
US6829579B2 (en) * 2002-01-08 2004-12-07 Dilithium Networks, Inc. Transcoding method and system between CELP-based speech codes
US7133368B2 (en) 2002-02-01 2006-11-07 Microsoft Corporation Peer-to-peer method of quality of service (QoS) probing and analysis and infrastructure employing same
AU2003209553A1 (en) * 2002-03-13 2003-09-22 Imax Corporation Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data
US7136417B2 (en) * 2002-07-15 2006-11-14 Scientific-Atlanta, Inc. Chroma conversion optimization
US6919892B1 (en) * 2002-08-14 2005-07-19 Avaworks, Incorporated Photo realistic talking head creation system and method
US7379496B2 (en) * 2002-09-04 2008-05-27 Microsoft Corporation Multi-resolution video coding and decoding
JP4762486B2 (ja) 2002-09-04 2011-08-31 マイクロソフト コーポレーション マルチ・リゾルーション・ビデオ符号化および復号化
JP3946177B2 (ja) * 2003-08-29 2007-07-18 三洋電機株式会社 動画像符号化装置及び復号装置
KR20060135667A (ko) 2003-12-17 2006-12-29 코닌클리케 필립스 일렉트로닉스 엔.브이. 이미지 포맷 변환
US8130825B2 (en) * 2004-05-10 2012-03-06 Nvidia Corporation Processor for video data encoding/decoding
US8442108B2 (en) * 2004-07-12 2013-05-14 Microsoft Corporation Adaptive updates in motion-compensated temporal filtering
US8340177B2 (en) * 2004-07-12 2012-12-25 Microsoft Corporation Embedded base layer codec for 3D sub-band coding
KR100648308B1 (ko) * 2004-08-12 2006-11-23 삼성전자주식회사 해상도 변환방법 및 장치
US8112399B2 (en) * 2005-11-07 2012-02-07 International Business Machines Corporation Method and apparatus for configurable data aggregation in a data warehouse
US8738565B2 (en) * 2005-11-07 2014-05-27 International Business Machines Corporation Collecting data from data sources
US20070112876A1 (en) * 2005-11-07 2007-05-17 Blaisdell Russell C Method and apparatus for pruning data in a data warehouse
US20070127824A1 (en) * 2005-12-07 2007-06-07 Trw Automotive U.S. Llc Method and apparatus for classifying a vehicle occupant via a non-parametric learning algorithm
GB2435140B (en) * 2006-02-13 2011-04-06 Snell & Wilcox Ltd Sport action coding
EP1837826A1 (en) * 2006-03-20 2007-09-26 Matsushita Electric Industrial Co., Ltd. Image acquisition considering super-resolution post-interpolation
US20070258012A1 (en) * 2006-05-04 2007-11-08 Syntax Brillian Corp. Method for scaling and cropping images for television display
US20080043832A1 (en) * 2006-08-16 2008-02-21 Microsoft Corporation Techniques for variable resolution encoding and decoding of digital video
US7903894B2 (en) * 2006-10-05 2011-03-08 Microsoft Corporation Color image coding using inter-color correlation
JP2008123086A (ja) 2006-11-09 2008-05-29 Matsushita Electric Ind Co Ltd 画像処理装置及び画像処理方法
US20080120676A1 (en) * 2006-11-22 2008-05-22 Horizon Semiconductors Ltd. Integrated circuit, an encoder/decoder architecture, and a method for processing a media stream
US8315466B2 (en) * 2006-12-22 2012-11-20 Qualcomm Incorporated Decoder-side region of interest video processing
KR20080071452A (ko) * 2007-01-30 2008-08-04 삼성전자주식회사 리스트 생성용 ｇｕｉ 제공방법 및 이를 적용한 영상기기
US20090299990A1 (en) * 2008-05-30 2009-12-03 Vidya Setlur Method, apparatus and computer program product for providing correlations between information from heterogenous sources
EP2157799A1 (en) * 2008-08-18 2010-02-24 Panasonic Corporation Interpolation filter with local adaptation based on block edges in the reference frame
US8543884B2 (en) * 2009-06-16 2013-09-24 Qualcomm Incorporated Communications channel parallel interleaver and de-interleaver

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
EP2335419A1 (en) 2011-06-22
CN102217314A (zh) 2011-10-12
KR20110059766A (ko) 2011-06-03
BRPI0918044A2 (pt) 2015-12-01
KR20170102033A (ko) 2017-09-06
KR101915425B1 (ko) 2018-11-05
US20110170615A1 (en) 2011-07-14
US9571857B2 (en) 2017-02-14
TW201031215A (en) 2010-08-16
JP2014060805A (ja) 2014-04-03
TWI499310B (zh) 2015-09-01
WO2010033151A1 (en) 2010-03-25
KR20160114186A (ko) 2016-10-04
JP2012503416A (ja) 2012-02-02
CN102217314B (zh) 2017-07-28
JP5907941B2 (ja) 2016-04-26

Similar Documents

Publication Publication Date Title
EP2335419B1 (en) Methods and apparatus for video imaging pruning
JP7228639B2 (ja) 省略符号化を用いた映像符号化及び復号化装置及びその方法
KR101545005B1 (ko) 이미지 압축 및 압축해제
US7379496B2 (en) Multi-resolution video coding and decoding
EP2422521B1 (en) Filter selection for video pre-processing in video applications
EP2141927A1 (en) Filters for video coding
US20110069752A1 (en) Moving image encoding/decoding method and apparatus with filtering function considering edges
CN114521327A (zh) 视频译码中的残差的量化
Vo et al. Selective data pruning-based compression using high-order edge-directed interpolation
US20240040160A1 (en) Video encoding using pre-processing
JP4762486B2 (ja) マルチ・リゾルーション・ビデオ符号化および復号化
KR102668077B1 (ko) 영상 부호화 및 복호화 장치 및 그 방법
WO2024078635A1 (en) Down-sampling methods and ratios for super-resolution based video coding
KR20240073227A (ko) 영상 부호화 및 복호화 장치 및 그 방법
EP4049449A1 (en) Weighted psnr quality metric for video coded data
Huang et al. A multi-frame post-processing approach to improved decoding of H. 264/AVC video
Vo et al. Data pruning-based compression using high order edge-directed interpolation
KR20140038322A (ko) 다중 계층 비디오를 위한 계층 간 필터링 장치 및 방법

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20110407

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA RS

RIN1 Information on inventor provided before grant (corrected)

Inventor name: ZHENG, YUNFEI

Inventor name: YIN, PENG

Inventor name: SOLE, JOEL

Inventor name: VO, DUNG, TRUNG

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: THOMSON LICENSING DTV

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20171128

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: INTERDIGITAL MADISON PATENT HOLDINGS

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602009063008

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04N0007460000

Ipc: H04N0019590000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 19/59 20140101AFI20191203BHEP

Ipc: H04N 19/85 20140101ALI20191203BHEP

INTG Intention to grant announced

Effective date: 20200103

RIN1 Information on inventor provided before grant (corrected)

Inventor name: SOLE, JOEL

Inventor name: VO, DUNG TRUNG

Inventor name: ZHENG, YUNFEI

Inventor name: YIN, PENG

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTC Intention to grant announced (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20200529

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1332349

Country of ref document: AT

Kind code of ref document: T

Effective date: 20201115

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602009063008

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20201104

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1332349

Country of ref document: AT

Kind code of ref document: T

Effective date: 20201104

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210204

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210304

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210205

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210204

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210304

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602009063008

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20210805

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210304

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210901

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210901

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210930

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20090901

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230514

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230926

Year of fee payment: 15

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230926

Year of fee payment: 15

Ref country code: DE

Payment date: 20230928

Year of fee payment: 15

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201104