US8554001B2 - Image encoding/decoding system using graph based pixel prediction and encoding system and method - Google Patents
Image encoding/decoding system using graph based pixel prediction and encoding system and method Download PDFInfo
- Publication number
- US8554001B2 US8554001B2 US13/016,373 US201113016373A US8554001B2 US 8554001 B2 US8554001 B2 US 8554001B2 US 201113016373 A US201113016373 A US 201113016373A US 8554001 B2 US8554001 B2 US 8554001B2
- Authority
- US
- United States
- Prior art keywords
- pixel
- graph
- pixels
- prediction
- prediction value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
- H04N19/19—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding using optimisation based on Lagrange multipliers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/11—Selection of coding mode or of prediction mode among a plurality of spatial predictive coding modes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
- H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/146—Data rate or code amount at the encoder output
- H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/593—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N2213/00—Details of stereoscopic systems
- H04N2213/003—Aspects relating to the "2D+depth" image format
Definitions
- One or more embodiments relate to an image encoding/decoding system and method using graph-based pixel prediction, and a depth map encoding system and method.
- a method of performing a prediction-encoding on the images using pixels neighboring a pixel to be encoded has been used. For example, when an image of a single frame is divided into blocks each having a pixel size of 4 ⁇ 4 in an H.264 of a digital video codec standard, a pixel value of a current block may be predicted, in a case of encoding a single block, using a pixel value of each of neighboring blocks adjacent to the current block, and a difference between the predicted pixel value and the pixel value of the current block may be encoded, thereby improving compression efficiency.
- shape information of the current block may be additionally provided so that the pixel value of the current block may be more accurately predicted.
- a method where the current block is divided into two regions using a single line, and a representative value of each of the two regions is encoded may be used.
- an encoding system including: an edge map generation unit to generate an edge map based on a pixel value difference between neighboring pixels in an inputted image; a graph generation unit to generate a graph by connecting pixels through the generated edge map; a prediction value selection unit to select a prediction value for at least one pixel based on the generated graph; and a prediction encoding unit to perform a prediction encoding on the at least one pixel based on the at least one pixel and the prediction value.
- the edge map generation unit may determine that an edge between the neighboring pixels exists, and mark, on the edge map, a value corresponding to a determined result to generate the edge map.
- the threshold value may be adjusted when generating the edge so that an edge generated amount is adjusted in accordance with a target bit rate of the inputted image to adjust a bit rate of an encoded edge map.
- the encoding system may further include an edge map encoding unit to encode the generated edge map.
- the edge map encoding unit may encode the edge map by compressing the edge map using a binary arithmetic encoding scheme, and the encoded edge map may be included in a bit string obtained by encoding the inputted image.
- the graph generation unit may generate the graph by connecting a plurality of pixels being inseparable from each other by an edge of the edge map.
- an encoding method including: generating an edge map based on a pixel value difference between neighboring pixels in an inputted image; generating a graph by connecting pixels through the generated edge map; selecting a prediction value for at least one pixel based on the generated graph; and performing a prediction encoding on the at least one pixel based on the at least one pixel and the prediction value.
- a decoding system including: an edge map decoding unit to decode an edge map in an inputted bit string; a graph generation unit to generate a graph by connecting pixels through the decoded edge map; a prediction value selection unit to select a prediction value for at least one pixel based on the generated graph; and a prediction compensation unit to restore the at least one pixel based on the prediction value and pixel information of the inputted bit string.
- a decoding method including: decoding an edge map from an inputted bit string; generating a graph by connecting pixels through the decoded edge map; selecting a prediction value for at least one pixel based on the generated graph; and restoring the at least one pixel based on the prediction value and pixel information of the inputted bit string (stream).
- a depth map encoding method including: predicting, based on a depth map and a reference image, an image quality of an image to be synthesized using the depth map; and selecting an encoding mode based on the predicted image quality.
- a depth map encoding system including: an image quality prediction unit to predict, based on a depth map and a reference image, an image quality of an image to be synthesized using the depth map; and an encoding mode selection unit to select an encoding mode based on the predicted image quality.
- FIG. 1 is a block diagram illustrating an internal configuration of an encoding system according to an embodiment
- FIG. 2 is a block diagram illustrating an internal configuration of a decoding unit according to an embodiment
- FIG. 3 illustrates an example of pixels of an image according to an embodiment
- FIG. 4 is a flowchart illustrating an encoding method according to an embodiment
- FIG. 5 is a block diagram illustrating an internal configuration of a decoding system according to an embodiment
- FIG. 6 is a flowchart illustrating a decoding method according to an embodiment
- FIG. 7 is a block diagram illustrating an internal configuration of a depth map encoding system according to an embodiment
- FIG. 8 is a flowchart illustrating a depth map encoding method according to an embodiment.
- FIG. 9 illustrates an example where a distortion generated in a synthesized image is estimated using a distortion of a depth map according to an embodiment.
- FIG. 1 is a block diagram illustrating an internal configuration of an encoding system according to an embodiment.
- the encoding system may include an edge map generation unit 101 , a graph generation unit 102 , a prediction value selection unit 103 , a prediction encoding unit 104 , a transformation and quantization unit 105 , an entropy encoding unit 106 , a decoding unit 107 , and an edge map encoding unit 109 .
- the edge map generation unit 101 may generate an edge map based on a pixel value difference between neighboring pixels in an inputted image. In this instance, when the pixel value difference is equal to or more than a predetermined threshold value, the edge map generation unit 101 may determine that an edge between the neighboring pixels exists, and mark, on the edge map, a value corresponding to a determined result to generate the edge map. Also, when the pixel value difference is less than the predetermined threshold value, the edge map generation unit 101 may determine that the edge between the neighboring pixels does not exist, and mark, on the edge map, a value corresponding to a determined result to generate the edge map.
- the threshold value may be adjusted when generating the edge so that an edge generated amount is adjusted in accordance with a target bit rate of the inputted image to adjust a bit rate of an encoded edge map.
- the graph generation unit 102 may generate a graph by connecting pixels through the generated edge map.
- the graph generation unit 102 may generate the graph by connecting a plurality of pixels being inseparable from each other by an edge of the edge map. Specifically, when an edge exists between the two neighboring pixels based on the edge map, the graph generation unit 102 may not connect two neighboring pixels to generate the graph, and when the edge does not exist between the two neighboring pixels, the graph generation unit 102 may connect the neighboring pixels to generate the graph.
- Equation 1 Equation 1
- Equation 2 [( i,j ),( i′,j ′)] ⁇ E [Equation 2]
- each of the blocks may be divided into a plurality of regions by a corresponding graph. Specifically, mutually connected pixels may generate a single path to form a single region.
- the prediction value selection unit 103 may select a prediction value for at least one pixel based on the generated graph. In this instance, the prediction value selection unit 103 may select, as the prediction value, a pixel value of a pixel neighboring a block including the at least one pixel and closest to the at least one pixel from among pixels connected to the at least one pixel by the graph.
- the prediction value selection unit 103 may assign mutually different weights to pixel values of pixels connected to the at least one pixel by the graph, and may select, as the prediction value, an operation result between the pixel values where the mutually different weights are assigned.
- the pixels connected to the at least one pixel by the graph may neighbor a block including the at least one pixel.
- the weights may be determined based on a spectral distribution of the graph.
- the prediction value selection unit 103 may calculate an eigenvector of a Laplacian matrix of the graph for the purpose of using the spectral distribution, and determine the weights based on the eigenvector. For example, when calculating a Laplacian matrix from the graph, and then calculating an eigenvector, the spectral distribution may be obtained. In this instance, a higher frequency may be obtained in an increased order of the eigenvector.
- the pixel value x desired to be predicted may be obtained using Equation 4 below.
- V [ x ⁇ ⁇ a 2 ⁇ ⁇ a 3 ⁇ ⁇ ... ⁇ ⁇ a N ] T [ Equation ⁇ ⁇ 3 ]
- Equations 3 and 4 ‘e’ denotes an eigenvector, ‘a 2 a 3 . . . a N ’ denotes pixel values of pixels neighboring a block including a pixel desired to be predicted, ‘T’ denotes a transposing operation, and an operator ‘•’ denotes a vector inner-product operator.
- the prediction value selection unit 103 may substitute, in a plane equation, pixel values that neighbor a block including at least one pixel and are connected by a graph or pixel values where mutually different weights are applied, and select an obtained plane value as the prediction value.
- pixel values of decoded pixels may be used as the pixel values of the pixels neighboring the block.
- the pixel values of the decoded pixels it may be possible to enable encoded pixels to be restored using identical information even in a decoding system that will be described later.
- a method of using the pixel values of the decoded pixels neighboring the block will be further described later.
- the prediction encoding unit 104 may perform a prediction encoding on at least one pixel based on the at least one pixel and the prediction value. In this instance, the prediction encoding unit 104 may encode a difference value between a pixel value of the at least one pixel and the prediction value.
- the transformation and quantization unit 105 may transform and quantize the pixel where the prediction encoding is performed. Specifically, the encoded difference values with respect to the block may be lossy compressed through a transformation and a quantization, such as Discrete Cosine Transform (DCT).
- DCT Discrete Cosine Transform
- the entropy encoding unit 106 may perform an entropy encoding on the transformed and quantized pixel to generate a bit string.
- the entropy encoding unit 106 may compress, into a bit string, pixel values of the transformed and quantized pixels, that is, transformed and quantized difference values through the entropy encoding, such as a Huffman encoding or an arithmetic encoding.
- the decoding unit 107 may decode the transformed and quantized pixel based on the prediction value.
- restored neighboring pixels may denote the above described decoded pixels.
- pixel values of corresponding restored neighboring pixels from the decoding unit 107 may be used as the pixel values of the pixels neighboring the block including the at least one pixel.
- the restored neighboring pixels in the selection process of the prediction value it may be possible to enable an encoded pixel to be restored using identical information even in the decoding system.
- the edge map encoding unit 109 may encode the generated edge map.
- the edge map encoding unit 109 may encode the edge map using a binary arithmetic encoding method. In this manner, the encoded edge map may be included in the above described bit string where the inputted image is encoded.
- FIG. 2 is a block diagram illustrating an internal configuration of a decoding unit according to an embodiment.
- the decoding unit 107 includes an inverse-transformation and inverse-quantization unit 201 and a prediction compensation unit 202 .
- the inverse-transformation and inverse-quantization unit 201 may inverse-transform and inverse-quantize the transformed and quantized values obtained by the transformation and quantization unit 105 .
- the prediction compensation unit 202 may restore pixels by applying, to the inverse-transformed and inverse-quantized values, the prediction value selected in the prediction value selection unit 103 .
- the reason why the pixels are restored may be because encoded pixel may be restored using identical information even in the decoding system, by selecting the prediction value using the pixel values of the restored pixels in the prediction value selection unit 103 .
- FIG. 3 illustrates an example of pixels of an image according to an embodiment.
- An image 300 may illustrate a plurality of pixels, and each of alphabets including uppercase and lowercase letters may denote a pixel.
- pixels from ‘a’ to ‘p’ may include a single block having a pixel size of 4 ⁇ 4.
- the pixels may be separated into two regions by an edge, that is, a bold line of the image 300 .
- pixels ‘a’, ‘b’, ‘c’, ‘d’, ‘e’, ‘f’, ‘g’, and ‘i’ and pixels ‘h’, ‘j’, ‘k’, ‘l’, ‘m’, ‘n’, ‘o’, and ‘p’ may be separated from each other.
- a prediction value for each of ‘h’, ‘j’, ‘k’, ‘l’, ‘m’, ‘n’, ‘o’, and ‘p’ a pixel value of a pixel ‘L’ that neighbors a corresponding block and is closest to a corresponding pixel from among pixels connected to the corresponding pixel by a graph may be selected.
- a pixel value of a pixel that neighbors a corresponding block and is closest to a corresponding pixel from among pixels connected to the corresponding pixel by a graph may be respectively selected.
- a pixel value of the pixel ‘B’ may be used as a prediction value for the pixel b
- a pixel value of the pixel ‘C’ may be used as a prediction value of each of the pixels ‘c’ and ‘g’.
- a priority may be applied to a pixel existing in a specific direction.
- a pixel value of the pixel positioned in an upper end of the image 300 may be selected as the prediction value.
- the pixel value of the selected pixel may be directly used as the prediction value, however, when the selected pixel is connected to the plurality of pixels described in FIG. 1 , mutually different weights may be applied to the pixel values of the connected plurality of pixels, and a sum of the pixel values where the weights are applied may be used as the prediction value.
- the pixels ‘a’, ‘b’, ‘c’, ‘d’, ‘e’, ‘f’, ‘g’, and ‘i’ within the block having the pixel size of 4 ⁇ 4 may be included in a single path of a graph, and may be connected to the pixels ‘A’, ‘B’, ‘C’, ‘D’, ‘I’, ‘J’, and ‘K’ neighboring the block.
- the prediction value of the pixel ‘a’ may be obtained using the following Equation 5.
- a — pred A*w — A+B*w — B+C*w — C+D*w — D+I*w — I+J*w — J+K*w — K
- ‘a_pred’ denotes the prediction value of the pixel ‘a’.
- FIG. 4 is a flowchart illustrating an encoding method according to an embodiment.
- the encoding method may be performed by the encoding system of FIG. 1 .
- the encoding method will be described based on a process where each of the operations is performed by the encoding system.
- the encoding system may generate an edge map based on a pixel value difference between neighboring pixels in an inputted image.
- the encoding system may determine that an edge between the neighboring pixels exists, and mark, on the edge map, a value corresponding to a determined result to generate the edge map.
- the encoding system may determine that the edge between the neighboring pixels does not exist, and mark, on the edge map, a value corresponding to a determined result to generate the edge map.
- the threshold value may be adjusted when generating the edge so that an edge generated amount is adjusted in accordance with a target bit rate of the inputted image to adjust a bit rate of an encoded edge map.
- the encoding system may generate a graph by connecting pixels through the generated edge map.
- the encoding system may generate the graph by connecting a plurality of pixels being inseparable from each other by an edge of the edge map. Specifically, when an edge exists between the two neighboring pixels based on the edge map, the encoding system may not connect two neighboring pixels to generate the graph, and when the edge does not exist between the two neighboring pixels, the encoding system may connect the neighboring pixels to generate the graph.
- the graph may be expressed as Equation 1 described above.
- each of the blocks may be divided into a plurality of regions by a corresponding graph. Specifically, mutually connected pixels may generate a single path to form a single region.
- the encoding system may select a prediction value for at least one pixel based on the generated graph. In this instance, the encoding system may select, as the prediction value, a pixel value of a pixel neighboring a block including the at least one pixel and closest to the at least one pixel from among pixels connected to the at least one pixel by the graph.
- the encoding system may assign mutually different weights to pixel values of pixels connected to the at least one pixel by the graph, and may select, as the prediction value, an operation result between the pixel values where the mutually different weights are assigned.
- the pixels connected to the at least one pixel by the graph may neighbor a block including the at least one pixel.
- the weights may be determined based on a spectral distribution of the graph.
- the encoding system may calculate an eigenvector of a Laplacian matrix of the graph for the purpose of using the spectral distribution, and determine the weights based on the eigenvector. For example, when calculating a Laplacian matrix from the graph, and then calculating an eigenvector, the spectral distribution may be obtained. In this instance, a higher frequency may be obtained in an increased order of the eigenvector.
- the pixel value x desired to be predicted may be obtained using Equation 4 described above.
- the encoding system may substitute, in a plane equation, pixel values that neighbor a block including at least one pixel and are connected by a graph or pixel values where mutually different weights are applied, and select an obtained plane value as the prediction value.
- pixel values of decoded pixels may be used as the pixel values of the pixels neighboring the block.
- the pixel values of the decoded pixels it may be possible to enable encoded pixels to be restored using identical information even in a decoding system that will be described later.
- a method of using the pixel values of the decoded pixels neighboring the block will be further described later.
- the encoding system may perform a prediction encoding on at least one pixel based on the at least one pixel and the prediction value.
- the encoding system may encode a difference value between a pixel value of the at least one pixel and the prediction value.
- the encoding system may transform and quantize the pixel where the prediction encoding is performed.
- the encoded difference values with respect to the block may be lossy compressed through a transformation and a quantization such as a DCT.
- the encoding system may perform an entropy encoding on the transformed and quantized pixel to generate a bit string.
- the entropy encoding unit 106 may compress, into a bit string, pixel values of the transformed and quantized pixels, that is, transformed and quantized difference values through the entropy encoding such as a Huffman encoding or an arithmetic encoding.
- the encoding system may encode the generated edge map.
- the encoding system may encode the edge map using a binary arithmetic encoding scheme.
- the encoded edge map may be included in the above described bit string where the inputted image is encoded.
- the encoding system may decode the transformed and quantized pixel based on the prediction value. Specifically, in operation 430 , the encoding system may use pixel values of corresponding restored neighboring pixels as the pixel values of the pixels neighboring the block including the at least one pixel. As described above, by using the restored neighboring pixels in the selection process of the prediction value, it may be possible to enable an encoded pixel to be restored using identical information even in a decoding method.
- FIG. 5 is a block diagram illustrating a decoding system according to an embodiment.
- the decoding system includes an edge map decoding unit 501 , a graph generation unit 502 , a prediction value selection unit 503 , an entropy decoding unit 504 , an inverse-transformation and inverse-quantization unit 505 , and a prediction compensation unit 507 .
- the edge map decoding unit 501 may decode an edge map from an inputted bit string. In this instance, the edge map decoding unit 501 may decode the edge map from the inputted bit string using a binary arithmetic decoding method.
- the graph generation unit 502 may generate a graph by connecting pixels through the decoded edge map.
- the graph generation unit 502 may generate the graph by connecting a plurality of pixels being inseparable from each other by an edge of the edge map.
- the method of generating the graph may be the same as those of FIG. 1 and FIG. 4 .
- the prediction value selection unit 503 may select a prediction value for at least one pixel based on the generated graph.
- an image corresponding to the inputted bit string may be classified into blocks each having a predetermined pixel size, and the prediction value selection unit 503 may select, as the prediction value, a pixel value of a pixel neighboring a block including the at least one pixel and closest to the at least one pixel from among pixels connected to the at least one pixel by the graph.
- the prediction value selection unit 503 may assign mutually different weights to pixel values of pixels connected to the at least one pixel by the graph, and may select, as the prediction value, an operation result between the pixel values where the mutually different weights are assigned.
- the pixels connected to the at least one pixel may neighbor a block including the at least one pixel.
- the method of selecting the prediction value may be the same as those of FIGS. 1 and 4 .
- the entropy decoding unit 504 may perform an entropy decoding on the inputted bit string to decode a pixel, and the inverse-transformation and inverse-quantization unit 505 may inverse-transform and inverse-quantize the pixel where the entropy decoding is performed.
- the decoding system may restore an original pixel value prior to the entropy encoding, the transformation, and the quantization described in FIGS. 1 and 4 by performing the entropy decoding, the inverse-transformation, and the inverse-quantization on the inputted bit string.
- the prediction value selection unit 503 may select a prediction value for the at least one pixel based on the pixels neighboring a block including the at least one pixel from among the inverse-transformed and inverse-quantized pixels and based on the graph. Specifically, as the pixels neighboring the block, corresponding pixels from among the inverse-transformed and the inverse-quantized pixels may be used.
- the restored neighboring pixels of FIG. 5 may denote the pixels neighboring the block including the at least one pixel from among the inverse-transformed and inverse-quantized pixels.
- the prediction compensation unit 507 may restore the at least one pixel based on the prediction value and pixel information of the inputted bit string.
- the pixel information of the bit string may be a value including a difference value between an original pixel value and the prediction value, and the prediction compensation unit 507 may add the prediction value to the difference value, thereby restoring the at least one pixel.
- FIG. 6 is a flowchart illustrating a decoding method according to an embodiment.
- the decoding method may be performed by the decoding system of FIG. 5 .
- the decoding method may be described based on a process where each of operations is performed by the decoding system.
- the decoding system may perform an entropy decoding on an inputted bit string to decode a pixel.
- the decoding system may decode an edge map in the inputted bit string.
- the decoding system may decode the edge map from the inputted bit string using a binary arithmetic decoding method.
- the decoding system may inverse-transform and inverse-quantize the pixel where the entropy decoding is performed. Specifically, the decoding system may restore an original pixel value prior to the entropy encoding, the transformation, and the quantization described in FIGS. 1 and 4 by performing the entropy decoding, the inverse-transformation, and the inverse-quantization on the bit string inputted through operations 610 and 630 .
- the decoding system may generate a graph by connecting pixels using a decoded edge map.
- the decoding system may generate the graph by connecting a plurality of pixels being inseparable from each other by an edge of the edge map. That is, a method of generating the graph may be the same as those described in FIGS. 1 and 4 .
- the decoding system may select a prediction value for at least one pixel based on the generated graph.
- an image corresponding to the inputted bit string may be classified into blocks each having a predetermined pixel size, and the decoding system may select, as the prediction value, a pixel value of a pixel neighboring a block including the at least one pixel and closest to the at least one pixel from among pixels connected to the at least one pixel by the graph.
- the decoding system may assign mutually different weights to pixel values of pixels connected to the at least one pixel by the graph, and select, as the prediction value, an operation result between the pixel values where the mutually different weights are assigned.
- the pixels connected to the at least one pixel may neighbor a block including the at least one pixel.
- a method of selecting the prediction value may be the same as those described in FIGS. 1 and 4 .
- the decoding system may select the prediction value for the at least one pixel based on pixels neighboring a block including the at least one pixel from among the inverse-transformed and inverse-quantized pixels. Specifically, as the pixels neighboring the block, corresponding pixels, from among the inverse-transformed and inverse-quantized pixels, may be used.
- the decoding system may restore the at least one pixel based on the prediction value and pixel information of the inputted bit string.
- the pixel information of the bit string may be a value including a difference value between an original pixel value and the prediction value, and the decoding system may add the prediction value to the difference value, thereby decoding the at least one pixel.
- FIGS. 5 and 6 may refer to the descriptions of FIGS. 1 and 4 .
- the encoding/decoding system and method according to embodiments may be effectively used in compressing the depth map.
- an optimal prediction method may need to be determined to be used.
- a distortion value generated by lossy encoding the depth map may not be directly used to calculate the distortion value, an influence exerted on a synthesized image due to the distortion value may be predicted to obtain a prediction value, and the prediction value may be used. This is because the depth map may not be displayed to a user when the depth map is applied to a 3D image, and the depth map may be used only for synthesizing the image. Thus, preferably, the distortion value generated in the synthesized image displayed to the user may be used to perform an optimization.
- a method of calculating the distortion value by synthesizing the image with respect to each of the prediction methods may have limitations due to its complexity when being actually utilized.
- the distortion generated in the synthesized image may be predicted using the distortion generated in the depth map to obtain a prediction value, so that a more accurate optimization may be performed using the prediction value.
- the distortion value generated in the synthesized image may be predicted using a difference between a pixel value of an image positioned in the same position as a pixel position of the depth map to be encoded and a pixel value moved, from the same position, by a displacement generated by multiplying a weight by a distortion value generated in a pixel value of the depth map.
- the weight used for calculating the displacement may be obtained based on an inertial coefficient such as a focal distance of a camera used for generating a 3D image, an external coefficient indicating a position and an angle of the camera, information about a maximal depth and a minimal depth of the generated 3D image, and the like.
- an optimal prediction method may be determined using the prediction value obtained by predicting the distortion value generated in the synthesized image. Specifically, when calculating the rate-distortion cost with respect to each of the prediction methods using the Lagrange optimal method, the rate-distortion cost may be calculated using the predicted distortion value, and the prediction method having the smallest rate-distortion cost may be selected as the optimal prediction method to be used.
- FIG. 7 is a block diagram illustrating an encoding system using a depth map according to an embodiment.
- the depth map encoding system includes an image quality prediction unit 701 , an encoding mode selection unit 702 , a graph based prediction unit 703 , a spatial prediction unit 704 , a temporal prediction unit 705 , a prediction encoding unit 706 , a transformation and quantization unit 707 , and an entropy encoding unit 708 .
- the image quality prediction unit 701 may predict an image quality of an image to be synthesized using the depth map based on a depth map and a reference image. In this instance, the image quality prediction unit 701 may calculate a distortion value generated in the image to be synthesized, using a difference between a first pixel value of the reference image positioned in the same position as a pixel position of the depth map and a second pixel value moved, from the same position, by a displacement generated by assigning a weight to a distortion value generated in a pixel value of the depth map.
- the weight may be obtained based on the inertial coefficient such as the focal distance of a camera used for generating the 3D image, the external coefficient indicating a position and an angle of the camera, information about a maximal depth and a minimal depth of the generated 3D image, and the like.
- the encoding mode selection unit 702 may select an encoding mode based on the predicted image quality. In this instance, the encoding mode selection unit 702 may calculate a rate-distortion cost based on the distortion value, and select the encoding mode having a value where the calculated rate-distortion cost is a minimum.
- the encoding mode may include at least two of a graph-based prediction mode, a spatial prediction mode, and a temporal prediction mode.
- the graph-based prediction mode is selected in the encoding mode selection unit 702
- the graph-based prediction unit 703 may be operated
- the spatial prediction unit 704 may be operated.
- the temporal prediction unit 705 may be operated.
- the graph-based prediction mode may generate an edge map based on a pixel value difference between neighboring pixels in an image to be synthesized, generate a graph by connecting pixels using the generated edge map, and select a prediction value for at least one pixel based on the generated graph.
- the graph based prediction unit 703 may further include the edge map generation unit 101 , the graph generation unit 102 , and the prediction value selection unit 103 which are described in FIG. 1 .
- the graph based prediction unit 703 may further include the edge map encoding unit 109 .
- the prediction encoding unit 706 , the transformation and quantization unit 707 , and the entropy encoding unit 708 may respectively correspond to the prediction encoding unit 104 , the transformation and quantization unit 105 , the entropy encoding unit 106 .
- FIG. 8 is a flowchart illustrating a depth map encoding method according to an embodiment.
- the depth map encoding method may be performed by the encoding system of FIG. 7 .
- the depth map encoding method will be described based on a process where each of operations is performed by the depth map encoding system.
- the depth map encoding system may predict an image quality of an image to be synthesized, using a depth map based on a depth map and a reference image.
- the depth map encoding system may calculate a distortion value generated in the image to be synthesized, using a difference between a first pixel value of the reference image positioned in the same position as a pixel position of the depth map and a second pixel value moved, from the same position, by a displacement generated by assigning a weight to a distortion value generated in a pixel value of the depth map.
- the weight used for calculating the displacement may be obtained based on an inertial coefficient such as a focal distance of a camera used for generating a 3D image, an external coefficient indicating a position and an angle of the camera, information about a maximal depth and a minimal depth of the generated 3D image, and the like.
- an inertial coefficient such as a focal distance of a camera used for generating a 3D image, an external coefficient indicating a position and an angle of the camera, information about a maximal depth and a minimal depth of the generated 3D image, and the like.
- the depth map encoding system may select an encoding mode based on the predicted image quality.
- the depth map encoding system may calculate a rate-distortion cost based on the distortion value, and select the encoding mode having a value where the calculated rate-distortion cost is a minimum.
- the encoding mode may include at least one of a graph-based prediction mode, a spatial prediction mode, and a temporal prediction mode.
- the depth map encoding system may determine whether the selected encoding mode is the graph-based prediction mode. Specifically, when the selected encoding mode is the graph-based prediction mode, the depth map encoding system may perform operation 840 , and when the selected encoding mode is not the graph-based prediction mode, the depth map encoding system may perform operation 850 .
- the depth map encoding system may perform a graph-based prediction encoding, and in operation 850 , the depth map encoding system may perform a spatial prediction encoding or a temporal prediction encoding.
- operation 840 for the graph-based prediction encoding may include operations 401 to 403 described with reference to FIG. 4 .
- the depth map encoding system may transform and quantize a pixel where the prediction encoding is performed, and in operation 870 , the depth map encoding system may perform an entropy encoding on the transformed and quantized pixel to generate a bit string.
- Operations 860 and 870 may respective correspond to operations 450 and 460 described with reference to FIG. 4 .
- FIGS. 7 and 8 may refer to the descriptions of FIGS. 1 and 6 .
- FIG. 9 illustrates an example where a distortion generated in a synthesized image is estimated using a distortion of a depth map according to an embodiment.
- a pixel of a first position (x im , y im ) 905 may be mapped to a second position (x′ im , y′ im ) 906 in a case where an error in the depth map does not occur.
- the pixel of the first position (x im , y im ) 905 may be moved by a displacement (tx, ty) to be mapped to a third position (x′ im+tx , y′ im+ty ) 907 .
- a distortion value may be calculated using a pixel value difference between the first position (x im , y im ) 905 and the third position (x′ im+tx , y′ im+ty ) 907 in the image obtained from the position of the camera (p) 901 .
- a pixel prediction may be effectively performed based on information about regions of an image, using the graph, to improve compression efficiency, so that an image quality of the restored image may be improved while reducing a compressed bit rate. Also, when encoding a 3D image in an image synthesizing scheme performed using the depth map, an influence exerted on the synthesized image due to the distortion value generated when the depth map is lossy encoded may be predicted to determine an optimized encoding mode.
- the methods according to the above-described embodiments may be recorded in computer-readable non-transitory storage media including program instructions to implement various operations embodied by a computer.
- the media may also include, alone or in combination with the program instructions, data files, data structures, and the like.
- Examples of computer-readable non-transitory media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks and DVDs; magneto-optical media such as optical disks; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like.
- Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.
- the described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, or vice versa.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Compression Of Band Width Or Redundancy In Fax (AREA)
Abstract
Description
G=(V,E) [Equation 1]
[(i,j),(i′,j′)]εE [Equation 2]
a — pred=A*w — A+B*w — B+C*w — C+D*w — D+I*w — I+J*w — J+K*w — K [Equation 5]
Here, ‘a_pred’ denotes the prediction value of the pixel ‘a’.
L=D+λR [Equation 6]
Claims (8)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/016,373 US8554001B2 (en) | 2010-02-12 | 2011-01-28 | Image encoding/decoding system using graph based pixel prediction and encoding system and method |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US30382610P | 2010-02-12 | 2010-02-12 | |
KR1020100018720A KR101598857B1 (en) | 2010-02-12 | 2010-03-02 | Image/video coding and decoding system and method using graph based pixel prediction and depth map coding system and method |
KR10-2010-0018720 | 2010-03-02 | ||
US13/016,373 US8554001B2 (en) | 2010-02-12 | 2011-01-28 | Image encoding/decoding system using graph based pixel prediction and encoding system and method |
Publications (2)
Publication Number | Publication Date |
---|---|
US20110206288A1 US20110206288A1 (en) | 2011-08-25 |
US8554001B2 true US8554001B2 (en) | 2013-10-08 |
Family
ID=43827087
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/016,373 Active 2031-08-18 US8554001B2 (en) | 2010-02-12 | 2011-01-28 | Image encoding/decoding system using graph based pixel prediction and encoding system and method |
Country Status (2)
Country | Link |
---|---|
US (1) | US8554001B2 (en) |
EP (1) | EP2360927A3 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9571468B2 (en) * | 2014-01-05 | 2017-02-14 | Whovoo, Inc. | Encoding data using a variable number of pixels and bits based on entropy |
US10984560B1 (en) * | 2019-03-29 | 2021-04-20 | Amazon Technologies, Inc. | Computer vision using learnt lossy image compression representations |
Families Citing this family (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2360927A3 (en) * | 2010-02-12 | 2011-09-28 | Samsung Electronics Co., Ltd. | Image encoding/decoding system using graph based pixel prediction and encoding system and method |
JP5281623B2 (en) * | 2010-09-29 | 2013-09-04 | 日本電信電話株式会社 | Image encoding method, image decoding method, image encoding device, image decoding device, and programs thereof |
JP5281624B2 (en) * | 2010-09-29 | 2013-09-04 | 日本電信電話株式会社 | Image encoding method, image decoding method, image encoding device, image decoding device, and programs thereof |
JP5357199B2 (en) * | 2011-03-14 | 2013-12-04 | 日本電信電話株式会社 | Image encoding method, image decoding method, image encoding device, image decoding device, image encoding program, and image decoding program |
PL397010A1 (en) * | 2011-11-17 | 2013-05-27 | Politechnika Poznanska | Method for predicting the shape of the coded area using depth maps |
JP5934380B2 (en) * | 2011-12-29 | 2016-06-15 | インテル コーポレイション | Variable depth compression |
CN102685530B (en) * | 2012-04-23 | 2014-05-07 | 北京邮电大学 | Intra depth image frame prediction method based on half-pixel accuracy edge and image restoration |
US20140270409A1 (en) * | 2013-03-15 | 2014-09-18 | Eyelock, Inc. | Efficient prevention of fraud |
US9495526B2 (en) | 2013-03-15 | 2016-11-15 | Eyelock Llc | Efficient prevention of fraud |
WO2016047897A1 (en) * | 2014-09-26 | 2016-03-31 | 엘지전자(주) | Method and device for processing graph-based signal using geometric primitives |
WO2016064185A1 (en) * | 2014-10-21 | 2016-04-28 | 엘지전자(주) | Method and apparatus for performing graph-based prediction by using optimization function |
EP3211893B1 (en) | 2014-10-24 | 2019-09-04 | LG Electronics Inc. | Method and apparatus for decoding/encoding video signal using transform derived from graph template |
KR102332782B1 (en) * | 2014-12-15 | 2021-11-30 | 삼성전자주식회사 | Video data compression considering a vision system |
WO2016129872A1 (en) * | 2015-02-12 | 2016-08-18 | 엘지전자(주) | Method and apparatus for processing video signal using graph-based transform |
WO2017043941A1 (en) * | 2015-09-10 | 2017-03-16 | 엘지전자(주) | Method and apparatus for processing video signal using graph-based transformation based on prediction angle |
ITUB20153724A1 (en) | 2015-09-18 | 2017-03-18 | Sisvel Tech S R L | METHODS AND EQUIPMENT TO CODIFY AND DECODE DIGITAL VIDEO IMAGES OR FLOWS |
ITUB20153912A1 (en) | 2015-09-25 | 2017-03-25 | Sisvel Tech S R L | METHODS AND EQUIPMENT TO CODIFY AND DECODE DIGITAL IMAGES BY SUPERPIXEL |
IT201600122898A1 (en) * | 2016-12-02 | 2018-06-02 | Ecole Polytechnique Fed Lausanne Epfl | METHODS AND EQUIPMENT TO CODIFY AND DECODE DIGITAL VIDEO IMAGES OR FLOWS |
IT201700024294A1 (en) | 2017-03-03 | 2018-09-03 | Sisvel Tech S R L | METHOD AND APPARATUS TO CODIFY AND DECODE DIGITAL VIDEO IMAGES OR FLOWS |
IT201700024221A1 (en) * | 2017-03-03 | 2018-09-03 | Sisvel Tech S R L | METHODS AND APPARATUSES FOR ENCODING AND DECODING SUPERPIXEL BORDERS |
US10638165B1 (en) * | 2018-11-08 | 2020-04-28 | At&T Intellectual Property I, L.P. | Adaptive field of view prediction |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5978514A (en) * | 1994-11-10 | 1999-11-02 | Kabushiki Kaisha Toshiba | Image data coding and decoding system for efficiently compressing information using the shape and position of the image content |
US6173077B1 (en) * | 1996-11-13 | 2001-01-09 | U.S. Philips Corporation | Image segmentation |
US20100278232A1 (en) * | 2009-05-04 | 2010-11-04 | Sehoon Yea | Method Coding Multi-Layered Depth Images |
US20100303150A1 (en) * | 2006-08-08 | 2010-12-02 | Ping-Kang Hsiung | System and method for cartoon compression |
US20110075729A1 (en) * | 2006-12-28 | 2011-03-31 | Gokce Dane | method and apparatus for automatic visual artifact analysis and artifact reduction |
US20110206288A1 (en) * | 2010-02-12 | 2011-08-25 | Samsung Electronics Co., Ltd. | Image encoding/decoding system using graph based pixel prediction and encoding system and method |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2012089905A (en) * | 2009-01-13 | 2012-05-10 | Hitachi Ltd | Image encoder and image encoding method, and image decoder and image decoding method |
-
2010
- 2010-12-08 EP EP10194221A patent/EP2360927A3/en not_active Withdrawn
-
2011
- 2011-01-28 US US13/016,373 patent/US8554001B2/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5978514A (en) * | 1994-11-10 | 1999-11-02 | Kabushiki Kaisha Toshiba | Image data coding and decoding system for efficiently compressing information using the shape and position of the image content |
US6173077B1 (en) * | 1996-11-13 | 2001-01-09 | U.S. Philips Corporation | Image segmentation |
US20100303150A1 (en) * | 2006-08-08 | 2010-12-02 | Ping-Kang Hsiung | System and method for cartoon compression |
US20110075729A1 (en) * | 2006-12-28 | 2011-03-31 | Gokce Dane | method and apparatus for automatic visual artifact analysis and artifact reduction |
US20100278232A1 (en) * | 2009-05-04 | 2010-11-04 | Sehoon Yea | Method Coding Multi-Layered Depth Images |
US20110206288A1 (en) * | 2010-02-12 | 2011-08-25 | Samsung Electronics Co., Ltd. | Image encoding/decoding system using graph based pixel prediction and encoding system and method |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9571468B2 (en) * | 2014-01-05 | 2017-02-14 | Whovoo, Inc. | Encoding data using a variable number of pixels and bits based on entropy |
US10984560B1 (en) * | 2019-03-29 | 2021-04-20 | Amazon Technologies, Inc. | Computer vision using learnt lossy image compression representations |
Also Published As
Publication number | Publication date |
---|---|
US20110206288A1 (en) | 2011-08-25 |
EP2360927A2 (en) | 2011-08-24 |
EP2360927A3 (en) | 2011-09-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8554001B2 (en) | Image encoding/decoding system using graph based pixel prediction and encoding system and method | |
US11509899B2 (en) | Method and device for encoding intra prediction mode for image prediction unit, and method and device for decoding intra prediction mode for image prediction unit | |
US8208544B2 (en) | Image coding method and apparatus using side matching process and image decoding method and apparatus for the same | |
US11451768B2 (en) | Method for image processing and apparatus for implementing the same | |
US9277224B2 (en) | Method and apparatus for encoding video, and method and apparatus for decoding video | |
US10798416B2 (en) | Apparatus and method for motion estimation of three dimension video | |
US8374243B2 (en) | Method and apparatus for encoding and decoding based on intra prediction | |
US11756233B2 (en) | Method for image processing and apparatus for implementing the same | |
KR101598857B1 (en) | Image/video coding and decoding system and method using graph based pixel prediction and depth map coding system and method | |
US8363967B2 (en) | Method and apparatus for intraprediction encoding/decoding using image inpainting | |
WO2010001917A1 (en) | Image processing device and method | |
US9936223B2 (en) | Method and device for encoding video to improve intra prediction processing speed, and method and device for decoding video | |
US20080240245A1 (en) | Image encoding/decoding method and apparatus | |
US20170366807A1 (en) | Coding of intra modes | |
US20130170761A1 (en) | Apparatus and method for encoding depth image by skipping discrete cosine transform (dct), and apparatus and method for decoding depth image by skipping dct | |
US20100329336A1 (en) | Method and apparatus for encoding and decoding based on inter prediction using image inpainting |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: UNIVERSITY OF SOUTHERN CALIFORNIA, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LEE, JAEJOON;WEY, HOCHEON;SHEN, GODWIN;AND OTHERS;SIGNING DATES FROM 20101112 TO 20110120;REEL/FRAME:025733/0245 Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LEE, JAEJOON;WEY, HOCHEON;SHEN, GODWIN;AND OTHERS;SIGNING DATES FROM 20101112 TO 20110120;REEL/FRAME:025733/0245 |
|
AS | Assignment |
Owner name: SOUTHERN CALIFORNIA UNIVERSITY OF, CALIFORNIA Free format text: RE-RECORD TO CORRECT SECOND ASSIGNEE'S ADDRESS PREVIOUSLY RECORDED AT REEL 025733, FRAME 0245;ASSIGNORS:LEE, JAEJOON;WEY, HOCHEON;SHEN, GODWIN;AND OTHERS;SIGNING DATES FROM 20101112 TO 20110120;REEL/FRAME:030696/0011 Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: RE-RECORD TO CORRECT SECOND ASSIGNEE'S ADDRESS PREVIOUSLY RECORDED AT REEL 025733, FRAME 0245;ASSIGNORS:LEE, JAEJOON;WEY, HOCHEON;SHEN, GODWIN;AND OTHERS;SIGNING DATES FROM 20101112 TO 20110120;REEL/FRAME:030696/0011 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |