JP5190469B2 - Boundary region processing in images - Google Patents

Description

  The present invention generally relates to image processing, and more particularly to image boundary region processing.

  Frame rate up-conversion is performed because it is generally considered that a higher frame rate video sequence provides a higher quality experience than a lower frame rate video sequence. The frame rate of the video sequence is increased by inserting expected frames between existing frames. A suitable method is to predict intermediate frames using bi-directional block motion estimation, thereby searching for linear motion between the preceding and succeeding frames of the input video sequence. It is possible to use non-linear methods that can represent acceleration, but linear methods are used because they are simple and not so complex. The intermediate frame is divided into blocks, and each block needs to be assigned a motion vector in some way.

  There is a problem with frame rate up conversion performed along image boundaries. In many recorded video sequences, there are black lines along one or more boundaries. This is seen along the right border of the left image 50 of FIG. In the conventional rate-up conversion algorithm (Non-Patent Documents 1 to 3), there is no special processing of such a boundary. As a result, when a motion vector pointing to the boundary region is assigned, the line moves from the edge into the picture following the background motion, as shown in the central image 60 of FIG.

Zhai, J., Yu, K., Li, J., & Li, S., 2005, A Low Complexity Motion Compensated Frame Interpolation Method, The 2005 IEEE International Symposium on Circuits and Systems (ISCAS2005), Kobe, Japan, May 23-26, 2005 Chen, Y.-K., Vetro, Y.-K., Sun H., & Kung S.-Y., 1998, Frame Rate Up-Conversion Using Transmitted True Motion ), To appear in Proc. Of 1998 Workshop on Multimedia Signal Processing, December 1998 Choi, BT, Lee, SH, & Ko, SJ, 2000, New frame rate up--conversion using bi-directional motion estimation, IEEE Trans. Consum. Electron ., Volume 46, Number 3, pages 603-609

  The present invention overcomes the above and other disadvantages of prior art configurations.

  A general object of the present invention is to provide boundary region identification in an image or frame.

  A particular object of the present invention is to provide frame interpolation / extrapolation that utilizes boundary region identification in determining image element property values.

  These and other objects are achieved by the present invention as defined by the appended claims.

  Briefly described, the present invention includes identifying a border region of an image or frame that includes rows and columns of image elements. The average value of the property values of the image elements in the row or column being tested is calculated. Further, the property value difference for each image element in a row or column is in the same column or row of adjacent rows or columns that are preferably closer to the center of the image than the image element property value and the column or row being tested. It is determined between the property values of adjacent image elements. The average difference value is calculated from the determined difference. The image elements in the current row or column are classified as belonging to the border region of the image or to the internal image region based on the average value and the average difference.

  In a preferred embodiment, the classification parameters are calculated based on the average value and the average difference, preferably as a weighted sum of the average value and the average difference. This parameter is compared to a threshold and classification is performed based on the comparison. In the case of a black border, the row or column being tested is classified as a border row or column if its classification parameter is below the threshold, otherwise it belongs to the inner region of the image.

  If the current row or column is classified as a boundary row or column, the adjacent row or column that is closer to the center of the image is tested until the row or column being examined is classified as an internal row or column. Is preferred. The procedure is preferably repeated for all edges of the image.

  The present invention further includes utilizing image element classification when determining image or frame property values to be interpolated or extrapolated from available images in a video sequence. In such a case, the group of image elements of the interpolated / extrapolated image is based on the property value of the first group of the first image of the video sequence and the property value of the second group of the second image of the video sequence. Is determined. However, when calculating the property values of the interpolation / extrapolation group, only the first group and second group of image elements that are not identified as boundary image elements are used. This prevents the boundary area from moving within the internal area of the constructed image.

  The present invention further includes a device for identifying a border region of the image and estimating a property value of the constructed image.

The invention will be best understood with further objects and advantages of the invention by reference to the following description taken in conjunction with the accompanying drawings.
It is a figure which shows the problem of the boundary line which appears in the interpolation and extrapolation image which concern on a prior art. It is a flowchart which shows the boundary area identification method which concerns on one Embodiment of this invention. It is a flowchart which shows one Embodiment of the classification | category step of the identification method of FIG. It is an enlarged view which shows a part of image related to a boundary area | region. 3 is a flowchart showing additional steps of the identification method of FIG. 3 is a flowchart showing additional steps of the identification method of FIG. 4 is a flowchart illustrating a method for determining a pixel value according to an embodiment of the present invention. 8 is a flowchart illustrating in more detail one embodiment of the providing and identifying steps of FIG. It is a figure which shows roughly the interpolation of the pixel value of a preceding image and a subsequent image. It is a flowchart which shows the additional step of the determination method of FIG. It is a schematic block diagram which shows the area | region discriminator which concerns on one Embodiment of this invention. It is a schematic block diagram which shows the group determination device which concerns on one Embodiment of this invention. FIG. 2 shows the diagram of FIG. 1 with the central diagram interpolated according to the present invention.

  In the figures, the same reference symbols are used for corresponding or similar elements.

  The present invention relates generally to image processing, and more particularly to a method and device for identifying border regions of an image, such as an image frame of a video sequence.

  As is well known in the prior art, many images and the like often include a boundary on one or more sides thereof. This phenomenon is shown in the image 50 on the left side of FIG. As can be seen, there is a black line along the right vertical side of the image. Such boundaries exist because of image recording processes or inherent features of the device. In addition, boundaries are added to the image during transcoding between different image formats, such as when going from a “normal” screen to a widescreen or from a widescreen to a “normal” screen.

  The inclusion of at least one boundary in the image 50 creates problems in the further processing of the image 50 or a frame / video sequence that includes the image 50. For example, during frame rate up conversion, the intermediate image 60 is generated based on the original images 40, 50 in the video sequence, increasing the frame rate. Such pixel values of the intermediate image 60 are determined from the corresponding pixel values of the adjacent images 40, 50, ie, interpolated or extrapolated. As shown in FIG. 1, since the boundary is included in at least one image 50 used in this pixel value determination, the boundary follows the movement of the background when moving between the images 50, 60, 40. May appear randomly in the interpolated / extrapolated image 60.

  Thus, the identification of border pixels and border regions of an image or frame is important, which provides significant advantages in connection with, for example, frame rate up conversion of a video sequence that includes at least one image or frame that includes a boundary.

  In the present invention, a video or frame sequence comprises a plurality, ie at least two frames or images. Also, such a frame is considered to be composed of a series of one or more slices, such a slice being composed of one or more macroblocks of image elements or pixels. In the present invention, the expression “image element” is used to indicate the smallest element of a frame or image in a sequence. Such image elements have associated image element properties such as color (red, green, blue, i.e. in RGB space) or luminance (Y) and chrominance (sometimes indicated by Cr, Cb or U, V). Have. A common example of an image element is a frame or picture pixel. The present invention is particularly adapted to video sequences that include a plurality of consecutive frames of a predetermined frame rate.

  The image elements are organized into groups of image elements. The expression “group of image elements” refers to any conventional well-known partition of frames and slices into a set of image elements that are processed together during decoding and encoding. In general, such groups are rectangular (M × N) or square (M × M) groups of image elements. An example of such a grouping is a macroblock in the video compression standard. Such macroblocks typically have a size of 16x16 image elements. A macroblock consists of a plurality of so-called sub-macroblock partitions such as 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, and 4 × 4 image elements. The 8 × 8 sub-macroblock partition is often indicated as a sub-macroblock or sub-block, and the 4 × 4 partition is often indicated as a block.

  An image having a boundary according to the present invention has boundary lines at all edges, and can basically constitute a frame around the image. Alternatively, the boundary line may exist at the opposite edges of the image when switching between the normal format and the wide screen format. In addition, the boundary line may exist along a single image edge as shown in FIG. Accordingly, the border region of an image according to the present invention includes one or more image element rows, one or more image element columns, or one or more image element rows and columns present at one or more edges of the image.

As is well known in the prior art, the border region or border is generally black or a very dark color. That is, it has zero luminance or low luminance. According to the present invention, a border region is any region along one or more edges of an image and having a relatively similar color or brightness value. In most practical applications, the border region is black (zero luminance) or at least a dark color (low luminance value). However, the teachings of the present invention is also used to identify regions of white other "color" of the (generally maximum brightness is 255 in the case of 2 ⁸ luminance levels) or bright colors (high luminance value) e.g. .

  FIG. 2 is a flowchart illustrating a method for identifying a border region of an image, such as a frame or video sequence image. The image includes a plurality of rows and columns of image elements. The method starts in step S1, which includes calculating an average value of property values of image elements in the first row or first column of the image. The property value of the image element used in the calculation of the average value is preferably a luminance value as described above, but the present invention is not limited to this. In the following, the invention will be described in more detail in connection with the identification of boundary regions along a row of image elements of an image. As a result, step S1 includes calculating an average value of the property values of the first column of the image. However, the same procedure can be applied additionally or alternatively to the row of image elements of the image. In such a case, the expression “column” in the following description is simply replaced with “row” and the expression “row” is replaced with “column”.

The first column is preferably the outermost column along the left or right edge of the image. If y _ij indicates the property value of the image element located in row i of column j, the average value AV _j for column j is calculated in step S1 as follows:

Where column j includes M image elements. The average value AV _j is preferably a representation of the average luminance value of the column j.

  The next step S2 determines each difference in property values for each image element in column j and the corresponding image element in adjacent column j + 1 or j-1. The difference between property values is calculated for each pair of image elements existing in the same row of column j and adjacent column j ± 1. Since the column j and the adjacent column j ± 1 include M image elements, the difference between the M property values is determined in step S2.

  The adjacent column j ± 1 is preferably an adjacent column located closer to the center of the image than the column j. For example, when the current column j is the first column when proceeding from the left side to the right side of the image, the adjacent column j + 1 is the second column. Similarly, when the current column j is the last column when proceeding from the left side to the right side of the image, the adjacent column j-1 is the second column from the last.

In step S3, the average difference value is calculated based on the M differences calculated in step S2. Therefore, the average difference AD _j for column j is calculated as follows:

This is the case when the next column j + 1 is located to the right of the current column j (towards the center of the image). Or:

This is the case when the next column j-1 is located to the left of the current column j (towards the center of the image). The average difference AD _j is a representation of the average amount of change in property values, preferably luminance values, between two adjacent columns.

Thereafter, step S4 uses the calculated average value AV _j and average difference AD _j from step S1 to classify the image elements in column j as belonging or not belonging to the border region of the image.

In step S4, if the current column j is classified as belonging to the boundary region of the image, the loop of steps S1 to S4 causes the adjacent column to investigate whether the image element of column j ± 1 also belongs to the boundary region. Preferably it is performed on j ± 1. This is schematically indicated by line L1. If this column j ± 1 is classified as a boundary region based on the average value AV _{j ± 1} and the average difference AD _{j ± 1} , the method will continue until the column to be tested is no longer classified as a boundary region column. Applies to the next column j ± 2 located closer to the center of the image than 1.

  In a preferred embodiment, the method defined by the operation steps S1-S4 is preferably applied to the first column or first row, which is the outermost column or row along one of the image edges. . If the column or row is classified as a border region column or row, the method continues by examining the next column or row closer to the center of the image than the previously tested and classified column or row. The method continues until a column or row is reached that is not classified as belonging to the boundary region based on the calculated average value and average difference. The method is preferably also applied to the outermost column or row of the opposite image edge. When image elements along those two image edges are tested, the method is preferably applied to the remaining two image edges. If the image element array that has already been tested is a column (or row), the remaining two image edges are rows (or columns). In other words, the boundary region identification method of the present invention disclosed in FIG. 2 is preferably applied to examine rows and columns along all four image edges. The particular order in which the image edges are tested, whether parallel or serial, is not important to the teachings of the present invention.

For example, if the current row i is tested, the average value AV _i of the property values of the image elements in row _i is calculated. The difference between the property values is determined for the image element in the current row i and the image element existing in the same column in the adjacent row i ± 1. Here, the adjacent row i ± 1 is closer to the center of the image than the current row i. Such a difference is determined for each image element in row i. The average difference AD _i is calculated based on those determined differences. The image element in the current row i is classified as belonging to the boundary region of the image or belonging to the inner region based on the calculated average value AV _i and average difference AD _i .

FIG. 3 is a flowchart showing a preferred embodiment of the classification step S4 of FIG. The method continues from step S3 of FIG. In the next step S10, the classification parameter CP _j is calculated based on the average value AV _j and the average difference AD _j for the current column j. The calculation of step S10 preferably includes calculating the parameters as a weighted sum of the average value and the average difference:
CP _j = αAV _j + βAD _j (4)
Where α and β are two non-zero weights, and in a preferred embodiment both are equivalent, ie α = β, more preferably α = β = 1. Combining Equation 1, Equation 2, and Equation 4 yields the following equation for the classification parameter:

By combining Equation 1, Equation 3 and Equation 4, the following corresponding equations are obtained:

In embodiments that utilize luminance as the preferred image element property, column j with image elements of low luminance, i.e. black or near black, has an average value AV _j that is relatively small, i.e. close to zero. Similarly, columns j with bright image elements or varying luminance values have a relatively large average value AV _j . Further, if the difference in luminance between the current column j and the adjacent column j ± 1 is small, that is, if the columns have the same or substantially the same luminance, the average difference AD _{j with} respect to the column _j is zero or zero. Close to. However, when the adjacent column j ± 1 has high luminance, the average difference AD _j is negative.

  Preferably, the classification parameter calculated in step S10 according to any of the above equations 4 to 6 is compared with the threshold T1 in step S11. The classification of the image elements in the current column j is performed based on this threshold comparison. In a preferred implementation, if the classification parameter is less than the threshold, column j is classified as a boundary region in step S12. In other cases, that is, when the classification parameter exceeds the threshold (or is equal to the threshold), the column j is classified as belonging to a non-boundary, that is, an internal region in step S13.

  Thus, an image having dark (low luminance) image elements and / or an average difference in luminance between column j and its adjacent column j ± 1 is negative, that is, adjacent column j ± 1 is relatively bright (high luminance). Column j with elements is classified as a boundary region depending on the actual value of the threshold. A column j having a bright column j (high luminance) and / or a relatively dark (low luminance) adjacent column j ± 1 is more likely to be classified as belonging to an internal region of the image.

  If column j is classified as a boundary column in step S12, continue to step S1 in FIG. 2 to classify the immediately adjacent column j + 1 or j-1 closer to the center of the image than the current column j by one step. Is preferred. This classification is preferably repeated for each column toward the center of the image until the column being tested is classified as a column in the inner region. The next column edge of the image is preferably tested in the same manner as the two edges of the row as described above. If the current column is classified as an internal column in step S13, the method ends when all image edges have been tested, or step S1 of FIG. 2 to test the other horizontal or vertical image edge. To continue.

  If the boundary region corresponds to a white line, i.e. a line with high brightness, the column (or row) is classified as a boundary column (or row), and if the calculated classification parameter is above a prescribed threshold, the column (or (Or row) is a column (or row) of the inner region.

  The threshold used in the comparison of step S11 is preferably determined through a test procedure, and the various input images are classified by the algorithm of the present invention. The threshold is usually adjusted or set by the operator to a specific value that appropriately classifies the columns and rows of the image as boundary or internal regions. Such a test procedure is performed and given a threshold of 24 as suitable candidates. However, the present invention is not limited to this specific value.

  In more complex embodiments, the threshold is adjusted or set based on the properties of the image elements of the current image. This procedure is shown in FIG. The method starts at step S20. In step S20, the average property value APV of the image elements is determined for the image:

Where the image includes M rows and N columns of image elements, each image element having an image element property value y _ij . The next step S21 determines or adjusts the threshold T1 based on the calculated average property value. Then, it continues to step S1 of FIG.

  This embodiment has the advantage of adjusting or setting the threshold based on specific property values of the image. This means that different thresholds are used for dark images with very low brightness image elements compared to brighter images. Here, the image element has a relatively large luminance value. In the previous example, the threshold is preferably small compared to the threshold used for brighter images.

In this embodiment, there are different sets of thresholds available that are adapted for different intervals of average property values. For example, if 0 ≦ APV <k ₁ , the first threshold is used, and if k ₁ ≦ APV <k ₂ , the larger second threshold is preferably used. Step S21 includes checking an interval that contains the current average property value and using a threshold associated with or assigned to that interval.

Another possible implementation is to have a default threshold, such as 24, and adjust that value based on the average property value determined as follows:
T1 ＝ DT−κ (DV−AVP) (8)
Or
T1 ＝ DT × κ × AVP / DV (9)
Where DT is the default threshold and κ is a non-zero weight that is one or other non-zero positive number. DV is the default value against which the calculated average property value of the image is compared in the threshold adjustment. In both Equation 8 and Equation 9, the darker image has a threshold that is adjusted smaller than the lighter image.

  FIG. 4 is a schematic diagram illustrating a portion of an image 40 having a border region 70 that is identifiable in accordance with the present invention. In the figure, five columns 10 to 16 of image elements 30 (four of the five are assigned reference numerals in the figure) are shown. The right column 10 is the edge column, that is, the rightmost column 10 of the image 40. As can be seen, the image elements 30 in column 10 are clearly part of the black line that forms the border region 70. This is evident because all image elements 30 in column 10 are black or very close to black. Further, the image elements in adjacent column 12 have a larger luminance value when compared to image elements in the same row. As a result, the classification parameter for column 10 is actually less than zero. The second column 12 from the outside does not have the same zero luminance as the previous matrix 10. However, the image element 30 in that column 12 is still classified as belonging to the border region 70 because it gives a negative average difference when compared to the next column 14. As a result, the classification parameter for column 12 is also less than the threshold.

  However, the next column 14 has brighter image elements 30. Furthermore, the pairwise difference in luminance values for column 14 and adjacent column 16 that is one step closer to the center of the image is sufficient to reduce the average luminance of column 14 such that the classification parameter of column 14 is less than the threshold. Not negative. As a result, this column 14 is classified as belonging to the internal image 75 of the image 40.

  Therefore, the present invention can identify and classify columns 12 (or rows) of image elements 30 as belonging to boundary region 70, although not all image elements 30 are black, using the average difference value of the classification parameters. Although this entire column 12 is not black, it is important that the column 12 be identified and labeled as part of the black line 70. If not so identified and labeled, it may cause artifacts in the intermediate image during frame rate up conversion because it is darker than the image content.

FIG. 6 shows additional steps of the identification method of FIG. The method continues from step S4 of FIG. 2 where a column (or row) has been identified as belonging to the border region of the image. The next step S30 calculates a first difference FD between the property values of the current image element in column j and the adjacent image element in the same column j:
FD = y _ij −y _{(i ± 1) j} (10)
Thus, if an image element is in row i of column j, an adjacent image element is in row i + 1 or i-1 of column j. In step S31, the second difference SD is calculated between the property value of the image element and the average value AV _j of the property values of the column j:

  If the first difference is different from the second difference by more than the maximum threshold, the current image element is reclassified as belonging to the internal region instead of the boundary region. For example, this comparison is performed as shown in step S32. In step S32, the absolute value of the difference between the first difference and the second difference is compared with the maximum threshold T2. If the absolute value is below the threshold value, the image element is still considered a border image element and the method ends. However, if the absolute value exceeds the threshold, then continue to step 33 and the image element is reclassified as belonging to the internal image area. A further possible comparison is to calculate the quotient of the first difference and the second difference or the absolute value of the first difference and the absolute value of the second difference. The quotient is then compared to a threshold value.

  In another embodiment of step S33, not only the examined image element, but the entire column is reclassified as belonging to the internal image region. This is because if the column previously classified as the boundary region column includes at least one image element to be reclassified in step S33, or at least the minimum multiple image element, the entire column is reclassified as the inner region column. Means that. This prevents intermittent columns where some of the image elements in the column are classified as boundary regions and other image elements are considered as part of the internal image region.

  This optional preferred reclassification embodiment is used to distinguish local changes associated with image edges. Thus, the inner area of the image close to the image edge may be relatively dark, such as showing the night sky, but a portion of this image contains very bright image elements such as image elements corresponding to dark sky stars or the moon. . In such a case, the high brightness of one or more image elements in the column corresponding to a star or moon can be used to obtain a classification parameter above the threshold and classify the column as an internal region, especially if threshold adaptation is not used. It may not be enough. An additional check of the method described above and disclosed in FIG. 6 is utilized to correct such erroneous classifications that occur in connection with “difficult” dark images.

The method described above and presented in FIG. 6 is preferably repeated for all image elements in the current column j. In a preferred embodiment, the method starts with the first y _0j (or last y _{(M−1) j} ) image element in column j, which is the next image element in column j in step S30. Compared with y _1j (or y _{(M-2) j} ). As with the entire column j being reclassified, once the image elements that require reclassification according to FIG. 6 have been identified, the investigation of the image elements in the current column j ends as necessary. Further, the operational steps shown in FIG. 6 are preferably repeated for all columns and / or rows initially classified as belonging to the boundary region.

  In particular, the classification of the present invention has the advantages associated with frame rate up conversion to prevent border regions or black lines from appearing in interpolated or extrapolated images or frames.

  FIG. 7 is a flowchart illustrating a method for estimating a property value of a group of at least one image element of an image or frame associated with a time of a video sequence. This estimation is preferably performed as part of a frame rate up-conversion procedure for adding one or more frames to the video sequence by frame interpolation or extrapolation.

The method starts with an optional step S40 and at least two images of the video sequence used for estimation are identified. In one general embodiment, one image corresponds to the preceding time for the interpolated intermediate image and the other image corresponds to the subsequent time in the sequence. In a preferred embodiment, the two images are images located immediately before and immediately after the intermediate image with respect to time. In other words, the image can be considered as an adjacent image. In this image interpolation, three or more images are actually used. For example, in order to interpolate an image at time t _k, the time _{t k-1, t k-} 3,. . . , T _{k + 1-2P} corresponding to P preceding images and times t _{k + 1} , t _{k + 3 ,.} . . , T subsequent images corresponding to t _{k-1 + 2Q} are used.

Similarly, when extrapolating the image or frame at time t _k, the time _{t k-1, t k-} 3,. . . , T _{k + 1-2P} two or more preceding images or times t _{k + 1} , t _{k + 3 ,.} . . , T _{k-1 + 2Q} , two or more subsequent images are used.

  A next step S41 identifies boundary image elements of at least a first image of at least two images used as reference images in interpolation or extrapolation. This step S41 is performed as described above in this specification, and any image element related to zero or one or more image edges of the two images that form part of the black line or other border region Identify.

  A next step S42 provides a first group of a plurality of image elements of the classified first image. The first group includes at least one image element identified as belonging to the boundary area in step S41, and at least one image element identified as not belonging to the boundary area of the first image in step S41. including. For example, if an image element group includes 4 × 4 image elements, one, two or three column and / or row image elements are classified as boundary image elements, and the remaining image elements are internal image elements. .

  Step S43 identifies a second group of at least one image element of a second (previous or subsequent) image associated with a preceding or subsequent time of the video sequence. The second group is preferably identified based on a displacement vector associated with the first group. Thus, the displacement vector points in the direction of the second group when applied from the corresponding position that the first group had in the second image.

  Finally, the property values of the group to be interpolated / extrapolated are based on the property values of the first group of image elements and the second group of property values identified as not belonging to the boundary region of the first image. Identified. If the boundary identification of step S41 is also applied to the second image, step S44 is an image identified as not belonging to the boundary area of the first image to determine the group to be interpolated / extrapolated. It only involves utilizing the image element identified as not belonging to the border region of the element and the second image.

  Accordingly, the classification and identification of the image elements of the present invention is utilized to exclude image elements that are identified as forming part of the boundary region from the interpolation / extrapolation of the image elements. On the other hand, when calculating the property value for the image element of the group to be determined, only the image element that is not identified as belonging to the boundary region can be used as the reference image element. This exclusion of border image elements reduces the likelihood that border regions (black lines) will appear in the interpolated / extrapolated image and greatly reduces the occurrence of such artifacts.

  FIG. 8 shows in more detail a possible implementation of the providing and identifying steps of FIG. The method continues from step S41 of FIG. The next step S50 provides a plurality of first images, ie a set of at least two candidate groups. Each of these candidate groups includes at least one image element, typically a plurality of image elements, and is associated with each displacement representation or vector. These displacement vectors are taken from the inter-coding of the frame, ie the video codec, or determined from a motion estimation search.

  Step S51 identifies each reference group of at least one image element of the second image for each candidate group as schematically indicated by line L2. The reference group associated with the candidate group is preferably identified based on a displacement vector associated with the candidate group. Therefore, the displacement vector indicates the direction of the reference group when applied from the corresponding position that the candidate group had in the second image.

  The next step S52 calculates a difference criterion representing the difference between the property values of the image elements of the candidate group and its identified reference group for each candidate group as schematically indicated by the line L2. In a preferred embodiment, the criterion is calculated based on the absolute value of the property value difference for image elements occupying corresponding positions in the candidate group and the reference group. Suitable examples of such difference criteria include absolute value difference sum (SAD) and square difference sum (SSD) as known in the prior art.

  A next step S53 selects a candidate group from the set of groups provided based on the calculated difference criterion. That is, it is preferable to select a candidate group that results in a minimum SAD or SSD criterion. Continuing to step S44 of FIG. 7 which determines the property value of the group of the interpolated / extrapolated frame based on the selected candidate group and the property value of the reference group associated with the selected candidate group. In a preferred embodiment, the image element property value is determined as a linear combination of the selected candidate group and the associated reference group property values. Arbitrary weights applied to the property values of the selected group and arbitrary weights of the associated group property values are respectively the time difference and interpolation between the interpolated / extrapolated frame and the first frame containing the selected candidate group. / It is preferable to determine based on the time difference between the extrapolated frame and the second frame. In other words, greater weight is used when the passage of time is short compared to the passage of longer time. Further, the frame weight value is used to reflect acceleration as is well known in the prior art. Note that only image elements that are not classified as boundary image elements according to the present invention of candidate groups and / or reference groups are used in the determination in step S44.

  In the embodiments described above, the reference group associated with the candidate group is identified or estimated relative to the candidate group based on the assigned displacement vector. This assumes that the same vector is used to move from the candidate group to the group to be interpolated / extrapolated, as when proceeding from the determined group to the reference group. However, the present invention is not limited to this.

  In another embodiment, a first set of a plurality of first candidate groups in the first frame is provided, and a second set of the plurality of second candidate groups present in the second frame is Provided. Each of these second candidate groups includes at least one image element, and preferably each has a displacement vector. A difference criterion is calculated for each pair of one first candidate group of the first set and one second candidate group of the second set. Alternatively, not all combinations of the first candidate and the second candidate are tested, only a limited portion of them is tested, and at the same frame position of the first frame and the second frame. Reflects reasonable combinations of candidate groups, such as groups with associated displacement vectors that identify existing groups and other candidate groups in other frames.

  Thereafter, the first candidate group and the second candidate group are selected based on a difference criterion. The image element property values of the groups to be determined are calculated based on the property values of those selected candidate groups as described above, but the boundary image elements of the two candidate groups are excluded.

  FIG. 9 shows a portion of video sequence 1 having a first frame 40, a second frame 50 and an intermediate frame 60 determined during frame rate up conversion. The group 62 of image elements to be determined is shown in the intermediate frame 60. A suitable first candidate group 42, 44, 46 is shown in the first frame 40 and a corresponding second candidate group 52, 54, 56 is shown in the second frame 50. Generally, these candidate groups include groups 42 and 52 that have positions in the first frame 40 and the second frame 50 that correspond to the positions that the group 62 has in the intermediate frame 60. The displacement vectors 41, 51 of those groups 42, 52 are shown and pass through the group 62 to be determined. Other candidates include adjacent groups 44, 54 of candidate groups 42, 52 having corresponding positions. Also, the groups 46 and 56 having the associated displacement vectors 43 and 53 passing through the group 62 of the intermediate frame 60 are suitable candidate groups according to the present invention.

  FIG. 10 is a flowchart illustrating additional steps of the estimation method of FIG. The method continues from step S41 of FIG. In a next step S60, the relative size of the boundary area identified in the first image is compared with the size of the boundary area identified in the second image. Furthermore, these two border regions correspond to the same image edge of the first image and the second image, ie the upper horizontal edge, the lower horizontal edge, the left vertical edge or the right vertical edge. Step S60 includes calculating a size difference SD between the two boundary regions. The difference in size can represent the difference in the number of columns (or the number of rows) between the two boundary regions. Alternatively, the size difference is a representation of the difference in the number of image elements between two corresponding boundary regions.

  If the size difference SD between the two boundary regions is equal to or smaller than the threshold T3 in the comparison in step S61, the process continues to step S42 in FIG. However, if the size difference exceeds the threshold value in step S61, the process continues to step S62, and the re-classification of the image elements of the second image (or first image) is the first image (or second image). This is performed based on the size of the boundary area identified in. In such a case, the second image having a position corresponding to the image element of the first image (or second image) identified as belonging to the boundary region of the first image (or second image). The image element of (or the first image) is classified as belonging to the boundary region of the second image (or the first image). Then, it continues to step S42.

  Therefore, this check in steps S60 to S62 investigates whether the size of the corresponding image edge boundary region is greatly increased or decreased when moving from one image to the next in the video sequence. For example, the size of the boundary region may be two rows along one edge of the first image. However, the next image in the video sequence has a border region that is classified as containing 7 columns along the same image edge. In such a case, a part of the internal image element is assumed to be a boundary image element, and thus may be classified incorrectly. This is particularly the case when threshold adjustment is not performed, especially when the image is very dark, such as showing the night sky, and therefore there is little difference in luminance values between the internal image elements and the boundary image elements. By comparing the size of the identified border region with the corresponding size of the previously classified image or the previously classified images, such incorrect classification is identified according to the procedure described above and shown in FIG. And corrected. In the above example, this means that the size of the boundary region of the second image is reset so as to have a size corresponding to the first image, that is, two columns. The five “misclassified” columns of image elements are reclassified as internal image elements rather than border image elements in step S62. This means that those five “misclassified” columns of image elements can now be used as reference image elements in interpolation / extrapolation because they are considered internal image elements at this point.

  FIG. 13 shows the result of frame rate up conversion using image classification according to the present invention. As can be seen from the interpolated image 60, the boundary present at the right edge of the left image 50 does not appear in the internal region of the interpolated image 60. In sharp contrast, by labeling the image elements in the outermost right column as the border image elements of the present invention, those image elements are removed from the interpolation and the middle image of FIG. 1 utilizing the prior art. Are prevented from appearing in the image 60 of FIG. In sharp contrast, dark border images are processed separately by simply assigning property values (brightness) of the border image elements to image elements occupying corresponding positions in the interpolated image 60, as is done in the figure. Is done. More specifically, a boundary having a width equal to the maximum value of the boundary between the two reference frames used for interpolation / extrapolation is created. An average (in the case of interpolation) image element value of the reference image is filled in the created image element of the boundary region. That is, motion compensation is not necessary. The image element at the position (i, j) at the boundary of the interpolated image is simply determined as the average value of the property values of the two image elements occupying the same position in the two (previous and subsequent) reference frames.

  The quality of the constructed frame or image in the frame rate up-conversion scheme is improved by applying the present invention. This means that the area close to the image boundary is presented more accurately. Furthermore, the present invention prevents the (black) lines from moving into the constructed image, and an overall higher quality is experienced.

  FIG. 11 is a schematic block diagram showing a region discriminator 100 according to an embodiment of the present invention. The area identifier 100 includes a calculator 110 that calculates an average value of property values of image elements in the current row or column (row / column). The difference determiner 120 causes the discriminator 100 to determine the difference between the property values of the corresponding image element existing in the same column / row of the image element and the adjacent row / column of the image for each row / column image element. Composed. Calculator 130 uses the determined difference value to calculate the average difference for the current row / column.

  Region identifier 100 is a classification configured to classify the current row / column image element as belonging to a boundary region or an internal image region based on the average value and average difference from the two calculators 110, 130. The apparatus 140 further includes a container 140.

  In a preferred embodiment, the discriminator 100 includes a parameter calculator 160 that calculates the classification parameters based on the average value and the average difference, ie, the sum or weighted sum of the average value and the average difference, as disclosed above. A comparator 170 is provided to the discriminator 100 for comparing thresholds such as those retrieved from the associative memory 150 with the classification parameters. In the present embodiment, the classifier 140 is configured to classify the current row / column image element as belonging to the border region or not, based on the comparison. In a preferred implementation applicable in connection with black lines / boundaries, if the classification parameter is below the threshold, the current row / column is classified as belonging to the boundary region of the image.

  One optional embodiment of the region identifier 100 includes a threshold determiner or adjuster 180. In such a case, the average value calculator 110 is further configured to calculate an average value of the property values of the image or at least most of the image. The determiner 180 uses the average property value to determine, select or adjust the threshold used by the comparator 170 along with the classification parameter.

  In another optional embodiment, the difference determiner 120 may include a first difference between property values of a current image element of a row / column classified as a boundary row / column and an adjacent image element of the same row / column. Is configured to determine. The determiner 120 further calculates a second difference between the property value of the current image element and the average value calculated by the average value calculator 110. In this embodiment, if the first difference differs from the second difference by more than the maximum threshold as retrieved from the memory 150, the classifier 140 may determine whether the current image element or actually the boundary row / column to which the image element belongs. Are reclassified as non-boundary image elements or non-boundary rows / columns. This procedure is performed for each image element in the current row / column, unless reclassification of the entire row / column is performed by the classifier 140 when an “incorrectly classified” image element is detected. Furthermore, the procedure is preferably applied to all rows / columns of the image classified as boundary rows / columns.

  The region identifier 100 examines all the outermost columns and rows of the image, and if the outer row or column is classified as a boundary row or column according to the present invention, it further continues to investigate the inner rows and columns. It is preferable to do this.

  The unit of the area identifier 100 can be provided in hardware, software, and / or a combination of hardware and software. The unit can be implemented in a video or image processing terminal or server as implemented at or connected to a node of a wired or wireless communication system. Alternatively, the unit of region identifier 100 can be configured in a user terminal, such as a TV decoder, computer, mobile phone, or other user equipment that has or is connected to an image rendering device.

FIG. 12 is a schematic block diagram illustrating a device 200 that determines a group by estimating a property value of at least one image element of the group of image elements. Device 200 optionally includes a frame or image identifier 210 that identifies at least a first image and a second image of the video sequence. These two images are associated with different times in the sequence compared to the current image containing the group to be determined. In the case of group interpolation, the first image is a previous or subsequent image, and the second image is a subsequent or previous image. In the case of group extrapolation, both the first image and the second image are preceding or succeeding images relative to the current image.

  The determination device 200 includes the region identifier 100 according to the present invention described above and shown in FIG. The region identifier 100 is used to identify the first image and preferably the image elements of the second image that are classified as belonging to each border region of the image.

  Group provider 220 is configured in device 200 to provide a first group of a plurality of image elements of a first image. This group includes at least one image element classified as a boundary image element by the classifier 100 and at least one image element classified as a non-boundary image element by the classifier 100. The device 200 further includes a group identifier 230 that identifies at least one image element of the second image, preferably a second group of a plurality of image elements. The value determiner 260 determines the property value of the group determined based on the property value of the image element that is not classified as the boundary image element in the two groups from the provider 220 and the identifier 230.

  The value determiner 260 preferably further determines the property values of the image elements present in the border region separately from interpolation / extrapolation using motion compensation. In a preferred embodiment, the size comparator 270 of the device 200 compares the relative boundary sizes of the two reference images used for image interpolation / extrapolation for each image edge. The edge boundary of the constructed image is selected as the maximum boundary of the edge of the reference image. The property value of the boundary image element is calculated based on the property value of the image element that occupies the same position in the reference image. The image element at the boundary position (i, j) obtains a property value equal to the average value of the property values of the image elements occupying the same position (i, j) in the reference frame.

  In an optional embodiment, the set provider 250 is configured in the device 200 to provide a set of candidate groups for the first image. Each candidate group includes at least one image element and has a respective displacement vector. The group identifier identifies each second group of the second image for each candidate group. This identification is preferably performed based on a displacement vector associated with the candidate group. The reference calculator 240 calculates a difference reference for each candidate group. Here, the reference represents the difference between the property values of the candidate group and the identified second group. The standard is preferably an SAD or SSD standard. That is, it is preferably based on the absolute value of the difference between image elements.

  In this embodiment, the group provider 220 selects a candidate group from the set provided based on the difference criterion by the calculator 240, i.e., preferably selects a candidate group having a minimum (SAD or SSD) difference criterion. Configured to do. The value determiner 260 is based on the selected candidate group and the associated second group property values, typically based on a linear combination of the image element values of the groups excluding the boundary image elements in the two groups. Determine the property value of the current group.

  An optional size comparator 270 is preferably provided to the decision device 200 to compare the relative sizes of the boundary area of the first image and the boundary area associated with the same image edge of the second image. The classifier 280 uses the size comparison to reclassify the image elements of the second image previously classified as boundary image elements into internal image elements when the size difference exceeds a threshold. The reclassified image element occupies an image position that is classified as an internal region of the first image. However, they have been classified by the area discriminator 100 as boundary areas of the second image.

  The unit of the determination device 200 can be provided in hardware, software, and / or a combination of hardware and software. The unit can be implemented in a video or frame processing terminal or server as implemented at or connected to a node of a wired or wireless communication system. Alternatively, the unit of decision device 200 can be configured in a user terminal such as a TV decoder, computer, mobile phone, or other user equipment having or connected to an image rendering device.

  It will be appreciated by those skilled in the art that various modifications and variations can be made to the present invention without departing from the scope of the invention as defined by the appended claims.

Claims (20)

A method for identifying a border region of an image edge comprising rows and columns of image elements, comprising:
Calculating an average value of property values of image elements in the image row or column;
Determining, for each image element in the row or column, a difference in property values between the image element and an image element existing in the same column or row of adjacent rows or columns of the image;
-Calculating an average difference that is an average of the differences in a row or column direction ;
-Calculating a classification parameter as a weighted sum of the average value and the average difference;
-Comparing said classification parameter with a threshold;
Categorizing the row or column image elements as belonging to the boundary region if the classification parameter is below the threshold . Determining an average property value of the image elements of the image;
- The method according to claim 1, further comprising a determining the threshold value based at least in part on the average value of said property values of said image elements of the image. 3. The method of claim 1 or 2 , wherein the adjacent row or column is an adjacent row or column located closer to the center of the image compared to the row or column. When the row or column is classified as belonging to the boundary region, for the adjacent row or column, calculating the average value, determining, calculating the average difference, and the classification The method according to any one of claims 1 to 3 , wherein the step of performing is repeated. Calculating a first difference between the image element in the row or column and the property value of the image element present in the same row or column of an adjacent column or adjacent row ;
Calculating a second difference between the property value of the image element and the average value;
- any of claims 1 to 4, characterized by further comprising a step of classifying as the first difference does not belong to the image element to the boundary area if different the second difference and the maximum threshold or The method according to claim 1. The classifying step classifies the image element in the row or column as not belonging to the boundary region when the first difference differs from the second difference by a maximum threshold or more. Item 6. The method according to Item 5 . The method according to any one of claims 1 6, characterized in that the property values of the image elements is a luminance value. A method for estimating a property value of a group of at least one image element in an interpolated intermediate image of a video sequence, comprising:
Identifying an image element belonging to a border region of an edge of a first image associated with a first different time of the video sequence according to any one of claims 1 to 7 ;
-At least one image element identified as belonging to the boundary region of the first image; and at least one image element identified as not belonging to the boundary region of the first image. Providing a group of said first image;
Identifying a second group of at least one image element in a second image associated with a second different time of the video sequence;
The interpolated intermediate image based on the property values of the image elements of the first group and the property values of the second group identified as not belonging to the boundary region of the first image; Determining the property value of the group of image elements in the method. 9. The method of claim 8 , wherein the step of identifying the second group comprises identifying the second group of the second image based on a displacement vector associated with the first group. . Further comprising identifying image elements belonging to a border region of an edge of the second image,
In the determining step, the property value of the image element of the first group identified as not belonging to the boundary region of the first image, and not belonging to the boundary region of the second group 10. The method of claim 8 or 9 , wherein the property value of the group is determined based on a property value of the second group of image elements identified as being. Calculating a difference in size between the boundary region of the first image and the boundary region of the second image;
An image element of the second image having a position corresponding to an image element of the first image identified as belonging to the boundary region of the first image when the size difference exceeds a threshold value ; 11. The method of claim 10 , further comprising the step of classifying the second image as belonging to the boundary region. A region identifier that identifies a border region of an image edge including rows and columns of image elements,
An average value calculating means for calculating an average value of property values of image elements in the row or column of the image;
A difference determination means for determining a difference between property values of image elements existing in the same column or row of the image element and an adjacent row or column of the image for each image element of the row or column;
An average difference calculating means for calculating an average difference that is an average of the differences in the row or column direction ;
-A parameter calculation means for calculating a classification parameter as a weighted sum of the average value and the average difference;
-A comparison means for comparing said classification parameter with a threshold;
A region classifier comprising: classifying means for classifying the row or column image elements as belonging to the boundary region when the classification parameter is below the threshold ; The average value calculating means is configured to calculate an average value of property values of the image elements of the image,
13. The region identification according to claim 12 , wherein the region identifier comprises threshold determination means for determining the threshold based at least in part on the average value of the property values of the image elements of the image. vessel. The average value calculating means is configured to calculate a corresponding average value for the adjacent row or column when the row or column is classified as belonging to the boundary region,
The difference determination means is configured to calculate a corresponding difference in property values for the adjacent row or column when the row or column is classified as belonging to the boundary region;
The average difference calculating means is configured to calculate a corresponding average difference for the adjacent row or column when the row or column is classified as belonging to the boundary region;
It said classification means, according to claim 12 or 13, wherein the row or column is configured to classify the image elements of the adjacent row or column when they are classified as belonging to the border region Area discriminator. The difference determining means determines i) a first difference between the image element of the row or column and a property value of an image element existing in the same row or column of an adjacent column or adjacent row , and ii) the image Configured to determine a second difference between the property value of the element and the average value;
The classification means is configured to reclassify the image element as not belonging to the boundary region when the first difference is different from the second difference by a maximum threshold or more. The area discriminator according to any one of 12 to 14 . The classifying means is configured to reclassify the image element in the row or column as not belonging to the boundary region when the first difference differs from the second difference by a maximum threshold or more. The area identifier according to claim 15, wherein A device for estimating a property value of a group of at least one image element of an intermediate image to be interpolated of a video sequence,
17. A region identifier as claimed in any one of claims 12 to 16 for identifying an image element belonging to a border region of an edge of a first image associated with a first different time of the video sequence;
-At least one image element identified as belonging to the boundary region of the first image; and at least one image element identified as not belonging to the boundary region of the first image. Group providing means for providing one group to the first image;
A group identification means for identifying a second group of at least one image element of a second image associated with a second different time of the video sequence;
- based on the property value of the first of the image element property values and the second group of the identified first group as being not belonging to the border area of the image, the intermediate image is the interpolation And a value determining means for determining the property value of the group of image elements . 18. The device of claim 17 , wherein the group identification means is configured to identify the second group of the second image based on a displacement vector associated with the first group. The region identifier is configured to identify an image element belonging to a boundary region of an edge of the second image;
The value determination means includes a property value of the image element of the first group identified as not belonging to the boundary area of the first image, and a value not belonging to the boundary area of the second group 19. A device according to claim 17 or 18 , wherein the device is configured to determine the property value of the group based on a property value of the image element of the second group identified as. -A size comparison means for calculating a difference in size between the boundary region of the first image and the boundary region of the second image;
An image element of the second image having a position corresponding to an image element of the first image identified as belonging to the boundary region of the first image when the size difference exceeds a threshold value; 20. The device according to claim 19 , further comprising classification means for classifying the second image as belonging to the boundary region.