JP2011517226A - System and method for enhancing the sharpness of an object in a digital picture - Google Patents

Abstract

The sharpness of an object in a digital picture is enhanced by comparing the input picture of the digital picture with stored information that represents the nature and characteristics of the object to identify the object and generate object positioning information that indicates its position. It is done. The sharpness of the object and the area where the object is located is enhanced by image processing, and the input video with such enhanced sharpness is encoded.

Description

  The present invention relates generally to the transmission of digital pictures, and in particular, the sharpness of interest of interest in digital pictures, in particular digital pictures displayed in units with low resolution and low bit rate video coding. Visibility).

  For example, there is an increasing demand for distributing video content to portable devices such as mobile phones and PDAs. Due to the small screen size, limited bandwidth, and limited decoder end processing power, the video is encoded at a low bit rate and low resolution.

  One of the main problems with low resolution, low bit rate encoding is the degradation or loss of objects important to perceived video quality. For example, it is cumbersome to watch a video clip of a soccer or tennis match when the ball is not clearly visible.

  Therefore, it is preferable to highlight the object of interest so as to improve the subjective display quality of the low resolution / low bit rate video.

  In various implementations of the present invention, the sharpness of an object of interest in a digital image is increased given the approximate location and size of the object in the image. Alternatively, the sharpness of the object is increased after refinement of the approximate position and size of the object. The subject enhancement provides at least two advantages. First, subject enhancement makes the subject easier to see and follow, thereby improving the user experience. Second, the enhancement of the object causes little degradation of the object during the encoding (ie, compression) phase. One of the main applications of the present invention is the distribution of video to portable devices such as mobile phones and PDAs, but the features, concepts and implementations of the present invention also include, for example, video over Internet protocols ( It is also useful for a variety of other applications, situations and environments, including low bit rate standard definition content).

  The present invention highlights objects of interest in the video to improve the subjective display quality of the low resolution, low bit rate video. The system and method of the present invention can handle objects of different characteristics and can operate in fully automatic mode, semi-automatic mode (ie with manual assistance) and fully manual mode. The subject enhancement may be performed in the pre-processing stage (ie, before or at the video encoding stage) or at the post-processing stage (ie, after video decoding).

  According to the present invention, the definition of a target in a digital picture provides an input video including the target, stores information representing the properties and characteristics of the target, and depends on the input video and information representing the characteristics and features of the target And by generating object positioning information that identifies the object and indicates the position of the object. An enhanced video of the portion of the input video including the target and the region where the target is located is generated from the input video according to the target positioning information, and the generated enhanced video is encoded.

1 is a block diagram according to a preferred embodiment of a system for enhancing the sharpness of an object in a digital video constructed according to the present invention. 2 represents the approximate location of an object provided by the system of FIG. A to D represent the workflow in the subject enhancement according to the present invention. 4 is a flowchart according to an object boundary estimation algorithm used to refine object identification information and object position information according to the present invention. Fig. 4 represents an implementation according to the concept of level setting estimation of an arbitrarily shaped object boundary according to the invention. Fig. 4 represents an implementation according to the concept of level setting estimation of an arbitrarily shaped object boundary according to the invention. Fig. 4 represents an implementation according to the concept of level setting estimation of an arbitrarily shaped object boundary according to the invention. Fig. 4 represents an implementation according to the concept of level setting estimation of an arbitrarily shaped object boundary according to the invention. 6 is a flowchart according to an object enlargement algorithm according to the present invention. A through C represent three possible subdivisions of a 16 × 16 macroblock useful for explaining the refinement of the object identification information and object position information during the encoding stage.

  Referring to FIG. 1, an object enhancing system configured in accordance with the present invention may span all components in transmitter 10, or the target enhancement component may be in receiver 20. There are three stages in the processing chain where the highlighting of the object takes place. (1) pre-processing in which the object is complemented by the transmitter 10 before the encoding (ie compression) stage; (2) the region of interest containing the object is transmitted to the transmitter 10 by refinement of information about the object and its position. (3) After the target is complemented at the receiver 20 after decoding using the side information about the target and its location sent from the transmitter 10 by the bitstream as metadata It is processing. An object enhancement system constructed in accordance with the present invention provides object highlighting in only one of the above stages, or in two of the above stages, or in all three of the above stages. May be arranged to do.

  The system of FIG. 1 for enhancing the sharpness of an object in a digital picture has means for providing an input video that includes the object of interest. The source of the digital picture that includes the object to be enhanced in sharpness may be a television camera of conventional construction and operation and is represented by arrow 12.

  The system of FIG. 1 further stores information (eg, a target template) that represents the properties and characteristics of the object of interest, and identifies the objects according to the input video and information that represents the characteristics and features of the target. And means for generating target positioning information indicating the position. Such means is shown in FIG. 1 as the object positioning module 14 and scans the input video frame by frame, and the object in the picture has the same properties and characteristics as the stored information representing the properties and characteristics of the object of interest. (I.e., what the subject is) and means for indicating the location of the subject (i.e., where the subject is). The object positioning module 14 may be a unit of conventional configuration and operation, scanning the digital picture of the input video every frame, and scanning the sector of the digital picture of the scanned input video to characterize and characterize the object of interest. A grid of digital pictures that identifies an object of interest when the information generated from a scan of a particular sector is the same as the stored information that represents the nature and characteristics of the object as compared to the stored information that represents The position is indicated by coordinates.

  In general, the object positioning module 14 performs one or more of the following methods in identifying and positioning an object of interest.

  Target tracking: The purpose of the target tracker is to locate a moving object in the video. Normally, the tracker estimates target parameters (eg, position, size) in the current frame in view of the moving target's history obtained from the previous frame. The tracking approach may be based on, for example, template matching, optical flow, Kalman filter, average shift analysis, hidden Markov model, and special filters.

  Object detection: The purpose in object detection is to detect the presence and position of an object in an image or video frame based on preliminary knowledge about the object. The object detection method generally uses a combination of a top-down approach and a bottom-up approach. In a top-down approach, the object detection method is based on rules derived from human knowledge of the detected object. In a bottom-up approach, the object detection method associates the object with low-level structural features or patterns and locates the object by looking for these features or patterns.

  Object segmentation. With this approach, an image or video is broken down into the "objects" that make it up. These “objects” may have semantic entities or visual structures such as color patches, for example. Such decomposition is generally based on object motion, color and texture attributes. Target segmentation may have several applications including compact video coding, automatic and semi-automatic content-based description, film post-production, and scene description. Specifically, segmentation simplifies the problem of object positioning by providing an object-based description of the scene.

  FIG. 2 represents the approximate object position provided by the object positioning module 14. For example, the user draws an ellipse around the area where the object is located to roughly indicate the position of the object. Finally, the approximate target positioning information (i.e., ellipse center point, long axis and short axis parameters) is refined.

  Ideally, the target positioning module 14 operates in a fully automatic mode. In particular, however, some manual assistance is required to correct errors caused by the system or minimally so as to define important objects by the system where the position is to be found. Non-target area enhancements can confuse viewers and miss actual movements. To eliminate or minimize this problem, the user can draw an ellipse around the object, as described above, and then the system can track the object from the identified location. If the object is successfully positioned in the frame, the object positioning module 14 outputs the corresponding ellipse parameters (ie, center point, major axis, and minor axis). Ideally, the contour of the ellipse serving as the boundary coincides with the contour of the object.

  However, if the parameter is just an approximation and the object enhancement is applied without the resulting ellipse firmly containing the object, two problems can arise. First, because an ellipse does not contain the entire object, the object cannot receive enhancement as a whole. Second, non-target areas may be enhanced. Because these results are undesirable, it is useful to refine the target area before enhancement under such circumstances. The refinement of the object positioning information is discussed in more detail below.

  The system of FIG. 1 further includes means for generating an enhanced image of a portion of the digital picture that includes the object of interest and the region in which the object is located in response to the input image and the object positioning information received from the object positioning module 14. Have Such means is shown in FIG. 1 as the object enhancement module 16 and may be a unit of conventional construction and operation, and the conventional image processing operations are applied to the sharpness of the area of the digital picture containing the object of interest in that area. Enhance by applying. The object positioning information received from the object positioning module 14 for each frame includes grid coordinates of a predetermined size area where the object of interest is located. In addition, as described above, object enhancement helps to reduce object degradation during the encoding stage (described below) that follows the enhancement stage. The operation of the system of FIG. 1 so far corresponds to the above-described preprocessing operation mode.

  When performing object enhancement, object sharpness is improved by applying image processing operations in the region where the object of interest is located. These operations are performed inside the object (eg texture enhancement) and possibly even outside the object (eg contrast enhancement, blurring outside the object region) along the object boundary (eg edge sharpening). ) May be applied. For example, one way to attract the most attention to an object is to sharpen the edges along the object's contour within the object. This makes the subject's details clearer and makes the subject stand out from the background. Furthermore, sharper edges tend to get through encoding better. Other possibilities include, for example, by applying smoothing, sharpening and object refinement operations interactively (the order of application need not necessarily be in the order listed here). To enlarge.

  3A to 3D show a workflow in the target enhancement process. FIG. 3A is a single frame in a soccer video where the focused image is a soccer ball. FIG. 3B shows the output of the target positioning module 14, that is, the target positioning information of the soccer ball in the frame. FIG. 3C represents a region refinement step, which will be discussed in more detail below, in which the approximate object location information of FIG. 3B provides a more accurate estimate of the object boundary, ie a bright colored line surrounding the ball Is refined to produce FIG. 3D shows the result after applying the target enhancement, in this example edge sharpening. It should be noted that the soccer ball is sharpened in FIG. 3D and is sharper than in the original frame of FIG. 3A. The subject also has a higher contrast. Contrast generally refers to making dark colors darker and light colors lighter.

  Including targeted enhancements in the system of FIG. 1 provides significant advantages. Problems associated with incomplete tracking and distortion enhancement are eliminated. Incomplete tracking can make object positioning difficult. For each frame, the position of the object is slightly off, and each frame may be off slightly differently. This can cause flicker, for example, due to background fragments being enhanced in various frames and / or different portions of the subject being enhanced in various frames. Furthermore, common enhancement techniques may introduce distortion under certain circumstances.

  As described above, the refinement of the target positioning information before enhancement is such that the target positioning information only shows the target property and target position in each frame so as to avoid enhancement for features outside the boundary of the region where the target is located. May be needed when approximating.

  The generation of the target positioning information by the target positioning module 14 and the transmission of the target positioning information to the target enhancement module 16 may be fully automatic as described above. When the frame of the input video is received by the target positioning module 14, the target positioning information is updated by the target positioning module 14, and the updated target positioning information is transmitted to the target enhancement module 16.

  The generation of the target positioning information by the target positioning module 14 and the transmission of the target positioning information to the target enhancement module 16 may also be semi-automatic. Instead of transmitting the target positioning information directly from the target positioning module 14 to the target enhancement module 16, the target is manually positioned in the digital picture of the input video after the user has available target positioning information. Markings (eg, borders) that define the size area may be added.

  Generation of target positioning information and transmission of target positioning information to the target enhancement module 16 may also be fully manual. In such an operation, the user views the digital picture of the input video and manually adds a marking (for example, a boundary line) that defines an area of a predetermined size where the object is located to the digital picture of the input video. In practice, full manual operation is not recommended for live event compensation.

  Refinement of the target positioning information involves target boundary estimation when necessary or desirable, and an accurate target boundary is estimated. Accurate boundary estimation helps to increase the sharpness of an object without unnatural object representation and side effects of motion, and is based on several criteria. These approaches for object boundary estimation are disclosed.

  The first is an ellipse-based approach that determines or identifies an ellipse that shows the boundary of the object almost exactly by searching over a region of ellipse parameters. A second approach for target boundary estimation is level set based search, in which a level set representation of the target neighborhood is obtained and the search is performed for level set contours that roughly represent the target boundary . A third approach for object boundary estimation is a curve expansion method such as contours or snakes that are used to reduce or expand a curve with certain constraints to gather at the object boundary ( curve evolution methods). Only the first and second approaches for object boundary estimation are discussed in more detail below.

  With an ellipse-based approach, object boundary estimation is equivalent to determining an ellipse parameter that indicates the object boundary almost exactly. This approach searches over the area of the ellipse parameter around the initial value (ie, the output of the object positioning module 14) to determine the tightness with which each ellipse represents the boundary of the object. The output of the algorithm represented in FIG. 4 is the most rigid boundary ellipse.

The ellipse stiffness measurement is defined to be the average gradation of the image intensity along the edge of the ellipse. The rationale behind this measurement is that the hardest bounding ellipse should closely follow the object contour and the image gradient is usually high along the object contour (ie, the edge between the object and the background). A flowchart for the object boundary estimation algorithm is shown in FIG. A search range (Δ _x , Δ _y , Δ _a , Δ _b ) for refining the parameters is specified by the user.

  The flowchart of FIG. 4 starts by calculating the average gradation. The variables are then initialized, with a horizontal center point position, a vertical center point position, and four nested loops for the two axes. If the ellipse described by this center point and two axes yields a better (ie, larger) average gradient, this gradient value and this ellipse are known as the best so far. Next is looping through all four loops, ending with the best ellipse.

  The ellipse-based approach may be applied to environments where the boundary between the object and the background has a uniformly high gradation. However, this approach may also be applied to environments where the boundaries do not have a uniform high gradation. For example, this approach is also useful even when the object and / or background has an intensity change along the object / background boundary.

  The ellipse-based approach produces a best-fit ellipse description in a typical implementation. The description usually includes a center point, a major axis and a minor axis.

  Ellipse-based representations may be inappropriate for describing objects with arbitrary shapes. Even elliptical objects may appear with irregular shapes when motion-blur or partial occluding occurs. The level set representation facilitates estimation of the boundary of the arbitrarily shaped aspect.

5A-5D represent concepts related to a level set approach for target boundary estimation. The intensity image I (x, y) is, for example, a continuous intensity surface as shown in FIG. 5B and is not a discrete intensity grid as shown in FIG. 5A. The level set at intensity value i is a closed contour set defined by I _l (i) = {(x, y) | I (x, y) = i}. A closed contour may be represented by a continuous curve or by a sequence of discrete pixels following the curve. The level set representation of image I is a set of level sets with different intensity level values (ie, L _l (M) = {I _l (i) | i∈M}). For example, M = {0,. . . , 255} or M = {50.5, 100.5, 200.5}. The level set can be extracted from the image in several ways. One such method applies bilinear interpolation between a set of four pixels at a time to convert a discrete intensity grid into a continuous intensity surface in both space and intensity values. That is. Thereafter, a level set, such as shown in FIG. 5D, calculates the intersection of the surface with one or more level surfaces (ie, horizontal surfaces at a particular level), eg, as shown in FIG. 5C. Is taken out.

  Level set representations are variously similar to topographical maps. Typically, topographic maps have closed contours for various elevation values.

Indeed, image I may be a sub-image that includes an object having a boundary to be estimated. The level set representation L ₁ (M) (M = {i ₁ , i ₂ ,..., I _N }) is retrieved. The set M may be constructed based on the estimated intensity of the target pixel, or may simply span the entire intensity range by a fixed step (eg, M = {0.5, 1.5,..., 254...). 5, 255.5}). Then, all level set curves (ie closed contours) C _j included in the set L ₁ (M) are considered. Object boundary estimation is assigned the problem of determining a level set curve C ^* that best satisfies a number of criteria associated with the object. Such criteria may have the following variables, among others:
The average gradient along · _{C j,}
The area in C _j ,
The length of C _j ,
The position of the center of C _j ,
The average and / or variance of the intensities of the pixels contained in C _j

  The criteria may impose constraints on these variables based on prior knowledge about the subject. In the following, a specific implementation of target boundary estimation by level set is described.

ここで、Ｎはプリセット値（例えば、１０）である。留意すべきは、

は、整数フローリング演算（integer flooring operation）を表す点である。 Let m _ref , s _ref , a _ref and the vector x _ref = (x _ref , y _ref ) be the reference values for the mean intensity, standard deviation of intensity, area and center of the object, respectively. These may be initialized based on preliminary knowledge about the object (eg, object parameters from the object positioning module 14 are obtained from an ellipse). A set of levels M is then constructed as M = {i _min , i _min + Δ _l , i _min + 2Δ _l ,..., I _max }. here,

Here, N is a preset value (for example, 10). It should be noted that

Is a point representing an integer flooring operation.

ここで、Ｓａ及びＳｘは、範囲［０，１］にある出力値を有する類似関数（similarity function）であって、値が大きいほど、基準値と測定値との間のより良い一致を示す。例えば、以下の通りである。 For a particular level set curve C _j , m _j , s _j , a _j and vector x _j = (x _j , y _j ) are respectively the average intensity, the standard deviation of the intensity, and the area of the image area included in C _j. And the center measurement. The average gradient _G avg _{(C j)} along the _{C j} are calculated. In other words, G _avg (C _j ) is the average of the gradation levels at each pixel on C _j . For each C _j , in this case the score is calculated as follows:

Here, Sa and Sx are similarity functions having output values in the range [0, 1], and a larger value indicates a better match between the reference value and the measured value. For example, it is as follows.

次いで、対象境界Ｃ^＊は、このスコアを最大とする曲線として推定される（すなわち、Ｃ^＊＝ａｒｇ_Ｃｊｍａｘ［Ｓ（Ｃ_ｊ）］）。

The target boundary C ^* is then estimated as the curve that maximizes this score (ie, C ^* = arg _Cj max [S (C _j )]).

After estimating the target boundary, the reference values m _ref , s _ref , a _ref and the vector x _ref may be updated by a learning factor αε [0, 1] (eg, m _ref ^new = αm _j + (1-α) m _ref ). In the case of a video sequence, the coefficient α may be a function of time (eg, frame index) t starting at a high value, decreasing from frame to frame, and eventually saturating to a fixed low value α _min .

  In object enhancement, object definition is improved by applying image processing operations in the vicinity of the object. Such operations may be applied along the object boundary (eg, edge sharpening), inside the object (eg, texture enhancement), and even even outside the object (eg, contrast enhancement). In the implementation described herein, a number of methods for target enhancement are proposed. The first is to sharpen the edges along the contour inside the object. Second, the object is enlarged by interactively applying smoothing, sharpening and boundary estimation operations (the order of application need not necessarily be in the order listed here). That is. Other possibilities include the use of morphological filters and object replacement.

  One way to attract more attention to the object is to sharpen the edges along the object's outline within the object. This makes the subject's details clearer and makes the subject stand out from the background. Furthermore, sharper edges tend to better survive compression. The algorithm for object enhancement by sharpening operates on the object one frame at a time, with its intensity image I (x, y) as input and the object parameters provided by the object positioning module 14 (ie, position, Size). The algorithm has three steps as follows.

  -Estimate the boundary O of the object.

· Applying a sharpening filter F _alpha to all pixels in the image I on the target within the boundary and the object boundary. This gives a new sharpening value I _sharp (x, y) for all pixels contained in O. Here, I _sharp (x, y) = (I * F _α ) (x, y), and (I * F _α ) indicates convolution of the image I by the sharpening filter F _α .

Replace pixel I (x, y) with I _sharp (x, y) for all (x, y) inside and above O.

The sharpening filter F _α is defined as the difference between the Kronecker delta function and the discrete Laplacian operator ∇ _α ² :
F _α (x, y) = δ (x, y) −∇ _α ² (x, y).

拡大（enlargement）による対象エンハンスメントは、スムージング、鮮鋭化及び境界推定の各動作をインタラクティブに適用することによって（必ずしも、適用される順序はここに挙げられた順でなくてもよい。）、対象の輪郭を拡大（enlarge）しようとする。対象拡大アルゴリムの具体的な実施形態に係るフローチャートは図６に示されている。このアルゴリズムは、その入力として強度画像Ｉ（ｘ，ｙ）と、対象位置決めモジュール１４によって提供される対象パラメータとをとる。第１に、対象周囲に十分なマージンを有して対象を含む領域（サブ画像Ｊ）が分離されて、ガウスフィルタにより平滑化される。この動作は、幾つかの画素だけ対象境界を外側に広げる。その後、上記の鮮鋭化動作が、エッジをより明らかにするために適用される。目下推定される対象境界と、平滑化され且つ鮮鋭化されたサブ画像（Ｊ_{ｓｍｏｏｔｈｓｈａｒｐ}）とを用いて、境界推定アルゴリズムは、対象境界Ｏの新たな推定を得るために適用される。最終的に、Ｏに含まれる画像における全ての画素が、サブ画像Ｊ_{ｓｍｏｏｔｈｓｈａｒｐ}内の対応する画素によって置換される。 The parameter αε [0,1] controls the shape of the Laplacian operator. In practice, a 3 × 3 filter kernel is constructed with the kernel center at the origin (0,0). An example of such a kernel is shown below:

Object enhancement by enlargement involves applying smoothing, sharpening and boundary estimation operations interactively (the order of application need not necessarily be in the order listed here). Try to enlarge the contour. A flow chart according to a specific embodiment of the subject expansion algorithm is shown in FIG. This algorithm takes as its input an intensity image I (x, y) and object parameters provided by the object positioning module 14. First, a region (sub-image J) including a target with a sufficient margin around the target is separated and smoothed by a Gaussian filter. This action extends the object boundary outward by a few pixels. The above sharpening operation is then applied to make the edges more obvious. The boundary estimation algorithm is applied to obtain a new estimate of the target boundary O, using the target boundary currently estimated and the smoothed and sharpened sub-image (J _smoothsharp ). Eventually, all pixels in the image contained in O are replaced by corresponding pixels in the sub-image J _smoothshap .

パラメータσ＜０はガウス関数の形を制御し、値が大きいほどより滑らかになる。実際に、３×３のフィルタカーネルは、原点（０，０）であるカーネルの中心を有して構成される。このようなカーネルの一例が以下に示される：

図１のシステムは、更に、対象エンハンスメントモジュール１６から出力されたエンハンスト映像をエンコードする手段を有する。この手段は、オブジェクトアウェア（object-aware）のエンコーダモジュール１８として図１に示され、従来の構成及び動作のモジュールであってよく、例えば関心領域に更なるビットを割り当てることによって関心のある対象を含む関心領域に特別の扱いを与えることで最小限の劣化をもってエンハンスト映像を重要な対象にまで圧縮し、あるいは、対象をより良く保つモード決定を行う。このようにして、オブジェクトアウェアのエンコーダモジュール１８は、対象の高められた鮮明度を利用して、高い忠実度（fidelity）を有して対象をエンコードする。 The smoothing filter _Gσ is a two-dimensional Gaussian function as follows:

The parameter σ <0 controls the shape of the Gaussian function, and the larger the value, the smoother. In practice, a 3 × 3 filter kernel is constructed with the kernel center at the origin (0,0). An example of such a kernel is shown below:

The system of FIG. 1 further includes means for encoding the enhanced video output from the target enhancement module 16. This means is shown in FIG. 1 as an object-aware encoder module 18 and may be a conventional configuration and operation module, for example to target objects of interest by assigning more bits to the region of interest. By giving special treatment to the region of interest to be included, the enhanced video is compressed to an important target with minimal degradation, or a mode decision for keeping the target better is performed. In this way, the object-aware encoder module 18 uses the increased sharpness of the object to encode the object with high fidelity.

  In order to optimize the enhancement of the input video, the object-aware encoder module 18 receives the target positioning information from the target positioning module 14 and better keeps the enhancement of the region in which the target is located, and consequently the target. Whether or not enhancement is preserved, the region where the object is located is better preserved than if there was no encoding by the object-aware encoder 18. However, enhancement also minimizes object degradation during compression. This optimized enhancement is achieved by appropriately managing encoding decisions and resource (eg, bit) allocation.

  The object-aware encoder 18 may be arranged to make an “object-friendly” macroblock (MB) mode decision, ie, an MB mode decision that causes little degradation of the object. Such an arrangement may have object-friendly partitioning of MBs for prediction, as shown, for example, in FIGS. 7 (A)-(C). Another approach is to force finer quantization, ie more bits, on the MB containing the object. This results in the subject getting more bits. Yet another approach targets the object itself for additional bits. Yet another approach uses weighted distortion metrics during the speed distortion optimization process. At this time, the pixels belonging to the region of interest have a higher weight than the pixels outside the region of interest.

  Referring to FIGS. 7A-7C, three possible subdivisions of a 16 × 16 macroblock are shown. Such subdivision is part of the mode decision performed by the encoding to determine how to encode the MB. One key metric is that the target is hardly degraded during encoding if the target occupies a high percentage of subdivision areas. This is because the degradation of the object degrades the quality of more parts of the subdivision. Thus, in FIG. 7C, each object constitutes only a small portion of 16 × 8 subdivisions, which is not considered a good subdivision. Object-aware encoders in various implementations know where the object is located and incorporate this position information into its mode decision. Such object-aware encoders prefer subdivisions that result in the subject occupying a larger part of the subdivision. Overall, the purpose of the object-aware encoder 18 helps to minimize object degradation as much as possible during the encoding process.

  As shown in FIG. 1, a target positioning module 14, a target enhancement module 16, and an object-aware encoder module 18 receive an input video of a digital picture that includes a target of interest, and compression with enhanced definition of the target. It is a component of the transmitter 10 which transmits a video stream. The transmission of the compressed video stream is received by a receiver 20 such as a mobile phone or a PDA.

  Accordingly, the system of FIG. 1 further comprises means for decoding the enhanced video in the compressed video stream received by the receiver 20. Such means is shown in FIG. 1 as a decoder module 22 and may be a conventional configuration and operation module, for example special to regions of interest including objects of interest by assigning further bits to the region of interest. By giving treatment, the enhanced video is restored to an important target with minimal degradation, or a mode decision is made to better maintain the enhanced target sharpness.

  As shown by the dotted lines in FIG. 1, when the object-aware post-processing module 24 is temporarily ignored, the decoded video output from the decoder module 22 has a digital picture with an enhanced target sharpness. In order to watch the video, the user is guided to a display component 26 such as a screen of a mobile phone or PDF.

  The mode of operation of the system of FIG. 1 above is considered preprocessing in that the subject is enhanced by the subject enhancement module 16 prior to the encoding operation. The sequence is changed before being compressed.

  Instead of increasing the object's definition before encoding as described above, the input video is directly directed to the object-aware encoder module 18 as represented by the dotted line 19 to increase the object's definition. Encoded without enhancement and has enhancements made by the object-aware post-processing module 24 at the receiver 20. Such a mode of operation of the system of FIG. 1 is considered post-processing in that the sharpness of the object is increased after the encoding and decoding stages, and side information about the object transmitted by the bitstream as metadata ( For example, the position and size of the object may be used. Post-processing mode operation has the disadvantage of increasing receiver complexity. In the post-processing mode operation, the object-aware encoder 18 in the transmitter 10 uses only the target position information when the target sharpness is enhanced at the receiver.

  As mentioned above, one of the advantages of the transmitter end object highlighting system (ie, pre-processing mode of operation) is to avoid the need to increase the complexity of the receiver end, which is typically a low power device. It is. In addition, the pre-processing mode of operation allows the use of a standard video decoder. This facilitates system deployment.

  The described implementations may be implemented, for example, on a method or process, an apparatus, or a software program. Although discussed only in connection with a single embodiment (e.g., discussed only as a method), the discussed implementations or features may be implemented in other forms (e.g., an apparatus or program). The device may be implemented, for example, with suitable hardware, software, and firmware. The method may be implemented on an apparatus such as a computer or other processing device, for example. Further, the method may be implemented by instructions executed by a processing device or other apparatus, such instructions stored on a computer readable medium such as a CD or other computer readable storage device or integrated circuit, for example. May be.

  As will be apparent to those skilled in the art, implementations may also generate signals that are formatted to carry information that is stored or transmitted, for example. The information may comprise, for example, instructions for performing the information or data generated by one of the described implementations. For example, the signal may be formatted to carry various types of target information (ie, position, shape) as data and / or carry encoded image data as data.

  Although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications may be made without departing from the invention within the scope defined by the claims.

[Cross-reference of related applications]
This application claims the priority of US Provisional Application No. 61/123844 (Attorney Docket Number PU080054) entitled “PROCESSING IMAGES HAVING OBJECTS” filed on April 11, 2008, The prior application is incorporated herein by reference in its entirety.

Claims (15)

A system that increases the sharpness of objects in digital pictures,
Means for providing an input video including the object;
(A) storing information representing the nature and characteristics of the object; and (b) identifying the object and indicating the position of the object according to the input video and information representing the nature and characteristics of the object. Means for generating target positioning information;
Means for generating an enhanced video of a portion of the input video including the target and an area of a digital picture in which the target is located, according to the input video and the target positioning information;
Means for encoding the enhanced video. (A) means for transmitting the encoded enhanced video;
(B) means for decoding the encoded enhanced video;
The system according to claim 1, further comprising: (c) means for displaying the enhanced video. The means for generating the target positioning information includes:
(A) means for scanning a sector of the input video;
(B) comparing the scanned sector of the input video with the stored information representative of the properties and characteristics of the object and the same properties and characteristics as the stored information representative of the characteristics and characteristics of the object; The system of claim 1, comprising: means for identifying an object in the digital picture having and determining a position of the object. (A) the object positioning information approximates only the identity and position of the object;
(B) The means for encoding the enhanced video comprises:
(1) receiving the target positioning information; and (2) having means for refining the target positioning information.
The system according to claim 3.   The system of claim 4, wherein the means for refining the object positioning information comprises: (a) estimating a boundary of the object; and (b) enhancing the object. (A) the object positioning information approximates only the identity and position of the object;
(B) means for generating an enhanced video of a portion of the input video including the target and a digital picture area where the target is located has means for refining the target positioning information;
The system according to claim 3.   The system of claim 6, wherein the means for refining the object positioning information comprises: (a) estimating a boundary of the object; and (b) enhancing the object. A method for increasing the sharpness of an object in a digital picture,
Providing an input video including the target;
Storing information representing the properties and characteristics of the object;
Generating target positioning information that identifies the target and indicates the position of the target according to the input video and information representing the nature and characteristics of the target;
Generating an enhanced video of a portion of the input video that includes the target and a region of a digital picture in which the target is located, according to the input video and the target positioning information;
Encoding the enhanced video;
Transmitting the encoded enhanced video. (A) receiving the encoded enhanced video;
(B) decoding the encoded enhanced video;
The method of claim 8, further comprising: (c) displaying the enhanced video. The step of generating the target positioning information includes:
(A) scanning the sector of the input video;
(B) comparing the scanned sector of the input video with the stored information representative of the properties and characteristics of the object and the same properties and characteristics as the stored information representative of the characteristics and characteristics of the object; And identifying a target in the digital picture having and determining the position of the target. (A) the object positioning information approximates only the identity and position of the object;
(B) encoding the enhanced video comprises:
(1) receiving the target positioning information;
The method according to claim 10, further comprising: refining the target positioning information. The step of refining the target positioning information includes:
(A) estimating a boundary of the object;
The method of claim 11, comprising: (b) enhancing the object. (A) the object positioning information approximates only the identity and position of the object;
(B) generating an enhanced image of the portion of the input image that includes the object and a region of the digital picture in which the object is located comprises refining the object positioning information;
The method of claim 10. The step of refining the target positioning information includes:
(A) estimating a boundary of the object;
14. The method of claim 13, comprising the step of: (b) enhancing the object. A system that increases the sharpness of objects in digital pictures,
Means for providing an input video including the object;
(A) storing information representing the nature and characteristics of the object; and (b) identifying the object and indicating the position of the object according to the input video and information representing the nature and characteristics of the object. Means for generating target positioning information;
Means for encoding the input video according to the input video and the target positioning information.

Images