201015492 九、發明說明： 【發明所屬之技術領域】 本發明係有關影像強化（enhancement )，特別是關 於深度相關（depth-based)的影像或視訊強化。 【先前技術】 鲁 當三維物體藉由照相機或攝影機而投影映射至二維景^ 像平面時，由於此種投射係為非唯一的多對一轉換，因此 會失去立體深度資訊。換句話說，無法藉由投射後的影像 點來決定其深度。為了得到一個完整重現或近似的立體表 $見，必須恢復或產生這些立體深度資訊，用以進行影像強 化（enhancement)、影像復原（restoration)、影像 ❸含成或影像的顯示。 影像強化是數位影像或視訊處理當中很重要的一種。 廣義來看’幾乎所有的數位影像處理工作都會應用到影像 強化。一方面來看’影像強化係用以改進影像的品質（例 如對比及亮度）；另一方面來看，影像強化係用以作為後 續影像處理之準備或前置處理。 201015492 影像邊界（edge)之強化或偵測為一種重要的影像強 化操作。影像邊界強化會加強影像的邊界或顯示出邊界的 細節（例如影像中物體的輪廓），而保持或甚至壓制影像 的其他部分。 傳統的邊界加強通常對二雄影像進行二維頻率轉換。 頻率轉換係將影像自空間域的亮度表示轉換為頻域的頻率 β 成分表示Q經頻率轉換之高頻成分或邊界資訊，其強度會 增加或加強，因而產生邊界加強之影像β然而，傳統方法 對於不同深度之邊界並沒有作區別，因此，單調地分派相 同的深度值給不同的邊界。 ! ~ 鑑於上述傳統方法未能忠實地進行影像之加強，因此 盈需提出一種深度相關之影像/視訊加強系統及方法，除了 β 加強影像外，也考量了個別的立體深度資訊。 【發明内容】 鑑於上述，本發明的目的之一在於提出一種新穎之深度相關影 像/視訊加強纽及方法，使壯體深度f訊以忠實地改良影像品 201015492 . 根據實施例’本發明提供一種深度相關之影像加強系統及方 法。深度估算單元從二維影像產生立體深度資訊。接著，影像加 強單元根據立體深度資訊以加強二維影像。在本實施例的影像加 強單元中，深度相關邊界加強單元強調（emphasize)二維 影像之邊界細節。深度相關對比調整單元加強亮度（luma) 成分之對比特徵，而深度相關飽和度調整單元則加強色度 (chroma)成分之飽和度特徵。 【實施方式】 第一 A圖顯示本發明實施例之深度相關 (depth-based)影1/視訊加強（enhancement)系統 100 »為了便於暸解本發明，包含有原始影像、處理中影 像、結果影像之例示影像也同時附屬顯示於圖式中。第一 ® B圖顯示本發明實施例之深度相關影像/視訊加強方法的 流程步驟〇 輸入裝置10提供或接收一或多張二維（平面）輸入 影像（步驟20)，用以進行本實施例之影像/視訊處理之 用。輸入裝置10可以是一種光電裝置，用以將三維物體 投影映射至二維景>像平面。在本實施例中，輸入裝置1〇 可以是照相機，用以取像得到二維影像；或者可以是攝影 8 201015492 多幅•在另-實施例中，輸入裝置Η) =: 置，進行-或多個影像處理工作， 成‘ 像彳影像分析、影像_或影 成:再者，輸入裝£ 10可更包含-儲存裝置(例如半導201015492 IX. Description of the Invention: [Technical Field of the Invention] The present invention relates to image enhancement, particularly to depth-based image or video enhancement. [Prior Art] When a three-dimensional object is projected onto a two-dimensional scene by a camera or a camera, since the projection is a non-unique multi-to-one transformation, the stereo depth information is lost. In other words, the depth of the image point cannot be determined by the projected image point. In order to obtain a fully reproducible or approximate stereoscopic table, these stereoscopic depth information must be restored or generated for image enhancement, image restoration, image inclusion or image display. Image enhancement is an important part of digital imaging or video processing. In a broad sense, almost all digital image processing work is applied to image enhancement. On the one hand, 'image enhancement is used to improve the quality of images (such as contrast and brightness); on the other hand, image enhancement is used as a preparation or pre-processing for subsequent image processing. 201015492 Enhancement or detection of image edges is an important image enhancement operation. Image boundary enhancement enhances the boundaries of the image or reveals the details of the border (such as the outline of an object in the image) while maintaining or even suppressing other parts of the image. Traditional boundary enhancement usually performs two-dimensional frequency conversion on the two male images. The frequency conversion system converts the brightness representation of the image from the spatial domain to the frequency of the frequency domain. The β component represents the high frequency component or boundary information of the Q frequency conversion, and its intensity is increased or strengthened, thereby generating a boundary-enhanced image. However, the conventional method There is no difference between the boundaries of different depths, so the same depth value is assigned monotonically to different boundaries. ~ In view of the fact that the above traditional methods have not faithfully enhanced the image, Ying Ying proposed a depth-related image/video enhancement system and method. In addition to the β-enhanced image, individual stereo depth information was also considered. SUMMARY OF THE INVENTION In view of the above, one of the objects of the present invention is to provide a novel depth-dependent image/video enhancement button and method for faithfully improving image product 201015492. According to an embodiment, the present invention provides a Depth-related image enhancement system and method. The depth estimation unit generates stereo depth information from the two-dimensional image. Then, the image enhancement unit enhances the two-dimensional image based on the stereoscopic depth information. In the image intensifying unit of this embodiment, the depth-dependent boundary enhancement unit emphasizes the boundary details of the two-dimensional image. The depth-dependent contrast adjustment unit enhances the contrast characteristics of the luma component, while the depth-dependent saturation adjustment unit enhances the saturation characteristic of the chroma component. [Embodiment] FIG. 1A shows a depth-based shadow/enhancement enhancement system 100 according to an embodiment of the present invention. In order to facilitate the understanding of the present invention, an original image, a processed image, and a resultant image are included. The exemplified image is also attached to the drawing. The first page B shows the flow of the depth-dependent image/video enhancement method of the embodiment of the present invention. The input device 10 provides or receives one or more two-dimensional (planar) input images (step 20) for performing the image of the embodiment. / Video processing. Input device 10 can be an optoelectronic device for mapping a three-dimensional object projection to a two-dimensional scene > image plane. In this embodiment, the input device 1〇 may be a camera for taking images to obtain a 2D image; or may be a photography 8 201015492 multiple frames • In another embodiment, the input device Η) =: set, perform - or Multiple image processing jobs, such as image analysis, image _ or shadow formation: again, the input can be more than 10 storage devices (such as semi-guide
St::或硬碟i:用以儲存經前置處理裝置所處理的影 。刚所述，當三維物體投影_至二維影像平面時了 ❹ 會失去立體深度資訊’因此，以下將詳述本發明實施例之 深度相關影像/視訊加強系統⑽的其他方塊是如何用來 處理輸入裝置10所提供的二維影像。 接下來，輸入裝置10所提供的二維影像經由立體深 度估算單元11處理，以產生或重現影像中物體的立體深 度資訊（步驟21)。在本說明書中，”單元”—詞可用以表 示一電路、一程式或其組合。立體深度估算單元11可以 ⑩是傳統的電路或演算法，用以自二維影像產生立體深度資 訊。或者，立體深度估算單元n也可以是新穎的電路或 演算法，其揭露於本案申請人的下列專利申請案中··”偵 測消失線以產生立體深度資訊”（發明人為陳良基等人）、” 區塊相關之材質密度分析以產生立體深度資訊”（發明人為 陳良基等人）、估算區域模糊程度以產生立體深度資訊” (發明人為陳良基等人）。 9 201015492 立體深度估算單元11所產生的立體深度資訊和輪入 裝置10所提供的原始二維影像共同饋至影像加強（或調 整（adjustment))單元12 (步驟22)β在本實施例中， 影像加強單元 以又進原始二維影像之邊界、對比及餘和 度（saturation)。再者，本發明實施例之影像加強單元 ❹ 12根據立體深度f訊來崎此影像纽，使得影像中的 界細節（例如個別區域之物體的輪廓）根據個別不同深产 而施予不同程度的強調（emphasize)或加強。例如^ 較小深度值（亦即，較靠近觀看者）之物體麵施予較大 程度的加強或使賴大加強權重，而具較大深度值（亦即， 較遠離觀看者）之物贿_施代小程㈣加強或 較小加強權重。影像加強之執行可以使轉統或未來 像技術，其可以屬於空間域加強處理或頻域加強處理， ^，，、斤货月貫施例之影像加強單元12 細方塊圖’而第二示本發明實施例之 = 驟22的詳細流程步驟。 、/ 強步 以強調顯示出 之執行係根據 f先，二維影像藉由邊界加強單元12〇 影像中的邊界細節。特_是，此邊界加強 201015492 深度資訊的。例如，具較小深度值（亦即，較靠近觀看者） 之物體使用較大加強權重，而具較大深度值（亦即，較遠 離觀看者）之物體則使用較小加強權重。權重值可以根據 個別應用需求而從實驗得到·。在本實施例中，邊界加強之 進行係藉由高通遽波技術，首先使用一空間域摺遮罩 (convolution mask)，接著以此遮罩針對一中心像素周 圍的像素群進行處理。 ❿ 邊界加強單元120所得到的邊界加強影像可藉由一轉 換器122將其自紅/綠/藍（RGB)顏色空間轉換至色調/ 飽和度/亮度（hue/saturation/intensity, HSI)顏色空 間（步骤222)。然而，如果經轉換之顏色資料可以事先提 供的話’則上述的轉換即可省略。再者，雖然本實施例使 用HSI顏色空間，然而也可以使用其他的顏色空間，例如 w HSL ( hue / saturation/lightness )或 HSV (hue/saturation/value)。經轉換的影像資料基本上包 含亮度（luma)成分及色度（chroma)成分。 在本實施例中，使用對比調整單元124對亮度成分進 行對比調整或加強（步驟224)，以加強其對比特性。特別 的是，此對比調整之執行係根據深度資訊的。例如，具較 11 201015492 小深度值（亦即’較靠近觀看者）之物體使用較大加強權 重’而具較大深度值（亦即，較遠離觀看者）之物體則使 用較小加強權重。權重值可以根據個别應用需求而從實驗 得到。在本實施例中’對比調整之進行係使用亮度分佈圖 伸展（histogram stretching)技術，其將分佈圖中灰階 值大於臨界值之像素值予以增加，而將分佈圖中灰階值小 於臨界值之像素值予以降低。 e 此外’使用飽和度調整單元126對色度成分進行飽和 度調整或加強（步驟226)，以加強其飽和度特性或顏色/ 色調之純度。特別的是，此飽和度調整之執行係根據深度 I ； 資訊的。例如，具較小深度值（亦即，較靠近觀看者）之 物體使用較大加強權重，而具較大深度值（亦即，較遠離 觀看者）之物體則使用較小加強權重。權重值可以根據個 ❹別應用需求而從實驗得到。 接下來，將對比調整單元124所得到的對比調整亮度 成分和飽和度調整單元126所得到的飽和度調整色度成分 從HSI顏色空間轉換回到RGB顏色空間（方塊128，步 驟228)。上述之顏色空間反轉換可以省略，或者暫時保持 不動。再者，在本實施例中，雖然邊界之調整120係先於 12 201015492 對比/飽和度之調整124/126,然而，其順序也可以予以 顛倒。另外，可以根據個別應用需求而將邊界調整120、 對比調整124、飽和度調整126其中的一或多個予以省略。 輸出裝置13從影像加強單元12接收加‘影像，並產 生輸出影像（步驟23)。在一實施例中，輸出裝置13可以 為顯示裝置，用以顯示或供觀看所接收的加強影像。在另 ❹一實施例中，輸出裝置13可以為儲存裝置，例如半導體 記憶體或硬碟，用以儲存所接收的加強影像。再者，輸出 裝置13也可更包含一後置處理裝置，用以進行一或多種 影像處理，例如影像復原、影像分析、影像壓縮或影像合 成。 根據上述，本發明實施例可忠實地改良影像品質，例 ® 如邊界、對比或飽和度，或者為其他特徵（例如材質 (texture )特徵、空間特徵、雜訊或模糊程度 (blurriness))〇 以上所述僅為本發明之較佳實施例而已，並非用以限 定本發明之申請專利範圍；凡其它未脫離發明所揭示之精 13 201015492 神下所完成之等效改變或修飾，均應包含在下述之申請專 利範圍内。 【圖式簡單說明】 第一 A圖顯示本發明實施例之深度相關影像/視訊加強系 統。 ' ❹第一 B圖顯示本發明實施例之深度相關影像/視訊加強方 法的流程步驟。 第一 A圖顯示本發明實施例之影像加強單元的詳細方塊 圖。. 第一 B圖顯示本發明實施例之影像加強步驟的詳細流程步 驟。 _ 【主要元件符號說明】 1〇〇 10 11 12 13 20-23 120 深度相關影像/視訊加強系統 輪入裝置 立體深度估算單元 影像加強單元 輪出裝置 實施例之流程步驟 邊界加強單元 201015492 . 122 轉換器 124 對比調整單元 126 飽和度調整單元 128 顏色空間反轉換 220-228 影像加強步驟的詳細流程步驟 ❹ 參 15St:: or hard disk i: used to store the images processed by the pre-processing device. As just mentioned, when the three-dimensional object is projected onto the two-dimensional image plane, the stereoscopic depth information will be lost. Therefore, how the other blocks of the depth-dependent image/video enhancement system (10) of the embodiment of the present invention are processed will be described in detail below. The two-dimensional image provided by the device 10 is input. Next, the two-dimensional image provided by the input device 10 is processed via the stereoscopic depth estimating unit 11 to generate or reproduce the stereoscopic depth information of the object in the image (step 21). In this specification, a "unit" - word may be used to mean a circuit, a program, or a combination thereof. The stereoscopic depth estimating unit 11 may be a conventional circuit or algorithm for generating stereoscopic depth information from the two-dimensional image. Alternatively, the stereoscopic depth estimating unit n may also be a novel circuit or algorithm disclosed in the following patent application of the applicant: "Detecting the vanishing line to generate stereoscopic depth information" (invented by Chen Liangji et al) , “Dimensional material density analysis to generate stereoscopic depth information” (invented by Chen Liangji et al.), estimating the degree of regional blurring to generate stereoscopic depth information” (invented by Chen Liangji et al.) 9 201015492 Stereoscopic depth estimation unit 11 The generated stereoscopic depth information and the original two-dimensional image provided by the wheeling device 10 are fed together to an image enhancement (or adjustment) unit 12 (step 22). In this embodiment, the image enhancement unit is re-entered into the original The boundary, the contrast, and the saturation of the two-dimensional image. Furthermore, the image enhancement unit ❹ 12 according to the embodiment of the present invention extracts the image image according to the stereoscopic depth, so that the boundary details in the image (for example, individual regions) The contour of the object) is given different degrees of emphasis or enhancement depending on the individual deep product. For example, ^ a smaller depth value (ie The object surface closer to the viewer) is given a greater degree of reinforcement or the weight is increased, while the material with a larger depth value (ie, farther away from the viewer) is reinforced or less. Strengthening the weight. The implementation of image enhancement can make the conversion system or the future image technology, which can belong to the spatial domain enhancement processing or the frequency domain enhancement processing, ^,,, the image enhancement unit 12 of the monthly application example 2 shows the detailed flow steps of the embodiment 22 of the present invention. / / Strong step to emphasize that the execution system is based on f first, and the two-dimensional image is bounded by the boundary enhancement unit 12 in the image boundary details. This boundary enhances the depth information of 201015492. For example, objects with smaller depth values (ie, closer to the viewer) use larger enhancement weights, while objects with larger depth values (ie, farther away from the viewer) are Use smaller enhancement weights. The weight values can be obtained experimentally according to individual application requirements. In this embodiment, the boundary enhancement is performed by high-pass chopping technology, first using a spatial domain folding mask (convo Lution mask), and then the mask is processed for a pixel group around a central pixel. 边界 The boundary-enhanced image obtained by the boundary enhancement unit 120 can be self-red/green/blue (RGB) color by a converter 122. The space is converted to a hue/saturation/intensity (HSI) color space (step 222). However, if the converted color data can be provided in advance, then the above conversion can be omitted. This embodiment uses the HSI color space, however other color spaces may be used, such as w HSL ( hue / saturation/lightness ) or HSV (hue/saturation/value). The converted image data basically contains a luma component and a chroma component. In the present embodiment, the contrast adjustment unit 124 is used to contrast adjust or enhance the luminance component (step 224) to enhance its contrast characteristics. In particular, the execution of this contrast adjustment is based on depth information. For example, objects with smaller depth values than 11 201015492 (i.e., 'closer to the viewer' use larger reinforcement weights') objects with larger depth values (i.e., farther away from the viewer) use smaller enhancement weights. Weight values can be derived experimentally based on individual application needs. In the present embodiment, the 'contrast adjustment is performed using a histogram stretching technique, which increases the pixel value of the grayscale value larger than the critical value in the distribution map, and the grayscale value in the distribution map is smaller than the critical value. The pixel value is reduced. Further, the saturation component is 126 for saturation adjustment or enhancement (step 226) using the saturation adjustment unit 126 to enhance the saturation characteristics or the purity of the color/tone. In particular, the execution of this saturation adjustment is based on depth I; information. For example, objects with smaller depth values (i.e., closer to the viewer) use larger enhancement weights, while objects with larger depth values (i.e., farther away from the viewer) use smaller enhancement weights. The weight value can be obtained experimentally based on the identification of the application requirements. Next, the contrast adjustment luminance component obtained by the contrast adjustment unit 124 and the saturation adjustment chrominance component obtained by the saturation adjustment unit 126 are converted from the HSI color space back to the RGB color space (block 128, step 228). The color space inverse conversion described above may be omitted or temporarily maintained. Furthermore, in the present embodiment, although the boundary adjustment 120 is prior to 12 201015492 contrast/saturation adjustment 124/126, the order may be reversed. Additionally, one or more of boundary adjustment 120, contrast adjustment 124, and saturation adjustment 126 may be omitted depending on individual application requirements. The output device 13 receives the added image from the image enhancement unit 12 and produces an output image (step 23). In an embodiment, the output device 13 can be a display device for displaying or viewing the received enhanced image. In another embodiment, the output device 13 can be a storage device, such as a semiconductor memory or a hard disk, for storing the received enhanced image. Furthermore, the output device 13 may further comprise a post-processing device for performing one or more image processing, such as image restoration, image analysis, image compression or image synthesis. In accordance with the above, embodiments of the present invention can faithfully improve image quality, such as borders, contrast or saturation, or other features such as texture features, spatial features, noise, or blurriness. The above is only the preferred embodiment of the present invention, and is not intended to limit the scope of the patent application of the present invention; any other equivalent changes or modifications made by God 13 201015492, which are not disclosed in the invention, should be included in the following. Within the scope of the patent application. BRIEF DESCRIPTION OF THE DRAWINGS The first A diagram shows a depth-dependent image/video enhancement system in accordance with an embodiment of the present invention. The first B diagram shows the flow steps of the depth-dependent image/video enhancement method of the embodiment of the present invention. Figure 1A shows a detailed block diagram of an image enhancement unit in accordance with an embodiment of the present invention. The first B diagram shows the detailed flow steps of the image enhancement step of the embodiment of the present invention. _ [Description of main component symbols] 1〇〇10 11 12 13 20-23 120 Depth-related image/video enhancement system wheeling device stereo depth estimation unit Image enhancement unit wheeling device embodiment process step boundary enhancement unit 201015492 . 122 conversion 124 Comparison Adjustment Unit 126 Saturation Adjustment Unit 128 Color Space Reverse Conversion 220-228 Detailed Process Steps for Image Enhancement Step ❹ Reference 15