WO2006082541A2 - Segmentation d'une image - Google Patents

Segmentation d'une image Download PDF

Info

Publication number
WO2006082541A2
WO2006082541A2 PCT/IB2006/050264 IB2006050264W WO2006082541A2 WO 2006082541 A2 WO2006082541 A2 WO 2006082541A2 IB 2006050264 W IB2006050264 W IB 2006050264W WO 2006082541 A2 WO2006082541 A2 WO 2006082541A2
Authority
WO
WIPO (PCT)
Prior art keywords
image
regions
signal
segment
images
Prior art date
Application number
PCT/IB2006/050264
Other languages
English (en)
Other versions
WO2006082541A3 (fr
Inventor
Fabian E. Ernst
Patrick P. E. Meuwissen
Original Assignee
Koninklijke Philips Electronics N.V.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics N.V. filed Critical Koninklijke Philips Electronics N.V.
Publication of WO2006082541A2 publication Critical patent/WO2006082541A2/fr
Publication of WO2006082541A3 publication Critical patent/WO2006082541A3/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/215—Motion-based segmentation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Definitions

  • the present invention is generally related to the enabling of conversion of images provided in two dimensions to three or more dimensions and more particularly to a method, device and computer program product for segmenting an image in a signal as well as to a device for providing a multi-dimensional image.
  • the images are made up of objects that move from image to image.
  • a segment from a previous image can then be moved into a next image, which means that earlier processing can be used.
  • This provides a consistency between different images.
  • objects that have been blocked in previous images are not taken care of, and new objects entering the scene are not handled either.
  • WO 2004057460 describes the division of an image into regions or tiles. This tiling is then used for motion estimation. Motion compensation can then be used for moving a segment from image to image.
  • One object of the present invention is therefore to provide an improved segmenting selection scheme, and in particular one where segmentation for an image is provided in a way which balances the requirements of accuracy and consistency as well as limits the computational power needed.
  • this object is achieved by a method of segmenting an image in a signal comprising a number of images comprising the steps of: - dividing a present image in the signal into a number of regions, selecting a limited number of regions, applying a segmentation scheme on said limited number of regions that provides at least one fresh segment, and providing at least one segment created for a previous image in the signal to the other regions of the present image.
  • this object is also achieved by a device for segmenting an image in a signal comprising a number of images and comprising: a segmenting unit arranged to apply a segmentation scheme to images, and - a control unit arranged to:
  • this object is also achieved by a device for providing a multi-dimensional image out of a signal comprising a number of two-dimensional images and comprising: - an image obtaining unit arranged to obtain the signal, a device for segmenting an image in the signal and comprising a segmenting unit arranged to apply segmentation schemes to images, and a control unit arranged to:
  • - provide at least one segment created for a previous image in the signal to the other regions of the present image, a motion estimation unit for generating motion vectors to be applied on created segments, a motion compensating unit arranged to motion compensate segments of images provided by the device for segmenting, and - a conversion unit arranged to code segmented images into a signal (X) having a format allowing the provision of multi-dimensional images.
  • this object is also achieved by a computer program product for segmenting an image in a signal comprising a number of images and comprising computer program code, to make a computer execute, when said program code is loaded in the computer: divide a present image in the signal into a number of regions, select a limited number of regions, apply a segmentation scheme on said selected regions that provides at least one fresh segment, and - provide at least one segment created for a previous image in the signal to the other regions of the present image.
  • the present invention there is provided a balance between the accuracy and consistency requirements of segmentation of an image while at the same time allowing a limiting of the required processing power.
  • the invention thus allows the provision of a good segmentation using limited computational power and fulfilling the latency requirements of the segmenting process.
  • the present invention is furthermore scalable, which allows changing the number of selected regions for fulfilling the latency requirements.
  • the invention furthermore allows the provision of devices for segmenting at a low cost.
  • the selection of the limited number of regions is based on computational resource restrictions where the number is set according to the computational resource restrictions.
  • This feature has the advantage of guaranteeing that the processing power is used as efficiently as possible while at the same time meeting the latency requirements.
  • the providing of at least one segment created for a previous image comprises motion compensating at least one segment created for said previous image and according to claim 4, a selection criterion is used that is based on the movement of segments of said previous image in relation to each other and to a region of the present image. This has the advantage of limiting the selection to regions, where there is known to have been changes in relation to a previous image.
  • the selection criterion is based on a motion compensated segment leaving an area of a region of the present image compared with the previous image. This has the advantage of selecting regions having areas that are not occupied by segments. These regions are regions that likely need a fresh segmentation.
  • the selection criterion is based on counting, in each region, pixels of all areas left by motion compensated segments and applying a segmentation scheme for regions with the highest count and according to claim 7 the count is reset for regions where a fresh segmentation has been performed. This has the advantage of guaranteeing that also regions where there are small changes, will receive a fresh segmentation from time to time.
  • segments provided at the borders of regions are stitched by combining them. This feature has the advantage of making the segments consistent from image to image, and, more importantly, reduce the effect due to the region boundaries, especially where a fresh segmentation has been performed.
  • the stitching comprises combining at least two segments in neighboring regions if a quality measure, after such a combining, is kept below a quality measure threshold.
  • the applying of a segmentation scheme or providing at least one segment created for a previous image is performed in parallel for all the regions of the image. This has the advantage of speeding up the segmentation processing and thus helps in meeting the latency requirements.
  • the basic idea of the invention is to divide a present image in a signal into a number of regions, select a limited number of regions, apply a segmentation scheme on the selected regions that provides at least one fresh segment and provide segments created for a previous image in the signal to the other regions of the present image. In this way the required computational power is reduced, while at the same time striking a balance between the accuracy and consistency requirements of the segmentation.
  • Fig. 2 schematically shows a block schematic of a device for providing a multi-dimensional image according to the invention
  • Fig. 3 schematically shows a block schematic of a device for segmenting an image according to the present invention
  • Fig.4 schematically shows a flow chart of a method according to the present invention
  • Fig. 5 schematically shows a computer program medium according to the present invention in the form of a CD Rom disc.
  • the present invention is generally directed towards segmenting of images, which is an important step when processing an image. This is of importance for instance when converting two dimensional images into three dimensional images. However if new segments are to be provided for each image, a lot of processing power is needed, which might not be at hand on all types of devices, especially if they are to be provided on a price sensitive consumer market. This might also reduce the consistency of the segmentation from image to image. It is furthermore possible to motion compensate an already provided segment from one frame to the other, which requires considerably less computational power, but then inconsistencies might also arise because of for instance (de-)occlusion of segments. An example of this is outlined in Figs. IA and B, where Fig.
  • the first image I 1 shows a number of objects that have been segmented.
  • a segment is normally provided by combining a number of pixel elements that have the same color.
  • Fig. IA there are shown six segments S 1 , S 2 , S 3 , S 4 , S 5 and S 6 , where a first and a second segment S 1 and S 2 make up a first object, a third segment S 3 makes up a second object, a fourth segment S 4 makes up a third object and a fifth and sixth segment S 5 and S 6 make up a fourth object.
  • the first and fourth objects are here moving, while the second and third objects are stationary. It should here be realized that the number of segments shown are very few in order to describe the functioning of the present invention. It should be realized that in reality there are normally provided more segments.
  • the image has furthermore been divided into six regions, where one has received reference numeral 10. In this embodiment the regions are in the form of tiles. The number of regions might also be more or fewer.
  • Fig. IB shows a next image I 2 where the objects from the same scene have been moved.
  • the first and second segments S 1 and S 2 are partially provided over the third segment S 3 .
  • a first overlapping area 12 associated with the first segment S 1 is indicated and a second overlapping area 16 associated with the second segments S 2 is indicated as well as a first area 14 that the first segment S 1 has left and a second area 18 that the second segment S 2 has left.
  • the same type of areas 22, 26 and 20, 24 are here provided for the fourth and fifth segments S 4 and S 5 .
  • a device 28 for providing multi-dimensional images according to the present invention is shown in a block schematic in Fig. 2.
  • the device 28 comprises an image obtaining unit 30 connected to a device 32 for segmenting images, the segmenting device 32 is in turn connected to a motion estimation unit 34, which in turn is connected to a motion compensating unit 36.
  • the motion compensating unit 36 is also connected to the segmenting device 32.
  • the segmenting device 32 is further connected to a conversion unit 38, which in turn is connected to a unit 40 for generating multi-view images.
  • the unit 40 is finally connected to a display unit 42.
  • FIG. 3 A block schematic of the segmentation device 32 is schematically shown in Fig. 3.
  • This device 32 includes a control unit 44, a segment library 46 and a segmenting unit 48 both separately connected to the control unit 44.
  • the control unit 44 is to be connected to the image obtaining unit 30, the motion estimation unit 34, the motion compensating unit 36 and the conversion unit 42.
  • image I 1 is the first image of a certain scene
  • this image which is here also called a previous image
  • the whole image is provided by the control unit 44 to the segmenting unit 48, which goes on and performs a fresh segmentation of the whole image according to a segmentation scheme, for instance based on color, like in the referenced article.
  • the different segments are then stored in the segment library 46 by the control unit 44.
  • the image obtaining unit 30 sends the second image h to the segmenting device 32, which segmenting device 32 thus receives this image h that is also denoted present image, step 50.
  • the control unit 44 then divides this image h into regions, step 52. Each region is preferably rectangular in shape, where the horizontal and vertical dimension is preferably an integer that is a multiple of eight pixels. In this way the regions can be used for other image based processing.
  • the control unit 44 provides the segments S 1 - S 6 , here collectively denoted S, to the motion estimation unit 34, which determines motion vectors V for the segments S based on information in the two images I 1 and h.
  • the motion vectors V are then provided from the motion estimation unit 34 to the motion compensating unit 36.
  • the segments S of the first image I 1 are motion compensated and their relevance for the regions of the second image are determined, step 54. Thereafter the control unit 44 compares each region with a selection criterion SC, step 56, which will be described further below. A limited number of regions that best match the selection criterion are then selected by control unit 44 for a fresh new segmentation, step 58.
  • the number of regions that are selected this way are chosen in dependence of the resource restrictions, i.e. the amount of processing power available in order to provide fresh segmentation in the time limit the image format allows. Normally there is a certain time or latency within which the segmentation process of an image has to be completed and the number of regions selected are decided based upon how well the device 32 can fulfill this requirement.
  • control unit 44 provides the selected regions to the segmenting unit 48 which applies a fresh segmentation on these regions according to the above-described segmentation scheme, step 60. For each region there is therefore provided at least one fresh segment and normally several fresh segments.
  • the control unit 44 thereafter provides the motion compensated segmentation, i.e. the previously provided segments that have been motion compensated to the rest of the regions, step 62. All the regions processed in this way are here processed in parallel.
  • the control unit 44 then stitches the region borders so that they are consistent between each other, step 64. Stitching is performed in such a way that segments on each side of a region border are combined if the union of these segments lead to a valid segment.
  • the control unit 44 then provides all segmented images of the input signal, including images I 1 and I 2 , to the conversion unit 38, which goes on and codes these image in an appropriate way and includes them in a signal X.
  • the signal X comprises coded images suitable for use as three-dimensional images, perhaps via suitable coding regarding depth of the segments, combinations of the segments into objects, focus as well as other properties.
  • the coding can here be made according to for instance the MPEG4 coding scheme.
  • the signal X is then provided to the unit 40.
  • There multi-view images corresponding to each coded image in signal X are generated.
  • the multi-view images are here provided as sets of images, where each set depicts the same content from different viewpoints. These multi-view images are then provided to the display 42 for display to a user.
  • the selection criterion described above can be based on those regions where segments have been moved in relation to each other and in relation to the previous image.
  • One such situation is where there is a de-occlusion or "hole" for a number of pixels of a region because of the movement of a segment from an area previously occupied by this segment, i.e. that there are many pixels that have no information because of segments being motion compensated. These regions are regions that likely need a fresh segmentation. The hole would however not appear if another segment of the previous image was moving into this area. If there are many such pixels the region in question is selected. If Fig. IB is taken as an example and the resource restriction specify that only two regions can get a fresh segmentation, it can be seen that the two regions furthermost to the left would be selected.
  • the present invention has the advantage of allowing limited resources to be used while at the same time allowing a reasonable quality to be obtained. Thus a balance between the accuracy and consistency requirements of segmentation of an image is struck.
  • the invention thus allows the provision of a good segmentation using limited computational power and fulfilling the latency requirements of the segmenting process.
  • the present invention is furthermore scalable, which allows changing the number of regions for which a segmentation scheme is applied that might be needed for fulfilling the latency requirements.
  • the invention furthermore allows the provision of low cost devices for segmenting. By basing the selection of regions on computational resource restrictions it is guaranteed that the processing power is used as efficiently as possible while at the same time meeting the latency requirements.
  • the selection of fresh segmentation is limited to regions, where there is known to have been changes in relation to a previous image.
  • the parallel processing of the regions has the further advantage of speeding up the segmentation processing and thus helps in meeting the latency requirements.
  • the selection criterion can be based on a quality measure of the previous segmentation, such as a variance measure of the average color, which is obtained by taking the root mean square value of for instance the color or brightness and select those regions for which the quality is low, i.e. the variance is high. It is furthermore possible to investigate the variation in motion provided for a region.
  • the quality measure can furthermore also be applied when determining what stitching is to be performed, i.e. to combine segments if the quality measure stays below a threshold after such stitching has been made. This has the advantage of only stitching those segments that can reasonably be expected to form the same segment, which further enhances the consistency of segments from image to image.
  • the regions in the embodiment described above were provided as tiles. They can however have any two-dimensional structure. When there is a scene change, there is no need for a time consistency requirement, as subsequent images are not correlated. Hence it is possible to then provide a fresh segmentation for a whole image. However it is also possible to apply the selection according to the present invention also in this case. Then the effects of a scene change will be taken care of in a number of frames. There are other variations that are possible to make to the present invention, where one such variation is that the display and possibly also the unit for multi- viewing are provided in another device, which the device according to the present invention is communicating with.
  • control unit and the segmenting unit are preferably provided in the form of a processor with associated program memory, which comprises program code for performing the method according to the present invention.
  • This program code can then be provided in the form of a computer program product which can be in the form of a CD Rom disc.
  • a computer program product which can be in the form of a CD Rom disc.
  • One such disc 66 is generally shown in Fig. 5. It should be realized that other types of products are also feasible like for instance memory sticks.
  • the program code can furthermore be downloaded into the device from a remote server.
  • All parts of the device for providing multi-dimensional images except for the display can furthermore be implemented in any suitable form including hardware, software, firmware or combinations of these.
  • the elements and components of an embodiment of the invention may furthermore be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or may be physically and functionally distributed between different units and processors.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Analysis (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne un procédé, un dispositif et un programme informatique permettant de segmenter une image dans un signal ainsi qu'un dispositif fournissant une image multidimensionnelle. Dans ce dispositif (32) de segmentation d'image, une unité de segmentation (48) est agencée afin d'appliquer un mécanisme de segmentation à des images; une unité de commande (44) étant agencée afin de diviser une image actuelle (I2) du signal en un certain nombre de régions (10), de sélectionner un nombre limité de régions, de commander à l'unité de segmentation d'appliquer un mécanisme de segmentation aux régions sélectionnées, ce qui fournit au moins un nouveau segment, et de fournir au moins un segment (S) créé pour une image antérieure (I1) du signal à d'autres régions de l'image actuelle. De ce fait, la puissance informatique nécessaire est réduite tout en fournissant simultanément un équilibre entre les exigences de précision et de consistance temporelle aux segmentations.
PCT/IB2006/050264 2005-02-07 2006-01-25 Segmentation d'une image WO2006082541A2 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP05100815 2005-02-07
EP05100815.9 2005-02-07

Publications (2)

Publication Number Publication Date
WO2006082541A2 true WO2006082541A2 (fr) 2006-08-10
WO2006082541A3 WO2006082541A3 (fr) 2007-02-22

Family

ID=36675993

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2006/050264 WO2006082541A2 (fr) 2005-02-07 2006-01-25 Segmentation d'une image

Country Status (1)

Country Link
WO (1) WO2006082541A2 (fr)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6173077B1 (en) * 1996-11-13 2001-01-09 U.S. Philips Corporation Image segmentation

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6173077B1 (en) * 1996-11-13 2001-01-09 U.S. Philips Corporation Image segmentation

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
DUFAUX F ET AL: "SPATIO-TEMPORAL SEGMENTATION BASED ON MOTION AND STATIC SEGMENTATION" PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING. (ICIP). WASHINGTON, OCT. 23 - 26, 1995, LOS ALAMITOS, IEEE COMP. SOC. PRESS, US, vol. VOL. 1, 23 October 1995 (1995-10-23), pages 306-309, XP000624236 ISBN: 0-7803-3122-2 *
L ET AL: "Spatio-temporal segmentation of image sequences for object-oriented low bit-rate image coding" SIGNAL PROCESSING. IMAGE COMMUNICATION, ELSEVIER SCIENCE PUBLISHERS, AMSTERDAM, NL, vol. 8, no. 6, September 1996 (1996-09), pages 513-543, XP004047115 ISSN: 0923-5965 *

Also Published As

Publication number Publication date
WO2006082541A3 (fr) 2007-02-22

Similar Documents

Publication Publication Date Title
US20210099706A1 (en) Processing of motion information in multidimensional signals through motion zones and auxiliary information through auxiliary zones
US8928729B2 (en) Systems and methods for converting video
TWI572196B (zh) 用於次世代視訊編碼之投射式內插預測產生之技術
TWI814732B (zh) 影像編碼方法
US8106924B2 (en) Method and system for video rendering, computer program product therefor
RU2503062C2 (ru) Способ и система для кодирования сигнала трехмерного видео, кодер для кодирования сигнала трехмерного видео, кодированный сигнал трехмерного видео, способ и система для декодирования сигнала трехмерного видео, декодер для декодирования сигнала трехмерного видео
US20080106546A1 (en) Method and device for generating 3d images
KR20090071624A (ko) 이미지 개선
JP3055438B2 (ja) ３次元画像符号化装置
CN105474640B (zh) 三维视频编码的摄像机参数发信的方法和装置
US9100642B2 (en) Adjustable depth layers for three-dimensional images
WO2014045504A1 (fr) Dispositif de compression de vidéo animée, dispositif de traitement d'image, procédé de compression de vidéo animée, dispositif de traitement d'image et structure de données de fichier de compression de vidéo animée
Werner et al. Hardware implementation of a full HD real-time disparity estimation algorithm
Li et al. A real-time high-quality complete system for depth image-based rendering on FPGA
JP6461209B2 (ja) ビデオ符号化システム及びビデオを符号化する方法
Lu et al. A survey on multiview video synthesis and editing
WO2006082541A2 (fr) Segmentation d'une image
US9787980B2 (en) Auxiliary information map upsampling
CN114897681A (zh) 基于实时虚拟视角插值的多用户自由视角视频方法及系统
CN104704836B (zh) 三维视频、多视图视频及可缩放视频的编码方法及装置
CN114930812A (zh) 用于解码3d视频的方法和装置
WO2019162564A1 (fr) Appareil, procédé et programme d'ordinateur pour vidéo volumétrique
CN104350748A (zh) 使用低分辨率深度图的视图合成
CN105144714B (zh) 三维或多视图视频编码或解码的方法及装置
Kwak et al. An Improved View Synthesis of Light Field Images for Supporting 6 Degrees-of-Freedom

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application
NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 06710741

Country of ref document: EP

Kind code of ref document: A2

WWW Wipo information: withdrawn in national office

Ref document number: 6710741

Country of ref document: EP