CN103795961A - Video conference telepresence system and image processing method thereof - Google Patents

Video conference telepresence system and image processing method thereof Download PDF

Info

Publication number
CN103795961A
CN103795961A CN201210423899.9A CN201210423899A CN103795961A CN 103795961 A CN103795961 A CN 103795961A CN 201210423899 A CN201210423899 A CN 201210423899A CN 103795961 A CN103795961 A CN 103795961A
Authority
CN
China
Prior art keywords
image
background
color
depth
color image
Prior art date
Application number
CN201210423899.9A
Other languages
Chinese (zh)
Inventor
冯斌
Original Assignee
三亚中兴软件有限责任公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三亚中兴软件有限责任公司 filed Critical 三亚中兴软件有限责任公司
Priority to CN201210423899.9A priority Critical patent/CN103795961A/en
Publication of CN103795961A publication Critical patent/CN103795961A/en

Links

Abstract

The invention discloses a video conference telepresence system and an image processing method thereof. The method includes: obtaining a color image and a depth image of a current conference scene; performing initial segmentation on a background area and a foreground area in the depth image according to difference of the background image and the foreground area in the depth image, mapping a segmentation result of the depth image to the color image, and segmenting a background area from the color image by using color different of a foreground image and a background image in the color image; processing the background area obtained by segmentation of the color image; and performing video conference universal processing on the color image with the processed background area, and the video conference universal processing including coding compression and network sending. Through the video conference telepresence system and the image processing method thereof, user experience can be improved.

Description

会议电视网真系统及其图像处理方法 Conference television system and its true image processing method

技术领域 FIELD

[0001] 本发明涉及多媒体通信技术领域,具体而言,涉及一种会议电视网真系统及其图像处理方法。 [0001] The present invention relates to multimedia communication technologies, and particularly, to a television conference system and image processing method true.

背景技术 Background technique

[0002] 近年来红外光学测距技术开始兴起并逐步产品化,其中最典型的就是使用一个红外发射装置与一个红外摄像头来实现深度图像的采集。 [0002] In recent years, infrared optical ranging technology began to rise and gradually commercialized, the most typical is the use of an infrared emitting device implemented with an infrared camera to capture depth image. 其中,红外发射装置用来生成结构光,红外摄像头通过扫描结构光在物体上的投影形状来获取深度图像。 Wherein the means for generating infrared emitting structured light, infrared light cameras projected shape on the object by scanning the structure to acquire a depth image. 结构光是指一些具有特定模式的光,其投影图案可以是线、点、面等多种图形,并且随着距离的变化,光线投影的形状也发生变化。 Some structures light refers to light having a specific pattern, the pattern may be projected a variety of graphics lines, dots, and other surfaces, and as the change in the distance, the shape of the projected light is also changed. 实际场景中,由于前景物体与背景物体在距离摄像头距离上存在一定差异,其深度图像上将存在较明显的区域性。 Actual scene, since the foreground object and the background object distance there are some differences from the imaging head, the depth of the image on the presence of regional obvious. 利用深度图像的区域性以及普通光学摄像头得到的物体颜色区域性,能较好的区分背景与前景。 A color image of an object using the depth regional and regional ordinary optical camera obtained can better distinguish between background and foreground.

[0003] 视频会议系统尤其是网真系统的背景环境往往比较固定,在视频会议系统中显示时,视频会议的背景不能改变,因此,在传统会议电视中,如果需要显示附加信息,则需要双流分屏显示。 [0003] Video conferencing systems in particular context telepresence systems tend to be more fixed, is displayed in a video conference system, the background video conference can not be changed, therefore, the conventional video conference, if you need to display additional information, you need to double split-screen display. 并且,由于视频会议电视的背景也是固定不变的,不能根据用户需求进行替换,从而降低了用户体验。 Also, since the background video conferencing is also fixed, it can not be replaced according to user needs, thereby reducing the user experience.

发明内容 SUMMARY

[0004] 针对相关技术中视频会议的背景不能改变的问题,本发明提供了一种会议电视网真系统及其图像处理方法,以至少解决上述问题。 [0004] context of the relevant art problems for video conference can not be changed, the present invention provides a system and a real television image processing method meeting to at least solve the above problems.

[0005] 根据本发明的一个方面,提供了一种会议电视网真系统,包括:虚拟背景生成装置、编码发送装置和接收解码装置。 [0005] In accordance with one aspect of the invention, there is provided a real television conference system comprising: a virtual background generation means, receiving and decoding the encoding apparatus transmitting apparatus. 其中,所述虚拟背景生成装置包括:彩色图像与深度图像获取装置,用于获取会议场景的彩色图像与深度图像;背景分割装置,用于根据背景图像与前景图像在所述深度图像中的差异,对所述深度图像中的背景区域和前景区域进行初步分割,并将对所述深度图像的分割结果映射到所述彩色图像中,利用所述彩色图像中前景图像与背景图像的颜色差异,从所述彩色图像中分割出背景区域;背景生成装置,用于对所述背景分割装置从所述彩色图像分割得到的背景区域进行处理;编码发送装置,用于对经过所述背景生成装置处理后的所述彩色图像进行会议电视通用处理,其中,所述会议电视通用处理包括:编码压缩以及网络发送;接收解码装置,用于对网络媒体数据进行接收并解码显示。 Wherein the virtual background generating apparatus comprising: a color image and a depth image acquisition means for acquiring a color image and a depth image of the conference scene; background segmentation means for differences in the depth image based on the background image and the foreground image , the depth image background area and the foreground area initial segmentation, and the color image is mapped to the divided result of the depth image, a color image using the color differences in the foreground image and the background image, dividing the color image from the background area; background generation means, dividing means for processing the background from the background color of the image segmentation region obtained; encipher transmission means, processing means for generating said background through the color image after the TV conference general processing, wherein the video conferencing general processing comprising: a compression encoding and transmitting network; receiving-decoding means, for the network receives and decodes the media data display.

[0006] 优选地,彩色图像与深度图像获取装置包括:光学摄像头,用于采集会议场景的所述彩色图像;红外摄像头,用于采集会议场景的所述深度图像。 [0006] Preferably, the color image and depth image acquisition apparatus comprising: an optical camera for the color image acquisition conference scene; the infrared camera, the depth image acquisition session for a scene.

[0007] 优选地,所述背景生成装置对从所述彩色图像分割得到的背景区域进行的处理包括以下至少之一:虚拟场景替换、动画播放、横幅叠加、及图形演示文稿双流。 Processing [0007] Preferably, the means for generating a background obtained from the background region color image segmentation comprises at least one of the following: the virtual scene Alternatively, animation, banner overlay, and the presentation graphics double.

[0008] 优选地,所述背景分割装置包括:深度图像处理装置,用于对所述深度图像进行边缘检测算子边缘化,得到形状不规则的背景墙壁区域、前景区域、及噪声区域;深度图像分割装置,用于对所述深度图像处理装置处理后的所述深度图像进行膨胀处理和腐蚀处理,消除所述噪声区域,然后从消除所述噪声区域后的所述深度图像中得到规则的背景区域;彩色图像分割装置,用于将所述深度图像的规则的背景区域映射到所述彩色图像中,得到所述彩色图像的规则背景区域,并结合当前的所述彩色图像的色彩信息对所述彩色图像的规则背景区域中的前景区域进行精确化处理。 [0008] Preferably, the background segmentation apparatus comprising: a depth image processing apparatus, for the depth image marginal edge detection operator to obtain an irregular shape wall background area, the foreground area, and a noise region; Depth image segmentation means for the depth of the depth image processing apparatus for processing image expansion process and etching process, to eliminate the noise region and the depth to give rules to eliminate from the image noise region background region; color image segmentation means, for the background region depth image mapping rule to the color image, the background area of ​​the rule to obtain a color image, the color information of the current combined image for the color regioregular foreground area of ​​the background color image in precise process.

[0009] 根据本发明的另一个方面,提供了一种会议电视网真系统的图像处理方法,包括:获取当前会议场景的彩色图像与深度图像;根据背景图像与前景图像在所述深度图像中的差异,对所述深度图像中的背景区域和前景区域进行初步分割,并将对所述深度图像的分割结果映射到所述彩色图像中,利用所述彩色图像中前景图像与背景图像的颜色差异,从所述彩色图像中分割出背景区域;从所述彩色图像分割得到的背景区域进行处理;对背景区域经过处理的所述彩色图像进行会议电视通用处理,其中,所述会议电视通用处理包括:编码压缩以及网络发送。 [0009] According to another aspect of the invention, there is provided an image processing method for a conference television system true, comprising: obtaining a color image of the current conference scene depth image; foreground image and the background image in the depth image difference, the depth image background area and the foreground area preliminary segmentation, segmentation results and map the depth image to the color image, a color image using the color image in the foreground and the background image difference, dividing the color image from the background area; processing region from the background color of the image segmentation obtained; background region through videoconferencing general processing of the color image processing, wherein said general processing videoconferencing comprising: a compression encoding, and network transmission.

[0010] 优选地,所述方法还包括:对网络媒体数据进行接收并解码显示。 [0010] Preferably, the method further comprising: the network receives and decodes the media data display.

[0011] 优选地,获取当前会议场景的彩色图像与深度图像,包括:通过光学摄像头采集会议场景的所述彩色图像;通过红外摄像头采集会议场景的所述深度图像。 [0011] Preferably, the current conference scene obtain a color image and a depth image, comprising: acquiring a color image of the conference scene through an optical camera; the depth image acquired by the conference scene infrared camera.

[0012] 优选地,对从所述彩色图像分割得到的背景区域进行的处理包括以下至少之一:虚拟场景替换、动画播放、横幅叠加、及图形演示文稿双流。 [0012] Preferably, the processing of the background region obtained from the color image segmentation comprises at least one of the following: the virtual scene Alternatively, animation, banner overlay, and the presentation graphics double.

[0013] 优选地,根据背景图像与前景图像在所述深度图像中的差异,对所述深度图像中的背景区域和前景区域进行初步分割,并将对所述深度图像的分割结果映射到所述彩色图像中,利用所述彩色图像中前景图像与背景图像的颜色差异,从所述彩色图像中分割出背景区域,包括:对所述深度图像进行边缘检测算子边缘化,得到形状不规则的背景墙壁区域、前景区域、及噪声区域;对经边缘检测算子边缘化后的所述深度图像进行膨胀处理和腐蚀处理,消除所述噪声区域,从消除所述噪声区域后的所述深度图像中得到规则的背景区域;将所述规则的背景区域映射到所述彩色图像中,得到所述彩色图像的规则背景区域,并结合当前的所述彩色图像的色彩信息对所述彩色图像的规则背景区域中的前景区域进行精确化处理。 [0013] Preferably, the difference between the background image and the foreground image in the depth image, the depth image background area and the foreground area initial segmentation and dividing the result of the depth image is mapped to the said color image, said color image using a color difference in the foreground image and the background image, the color image segmentation from the background region, comprising: a depth image of the edge detection operator marginalized to obtain irregular shape background wall area, the foreground area, and a noise region; the depth image by an edge detection operator is expanded marginalization process and etching process, to eliminate the noise region, the depth of the region from the elimination of the noise background image region obtained in the rule; background region of the mapping rule to the color image, the background area of ​​the rule to obtain a color image, combined with the current color of the color image of the color image information rules foreground area background area precise treatment.

[0014] 通过本发明,使用背景分割技术将会场背景图像进行分割提取,并对其进行后期处理后编码发送至对端,从而可以对会议视频的背景图像进行处理,进而使得在需要显示附加信息时无需进行双流分屏,并且,在对提取的背景图像进行后期处理时,可以根据用户的需求进行处理,使得用户可以定制自己的背景环境,提高了用户体验。 [0014] By the present invention, the segmentation technique using a background image segmentation of the conference background, and transmits them to the end of post-processing after the encoding, so that the background image can be processed in a video conference, and further such that additional information needs to be displayed when the need for double split-screen, and, when the background image is extracted post-processing can be processed according to the needs of users, allows users to customize their context, improving the user experience.

附图说明 BRIEF DESCRIPTION

[0015] 此处所说明的附图用来提供对本发明的进一步理解,构成本申请的一部分,本发明的示意性实施例及其说明用于解释本发明,并不构成对本发明的不当限定。 [0015] The drawings described herein are provided for further understanding of the present invention, constitute a part of this application, exemplary embodiments of the present invention are used to explain the present invention without unduly limiting the present invention. 在附图中: In the drawings:

[0016] 图1是根据本发明实施例的会议电视网真系统的结构示意图; [0016] FIG. 1 is a block schematic diagram of a television conference embodiment of the present invention, the true embodiment of the system;

[0017] 图2是根据本发明实施例的虚拟背景生成装置的结构示意图; [0017] FIG. 2 is a schematic structural diagram of a virtual background generating apparatus according to an embodiment of the present invention;

[0018] 图3是根据本发明优选实施例的背景分割装置的结构示意图; [0018] FIG. 3 is a schematic structural diagram of embodiment BACKGROUND dividing device according to a preferred embodiment of the present invention;

[0019] 图4是根据本发明实施例的会议电视网真系统的图像处理方法流程图; [0019] FIG. 4 is a flowchart illustrating an image processing method according to an embodiment of the television conference true system of the present invention;

[0020] 图5是根据本发明实施例的一种三屏网真布局的示意图;[0021] 图6是根据本发明优选实施例的会议电视网真系统的会议视频的发送方法的流程图; [0020] FIG. 5 is a schematic view of a three-panel embodiment of the present invention, network arrangement according to the true; [0021] FIG. 6 is a flowchart of a method of transmitting video conference television conference system according to an example of a preferred real embodiment of the present invention;

[0022] 图7是本发明优选实施例中经深度图Candy算子处理后的各区域图像的示意图; [0022] FIG. 7 is a schematic diagram of each area by the depth of the image after the processing in FIG Candy operator preferred embodiment of the present invention;

[0023] 图8是本发明优选实施例中深度图像的规则背景矩形区域的示意图; [0023] FIG. 8 is a schematic diagram of a background depth image rule rectangular areas preferred embodiment of the present invention;

[0024] 图9是本发明优选实施例中彩色图像经过映射、前景精确化处理后得到的规则区域的示意图。 [0024] FIG. 9 is a schematic diagram of a color image through mapping rules behind the foreground area of ​​the refinement process to obtain a preferred embodiment of the present invention.

具体实施方式 Detailed ways

[0025] 下文中将参考附图并结合实施例来详细说明本发明。 [0025] Hereinafter with reference to the accompanying drawings and embodiments of the present invention will be described in detail. 需要说明的是,在不冲突的情况下,本申请中的实施例及实施例中的特征可以相互组合。 Incidentally, in the case of no conflict, embodiments and features of the embodiments of the present application can be combined with each other.

[0026] 根据本发明实施例,提供了一种会议电视网真系统。 [0026] According to an embodiment of the present invention, there is provided a television conference system true.

[0027] 图1为根据本发明实施例的会议电视网真系统的结构示意图,如图1所示,该系统主要包括:虚拟背景生成装置2、编码发送装置4和接收解码装置6。 [0027] FIG. 1 is a schematic structural diagram of television conference system according to the true embodiment of the present invention embodiment, shown in FIG. 1, the system including: a virtual background generation means 2, the encoding transmitting apparatus and a receiving-decoding means 6 4. 本发明实施例对现有的会议电视网真系统进行了改进,加入了虚拟背景生成装置2,该虚拟背景生成装置2用于对当前会议场景的背景进行处理,以满足用户的需求。 Embodiment of the present invention really existing television network conference system has been improved by adding the virtual background generation means 2, the virtual background generation means 2 for background current conference scene is processed in order to meet the needs of users. 而其中的编码发送装置4和接收解码装置6可以采用相关技术中的会议电视系统通用模块,具体本发明实施例不作限定。 And wherein the transmission coding means 4 and decoding device 6 receives television conference system in the related art general module may be employed, specific embodiments of the present invention is not limited. 其中,编码发送装置4,用于对经过所述虚拟背景生成装置2处理后的彩色图像进行会议电视通用处理,其中,所述会议电视通用处理包括但不限于:编码压缩以及网络发送;接收解码装置6用于对网络媒体数据进行接收并解码显示。 Wherein the transmission coding means 4, for a color image processing apparatus 2 via the virtual context generated a general processing television conference, wherein said general processing video conference include but are not limited to: encoding and compressing transmission network; receiver decoder network means 6 for receiving and decoding media data display.

[0028] 图2本发明实施例中虚拟背景生成装置2的结构示意图,如图2所示,虚拟背景生成装置2可以包括:彩色图像与深度图像获取装置20,用于获取会议场景的彩色图像与深度图像;背景分割装置22,用于根据背景图像与前景图像在所述深度图像中的差异,对所述深度图像中的背景区域和前景区域进行初步分割,并将对所述深度图像的分割结果映射到所述彩色图像中,利用所述彩色图像中前景图像与背景图像的颜色差异,从所述彩色图像中分割出背景区域;背景生成装置24,用于对所述背景分割装置从所述彩色图像分割得到的背景区域进行处理。 [0028] Background of the virtual embodiment of the present invention FIG. 2 is a schematic structural diagram generating apparatus 2, as shown in FIG virtual background generation means 22 may comprise: a color image and a depth image acquisition means 20 for acquiring a color image of a scene meeting depth image; background segmentation means 22, according to the difference in the background image and the foreground image in the depth image, the depth image background area and the foreground area initial segmentation of the image and the depth mapping the segmentation result to the color image, the color image using a color difference in the foreground image and the background image, the background region from the segmented color image; background generation means 24, means for dividing the background from the background color image segmentation region obtained for processing.

[0029] 在一个实施方式中,彩色图像与深度图像获取装置20可以包括:光学摄像头,用于采集会议场景的所述彩色图像;红外摄像头,用于采集会议场景的所述深度图像。 [0029] In one embodiment, the color image and a depth image acquisition apparatus 20 may include: an optical camera for the color image acquisition conference scene; the infrared camera, the depth image acquisition session for a scene.

[0030] 在一个实施例方式中,背景生成装置24对从所述彩色图像分割得到的背景区域进行的处理包括以下至少之一:虚拟场景替换、动画播放、横幅叠加、及图形演示文稿(例如,PPT)双流。 [0030] In one embodiment, the background processing includes at least one area of ​​the background color image obtained from the segmentation means 24 generates: a virtual scene Alternatively, animation, banner overlay, and presentation graphics (e.g. , PPT) Shuangliu.

[0031] 在一个实施例中,背景分割装置22可以采取图3所示的结构实现,如图3所示,背景分割装置22可以包括:深度图像处理装置220,用于对所述深度图像进行边缘检测算子(例如,Candy算子)边缘化,得到形状不规则的背景墙壁区域、前景区域、及噪声区域;深度图像分割装置222,用于对所述深度图像处理装置处理后的所述深度图像进行膨胀处理和腐蚀处理,消除所述噪声区域,然后从消除所述噪声区域后的所述深度图像中得到规则的背景区域;彩色图像分割装置224,用于将所述深度图像的规则的背景区域映射到所述彩色图像中,得到所述彩色图像的规则背景区域,并结合当前的所述彩色图像的色彩信息对所述彩色图像的规则背景区域中的前景区域进行精确化处理。 [0031] In one embodiment, background segmentation means 22 may take the configuration shown in FIG. 3 achieved, as shown in FIG background segmentation unit 322 may include: a depth image processing apparatus 220, for the depth image edge detection operator (e.g., operator Candy) marginalized to obtain irregularly shaped wall background area, the foreground area, and a noise region; the depth image dividing means 222 for the depth of the rear image processing means depth image expansion processing and etching treatment to eliminate the noise region and the background region resulting from elimination rule the depth image after the noise areas; color image dividing means 224, the rules for depth image background area to the mapped color image, the background area of ​​the rule to obtain a color image, and color information in conjunction with the current rules of the color image of the background region color image in the foreground region refinement process. [0032] 通过上述实施例之一提供的会议电视网真系统,可以实现会议背景的虚拟现实。 [0032] True television conference system provided by one of the embodiments described above, the virtual reality may be achieved conference background. 将动画播放、横幅叠加、PPT双流等融和后,可以改变传统会议电视需要双流分屏显示附加信息的局面,背景的虚拟替换使得客户可以定制自己的背景环境,提高了用户体验。 The animation, banner overlay, PPT after double integration, etc., may need to change the traditional video conferencing double split-screen display additional information on the situation, replacing a virtual background enables customers to customize their context, improving the user experience.

[0033] 根据本发明实施例,还提供了一种会议电视网真系统的图像处理方法。 [0033] According to an embodiment of the present invention, there is provided an image processing method for real television conference system.

[0034] 图4是根据本发明实施例的会议电视网真系统的图像处理方法流程图,如图4所示,会议电视网真系统的图像处理方法可以包括以下步骤: [0034] FIG. 4 is an image processing method according to an embodiment of the present conference television system of the invention the true flowchart, the image processing method true television conference system 4 may comprise the steps:

[0035] 步骤S402,获取当前会议场景的彩色图像与深度图像; [0035] In step S402, acquires the current color image and the depth image conference scene;

[0036] 例如,可以通过光学摄像头采集会议场景的所述彩色图像,通过红外摄像头采集会议场景的所述深度图像。 The color image [0036] For example, the conference scene can be collected by the optical camera, the conference scene acquired by the infrared camera, the depth image.

[0037] 步骤S404,根据背景图像与前景图像在所述深度图像中的差异,对所述深度图像中的背景区域和前景区域进行初步分割,并将对所述深度图像的分割结果映射到所述彩色图像中,利用所述彩色图像中前景图像与背景图像的颜色差异,从所述彩色图像中分割出背景区域; [0037] step S404, the foreground image and the background based on the difference image in the depth image, the depth image background area and the foreground area initial segmentation and dividing the result of the depth image is mapped to the said color image, the color image using a color difference in the foreground image and the background image, the background area is divided from the color image;

[0038] 步骤S406,从所述彩色图像分割得到的背景区域进行处理; [0038] step S406, the processing from the background region obtained by dividing the color image;

[0039] 例如,可以对从所述彩色图像分割得到的背景区域进行的处理包括以下至少之一:虚拟场景替换、动画播放、横幅叠加、及图形演示文稿(例如,PPT)双流。 Processing [0039] For example, the background region obtained from the color image segmentation comprises at least one of the following: the virtual scene Alternatively, animation, banner overlay, and presentation graphics (e.g., the PPT) double.

[0040] 步骤S408,对背景区域经过处理的所述彩色图像进行会议电视通用处理,其中,所述会议电视通用处理包括:编码压缩以及网络发送。 [0040] step S408, the background region through videoconferencing general processing of the color image processing, wherein said general processing video conferencing comprising: code compression and network transmission.

[0041] 上述步骤是网真会议电视发送端在发送视频时的处理,在一个实施方式中,如果有网络媒体数据发送到该发送端,则所述方法还包括:对网络媒体数据进行接收并解码显 [0041] The above steps are telepresence television transmission side processing when transmitting the video, in one embodiment, if there is transmission data to the transmission media network side, the method further comprising: receiving media data network and decoding significantly

/Jn ο / Jn ο

[0042] 在一个实施方式中,步骤S404可以包括以下处理步骤: [0042] In one embodiment, the step S404 may include the following process steps:

[0043] 步骤I,对深度图像进行Candy算子边缘化,得到形状不规则的背景墙壁区域、前景区域、及噪声区域; [0043] Step I, the depth image Candy operator marginalized to obtain irregularly shaped wall background area, the foreground area, and a noise region;

[0044] 步骤2,对经Candy算子边缘化后的所述深度图像进行膨胀处理和腐蚀处理,消除所述噪声区域,从消除所述噪声区域后的所述深度图像中得到规则的背景区域; [0044] Step 2, the depth image of the operator by the marginalization Candy inflated and etching process, to eliminate the noise region, the background region resulting from elimination rule the depth image after the noise region ;

[0045] 步骤3,将所述规则的背景区域映射到所述彩色图像中,得到所述彩色图像的规则背景区域,并结合当前的所述彩色图像的色彩信息对所述彩色图像的规则背景区域中的前景区域进行精确化处理。 [0045] Step 3, the background region of the mapping rule to the color image, the background area of ​​the rule to obtain a color image, and color information in conjunction with the current rules of the color image of the color image background foreground region area precise process.

[0046] 需要注意上述步骤只是一种实施方式,其中的步骤I可以使用其它边缘检测算法(如sobel、prewitt等算子),步骤2也可以通过时间上连续多巾贞的深度信息序列,结合时空域静态和动态信息判定背景区域,而步骤3的精细化处理则可以通过双线性插值、三次样条插值等算法消除前面2个步骤没有准确定位的背景和前景的过渡区域。 [0046] Note that the above-described steps are only an embodiment, wherein in step I may use other edge detection algorithms (such as sobel, prewitt operator, etc.), Step 2 may be depth information by the time sequence of the continuous multi-towel Chen, binding when the static and dynamic information determination airspace background region, and the 3 step process is refined by bilinear interpolation, cubic spline interpolation algorithm eliminates two steps in front of the transition region without accurate positioning of the background and foreground.

[0047] 图5是本发明实施例的一种三屏网真布局的示意图,三屏网真是目前业界使用最广的网真布局方式。 [0047] FIG. 5 is a schematic diagram of a three-screen telepresence layout of an embodiment of the present invention, three-screen Telepresence is currently the most widely used in the industry real network layout. 这里的红外深度图像采集装置由红外发射装置与一个红外摄像头组成负责生成深度图。 Here the depth infrared image pickup device with an infrared emitting device composed of an infrared camera is responsible for generating a depth map. 每个屏都配有一个独立的高清彩色图像光学摄像头、一个红外深度图像采集装置,这样做是因为网真大视角需要多个摄像头分别采集各自范围内的图像,另外红外深度图像采集装置也需要相互分离来避免红外光干扰。 Each panel is equipped with a separate high-definition color image optical camera, an infrared depth image acquisition device, do so because telepresence large viewing by a plurality of cameras each captured image within the respective ranges, additional infrared depth image acquisition device also requires infrared light from each other to avoid interference. 各屏上的红外深度图像采集装置与彩色图像光学摄像头平行安装,这样得到的深度图像与彩色图像之间需要做一个水平位置匹配校正。 Infrared depth image acquisition device and the color image of the optical camera mounted in parallel on each screen needs to be done between a horizontal position and the depth image to match the color image thus obtained corrected. 下面以图5所示的布局为例,对本发明实施例提供的技术方案进行说明。 Below an example layout shown in FIG. 5, the present invention will be described the technical solutions provided by the embodiments.

[0048] 图6是根据本发明优选实施例的会议电视网真系统的会议视频的发送方法的流程图,如图6所示,主要包括以下步骤: [0048] FIG. 6 is a flowchart illustrating a method of transmitting a video conference session networks preferred embodiment of the present invention, the true system, shown in Figure 6, includes the following steps:

[0049] 步骤601,红外深度图像采集装置与彩色图像摄像头分别采集深度图像数据与彩色图像数据。 [0049] Step 601, the depth of the infrared image pickup device and the color image data of a camera image capture depth data and the color image, respectively.

[0050] 步骤602,对深度图像进行Candy算子边缘化,得到形状不规则的背景墙壁区域、前景区域、噪声区域,如图7所示。 [0050] Step 602, a depth image Candy operator marginalized to obtain irregularly shaped wall background area, the foreground area, the noise region, as shown in FIG.

[0051] 步骤603,对步骤602处理后的深度图进行膨胀和腐蚀处理消除噪声区域。 [0051] Step 603, the depth of view of the process step 602 is expanded and the etching treatment to eliminate noise region.

[0052] 步骤604,从步骤603处理后的深度图中得到规则的背景区域,如图8所示。 [0052] Step 604, the rules obtained from the background region in step 603 after processing the depth map, as shown in FIG.

[0053] 步骤605,将深度图规则背景区域映射到彩色图像中,得到彩色图规则背景区域。 [0053] Step 605, the depth map mapping rule to the color image background area, the background area obtained colored FIG rules. 由于两类摄像头水平并排放置且各自分辨率不同,映射关系为水平移位并做缩放处理。 Since two cameras placed side by side and each resolution level different mappings between a horizontal shift and make the scaling processing.

[0054] 步骤606,结合当前的彩色图的色彩信息对彩色图规则区域中的前景区域进行精确化处理,如图9所示。 [0054] Step 606, with the current color map color information in the color area in FIG rules foreground area precise process, as shown in FIG.

[0055] 步骤607,将彩色图规则区域中的背景部分图像使用用户指定的场景替换,前景部分保持不变。 [0055] Step 607, the background portion in the color image region in FIG rules specified by the user using the scenario Alternatively, the foreground remains unchanged. 规则区域的大小经过一定帧数后稳定不变,方便用户给定场景到固定目标宽高的缩放处理。 Area sizes rule after a certain number of frames constant and the user given the fixed target scene width and height scaling processing.

[0056] 步骤608,将经过了用户图像背景区域叠加处理后的彩色图像进行视频编码压缩、网络发送。 [0056] Step 608, after the color image of the background image after the user area superimposition processing video compression coding, network transmission.

[0057] 步骤609,对端接收码流后解码显示,收端用户可以看到经虚拟背景处理后的对方图像。 [0057] Step 609, after receiving the code stream of the decoder to the display, the user can see each other terminating the image processing by the virtual background.

[0058] 从以上的描述中,可以看出,通过上述实施例之一提供的技术方案,可以实现会议背景的虚拟现实。 [0058] From the above description, it can be seen that by the technical solution provided in one of the embodiments described above, the virtual reality may be achieved conference background. 将动画播放、横幅叠加、PPT双流等融和后,可以改变传统会议电视需要双流分屏显示附加信息的局面,背景的虚拟替换使得客户可以定制自己的背景环境,提高了用户体验。 The animation, banner overlay, PPT after double integration, etc., may need to change the traditional video conferencing double split-screen display additional information on the situation, replacing a virtual background enables customers to customize their context, improving the user experience.

[0059] 显然,本领域的技术人员应该明白,上述的本发明的各模块或各步骤可以用通用的计算装置来实现,它们可以集中在单个的计算装置上,或者分布在多个计算装置所组成的网络上,可选地,它们可以用计算装置可执行的程序代码来实现,从而,可以将它们存储在存储装置中由计算装置来执行,并且在某些情况下,可以以不同于此处的顺序执行所示出或描述的步骤,或者将它们分别制作成各个集成电路模块,或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。 [0059] Obviously, those skilled in the art should understand that the modules or steps of the present invention described above can be used general-purpose computing device, they can be integrated in a single computing device or distributed across multiple computing devices available on the Internet, optionally, they can be implemented with program codes executable by a computing device, so that, to be performed by a computing device stored in a storage means, and in some cases, may be different from this at step sequence shown or described, or they are made into integrated circuit modules, or by making them of a plurality of modules or steps in a single integrated circuit module. 这样,本发明不限制于任何特定的硬件和软件结合。 Thus, the present invention is not limited to any particular hardware and software combination.

[0060] 以上所述仅为本发明的优选实施例而已,并不用于限制本发明,对于本领域的技术人员来说,本发明可以有各种更改和变化。 [0060] The above description is only preferred embodiments of the present invention, it is not intended to limit the invention to those skilled in the art, the present invention may have various changes and variations. 凡在本发明的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本发明的保护范围之内。 Any modification within the spirit and principle of the present invention, made, equivalent substitutions, improvements, etc., should be included within the scope of the present invention.

Claims (9)

1.一种会议电视网真系统,其特征在于,包括:虚拟背景生成装置、编码发送装置和接收解码装置,其中, 所述虚拟背景生成装置包括: 彩色图像与深度图像获取装置,用于获取会议场景的彩色图像与深度图像; 背景分割装置,用于根据背景图像与前景图像在所述深度图像中的差异,对所述深度图像中的背景区域和前景区域进行初步分割,并将对所述深度图像的分割结果映射到所述彩色图像中,利用所述彩色图像中前景图像与背景图像的颜色差异,从所述彩色图像中分割出背景区域; 背景生成装置,用于对所述背景分割装置从所述彩色图像分割得到的背景区域进行处理; 编码发送装置,用于对经过所述背景生成装置处理后的所述彩色图像进行会议电视通用处理,其中,所述会议电视通用处理包括:编码压缩以及网络发送; 接收解码装置,用于对网 A true television conference system, characterized by comprising: a virtual background generation means, receiving and decoding the encoding apparatus transmitting apparatus, wherein the virtual background generating apparatus comprising: a color image and a depth image obtaining means for obtaining a color image and a depth image of the conference scene; background segmentation means for foreground and background image from the difference image in the depth image, the depth image background area and the foreground area initial segmentation, and of their said depth image segmentation result to the mapping of the color image, the color image using a color difference in the foreground image and the background image, the background region from the segmented color image; background generation means, background for the dividing means for processing the background color of the image segmentation region obtained; encipher transmission means for the color of the background image generation means through said television conference general processing process, wherein the process comprises a general conferencing : compression encoding and transmitting network; receiving-decoding means, for network 媒体数据进行接收并解码显示。 Receiving and decoding media data display.
2.根据权利要求1所述的会议电视网真系统,其特征在于,彩色图像与深度图像获取装置包括: 光学摄像头,用于采集会议场景的所述彩色图像; 红外摄像头,用于采集会议场景的所述深度图像。 2. The television conference system of the true claim 1, wherein the color image and depth image acquisition apparatus comprising: an optical camera, the color image acquisition session for a scene; infrared camera, for acquiring the conference scene the depth image.
3.权利要求1所述的会议电视网真系统,其特征在于,所述背景生成装置对从所述彩色图像分割得到的背景区域进行的处理包括以下至少之一:虚拟场景替换、动画播放、横幅叠加、及图形演示文稿双流。 True television conference system according to claim 1, wherein the processing means for generating said background obtained from the background region color image segmentation comprises at least one of the following: the virtual scene Alternatively, animation, banner overlay, and graphical presentations Shuangliu.
4.根据权利要求1至3中任一项所述的会议电视网真系统,其特征在于,所述背景分割装置包括: 深度图像处理装置,用于对所述深度图像进行边缘检测算子边缘化,得到形状不规则的背景墙壁区域、前景区域、及噪声区域; 深度图像分割装置,用于对所述深度图像处理装置处理后的所述深度图像进行膨胀处理和腐蚀处理,消除所述噪声区域,然后从消除所述噪声区域后的所述深度图像中得到规则的背景区域; 彩色图像分割装置,用于将所述深度图像的规则的背景区域映射到所述彩色图像中,得到所述彩色图像的规则背景区域,并结合当前的所述彩色图像的色彩信息对所述彩色图像的规则背景区域中的前景区域进行精确化处理。 The true television conference system according to any one of claims 1 to 3, characterized in that said background segmentation apparatus comprising: a depth image processing means for performing edge detection operator edge of the depth image , thereby obtaining irregular shaped wall background area, the foreground area, and a noise region; image segmentation means, for the depth of the depth image processing apparatus for processing image expansion process and etching process, to eliminate the noise region, and then obtain the depth image from the elimination of the background noise region regioregular; color image segmentation means, for mapping rule background area to the depth image of the color image, to give the rule background region color image, and color information in conjunction with the current rules of the color image of the color image of the background region foreground area precise process.
5.一种会议电视网真系统的图像处理方法,其特征在于,包括: 获取当前会议场景的彩色图像与深度图像; 根据背景图像与前景图像在所述深度图像中的差异,对所述深度图像中的背景区域和前景区域进行初步分割,并将对所述深度图像的分割结果映射到所述彩色图像中,利用所述彩色图像中前景图像与背景图像的颜色差异,从所述彩色图像中分割出背景区域; 从所述彩色图像分割得到的背景区域进行处理; 对背景区域经过处理的所述彩色图像进行会议电视通用处理,其中,所述会议电视通用处理包括:编码压缩以及网络发送。 A television conference system true image processing method comprising: obtaining a color image of the current conference scene depth image; difference in the depth image and the foreground image from the background image, the depth background area and the foreground area in the initial image is divided, and the color image is mapped to the divided result of the depth image, a color image using the color differences in the foreground image and the background image from the color image segmenting the background region; processing region from the background color of the image segmentation obtained; background region through videoconferencing general processing of the color image processing, wherein the video conferencing general processing comprising: a compression encoding and transmission network .
6.根据权利要求5所述的方法,其特征在于,所述方法还包括:对网络媒体数据进行接收并解码显示。 6. The method according to claim 5, characterized in that the method further comprises: the network receives and decodes the media data display.
7.根据权利要求5所述的方法,其特征在于,获取当前会议场景的彩色图像与深度图像,包括: 通过光学摄像头采集会议场景的所述彩色图像; 通过红外摄像头采集会议场景的所述深度图像。 7. The method as claimed in claim 5, characterized in that the current conference scene acquired color image and a depth image, comprising: the color image acquired by the optical conference scene camera; the depth of the conference scene acquired by the infrared camera image.
8.根据权利要求5所述的方法,其特征在于,对从所述彩色图像分割得到的背景区域进行的处理包括以下至少之一:虚拟场景替换、动画播放、横幅叠加、及图形演示文稿双流。 The method according to claim 5, characterized in that the processing of the background region obtained from the color image segmentation comprises at least one of the following: the virtual scene Alternatively, animation, banner overlay, and the presentation graphics double .
9.根据权利要求5至8中任一项所述的方法,其特征在于,根据背景图像与前景图像在所述深度图像中的差异,对所述深度图像中的背景区域和前景区域进行初步分割,并将对所述深度图像的分割结果映射到所述彩色图像中,利用所述彩色图像中前景图像与背景图像的颜色差异,从所述彩色图像中分割出背景区域,包括: 对所述深度图像进行边缘检测算子边缘化,得到形状不规则的背景墙壁区域、前景区域、及噪声区域; 对经边缘检测算子边缘化后的所述深度图像进行膨胀处理和腐蚀处理,消除所述噪声区域,从消除所述噪声区域后的所述深度图像中得到规则的背景区域; 将所述规则的背景区域映射到所述彩色图像中,得到所述彩色图像的规则背景区域,并结合当前的所述彩色图像的色彩信息对所述彩色图像的规则背景区域中的前景区域进行精确化 9. The method according to any one of any one of claim 8, wherein, based on the difference in the background image and the foreground image in the depth image, the depth image background area and the foreground area preliminary segmentation, segmentation results and map the depth image to the color image, the color image using a color difference in the foreground image and the background image, the color image segmentation from the background region, comprising: the said depth image marginal edge detection operator to obtain an irregularly shaped area wall background, foreground region and a noise region; the depth image by an edge detection operator is expanded marginalization process and etching process, to eliminate the said noise region, a background region from the elimination of the depth image obtained in the noise region rules; background region of the mapping rule to the color image, the background area of ​​the rule to obtain a color image, and combined the current color information of the color image is precise rules of the background region color image in the foreground region 理。 Management.
CN201210423899.9A 2012-10-30 2012-10-30 Video conference telepresence system and image processing method thereof CN103795961A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201210423899.9A CN103795961A (en) 2012-10-30 2012-10-30 Video conference telepresence system and image processing method thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201210423899.9A CN103795961A (en) 2012-10-30 2012-10-30 Video conference telepresence system and image processing method thereof

Publications (1)

Publication Number Publication Date
CN103795961A true CN103795961A (en) 2014-05-14

Family

ID=50671194

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201210423899.9A CN103795961A (en) 2012-10-30 2012-10-30 Video conference telepresence system and image processing method thereof

Country Status (1)

Country Link
CN (1) CN103795961A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104112275A (en) * 2014-07-15 2014-10-22 青岛海信电器股份有限公司 Image segmentation method and device
CN104168482A (en) * 2014-06-27 2014-11-26 中安消技术有限公司 Method and device for video coding and decoding
CN106447677A (en) * 2016-10-12 2017-02-22 广州视源电子科技股份有限公司 Image processing method and apparatus thereof
CN106469306A (en) * 2016-09-28 2017-03-01 深圳市优象计算技术有限公司 Multi-person image real-time extraction and synthesis method based on infrared structured light
CN106485720A (en) * 2016-11-03 2017-03-08 广州视源电子科技股份有限公司 Image processing method and apparatus

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07250312A (en) * 1994-03-08 1995-09-26 Fujitsu Ltd Portrait segmentation transmission method
US20020061131A1 (en) * 2000-10-18 2002-05-23 Sawhney Harpreet Singh Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery
US20080077953A1 (en) * 2006-09-22 2008-03-27 Objectvideo, Inc. Video background replacement system
CN101261722A (en) * 2008-01-17 2008-09-10 北京航空航天大学 Electronic police background intelligent management and automatic implementation system
CN101276476A (en) * 2008-05-14 2008-10-01 清华大学 Process for the separating prospect background of 2D cartoon animation
CN101459857A (en) * 2007-12-10 2009-06-17 深圳华为通信技术有限公司 Communication terminal and information system
WO2009146407A1 (en) * 2008-05-30 2009-12-03 General Instrument Corporation Replacing image information in a captured image
CN101610421A (en) * 2008-06-17 2009-12-23 深圳华为通信技术有限公司 Video communication method, video communication device and video communication system
CN102725773A (en) * 2009-12-02 2012-10-10 惠普发展公司,有限责任合伙企业 System and method of foreground-background segmentation of digitized images

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07250312A (en) * 1994-03-08 1995-09-26 Fujitsu Ltd Portrait segmentation transmission method
US20020061131A1 (en) * 2000-10-18 2002-05-23 Sawhney Harpreet Singh Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery
US20080077953A1 (en) * 2006-09-22 2008-03-27 Objectvideo, Inc. Video background replacement system
CN101459857A (en) * 2007-12-10 2009-06-17 深圳华为通信技术有限公司 Communication terminal and information system
CN101261722A (en) * 2008-01-17 2008-09-10 北京航空航天大学 Electronic police background intelligent management and automatic implementation system
CN101276476A (en) * 2008-05-14 2008-10-01 清华大学 Process for the separating prospect background of 2D cartoon animation
WO2009146407A1 (en) * 2008-05-30 2009-12-03 General Instrument Corporation Replacing image information in a captured image
CN101610421A (en) * 2008-06-17 2009-12-23 深圳华为通信技术有限公司 Video communication method, video communication device and video communication system
CN102725773A (en) * 2009-12-02 2012-10-10 惠普发展公司,有限责任合伙企业 System and method of foreground-background segmentation of digitized images

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104168482A (en) * 2014-06-27 2014-11-26 中安消技术有限公司 Method and device for video coding and decoding
CN104112275A (en) * 2014-07-15 2014-10-22 青岛海信电器股份有限公司 Image segmentation method and device
CN104112275B (en) * 2014-07-15 2017-07-04 青岛海信电器股份有限公司 Method and apparatus for generating viewpoint
CN106469306A (en) * 2016-09-28 2017-03-01 深圳市优象计算技术有限公司 Multi-person image real-time extraction and synthesis method based on infrared structured light
CN106447677A (en) * 2016-10-12 2017-02-22 广州视源电子科技股份有限公司 Image processing method and apparatus thereof
CN106485720A (en) * 2016-11-03 2017-03-08 广州视源电子科技股份有限公司 Image processing method and apparatus

Similar Documents

Publication Publication Date Title
Chen et al. Efficient depth image based rendering with edge dependent depth filter and interpolation
Kauff et al. An immersive 3D video-conferencing system using shared virtual team user environments
Gross et al. blue-c: a spatially immersive display and 3D video portal for telepresence
US9628755B2 (en) Automatically tracking user movement in a video chat application
CN102084650B (en) Telepresence system, method and video capture device
US9215408B2 (en) Method and apparatus maintaining eye contact in video delivery systems using view morphing
JP5107338B2 (en) Adaptive rendering of video content based on a further frame of the content
US7894633B1 (en) Image conversion and encoding techniques
US8471898B2 (en) Medial axis decomposition of 2D objects to synthesize binocular depth
US20100182403A1 (en) File format for encoded stereoscopic image/video data
CN102939763B (en) Calculating disparity for three-dimensional images
Huynh-Thu et al. The importance of visual attention in improving the 3D-TV viewing experience: Overview and new perspectives
Isgro et al. Three-dimensional image processing in the future of immersive media
EP2328125A1 (en) Image splicing method and device
JP5132690B2 (en) System and method for synthesizing the text and the three-dimensional content
US8279254B2 (en) Method and system for video conferencing in a virtual environment
US8588514B2 (en) Method, apparatus and system for processing depth-related information
US8908008B2 (en) Methods and systems for establishing eye contact and accurate gaze in remote collaboration
US20120011454A1 (en) Method and system for intelligently mining data during communication streams to present context-sensitive advertisements using background substitution
KR101468267B1 (en) Intermediate view synthesis and multi-view data signal extraction
US9094660B2 (en) Hierarchical hole-filling for depth-based view synthesis in FTV and 3D video
US20110080466A1 (en) Automated processing of aligned and non-aligned images for creating two-view and multi-view stereoscopic 3d images
ES2676055T3 (en) Effective image receiver for multiple views
CN101610421B (en) Video communication method, apparatus and system for
JP5654138B2 (en) 3d hybrid reality for the human-machine interface

Legal Events

Date Code Title Description
C06 Publication
C10 Entry into substantive examination
RJ01