CN114391259B - Information processing method, terminal device and storage medium - Google Patents

Information processing method, terminal device and storage medium Download PDF

Info

Publication number
CN114391259B
CN114391259B CN201980100362.9A CN201980100362A CN114391259B CN 114391259 B CN114391259 B CN 114391259B CN 201980100362 A CN201980100362 A CN 201980100362A CN 114391259 B CN114391259 B CN 114391259B
Authority
CN
China
Prior art keywords
depth information
video image
original depth
information
code stream
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201980100362.9A
Other languages
Chinese (zh)
Other versions
CN114391259A (en
Inventor
贾玉虎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Oppo Mobile Telecommunications Corp Ltd
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp Ltd filed Critical Guangdong Oppo Mobile Telecommunications Corp Ltd
Publication of CN114391259A publication Critical patent/CN114391259A/en
Application granted granted Critical
Publication of CN114391259B publication Critical patent/CN114391259B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

The invention discloses an information processing method, which comprises the following steps: acquiring original depth information corresponding to the depth information under the condition that the depth information of a target object is acquired through a depth information sensor, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information; acquiring video image data of the target object through an image sensor; and carrying out merging coding on the original depth information and the video image data to obtain a video image code stream, and outputting the video image code stream. The invention also discloses another information processing method, terminal equipment and a storage medium.

Description

信息处理方法、终端设备及存储介质Information processing method, terminal device and storage medium

技术领域Technical Field

本发明涉及计算机技术,尤其涉及一种信息处理方法、终端设备及存储介质。The present invention relates to computer technology, and in particular to an information processing method, terminal equipment and storage medium.

背景技术Background technique

在当今社会,越来越多的终端上都设置有摄像装置,从而方便用户可以随时随地的拍照或拍视频。在实际运用中,编码端通过现有的摄像装置采用飞行时间技术(Time OfLight,TOF)摄像头、双目摄像头等深度信息传感器获取目标对象的深度信息,在解码端通过深度信息进行目标对象的深度图像的恢复。但深度图像仅提供了目标对象的深度信息,并不能提高目标对象的视频图像的图像质量。In today's society, more and more terminals are equipped with cameras, so that users can take photos or videos anytime and anywhere. In actual use, the encoding end uses the existing camera device to use a time of flight technology (TOF) camera, a binocular camera and other depth information sensors to obtain the depth information of the target object, and the decoding end uses the depth information to restore the depth image of the target object. However, the depth image only provides the depth information of the target object and cannot improve the image quality of the video image of the target object.

发明内容Summary of the invention

本发明实施例提供一种信息处理方法、终端设备及存储介质,能够提高目标对象的视频图像的图像质量。The embodiments of the present invention provide an information processing method, a terminal device and a storage medium, which can improve the image quality of a video image of a target object.

第一方面,本发明实施例提供一种信息处理方法,包括:In a first aspect, an embodiment of the present invention provides an information processing method, including:

通过深度信息传感器获取目标对象的深度信息的情况下,获取所述深度信息对应的原始深度信息,所述原始深度信息表征所述深度信息传感器采集所述深度信息的采集状态或采集到的所述深度信息以外的信息;When the depth information of the target object is acquired through the depth information sensor, original depth information corresponding to the depth information is acquired, where the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information;

通过图像传感器获取所述目标对象的视频图像数据;Acquiring video image data of the target object through an image sensor;

对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流,并输出所述视频图像码流。The original depth information and the video image data are combined and encoded to obtain a video image code stream, and the video image code stream is output.

第二方面,本发明实施例提供一种信息处理方法,包括:In a second aspect, an embodiment of the present invention provides an information processing method, including:

接收视频图像码流,所述视频图像码流为对原始深度信息和视频图像数据进行合并编码得到的,所述原始深度信息是通过深度信息传感器获取目标对象的深度信息的情况下获取的,所述视频图像数据是通过图像传感器获取的所述目标对象的,所述原始深度信息表征所述深度信息传感器采集所述深度信息的采集状态或采集到的所述深度信息以外的信息;receiving a video image code stream, the video image code stream being obtained by merging and encoding original depth information and video image data, the original depth information being obtained when a depth information sensor acquires depth information of a target object, the video image data being acquired by an image sensor of the target object, the original depth information representing an acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information;

对所述视频图像码流进行解码,得到所述原始深度信息和所述视频图像数据对应的视频图像;Decoding the video image code stream to obtain the original depth information and the video image corresponding to the video image data;

对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像。Image processing is performed on the original depth information and the video image to obtain a target video image.

第三方面,本发明实施例提供一种终端设备,包括:In a third aspect, an embodiment of the present invention provides a terminal device, including:

第一获取单元,配置为通过深度信息传感单元获取目标对象的深度信息的情况下,获取所述深度信息对应的原始深度信息,所述原始深度信息表征所述深度信息传感单元采集所述深度信息的采集状态或采集到的所述深度信息以外的信息;A first acquisition unit is configured to acquire original depth information corresponding to the depth information when the depth information of the target object is acquired through the depth information sensing unit, wherein the original depth information represents an acquisition state of the depth information acquired by the depth information sensing unit or information other than the acquired depth information;

第二获取单元,配置为通过图像传感单元获取所述目标对象的视频图像数据;A second acquisition unit, configured to acquire video image data of the target object through an image sensing unit;

编码单元,配置为对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流;An encoding unit configured to combine and encode the original depth information and the video image data to obtain a video image code stream;

输出单元,配置为输出所述视频图像码流。The output unit is configured to output the video image code stream.

第四方面,本发明实施例提供一种终端设备,包括:In a fourth aspect, an embodiment of the present invention provides a terminal device, including:

接收单元,配置为接收视频图像码流,所述视频图像码流为对原始深度信息和视频图像数据进行合并编码得到的,所述原始深度信息是通过深度信息传感单元获取目标对象的深度信息的情况下获取的,所述视频图像数据是通过图像传感单元获取的所述目标对象的;所述原始深度信息表征所述深度信息传感单元采集所述深度信息的采集状态或采集到的所述深度信息以外的信息;a receiving unit configured to receive a video image code stream, wherein the video image code stream is obtained by merging and encoding original depth information and video image data, wherein the original depth information is obtained when a depth information sensing unit obtains depth information of a target object, and the video image data is obtained by an image sensing unit of the target object; the original depth information represents a state of the depth information sensing unit collecting the depth information or information other than the collected depth information;

解码单元,配置为对所述视频图像码流进行解码,得到所述原始深度信息和所述视频图像数据对应的视频图像;A decoding unit, configured to decode the video image code stream to obtain the original depth information and the video image corresponding to the video image data;

图像处理单元,配置为对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像。The image processing unit is configured to perform image processing on the original depth information and the video image to obtain a target video image.

第五方面,本发明实施例提供一种终端设备,包括处理器和配置为存储能够在处理器上运行的计算机程序的存储器,其中,所述处理器配置为运行所述计算机程序时,执行上述终端设备执行的信息处理方法的步骤。In a fifth aspect, an embodiment of the present invention provides a terminal device, comprising a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the information processing method executed by the above-mentioned terminal device when running the computer program.

第六方面,本发明实施例提供一种存储介质,存储有可执行程序,所述可执行程序被处理器执行时,实现上述终端设备执行的信息处理方法。In a sixth aspect, an embodiment of the present invention provides a storage medium storing an executable program, which, when executed by a processor, implements the information processing method executed by the above-mentioned terminal device.

本发明实施例提供的信息处理方法,包括:在编码端,通过深度信息传感器获取目标对象的深度信息的情况下,获取所述深度信息对应的原始深度信息;通过图像传感器获取所述目标对象的视频图像数据;对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流,并输出所述视频图像码流。在解码端,接收视频图像码流;对所述视频图像码流进行解码,得到所述原始深度信息和所述视频图像数据对应的视频图像;对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像。从而将深度传感器获得的原始深度信息,在编码端直接写入视频图像码流,并在解码端进行解析,解析得到原始深度信息对图像传感器采集的图像数据得到的视频图像,得到目标视频图像,提高视频图像的质量,给用户带来更真实的图像视频体验。The information processing method provided by the embodiment of the present invention includes: at the encoding end, when the depth information of the target object is obtained through the depth information sensor, the original depth information corresponding to the depth information is obtained; the video image data of the target object is obtained through the image sensor; the original depth information and the video image data are combined and encoded to obtain a video image code stream, and the video image code stream is output. At the decoding end, the video image code stream is received; the video image code stream is decoded to obtain the original depth information and the video image corresponding to the video image data; the original depth information and the video image are image processed to obtain a target video image. Thus, the original depth information obtained by the depth sensor is directly written into the video image code stream at the encoding end, and parsed at the decoding end, and the original depth information is parsed to obtain the video image obtained by the image data collected by the image sensor, and the target video image is obtained, thereby improving the quality of the video image and bringing a more realistic image and video experience to the user.

附图说明BRIEF DESCRIPTION OF THE DRAWINGS

图1A为本发明实施例信息处理系统的一种可选的结构示意图;FIG1A is a schematic diagram of an optional structure of an information processing system according to an embodiment of the present invention;

图1B为本发明实施例编码端的一种可选的结构示意图;FIG1B is a schematic diagram of an optional structure of an encoding end according to an embodiment of the present invention;

图1C为本发明实施例解码端的一种可选的结构示意图FIG. 1C is a schematic diagram of an optional structure of a decoding end according to an embodiment of the present invention.

图2为本发明实施例信息处理方法的一种可选的处理流程示意图;FIG2 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

图3为本发明实施例信息处理方法的一种可选的处理流程示意图;FIG3 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

图4为本发明实施例信息处理方法的一种可选的处理流程示意图;FIG4 is a schematic diagram of an optional processing flow of an information processing method according to an embodiment of the present invention;

图5为本发明实施例信息处理方法的一种可选的处理流程示意图;FIG5 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

图6为本发明实施例信息处理方法的一种可选的处理流程示意图;FIG6 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

图7为本发明实施例信息处理方法的一种可选的处理流程示意图;FIG7 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

图8A为本发明实施例信息处理系统的一种可选的框架示意图;FIG8A is a schematic diagram of an optional framework of an information processing system according to an embodiment of the present invention;

图8B为本发明实施例信息处理系统的一种可选的框架示意图;FIG8B is a schematic diagram of an optional framework of an information processing system according to an embodiment of the present invention;

图9A为本发明实施例解码端的一种可选的框架示意图;FIG9A is a schematic diagram of an optional framework of a decoding end according to an embodiment of the present invention;

图9B为本发明实施例解码端的一种可选的框架示意图;FIG9B is a schematic diagram of an optional framework of a decoding end according to an embodiment of the present invention;

图9C为本发明实施例解码端的一种可选的框架示意图;FIG9C is a schematic diagram of an optional framework of a decoding end according to an embodiment of the present invention;

图9D为本发明实施例解码端的一种可选的框架示意图;FIG9D is a schematic diagram of an optional framework of a decoding end according to an embodiment of the present invention;

图10为本发明实施例对原始深度信息进行采样的采样示意图;FIG10 is a schematic diagram of sampling of original depth information according to an embodiment of the present invention;

图11为本发明实施的终端设备的一个可选的结构示意图;FIG11 is a schematic diagram of an optional structure of a terminal device implemented in the present invention;

图12是本发明实施例终端设备的一个可选的结构示意图;12 is a schematic diagram of an optional structure of a terminal device according to an embodiment of the present invention;

图13是本发明实施例提供的电子设备的一个可选的结构示意图。FIG. 13 is a schematic diagram of an optional structure of an electronic device provided in an embodiment of the present invention.

具体实施方式Detailed ways

为了能够更加详尽地了解本发明实施例的特点和技术内容,下面结合附图对本发明实施例的实现进行详细阐述,所附附图仅供参考说明之用,并非用来限定本发明实施例。In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present invention, the implementation of the embodiments of the present invention is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not intended to limit the embodiments of the present invention.

在对本发明实施例提供的信息处理方法进行详细说明之前,先对深度图像过程进行介绍。Before describing in detail the information processing method provided by the embodiment of the present invention, the depth image process is first introduced.

深度图像(depth image)也被称为距离影像(range image),是指将从图像传感器到场景中各点的距离(深度)作为像素值的图像,能够直接反映了目标对象可见表面的几何形状。深度图像经过坐标转换可以计算为点云数据,有规则及必要信息的点云数据也可以反算为深度图像数据。A depth image, also known as a range image, is an image that uses the distance (depth) from the image sensor to each point in the scene as a pixel value, and can directly reflect the geometry of the visible surface of the target object. The depth image can be calculated as point cloud data after coordinate transformation, and point cloud data with rules and necessary information can also be inversely calculated into depth image data.

这里,编码端对深度信息传感器捕获形成的深度图像进行视频编码,得到编码后的深度图像信息,解码器端仅能够根据编码后的深度图像信息将深度图像进行恢复。但是,深度信息传感器接收到的信息量远超深度图像的信息量。这些海量的信息在生成深度图像后,作为冗余进行抛弃。因此,在上述方案中,并未考虑到这些冗余信息的其他作用,如解码端的图像增强等。Here, the encoder performs video encoding on the depth image captured by the depth information sensor to obtain the encoded depth image information, and the decoder can only restore the depth image based on the encoded depth image information. However, the amount of information received by the depth information sensor far exceeds the amount of information in the depth image. After the depth image is generated, this massive amount of information is discarded as redundancy. Therefore, in the above scheme, other functions of this redundant information, such as image enhancement at the decoding end, are not taken into account.

基于上述问题,本发明实施例提供一种信息处理方法,本发明实施例的信息处理方法可以应信息处理系统,Based on the above problems, an embodiment of the present invention provides an information processing method. The information processing method of the embodiment of the present invention can be applied to an information processing system.

示例性的,本发明实施例应用的信息处理系统100,可为如图1A所示。该信息处理100可以包括编码端101和解码端102。编码端101用于对采集视频图像数据和原始深度信息,并对视频图像数据和原始深度信息进行编码,形成视频图像码流。解码端120用于对图像视频码流进行解码,得到视频图像数据和原始深度信息,并对视频图像数据和原始深度信息进行图像处理,得到目标视频图像。Exemplarily, the information processing system 100 applied in the embodiment of the present invention may be as shown in FIG1A. The information processing 100 may include an encoding end 101 and a decoding end 102. The encoding end 101 is used to collect video image data and original depth information, and encode the video image data and the original depth information to form a video image code stream. The decoding end 120 is used to decode the image video code stream to obtain the video image data and the original depth information, and perform image processing on the video image data and the original depth information to obtain the target video image.

编码端101和解码端102可包括包含台式计算机、移动计算装置、笔记本(例如,膝上型)计算机、平板计算机、机顶盒、智能电话等手持机、电视、相机、显示装置、数字媒体播放器、视频游戏控制台、车载计算机,或其类似者。The encoding end 101 and the decoding end 102 may include a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a handheld device such as a smart phone, a television, a camera, a display device, a digital media player, a video game console, a car computer, or the like.

如图1A所示,解码端102可经由链路103接收来自编码端101编码后的视频图像码流。链路103可包括能够将视频图像码流从编码端101移动到解码端102的一个或多个媒体及/或装置。As shown in FIG1A , the decoding end 102 may receive the encoded video image code stream from the encoding end 101 via the link 103. The link 103 may include one or more media and/or devices capable of moving the video image code stream from the encoding end 101 to the decoding end 102.

在一示例中,链路103可包括使编码端101能够实时地将编码后的视频数据直接发送到解码端102的一个或多个通信媒体。在此实例中,编码端101可根据通信标准(例如,无线通信协议)来调制视频图像码流,且可将调制后的视频图像码流发送到解码端102。In one example, the link 103 may include one or more communication media that enable the encoder 101 to directly send the encoded video data to the decoder 102 in real time. In this instance, the encoder 101 may modulate the video image code stream according to a communication standard (e.g., a wireless communication protocol), and may send the modulated video image code stream to the decoder 102.

在一示例中,链路103可包含存储有编码端101形成的视频图像码流的存储媒体。在此示例中,解码端102可经由磁盘存取或卡存取来存取存储媒体。存储媒体可包含多种本地存取式数据存储媒体,例如蓝光光盘、DVD、CD-ROM、快闪存储器,或用于存储视频图像码流的其它合适数字存储媒体。In one example, link 103 may include a storage medium storing a video image code stream formed by encoding end 101. In this example, decoding end 102 may access the storage medium via disk access or card access. The storage medium may include a variety of local access data storage media, such as a Blu-ray disc, DVD, CD-ROM, flash memory, or other suitable digital storage media for storing video image code streams.

在又一示例中,链路103可包含文件服务器或存储由编码端101形成的视频图像码流的另一中间存储装置。在此示例中,解码端102可经由流式传输或下载来存取存储于文件服务器或其它中间存储装置处的视频图像码流。文件服务器可以是能够存储视频图像码流且将视频图像码流发送到解码端102的服务器类型。文件服务器包含web服务器(例如,用于网站)、文件传送协议服务器、网络附加存储装置,及本地磁盘驱动器等。In another example, link 103 may include a file server or another intermediate storage device storing the video image code stream formed by encoding end 101. In this example, decoding end 102 may access the video image code stream stored at the file server or other intermediate storage device via streaming or downloading. The file server may be a server type capable of storing the video image code stream and sending the video image code stream to decoding end 102. The file server includes a web server (for example, for a website), a file transfer protocol server, a network attached storage device, and a local disk drive, etc.

解码端102可经由标准数据连接(例如,因特网连接)来存取视频图像码流。数据连接的实例类型包含适合于存取存储于文件服务器上的视频图像码流的无线链路(例如,Wi-Fi连接)、有线连接(例如,DSL、缆线调制解调器等),或两者的组合。The decoding end 102 can access the video image code stream via a standard data connection (e.g., an Internet connection). Example types of data connections include a wireless link (e.g., a Wi-Fi connection) suitable for accessing the video image code stream stored on a file server, a wired connection (e.g., DSL, cable modem, etc.), or a combination of the two.

如图1B所示,编码端101包括:深度信息传感器1011、图像传感器1012和视频图像编码器1013,深度信息传感器1011用于获取原始深度信息,图像传感器1012用于获取视频图像数据,视频图像编码器1013用于对原始深度信息和视频图像数据进行编码,形成视频图像码流。As shown in FIG. 1B , the encoding end 101 includes: a depth information sensor 1011, an image sensor 1012 and a video image encoder 1013. The depth information sensor 1011 is used to obtain original depth information, the image sensor 1012 is used to obtain video image data, and the video image encoder 1013 is used to encode the original depth information and video image data to form a video image code stream.

如图1C所示,解码端102包括:视频图像解码器1021和图像处理器1022,视频图像解码器1021用于对视频图像码流进行解码,得到原始深度信息和视频图像数据对应的视频图像,图像处理器1022用于对原始深度信息和视频图像进行处理,得到目标视频图像。这里,原始深度信息作用于视频图像,能够得到清晰度高、噪声低等高质量的视频图像。As shown in FIG1C , the decoding end 102 includes: a video image decoder 1021 and an image processor 1022. The video image decoder 1021 is used to decode the video image code stream to obtain the original depth information and the video image corresponding to the video image data. The image processor 1022 is used to process the original depth information and the video image to obtain the target video image. Here, the original depth information acts on the video image to obtain a high-quality video image with high definition and low noise.

在一示例中,如图1C所示,解码端102还包括:深度图像生成器1023,用于基于原始深度信息生成深度图像。In an example, as shown in FIG. 1C , the decoding end 102 further includes: a depth image generator 1023 for generating a depth image based on original depth information.

本发明实施例提供的信息处理方法的一种可选处理流程,应用于编码端,如图2所示,包括以下步骤:An optional processing flow of the information processing method provided in an embodiment of the present invention is applied to an encoding end, as shown in FIG2 , and includes the following steps:

S201,通过深度信息传感器获取目标对象的深度信息的情况下,获取所述深度信息对应的原始深度信息。S201 : When acquiring depth information of a target object through a depth information sensor, acquire original depth information corresponding to the depth information.

所述原始深度信息表征所述深度信息传感器采集所述深度信息的采集状态或采集到的所述深度信息以外的信息。The original depth information represents a state in which the depth information sensor collects the depth information or information other than the collected depth information.

深度信息传感器为能够采集目标对象的深度信息的传感器。在一示例中,深度信息传感器为采用TOF测距方法的TOF模组。在一示例中,深度信息传感器为双目摄像头。The depth information sensor is a sensor capable of collecting depth information of a target object. In one example, the depth information sensor is a TOF module using a TOF ranging method. In one example, the depth information sensor is a binocular camera.

本发明实施例中,编码端在深度信息传感器采集深度信息的情况下,通过深度信息传感器获取原始深度信息,所述原始深度信息包括以下至少之一:电荷信息、相位信息和所述深度信息传感器的属性参数。其中,电荷信息、相位信息为深度信息传感器采集到的所述深度信息以外的信息,深度信息传感器的属性参数表征所述深度信息传感器的采集所述深度信息的采集状态。In the embodiment of the present invention, when the depth information sensor collects depth information, the encoder obtains original depth information through the depth information sensor, and the original depth information includes at least one of the following: charge information, phase information, and attribute parameters of the depth information sensor. Among them, the charge information and phase information are information other than the depth information collected by the depth information sensor, and the attribute parameters of the depth information sensor represent the collection state of the depth information sensor in collecting the depth information.

以原始深度信息为电荷信息为例,一个时间点的电荷信息可体现为一幅电荷图像。这里,获取深度信息传感器采集深度信息时接收的光信号,并将光信号通过光电转换转换为电信号,电信号经过量化后生成电荷图像。Taking the original depth information as charge information as an example, the charge information at a time point can be reflected as a charge image. Here, the depth information sensor receives the light signal when collecting the depth information, and converts the light signal into an electrical signal through photoelectric conversion, and the electrical signal is quantified to generate a charge image.

以原始深度信息为相位信息为例,一个时间点的相位信息可体现为一幅相位图像。Taking the original depth information as phase information as an example, the phase information at a time point can be reflected as a phase image.

以原始深度信息为深度信息传感器的属性参数为例,原始深度信息可包括:温度、位姿等属性参数。Taking the original depth information as the attribute parameter of the depth information sensor as an example, the original depth information may include attribute parameters such as temperature and posture.

S202,通过图像传感器获取所述目标对象的视频图像数据。S202: Acquire video image data of the target object through an image sensor.

编码端在图像预览或视频拍摄场景下通过图像传感器获取目标对象的视频图像数据,这里,视频图像数据包括至少一帧图像帧。The encoding end obtains video image data of the target object through an image sensor in an image preview or video shooting scenario. Here, the video image data includes at least one image frame.

本发明实施例中,原始深度信息与视频帧一一对应。在一示例中,不同的电荷图像或相位图像分别对应不同的图像帧。In the embodiment of the present invention, the original depth information corresponds to the video frame one by one. In one example, different charge images or phase images correspond to different image frames.

S203,对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流,并输出所述视频图像码流。S203: Combine and encode the original depth information and the video image data to obtain a video image code stream, and output the video image code stream.

编码端通过视频图像编码器对原始深度信息和视频图像数据进行合并编码,视频图像编码器输出视频图像码流,并将视频图像编码器输出的视频图像码流输出至解码端,使得解码端基于原始深度信息对视频图像数据对应的视频图像进行图像处理。The encoding end combines and encodes the original depth information and the video image data through a video image encoder, and the video image encoder outputs a video image code stream, and outputs the video image code stream output by the video image encoder to the decoding end, so that the decoding end performs image processing on the video image corresponding to the video image data based on the original depth information.

可选地,视频图像编码器采用视频图像编解码协议,对视频图像帧或原始深度信息进行编码,得到视频图像码流信息;视频编解码协议可以为H.264、H.265、H.266、VP9或AV1等。Optionally, the video image encoder adopts a video image coding and decoding protocol to encode the video image frame or the original depth information to obtain video image code stream information; the video coding and decoding protocol may be H.264, H.265, H.266, VP9 or AV1, etc.

可选地,,采用视频图像编解码协议对原始深度信息和图像视频数据进行编码,此时,视频图像信息携带的数据不包括深度信息。Optionally, a video image coding and decoding protocol is used to encode the original depth information and the image video data. In this case, the data carried by the video image information does not include the depth information.

在本发明实施例中,在视频图像信息携带的数据不包括深度信息的情况下,编码端可仅获取深度信息传感器在获取深度信息时的原始深度信息,不获取深度信息传感器所采集到的深度信息,或将采集的深度信息丢弃。In an embodiment of the present invention, when the data carried by the video image information does not include depth information, the encoding end may only obtain the original depth information when the depth information sensor obtains the depth information, and may not obtain the depth information collected by the depth information sensor, or may discard the collected depth information.

可选地,采用视频图像编解码协议对原始深度信息和图像视频数据进行编码,并采用视频图像编解码协议对深度信息传感器采集的深度信息进行编码,此时,视频图像信息携带的数据包括:原始深度信息、深度信息和视频图像数据。Optionally, the original depth information and image video data are encoded using a video image encoding and decoding protocol, and the depth information collected by the depth information sensor is encoded using a video image encoding and decoding protocol. At this time, the data carried by the video image information includes: original depth information, depth information and video image data.

本发明实施例中,对深度信息传感器所采集的深度信息的处理不进行任何限定。In the embodiment of the present invention, no limitation is imposed on the processing of the depth information collected by the depth information sensor.

可选地,视频图像编码器采用行业标准或特定组织的特定标准,对视频图像帧或原始深度信息进行编码,得到视频图像码流。Optionally, the video image encoder adopts an industry standard or a specific standard of a specific organization to encode the video image frame or the original depth information to obtain a video image code stream.

编码端可将全部的原始深度信息输入视频图像编码器,以对全部的原始深度信息进行编码,也可仅将部分原始深度信息输入视频图像编码器,以对部分原始深度信息编码。可选地,部分原始深度信息为指定图像帧对应的原始深度信息。可选地,部分原始深度信息为指定图像位置对应的原始深度信息。The encoding end may input all the original depth information into the video image encoder to encode all the original depth information, or may input only part of the original depth information into the video image encoder to encode part of the original depth information. Optionally, the part of the original depth information is the original depth information corresponding to the specified image frame. Optionally, the part of the original depth information is the original depth information corresponding to the specified image position.

以部分原始深度信息为指定图像视频对应的原始深度信息为例,所述对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流,包括:对所述视频图像数据对应的图像帧中指定图像帧对应的原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流。Taking part of the original depth information as the original depth information corresponding to the specified image video as an example, the original depth information and the video image data are merged and encoded to obtain a video image code stream, including: merging the original depth information corresponding to a specified image frame in the image frame corresponding to the video image data and the video image data to obtain a video image code stream.

可选地,指定图像帧为视频图像数据对应的图像帧中的一个图像帧。可选地,指定图像帧包括视频图像数据对应的图像帧中的多个图像帧。Optionally, the designated image frame is one of the image frames corresponding to the video image data. Optionally, the designated image frame includes a plurality of image frames corresponding to the video image data.

本发明实施例对指定图像帧的数量不进行任何限制。The embodiment of the present invention does not impose any restriction on the number of designated image frames.

编码端仅对指定图像帧对应的原始深度信息和所述视频图像数据进行合并编码,对视频图像数据对应的图像帧中指定图像帧以外非指定视频帧对应的原始深度信息不进行编码。The encoding end only performs combined encoding on the original depth information corresponding to the designated image frame and the video image data, and does not encode the original depth information corresponding to non-designated video frames other than the designated image frame in the image frames corresponding to the video image data.

以部分原始深度信息为指定图像位置对应的原始深度信息为例,所述对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流,包括:对指定图像位置对应的原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流。Taking part of the original depth information as the original depth information corresponding to the specified image position as an example, the original depth information and the video image data are merged and encoded to obtain a video image code stream, including: the original depth information corresponding to the specified image position and the video image data are merged and encoded to obtain a video image code stream.

指定图像位置为图像采集范围内指定点所在的位置。可选地,指定图像位置为图像采集范围内指定区域所在的位置。本发明实施例对指定图像位置的范围大小或所在的位置不进行任何限定。The designated image position is the position of the designated point within the image acquisition range. Optionally, the designated image position is the position of the designated area within the image acquisition range. The embodiment of the present invention does not impose any limitation on the range size or position of the designated image position.

编码端仅对指定图像位置对应的原始深度信息和所述视频图像数据进行合并编码,对图像帧中指定图像位置以外非指定视频位置对应的原始深度信息不进行编码。The encoding end only performs combined encoding on the original depth information corresponding to the designated image position and the video image data, and does not encode the original depth information corresponding to the non-designated video position other than the designated image position in the image frame.

本发明实施例中,编码端对所述原始深度信息和所述视频图像数据进行合并编码的编码方式包括以下之一:In the embodiment of the present invention, the encoding mode in which the encoding end performs combined encoding on the original depth information and the video image data includes one of the following:

编码方式一、根据所述原始深度信息和所述视频图像数据的相关性,对所述原始深度信息和所述视频图像数据进行混合编码,得到视频图像码流;Coding method 1: performing mixed coding on the original depth information and the video image data according to the correlation between the original depth information and the video image data to obtain a video image code stream;

编码方式二、分别对所述原始深度信息和所述视频图像数据进行独立编码,得到包括第一码流和第二码流的图像视频码流,其中,所述第一码流为所述原始深度信息编码后得到的码流,所述第二码流为所述图像视频数据编码后得到的码流。Coding method 2: independently encode the original depth information and the video image data respectively to obtain an image video code stream including a first code stream and a second code stream, wherein the first code stream is a code stream obtained after encoding the original depth information, and the second code stream is a code stream obtained after encoding the image video data.

在编码方式一中,对原始深度信息和视频图像数据进行编码所采用编解码协议相同。In the first encoding method, the original depth information and the video image data are encoded using the same encoding and decoding protocol.

可选地,在编码方式一中,视频图像码流中的编码信息为对原始深度信和视频图像数据进行联合编码得到的混合编码信息。其中,视频图像编码器可利用原始深度信息和图像视频数据的空间相关性或时间相关性等,对原始深度信和视频图像数据进行联合编码。Optionally, in encoding mode 1, the encoding information in the video image code stream is mixed encoding information obtained by jointly encoding the original depth information and the video image data. The video image encoder may jointly encode the original depth information and the video image data by utilizing the spatial correlation or temporal correlation of the original depth information and the video image data.

可选地,在编码方式一中,将所述原始深度信息对应的第一编码信息写入所述视频图像数据对应的第二编码信息的指定位置处。可选地,指定位置可以为图像信息头、序列信息头、附加参数集或其他任意位置。Optionally, in encoding mode 1, the first encoding information corresponding to the original depth information is written into a designated position of the second encoding information corresponding to the video image data. Optionally, the designated position may be an image information header, a sequence information header, an additional parameter set or any other position.

可选地,在编码方式一中,利用原始深度信息和图像视频数据的空间相关性或时间相关性等,对原始深度信息进行编码,得到第一编码信息,并对视频图像数据进行编码,得到第二编码信息,并将第一编码信息写入第一编码信息的指定位置处,得到视频图像码流。Optionally, in encoding method one, the original depth information is encoded using spatial correlation or temporal correlation between the original depth information and the image video data to obtain first encoding information, and the video image data is encoded to obtain second encoding information, and the first encoding information is written into a specified position of the first encoding information to obtain a video image code stream.

在编码方式二中,对原始深度信息所采用编解码协议与对视频图像数据进行编码所采用编解码协议独立。可选地,对原始深度信息所采用编解码协议与对视频图像数据进行编码所采用编解码协议相同。可选地,对原始深度信息所采用编解码协议与对视频图像数据进行编码所采用编解码协议不同。In the second encoding method, the codec protocol used for the original depth information is independent of the codec protocol used for encoding the video image data. Optionally, the codec protocol used for the original depth information is the same as the codec protocol used for encoding the video image data. Optionally, the codec protocol used for the original depth information is different from the codec protocol used for encoding the video image data.

在一实施例中,如图3所示,在S203之前,包括:In one embodiment, as shown in FIG3 , before S203 , the following steps are included:

204A,对所述原始深度信息进行预处理。204A, pre-process the original depth information.

S203中对原始深度信息和视频图像数据进行合并编码,可执行为S203A:对经过预处理的原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流。The original depth information and the video image data are combined and encoded in S203, which can be executed as S203A: the pre-processed original depth information and the video image data are combined and encoded to obtain a video image code stream.

本发明是实施例中,预处理可以为滤波、去噪、信号放大等相位校准等方式中的一种或两种处理方式,还可以为其他处理方式,具体的预处理可根据实际情况确定,本发明实施例对此不做限定。In the embodiments of the present invention, the preprocessing may be one or two processing methods such as filtering, denoising, signal amplification, phase calibration, etc., or other processing methods. The specific preprocessing may be determined according to actual conditions, and the embodiments of the present invention do not limit this.

可选地,编码端通过深度信息传感器对原始深度信息进行预处理。Optionally, the encoding end preprocesses the original depth information through a depth information sensor.

在一实施例中,如图4所示,在S203之前,包括:In one embodiment, as shown in FIG4 , before S203 , the following steps are included:

204B,对所述原始深度信息进行冗余消除处理,以消除所述原始深度信息中的冗余信息。204B: Perform redundancy elimination processing on the original depth information to eliminate redundant information in the original depth information.

S203中对原始深度信息和视频图像数据进行合并编码,可执行为S203B:对经过冗余消除处理的原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流。The original depth information and the video image data are combined and encoded in S203, which can be executed as S203B: the original depth information after redundancy elimination processing and the video image data are combined and encoded to obtain a video image code stream.

编码端通过对原始深度信息进行冗余消除处理,能够消除原始深度信息中的冗余信息,从而压缩原始深度信息的信息量,使得减小视频数据码流的大小。The encoding end can eliminate redundant information in the original depth information by performing redundancy elimination processing on the original depth information, thereby compressing the information amount of the original depth information, thereby reducing the size of the video data code stream.

本发明实施例中,所述根据对所述原始深度信息进行冗余消除处理,包括以下至少之一:In the embodiment of the present invention, performing redundancy elimination processing on the original depth information includes at least one of the following:

基于相位相关性对所述原始深度信息冗余消除处理;Eliminate redundancy of the original depth information based on phase correlation;

基于空间相关性对所述原始深度信息进行冗余消除处理;Performing redundancy elimination processing on the original depth information based on spatial correlation;

基于时间相关性对所述原始深度信息进行冗余消除处理;Performing redundancy elimination processing on the original depth information based on time correlation;

基于指定深度对所述原始深度信息进行冗余消除处理;Performing redundancy elimination processing on the original depth information based on the specified depth;

基于频域相关性对所述原始深度信息进行冗余消除处理;Performing redundancy elimination processing on the original depth information based on frequency domain correlation;

基于编码二进制数据之间的相关性对所述原始深度信息的编码比特进行冗余消除处理。Redundancy elimination processing is performed on the coded bits of the original depth information based on the correlation between the coded binary data.

可选地,将原始深度信息转换到频域,基于频域相关性对转换为频域的所述原始深度信息进行冗余消除处理。Optionally, the original depth information is converted into a frequency domain, and redundancy elimination processing is performed on the original depth information converted into the frequency domain based on frequency domain correlation.

可选地,指定深度为目标对象所处的场景敏感的深度的范围,基于指定深度对所述原始深度信息进行冗余消除处理,将场景敏感的深度的范围之外的深度对应的原始深度信息作为冗余进行消除。Optionally, the designated depth is a scene-sensitive depth range where the target object is located, and redundancy elimination processing is performed on the original depth information based on the designated depth, and original depth information corresponding to depths outside the scene-sensitive depth range is eliminated as redundancy.

可选地,对原始深度信息进行熵编码,对基于编码二进制数据之间的相关性对原始深度信息的熵编码结果的编码比特进行冗余消除处理。Optionally, entropy coding is performed on the original depth information, and redundancy elimination processing is performed on coded bits of the entropy coding result of the original depth information based on the correlation between the coded binary data.

以基于空间相关性对所述原始深度信息进行冗余消除处理为例,编码端的原始深度信息对应至少一个视点;从至少一个视点中确定间隔视点,将与间隔视点对应的原始深度信息,作为间隔原始深度信息;将原始深度信息中间隔原始深度以外的原始深度信息作为冗余消除,对间隔原始深度信息和视频图像数据进行合并编码,得到视频数据码流。Taking the redundancy elimination processing of the original depth information based on spatial correlation as an example, the original depth information at the encoding end corresponds to at least one viewpoint; an interval viewpoint is determined from the at least one viewpoint, and the original depth information corresponding to the interval viewpoint is used as the interval original depth information; the original depth information other than the interval original depth in the original depth information is eliminated as redundancy, and the interval original depth information and the video image data are merged and encoded to obtain a video data code stream.

以基于时间相关性对所述原始深度信息进行冗余消除处理为例,编码端获取到一端时间内的原始深度信息,并基于采样间隔对获取到的原始深度信息进行采样,并保留采样后的原始深度信息,将获取的原始深度信息中采样后的原始深度信息以外的原始深度信息作为冗余消除,对采样得到的原始深度信息和视频图像数据进行合并编码,得到视频数据码流。Taking the redundancy elimination processing of the original depth information based on time correlation as an example, the encoding end obtains the original depth information within one end time, and samples the obtained original depth information based on the sampling interval, and retains the sampled original depth information, and eliminates the original depth information other than the sampled original depth information in the obtained original depth information as redundancy, and merges and encodes the sampled original depth information and the video image data to obtain a video data code stream.

本发明实施例提供的信息处理方法的一种可选处理流程,应用于解码端,如图5所示,包括以下步骤:An optional processing flow of the information processing method provided in an embodiment of the present invention is applied to a decoding end, as shown in FIG5 , and includes the following steps:

S501,接收视频图像码流。S501, receiving a video image code stream.

解码端通过链路接收编码端发送的视频图像码流。所述视频图像码流为对原始深度信息和视频图像数据进行合并编码得到的,所述原始深度信息是通过深度信息传感器获取目标对象的深度信息的情况下获取的,所述视频图像数据是通过图像传感器获取的所述目标对象的;所述原始深度信息表征所述深度信息传感器采集所述深度信息的采集状态或采集到的所述深度信息以外的信息。The decoding end receives the video image code stream sent by the encoding end through the link. The video image code stream is obtained by merging and encoding the original depth information and the video image data, the original depth information is obtained when the depth information sensor obtains the depth information of the target object, and the video image data is obtained by the image sensor of the target object; the original depth information represents the acquisition state of the depth information collected by the depth information sensor or information other than the acquired depth information.

S502,对所述视频图像码流进行解码,得到所述原始深度信息和所述视频图像数据对应的视频图像。S502: Decode the video image code stream to obtain the original depth information and the video image corresponding to the video image data.

这里,通过视频图像解码器对所述视频图像码流进行解码,得到所述原始深度信息和所述视频图像数据对应的视频图像。Here, the video image code stream is decoded by a video image decoder to obtain the original depth information and the video image corresponding to the video image data.

解码端将接收的视频图像码流发送至视频图像解码器,视频图像解码器对视频图像码流进行解码。The decoding end sends the received video image code stream to the video image decoder, and the video image decoder decodes the video image code stream.

可选地,视频图像解码器和编码端的视频编码器所支持的视频图像编解码协议相同。Optionally, the video image decoder and the video encoder at the encoding end support the same video image encoding and decoding protocol.

可选地,视频图像编码器对原始深度信息和视频图像数据进行混合编码的情况下,Optionally, when the video image encoder performs hybrid encoding on the original depth information and the video image data,

视频图像解码器对视频图像码流进行混合解码,得到原始深度信息和所述视频图像数据对应的视频图像。The video image decoder performs hybrid decoding on the video image code stream to obtain the original depth information and the video image corresponding to the video image data.

可选地,视频图像编码器对原始深度信息和视频图像数据独立进行编码的情况下,视频图像解码器对视频图像码流中的第一码流和第二码流进行独立解码,对视频图像数据的第一码流进行解码,得到原始深度信息,并对第二码流进行解码,得到所述视频图像数据对应的视频图像。这里,视频图像数据对应的视频图像也可称为原始视频图像。图像视频码流解码得到的原始视频图像可包括一帧或多帧原始视频图像。Optionally, when the video image encoder encodes the original depth information and the video image data independently, the video image decoder independently decodes the first code stream and the second code stream in the video image code stream, decodes the first code stream of the video image data to obtain the original depth information, and decodes the second code stream to obtain the video image corresponding to the video image data. Here, the video image corresponding to the video image data may also be referred to as the original video image. The original video image obtained by decoding the video code stream may include one or more frames of original video images.

S503,对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像。S503: Perform image processing on the original depth information and the video image to obtain a target video image.

通过图像处理器对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像。The original depth information and the video image are processed by an image processor to obtain a target video image.

解码端解码得到原始深度信息和视频图像数据后,通过图像处理器,将原始深度信息作用于视频图像,对视频图像进行图像处理,得到目标视频图像。目标视频图像的图像质量高于原始视频图像。After the decoding end obtains the original depth information and video image data through decoding, the original depth information is applied to the video image through the image processor, and the video image is processed to obtain the target video image. The image quality of the target video image is higher than that of the original video image.

可选地,解码端可基于相位相关性、空间相关性、时间相关性、指定深度、频域相关性、编码二进制数据之间的相关性对解码得到的原始深度信息进行冗余恢复,得到冗余恢复后的原始深度信息,基于冗余恢复后的原始深度信息对视频图像进行图像处理,得到目标视频图像。Optionally, the decoding end may perform redundant recovery on the original depth information obtained by decoding based on phase correlation, spatial correlation, temporal correlation, specified depth, frequency domain correlation, and correlation between encoded binary data to obtain the redundantly recovered original depth information, and perform image processing on the video image based on the redundantly recovered original depth information to obtain the target video image.

以基于空间相关性对解码得到的原始深度信息进行冗余恢复,得到冗余恢复后的原始深度信息为例,解码端对视频图像码流进行独立解码或混合解码,得到间隔视点的原始深度信息和至少一个视点的视频图像;对间隔视点的原始深度信息进行差值,得到至少一个视点中除了间隔视点以外的其他视点的原始深度信息;利用间隔视点的原始深度信息、其他视点的原始深度信息,对视频图像进行图像处理,得到目标视频图像。Taking the redundant recovery of the original depth information obtained by decoding based on spatial correlation to obtain the redundantly restored original depth information as an example, the decoding end independently decodes or hybrid decodes the video image bitstream to obtain the original depth information of the interval viewpoints and the video image of at least one viewpoint; the original depth information of the interval viewpoints is differentiated to obtain the original depth information of other viewpoints in at least one viewpoint except the interval viewpoint; the video image is processed using the original depth information of the interval viewpoints and the original depth information of other viewpoints to obtain the target video image.

以基于时间相关性对解码得到的原始深度信息进行冗余恢复,得到冗余恢复后的原始深度信息为例,解码端对视频图像码流进行独立解码或混合解码,得到采样后的原始深度信息,并基于时间相邻的采样后的原始深度信息恢复相邻的采样后的原始深度信息之间的原始深度信息,利用解码得到的原始深度信息和恢复的原始深度信息,对视频图像进行图像处理,得到目标视频图像。Taking the redundant recovery of the decoded original depth information based on time correlation to obtain the redundantly restored original depth information as an example, the decoding end independently decodes or hybrid decodes the video image bitstream to obtain the sampled original depth information, and restores the original depth information between adjacent sampled original depth information based on the temporally adjacent sampled original depth information. The video image is processed using the decoded original depth information and the restored original depth information to obtain the target video image.

可选地,所述视频图像解码器和所述图像处理器相互独立。可选地,所述图像处理器集成在所述视频图像解码器内。Optionally, the video image decoder and the image processor are independent of each other. Optionally, the image processor is integrated into the video image decoder.

在一示例中,以所述原始深度信息为电荷信息为例,所述对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像,包括:根据所述原始深度信息对所述视频图像进行去噪处理或白平衡调节,得到所述目标视频图像。In one example, taking the original depth information as charge information, performing image processing on the original depth information and the video image to obtain a target video image includes: performing denoising or white balance adjustment on the video image according to the original depth information to obtain the target video image.

在一示例中,以所述原始深度信息为相位信息为例,所述对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像,包括:根据所述原始深度信息对所述视频图像进行去模糊处理,得到所述目标视频图像。In one example, taking the original depth information as phase information, performing image processing on the original depth information and the video image to obtain a target video image includes: performing deblurring processing on the video image according to the original depth information to obtain the target video image.

解码端中的图像处理器解析每个相位信息,得到解析结果,利用解析结果,对与之对应的视频帧进行去模糊,得到目标视频图像。The image processor in the decoding end analyzes each phase information to obtain an analysis result, and uses the analysis result to deblur the corresponding video frame to obtain a target video image.

在一示例中,在高动态范围(HDR,High Dynamic Range)视频中,每一帧HDR图像是对1幅长曝光图像和1幅短曝光图像进行融合得到的,在当前时刻,针对同一个场景,控制图像传感器拍摄长曝光图像和短曝光图像,并控制深度信息传感器拍摄相位图像,将相位图像作为原始深度信息;对相位图像和长曝光图像进行混合编码或独立编码,对相位图像和短曝光图像进行混合编码或独立编码,得到视频图像码流;并将视频图像码流输出至解码端;解码端解从视频图像码流中解码出长曝光图像、短曝光图像和相位图像;再利用相位图像,对长曝光图像和短曝光图像分别进行去模糊,得到去模糊后的长曝光图像和去模糊后的短曝光图像;对去模糊后的长曝光图像和去模糊后的短曝光图像进行融合,得到一帧更为清晰的HDR图像。In one example, in a high dynamic range (HDR) video, each frame of HDR image is obtained by fusing a long exposure image and a short exposure image. At the current moment, for the same scene, the image sensor is controlled to capture the long exposure image and the short exposure image, and the depth information sensor is controlled to capture the phase image, and the phase image is used as the original depth information; the phase image and the long exposure image are mixed or independently encoded, and the phase image and the short exposure image are mixed or independently encoded to obtain a video image stream; and the video image stream is output to a decoding end; the decoding end decodes the long exposure image, the short exposure image and the phase image from the video image stream; the phase image is then used to deblur the long exposure image and the short exposure image, respectively, to obtain a deblurred long exposure image and a deblurred short exposure image; the deblurred long exposure image and the deblurred short exposure image are fused to obtain a clearer HDR image.

如图6所示,在S502之后,还包括:As shown in FIG. 6 , after S502 , the method further includes:

S504,对所述原始深度信息进行恢复,得到深度图像。S504: Restore the original depth information to obtain a depth image.

可选地,通过深度图像生成器对所述原始深度信息进行恢复,得到所述深度图像。Optionally, the original depth information is restored by a depth image generator to obtain the depth image.

需要说明的是,本发明实施例中,在图6中以S504位于S503之后为例对得到目标视频图像和得到深度图像的先后顺序进行示例性说明,在实际应用中,S504和S503的执行不分先后顺序。It should be noted that, in the embodiment of the present invention, FIG. 6 takes S504 after S503 as an example to illustrate the order of obtaining the target video image and obtaining the depth image. In actual applications, S504 and S503 are executed in any order.

可选地,深度图像生成器与视频图像解码器相互独立。可选地,深度图像生成器集成在视频图像解码器内。Optionally, the depth image generator is independent of the video image decoder. Optionally, the depth image generator is integrated into the video image decoder.

在一示例中,视频图像解码器、深度图像生成器和图像处理器相互独立,此时,将视频图像码流输入视频图像解码器,视频图像解码器输出原始深度信息和视频图像,并将原始深度信息和视频图像输入图像处理器,将原始深度信息输入深度图像生成器,图像处理器输出目标视频图像,深度图像生成器输出深度图像。In one example, the video image decoder, the depth image generator and the image processor are independent of each other. In this case, the video image code stream is input into the video image decoder, the video image decoder outputs the original depth information and the video image, and the original depth information and the video image are input into the image processor, the original depth information is input into the depth image generator, the image processor outputs the target video image, and the depth image generator outputs the depth image.

在一示例中,深度图像生成器和图像处理器集成在视频图像解码器中,此时,将视频图像码流输入视频图像解码器,视频图像解码器输出目标视频图像和深度图像。In one example, the depth image generator and the image processor are integrated in a video image decoder. In this case, the video image code stream is input into the video image decoder, and the video image decoder outputs a target video image and a depth image.

在一示例中,深度图像生成器集成在视频图像解码器内,图像处理器与视频图像解码器相互独立,此时,将视频图像码流输入视频图像解码器,视频图像解码器输出原始深度信息和目标视频图像,并将原始深度信息输入深度图像生成器,深度图像生成器输出深度图像。In one example, the depth image generator is integrated in the video image decoder, and the image processor and the video image decoder are independent of each other. At this time, the video image code stream is input into the video image decoder, the video image decoder outputs the original depth information and the target video image, and the original depth information is input into the depth image generator, and the depth image generator outputs the depth image.

在一示例中,图像处理器集成在视频图像解码器内,深度图像生成器与视频图像解码器相互独立,此时,将视频图像码流输入视频图像解码器,视频图像解码器输出原始深度信息、视频图像以及深度图像,并将原始深度信息和视频图像输入图像处理器,图像处理器输出目标视频图像。In one example, the image processor is integrated in the video image decoder, and the depth image generator and the video image decoder are independent of each other. At this time, the video image code stream is input into the video image decoder, the video image decoder outputs the original depth information, the video image and the depth image, and the original depth information and the video image are input into the image processor, and the image processor outputs the target video image.

本发明实施例还提供一种信息处理方法,应用于包括编码端和解码端的信息处理系统,如图7所示,包括:The embodiment of the present invention further provides an information processing method, which is applied to an information processing system including an encoding end and a decoding end, as shown in FIG7 , including:

S701,编码端在深度信息传感器采集目标对象的深度信息的情况下,获取深度信息对应的原始深度信息。S701: When a depth information sensor collects depth information of a target object, the encoding end obtains original depth information corresponding to the depth information.

所述原始深度信息表征所述深度信息传感器采集所述深度信息的采集状态或采集到的所述深度信息以外的信息;The original depth information represents a state in which the depth information sensor collects the depth information or information other than the collected depth information;

S702,编码端通过图像传感器获取目标对象的视频图像数据;S702, the encoding end obtains video image data of the target object through an image sensor;

S703,编码端对原始深度信息和视频图像数据进行合并编码,得到视频图像码流,并输出视频图像码流。S703, the encoding end combines and encodes the original depth information and the video image data to obtain a video image code stream, and outputs the video image code stream.

S704,解码端接收视频图像码流。S704: The decoding end receives the video image code stream.

S705,解码端对视频图像码流进行解码,得到原始深度信息和视频图像数据对应的视频图像。S705: The decoding end decodes the video image code stream to obtain the original depth information and the video image corresponding to the video image data.

S706,解码端对原始深度信息和视频图像进行图像处理,得到目标视频图像。S706: The decoding end performs image processing on the original depth information and the video image to obtain a target video image.

本发明实施例中,解码端接收包括原始深度信息的编码信息和图像视频信息的编码信息的视频图像码流,如此,解码端可以从视频图像码流中解码出原始深度信息和视频图像,进而,解码端不仅可以利用原始深度信息,恢复得到深度图像,还可以利用原始深度信息,对视频图像进行去噪、白平衡调整和去模糊等优化处理,提高了信息利用率,并且优化处理后得到的目标视频图像相较于原始视频图像,图像质量更高。In an embodiment of the present invention, a decoding end receives a video image code stream including encoding information of original depth information and encoding information of image video information. In this way, the decoding end can decode the original depth information and the video image from the video image code stream. Furthermore, the decoding end can not only use the original depth information to restore the depth image, but also use the original depth information to perform optimization processing such as denoising, white balance adjustment and deblurring on the video image, thereby improving information utilization. The target video image obtained after the optimization processing has higher image quality than the original video image.

下面,通过场景的示例对本发明实施例提供的信息处理方法进行举例说明。The information processing method provided by the embodiment of the present invention is described below by taking examples of scenarios.

本发明的信息系统的框架如图8A和图8B所示。视频图像编码器1013对深度信息传感器1011获取到的原始深度信息801和图像传感器1012采集到的视频图像数据802进行合并编码,形成视频图像码流803;视频图像解码器1021获取视频图像码流803后,对视频图像码流803解析,得到原始深度信息804和视频图像805,深度图像生成器1023对原始深度信息804恢复得到深度图像806,图像处理器1022通过原始深度信息804对视频图像解码器1021得到的视频图像805进行处理,得到目标视频图像807。其中,深度图像生成器1023、图像处理器1022和视频图像解码器1021可以分别独立,如图8A所示;也可以将深度图像生成器1023和图像处理器1022作为视频图像解码器1021的一个组成部分,如图8B所示。The framework of the information system of the present invention is shown in FIG8A and FIG8B. The video image encoder 1013 combines and encodes the original depth information 801 obtained by the depth information sensor 1011 and the video image data 802 collected by the image sensor 1012 to form a video image code stream 803; after the video image decoder 1021 obtains the video image code stream 803, it parses the video image code stream 803 to obtain the original depth information 804 and the video image 805, the depth image generator 1023 restores the original depth information 804 to obtain the depth image 806, and the image processor 1022 processes the video image 805 obtained by the video image decoder 1021 through the original depth information 804 to obtain the target video image 807. Among them, the depth image generator 1023, the image processor 1022 and the video image decoder 1021 can be independent, as shown in FIG8A; the depth image generator 1023 and the image processor 1022 can also be used as a component of the video image decoder 1021, as shown in FIG8B.

深度信息传感器输出的原始深度信息可以为深度信息传感器所获取得到的最初的数据信息即未经预处理的原始深度信息,也可以为最初的数据信息经预处理后所得到的中间数据信息即经过预处理的原始深度信息;当输出的信息为最初的数据信息,输出的信息可以为电荷信息或相位信息等经过光电转换后的电信号;当输出的信息为中间数据信息,输出的信息可以为对最初的数据信进行相位校准或其他方式等处理后的能够生成深度图像的中间视频图像数据。The original depth information output by the depth information sensor can be the initial data information acquired by the depth information sensor, that is, the original depth information without preprocessing, or it can be the intermediate data information obtained after preprocessing the initial data information, that is, the preprocessed original depth information; when the output information is the initial data information, the output information can be an electrical signal after photoelectric conversion such as charge information or phase information; when the output information is the intermediate data information, the output information can be intermediate video image data that can generate a depth image after phase calibration or other processing of the initial data information.

视频图像编码器对输入的原始深度信息进行编码形成视频图像码流。其中,编码方式包括:The video image encoder encodes the input original depth information to form a video image code stream. The encoding method includes:

编码方式1、利用视频图像数据和深度原始信息的相关性,将二者混合进行编码;Coding method 1: Utilize the correlation between video image data and original depth information and mix the two for encoding;

编码方式2、分别将视频图像数据和深度原始信息进行独立编码。Coding method 2: independently encode the video image data and the original depth information.

在编码方式1中,原始深度信息的编码信息在视频图像数据的编码信息的信息头、序列信息头、附加参数集或者其他等任意位置。In encoding method 1, the encoding information of the original depth information is in any position such as an information header, a sequence information header, an additional parameter set or other positions of the encoding information of the video image data.

在编码方式2中,利用原始深度信息的空间相关性或时间相关性等其他相关性,对原始深度信息自身进行单独的编码。In encoding method 2, the original depth information itself is encoded separately by using other correlations such as spatial correlation or temporal correlation of the original depth information.

在视频图像编码器中,可以对每一幅视频图像对应的原始深度信息都编码,也可以仅对指定图像或指定图像位置对应的原始深度信息进行编码,其他非指定图像或非指定图像位置的对应原始深度信息不进行编码。In the video image encoder, the original depth information corresponding to each video image may be encoded, or only the original depth information corresponding to a specified image or a specified image position may be encoded, and the original depth information corresponding to other non-specified images or non-specified image positions may not be encoded.

对于图像处理器,在拍照或预览场景中,对于景深的产生,可以直接利用原始深度信息作用于视频图像,形成具有景深的目标视频图像,而不用将深度图像和视频图像叠加处理产生具有景深的目标视频图像。For the image processor, in the photo-taking or previewing scene, to generate the depth of field, the original depth information can be directly used on the video image to form a target video image with depth of field, without superimposing the depth image and the video image to generate the target video image with depth of field.

编码端对原始深度信息进行编码的过程中,为了压缩数据量,可以利用且不限于如下相关性消除冗余:In the process of encoding the original depth information at the encoder, in order to compress the data volume, the following correlations can be used but are not limited to eliminate redundancy:

1、若原始深度信息包括多个视频图像的相位信息,利用相位之间的相关性消除相位数据冗余;若原始深度信息为其他数据,利用这些数据间的空间相关性等其相关性消除数据冗余;1. If the original depth information includes phase information of multiple video images, the correlation between the phases is used to eliminate phase data redundancy; if the original depth information is other data, the spatial correlation between these data and other correlations are used to eliminate data redundancy;

2、利用原始深度信息的时间相关性消除数据冗余;2. Eliminate data redundancy by using the temporal correlation of the original depth information;

3、利用指定深度消除基于场景的数据冗余;3. Eliminate scenario-based data redundancy using specified depth;

4、将原始深度信息转化为频域,利用频域相关性消除频域的数据冗余;4. Convert the original depth information into the frequency domain and use the frequency domain correlation to eliminate the data redundancy in the frequency domain;

5、利用编码二进制数据之间的相关性,消除编码的比特冗余;其中,这里的编码可为熵编码。5. Utilize the correlation between the encoded binary data to eliminate the bit redundancy of the encoding; wherein the encoding here can be entropy coding.

本发明实施例中,视频图像编码器形成的包含原始深度信息的视频图像码流中,原始深度信息和视频图像数据可独立解码,即视频图像码流具有可解耦性或独立性,使得采用各种视频图像标准编解码协议的视频图像解码器可以从该视频图像码流中仅提取视频图像,而不提取原始深度信息,也可以仅提取原始深度信息,而不提取视频图像。In an embodiment of the present invention, in a video image bitstream containing original depth information formed by a video image encoder, the original depth information and video image data can be independently decoded, that is, the video image bitstream has decoupling or independence, so that a video image decoder using various video image standard encoding and decoding protocols can extract only the video image from the video image bitstream without extracting the original depth information, or can extract only the original depth information without extracting the video image.

如图9A至9D所示,对于视频图像解码器、深度图像生成器和图像处理器,三者相互配合将视频图像码流按照视频图像标准编解码协议进行解码,并产生处理后的图像和原始深度信息;该视频图像标准编解码协议可以为厂商定制的私有标准,也可以为行业标准。视频图像解码器、深度图像生成器和图像处理器这三者的组成方式包括:As shown in Figures 9A to 9D, the video image decoder, the depth image generator and the image processor cooperate with each other to decode the video image code stream according to the video image standard codec protocol and generate the processed image and original depth information; the video image standard codec protocol can be a private standard customized by the manufacturer or an industry standard. The composition of the video image decoder, the depth image generator and the image processor includes:

组成方式1、如图9A所示,视频图像解码器1021、深度图像生成器1023和图像处理器1022这三者相互独立,视频图像解码器1021解析视频图像码流803得到视频图像805和原始深度信息804后,将原始深度信息804送入深度图像生成器1023产生深度图像806,并将视频图像805和原始深度信息804送入图像处理器1022产生处理后的目标视频图像807;Composition 1, as shown in FIG9A , the video image decoder 1021, the depth image generator 1023 and the image processor 1022 are independent of each other. After parsing the video image code stream 803 to obtain the video image 805 and the original depth information 804, the video image decoder 1021 sends the original depth information 804 to the depth image generator 1023 to generate the depth image 806, and sends the video image 805 and the original depth information 804 to the image processor 1022 to generate the processed target video image 807;

组成方式2、如图9B所示,深度图像生成器1023和图像处理器1022嵌入视频图像解码器1021内部,在视频图像解码器1021内部对视频图像码流803进行处理,直接输出深度图像806和处理后的目标视频图像807。Composition method 2, as shown in Figure 9B, the depth image generator 1023 and the image processor 1022 are embedded in the video image decoder 1021, and the video image code stream 803 is processed inside the video image decoder 1021 to directly output the depth image 806 and the processed target video image 807.

组成方式3、如图9C所示,深度图像生成器1023嵌入视频图像解码器1021内部,先在视频图像解码器1021内部对视频图像码流803进行处理,输出深度图像806和视频图像805,再将视频图像805和原始深度信息804送入图像处理器1022,输出处理后的目标视频图像807;Composition mode 3, as shown in FIG9C , the depth image generator 1023 is embedded in the video image decoder 1021, and the video image code stream 803 is first processed in the video image decoder 1021, and a depth image 806 and a video image 805 are output, and then the video image 805 and the original depth information 804 are sent to the image processor 1022, and the processed target video image 807 is output;

组成方式4、如图9D所示,图像处理器1022嵌入视频图像解码器1021内部,先在视频图像解码器1021内部对图像视频码流803进行处理,输出原始深度信息804和处理后的目标视频图像807,再将原始深度信息804送入深度图像生成器1023,输出深度图像806。Composition method 4, as shown in Figure 9D, the image processor 1022 is embedded in the video image decoder 1021, and the image video code stream 803 is first processed in the video image decoder 1021 to output the original depth information 804 and the processed target video image 807, and then the original depth information 804 is sent to the depth image generator 1023 to output the depth image 806.

本发明实施例提供的信息处理方法,在编码端,将通过深度信息传感器获取得到的原始深度信息,对原始深度信息进行视频图像编码,形成视频图像码流进行传输;在解码端,通过视频图像码流不仅能够恢复得到深度图像,而且可以通过解析得到的原始深度信息,对原始视频图像进行处理,得到图像质量更高的目标视频图像。The information processing method provided by the embodiment of the present invention, at the encoding end, obtains the original depth information through the depth information sensor, encodes the original depth information into a video image, and forms a video image code stream for transmission; at the decoding end, not only can the depth image be restored through the video image code stream, but also the original video image can be processed by parsing the original depth information to obtain a target video image with higher image quality.

在一示例中,原始深度信息为相位信息,可通过不同时间点采样得到的多幅相位图像恢复得到深度图像,且当因为运动造成原始视频图像模糊时,由于多幅相位图像可携带不同时间点的更多信息,可以基于相位信息通过运动估计将模糊的原始视频图像进行恢复,以得到更为清晰的目标视频图像。In one example, the original depth information is phase information, and the depth image can be restored by multiple phase images sampled at different time points. When the original video image is blurred due to motion, since the multiple phase images can carry more information at different time points, the blurred original video image can be restored by motion estimation based on the phase information to obtain a clearer target video image.

在又一示例中,深度信息传感器为TOF架构或模组,原始深度信息为电荷信息,不仅可以生成深度图像,而且可以根据电荷信息来判断拍摄场景的噪声和外部可见光,通过电荷信息进行原始视频图像的去燥和白平衡调节,以得到图像质量更好的视频图像,给用户更美更真实的图像视频体验。In another example, the depth information sensor is a TOF architecture or module, and the original depth information is charge information, which can not only generate a depth image, but also judge the noise and external visible light of the shooting scene based on the charge information. The charge information is used to denoise and adjust the white balance of the original video image to obtain a video image with better image quality, giving users a more beautiful and realistic image and video experience.

本发明实施例中,原始深度信息的获取方式包括但是不限于以下方式:In the embodiment of the present invention, the original depth information is obtained in the following ways, including but not limited to:

方式一method one

采用连续调制的TOF方法,在两种不同的发射信号频率下,通过控制积分时间,通过TOF传感器采样得到不同相位的共8组光信号,并对这8组光信号进行光电转换,得到8组电荷信号,再将这8组电荷信号进行10比特量化,生成8张原始电荷图像;解码端将这8张原始电荷图像和TOF传感器的温度等属性参数一起作为原始深度信息进行编码;或者对这8张原始电荷图像进行预处理,生成2幅过程深度数据和一幅背景数据,并将这2幅过程深度数据和一幅背景数据作为原始深度信息进行编码。Using the continuously modulated TOF method, under two different emission signal frequencies, by controlling the integration time, a total of 8 groups of light signals with different phases are obtained through TOF sensor sampling, and the 8 groups of light signals are photoelectrically converted to obtain 8 groups of charge signals, and then the 8 groups of charge signals are quantized by 10 bits to generate 8 original charge images; the decoding end encodes the 8 original charge images together with the temperature and other attribute parameters of the TOF sensor as the original depth information; or the 8 original charge images are pre-processed to generate 2 process depth data and one background data, and the 2 process depth data and one background data are encoded as the original depth information.

方式二Method 2

采用双目成像的原理,利用双目摄像头拍摄得到的两幅视频图像,根据两幅视频图像的位姿将计算得到视差等信息,将视差信息和摄像头参数等作为原始深度信息进行编码。The principle of binocular imaging is adopted, and two video images are taken with a binocular camera. The parallax and other information are calculated according to the postures of the two video images, and the parallax information and camera parameters are encoded as the original depth information.

本发明实施例中,以编解码协议3维高性能视频编码(3 Dimension HighEfficiency Video Coding,3D HEVC)为例,对原始深度信息进行编码时,作为一种可能的实现方式,每个视点以及对应的原始深度信息均编码;作为另一种可能的实现方式,可基于视点对原始深度信息进行间隔编码,即由于在同一时刻的不同视点之间,如相位图或电荷图像等原始深度信息存在很强的相关性,可以利用该相关性减少传输的视频图像码流数据量。在一示例中,对于三个视点的视频编码,在编码端,在视频图像码流中仅需要保留左右两个视点的原始深度数据,在解码端,可以通过对左右两个视点的原始深度信息进行插值处理得到中间视点的原始深度信息。In an embodiment of the present invention, taking the codec protocol 3D High Efficiency Video Coding (3D HEVC) as an example, when encoding the original depth information, as a possible implementation method, each viewpoint and the corresponding original depth information are encoded; as another possible implementation method, the original depth information can be interpolated based on the viewpoint, that is, since there is a strong correlation between the original depth information such as the phase map or the charge image between different viewpoints at the same time, the correlation can be used to reduce the amount of video image code stream data transmitted. In one example, for video encoding of three viewpoints, at the encoding end, only the original depth data of the left and right viewpoints need to be retained in the video image code stream, and at the decoding end, the original depth information of the middle viewpoint can be obtained by interpolating the original depth information of the left and right viewpoints.

本发明实施例中,以基于时间相关性对原始深度信息进行冗余消除为例,作为一种可能的实现方式,不需要对所有原始深度信息进行编码,而仅需要采用采样的方式对深度信息传感器采集到的原始深度信息采用固定步长进行采样,并通过视频图像编码器对这些采样信号进行编码;解码端恢复得到这些采样信号后,通过插值等方法恢复得到未被采样的原始深度信息。In an embodiment of the present invention, taking the redundancy elimination of original depth information based on time correlation as an example, as a possible implementation method, it is not necessary to encode all the original depth information, but only needs to adopt a sampling method to sample the original depth information collected by the depth information sensor with a fixed step size, and encode these sampled signals through a video image encoder; after the decoding end recovers these sampled signals, the unsampled original depth information is recovered through interpolation and other methods.

在一示例中,如图10所示,原始深度信息包括:编号分别为信号1、信号2、信号3、信号4......信号N,采用固定步长3对原始深度信息进行采样,得到采样后的原始深度信息包括:信号1、信号4、信号7......信号N,对于采样后的原始深度信息进行编码并解码,对于解码后的非采样信号,根据其相邻的采样信号进行恢复;如,对信号1和信号4进行插值恢复得到信号2,对信号2和信号4进行插值恢复得到信号3,以此类推。In an example, as shown in FIG10 , the original depth information includes: signal 1, signal 2, signal 3, signal 4…signal N, and the original depth information is sampled using a fixed step size of 3. The sampled original depth information is encoded and decoded, and the decoded non-sampled signal is restored based on its adjacent sampled signal; for example, signal 1 and signal 4 are interpolated and restored to obtain signal 2, and signal 2 and signal 4 are interpolated and restored to obtain signal 3, and so on.

本发明实施例中,在AR场景中,作为一种可能的实现方式,不需要对整幅深度图像对应的原始深度信息进行编码,而仅需要对部分画面进行编码,从而实现指定的局部原始深度信息的编码传输。In an embodiment of the present invention, in an AR scenario, as a possible implementation method, it is not necessary to encode the original depth information corresponding to the entire depth image, but only to encode part of the picture, so as to achieve the encoding and transmission of the specified local original depth information.

为实现上述信息处理方法,本发明实施例还提供一种终端设备,所述终端设备的组成结构,如图11所示,终端设备1100包括:To implement the above information processing method, an embodiment of the present invention further provides a terminal device. The component structure of the terminal device is shown in FIG11 . The terminal device 1100 includes:

第一获取单元1101,配置为通过深度信息传感单元获取目标对象的深度信息的情况下,获取所述深度信息对应的原始深度信息,所述原始深度信息表征所述深度信息传感单元采集所述深度信息的采集状态或采集到的所述深度信息以外的信息;A first acquisition unit 1101 is configured to acquire original depth information corresponding to the depth information when the depth information of the target object is acquired through the depth information sensing unit, wherein the original depth information represents an acquisition state of the depth information acquired by the depth information sensing unit or information other than the acquired depth information;

第二获取单元1102,配置为通过图像传感单元获取所述目标对象的视频图像数据;A second acquisition unit 1102 is configured to acquire video image data of the target object through an image sensing unit;

编码单元1103,配置为对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流;The encoding unit 1103 is configured to combine and encode the original depth information and the video image data to obtain a video image code stream;

输出单元1104,配置为输出所述视频图像码流。The output unit 1104 is configured to output the video image code stream.

本发明实施例中,编码单元1103,还配置为:In the embodiment of the present invention, the encoding unit 1103 is further configured as:

对所述视频图像数据对应的图像帧中指定图像帧视对应的原始深度信息和所述视频图像数据进行合并编码,得到所述视频图像码流。The original depth information corresponding to a designated image frame view in an image frame corresponding to the video image data and the video image data are combined and encoded to obtain the video image code stream.

本发明实施例中,编码单元1103,还配置为:In the embodiment of the present invention, the encoding unit 1103 is further configured as:

对指定图像位置对应的原始深度信息和所述视频图像数据进行合并编码,得到所述视频图像码流。The original depth information corresponding to the designated image position and the video image data are combined and encoded to obtain the video image code stream.

本发明实施例中,编码单元1103,还配置为:In the embodiment of the present invention, the encoding unit 1103 is further configured as:

根据所述原始深度信息和所述视频图像数据的相关性,对所述原始深度信息和所述视频图像数据进行混合编码,得到所述视频图像码流。According to the correlation between the original depth information and the video image data, the original depth information and the video image data are mixed encoded to obtain the video image code stream.

本发明实施例中,编码单元1103,还配置为:In the embodiment of the present invention, the encoding unit 1103 is further configured as:

对所述原始深度信息的进行编码,得到第一编码信息;Encoding the original depth information to obtain first encoded information;

将所述第一编码信息写入所述视频图像数据的指定位置;Writing the first encoding information into a specified position of the video image data;

对写入所述第一编码信息的视频图像数据进行编码,得到所述视频图像码流。The video image data written into the first coding information is encoded to obtain the video image code stream.

本发明实施例中,编码单元1103,还配置为:In the embodiment of the present invention, the encoding unit 1103 is further configured as:

对所述原始深度信息的进行编码,得到第一编码信息;Encoding the original depth information to obtain first encoded information;

对视频图像数据进行编码,得到第二编码信息;Encoding the video image data to obtain second encoding information;

将所述第一编码信息和所述第二编码信息进行合并,得到所述视频图像码流。The first coding information and the second coding information are combined to obtain the video image code stream.

本发明实施例中,所述终端设备还包括:In the embodiment of the present invention, the terminal device further includes:

预处理单元,配置为:Preprocessing unit, configured as:

在对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流之前,对所述原始深度信息进行预处理。Before the original depth information and the video image data are combined and encoded to obtain a video image code stream, the original depth information is preprocessed.

本发明实施例中,所述终端设备还包括:In the embodiment of the present invention, the terminal device further includes:

消除单元,配置为:Elimination unit, configured as:

在对所述原始深度信息和所述视频图像数据进行合并编码,得到视频图像码流之前,对所述原始深度信息进行冗余消除处理,以消除所述原始深度信息中的冗余信息。Before the original depth information and the video image data are combined and encoded to obtain a video image code stream, redundancy elimination processing is performed on the original depth information to eliminate redundant information in the original depth information.

本发明实施例中,所述消除单元,还配置为以下至少之一:In the embodiment of the present invention, the elimination unit is further configured as at least one of the following:

基于相位相关性对所述原始深度信息冗余消除处理;Eliminate redundancy of the original depth information based on phase correlation;

基于空间相关性对所述原始深度信息进行冗余消除处理;Performing redundancy elimination processing on the original depth information based on spatial correlation;

基于时间相关性对所述原始深度信息进行冗余消除处理;Performing redundancy elimination processing on the original depth information based on time correlation;

基于指定深度对所述原始深度信息进行冗余消除处理;Performing redundancy elimination processing on the original depth information based on the specified depth;

基于频域相关性对所述原始深度信息进行冗余消除处理;Performing redundancy elimination processing on the original depth information based on frequency domain correlation;

基于编码二进制数据之间的相关性对所述原始深度信息的编码比特进行冗余消除处理。Redundancy elimination processing is performed on the coded bits of the original depth information based on the correlation between the coded binary data.

本发明实施例中,所述原始深度信息包括以下至少之一:电荷信息、相位信息和所述深度信息传感单元的属性参数。In the embodiment of the present invention, the original depth information includes at least one of the following: charge information, phase information, and attribute parameters of the depth information sensing unit.

本发明实施例还提供一种终端设备,包括处理器和配置为存储能够在处理器上运行的计算机程序的存储器,其中,所述处理器配置为运行所述计算机程序时,执行上述终端设备执行的信息处理方法的步骤。An embodiment of the present invention also provides a terminal device, comprising a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the information processing method executed by the above-mentioned terminal device when running the computer program.

需要说明的是,本发明实施例中的深度信息传感单元、图像传感单元和视频图像编码单元可分别为深度信息传感器、图像传感器和视频图像编码器。It should be noted that the depth information sensing unit, the image sensing unit and the video image encoding unit in the embodiment of the present invention may be a depth information sensor, an image sensor and a video image encoder, respectively.

为实现上述信息处理方法,本发明实施例还提供一种终端设备,所述终端设备的组成结构,如图12所示,终端设备1200包括:To implement the above information processing method, an embodiment of the present invention further provides a terminal device. The component structure of the terminal device is shown in FIG12. The terminal device 1200 includes:

接收单元1201,配置为接收视频图像码流,所述视频图像码流为对原始深度信息和视频图像数据进行合并编码得到的,所述原始深度信息是通过深度信息传感单元获取目标对象的深度信息的情况下获取的,所述视频图像数据是通过图像传感单元获取的所述目标对象的;所述原始深度信息表征所述深度信息传感单元采集所述深度信息的采集状态或采集到的所述深度信息以外的信息;The receiving unit 1201 is configured to receive a video image code stream, where the video image code stream is obtained by merging and encoding original depth information and video image data, where the original depth information is obtained when a depth information sensing unit obtains depth information of a target object, and the video image data is obtained by an image sensing unit of the target object; the original depth information represents an acquisition state of the depth information sensing unit acquiring the depth information or information other than the acquired depth information;

解码单元1202,配置为对所述视频图像码流进行解码,得到所述原始深度信息和所述视频图像数据对应的视频图像;A decoding unit 1202 is configured to decode the video image code stream to obtain the original depth information and the video image corresponding to the video image data;

处理单元1203,配置为对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像。The processing unit 1203 is configured to perform image processing on the original depth information and the video image to obtain a target video image.

本发明实施例中,解码单元1202,还配置为通过视频图像解码单元对所述视频图像码流进行解码,得到所述原始深度信息和所述视频图像数据对应的视频图像;In the embodiment of the present invention, the decoding unit 1202 is further configured to decode the video image code stream through a video image decoding unit to obtain the original depth information and the video image corresponding to the video image data;

处理单元1203,还配置为通过述视频图像解码单元对所述原始深度信息和所述视频图像进行图像处理,得到目标视频图像。The processing unit 1203 is further configured to perform image processing on the original depth information and the video image through the video image decoding unit to obtain a target video image.

本发明实施例中,所述视频图像解码单元和所述图像处理单元相互独立,或所述图像处理单元集成在所述视频图像解码单元内。In the embodiment of the present invention, the video image decoding unit and the image processing unit are independent of each other, or the image processing unit is integrated into the video image decoding unit.

本发明实施例中,所述原始深度信息包括以下至少之一:电荷信息、相位信息和所述深度信息传感单元的属性参数。In the embodiment of the present invention, the original depth information includes at least one of the following: charge information, phase information, and attribute parameters of the depth information sensing unit.

本发明实施例中,处理单元1203,还配置为:In the embodiment of the present invention, the processing unit 1203 is further configured as follows:

当所述原始深度信息为电荷信息,根据所述电荷信息对所述视频图像进行去噪处理或白平衡调节,得到所述目标视频图像。When the original depth information is charge information, denoising or white balance adjustment is performed on the video image according to the charge information to obtain the target video image.

本发明实施例中,处理单元1203,还配置为:In the embodiment of the present invention, the processing unit 1203 is further configured as follows:

当所述原始深度信息为相位信息,根据所述相位信息对所述视频图像进行去模糊处理,得到所述目标视频图像。When the original depth information is phase information, the video image is deblurred according to the phase information to obtain the target video image.

本发明实施例中,所述终端设备还包括:In the embodiment of the present invention, the terminal device further includes:

生成单元,配置为对所述原始深度信息进行恢复,得到深度图像。The generating unit is configured to restore the original depth information to obtain a depth image.

本发明实施例中,所述生成单元,还配置为通过深度图像生成单元对所述原始深度信息进行恢复,得到深度图像,得到所述深度图像。In the embodiment of the present invention, the generating unit is further configured to restore the original depth information through a depth image generating unit to obtain a depth image, thereby obtaining the depth image.

本发明实施例还提供一种终端设备,包括处理器和配置为存储能够在处理器上运行的计算机程序的存储器,其中,所述处理器配置为运行所述计算机程序时,执行上述终端设备执行的信息处理方法的步骤。An embodiment of the present invention also provides a terminal device, comprising a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the information processing method executed by the above-mentioned terminal device when running the computer program.

需要说明的是,本发明实施例中的视频图像解码单元、图像处理单元和深度图像生成单元可分别为视频图像解码器、图像处理器和深度图像生成器。It should be noted that the video image decoding unit, the image processing unit and the depth image generating unit in the embodiment of the present invention may be a video image decoder, an image processor and a depth image generator, respectively.

图13是本发明实施例的电子设备(终端设备)的硬件组成结构示意图,电子设备1300包括:至少一个处理器1301、存储器1302和至少一个网络接口1304。电子设备1300中的各个组件通过总线系统1305耦合在一起。可理解,总线系统1305用于实现这些组件之间的连接通信。总线系统1305除包括数据总线之外,还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见,在图13中将各种总线都标为总线系统1305。FIG13 is a schematic diagram of the hardware composition structure of an electronic device (terminal device) according to an embodiment of the present invention. The electronic device 1300 includes: at least one processor 1301, a memory 1302, and at least one network interface 1304. The various components in the electronic device 1300 are coupled together via a bus system 1305. It is understood that the bus system 1305 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1305 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus systems 1305 in FIG13.

可以理解,存储器1302可以是易失性存储器或非易失性存储器,也可包括易失性和非易失性存储器两者。其中,非易失性存储器可以是ROM、可编程只读存储器(PROM,Programmable Read-Only Memory)、可擦除可编程只读存储器(EPROM,ErasableProgrammable Read-Only Memory)、电可擦除可编程只读存储器(EEPROM,ElectricallyErasable Programmable Read-Only Memory)、磁性随机存取存储器(FRAM,ferromagneticrandom access memory)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(CD-ROM,Compact Disc Read-Only Memory);磁表面存储器可以是磁盘存储器或磁带存储器。易失性存储器可以是随机存取存储器(RAM,Random Access Memory),其用作外部高速缓存。通过示例性但不是限制性说明,许多形式的RAM可用,例如静态随机存取存储器(SRAM,Static Random Access Memory)、同步静态随机存取存储器(SSRAM,SynchronousStatic Random Access Memory)、动态随机存取存储器(DRAM,Dynamic Random AccessMemory)、同步动态随机存取存储器(SDRAM,Synchronous Dynamic Random AccessMemory)、双倍数据速率同步动态随机存取存储器(DDRSDRAM,Double Data RateSynchronous Dynamic Random Access Memory)、增强型同步动态随机存取存储器(ESDRAM,Enhanced Synchronous Dynamic Random Access Memory)、同步连接动态随机存取存储器(SLDRAM,SyncLink Dynamic Random Access Memory)、直接内存总线随机存取存储器(DRRAM,Direct Rambus Random Access Memory)。本发明实施例描述的存储器1302旨在包括但不限于这些和任意其它适合类型的存储器。It can be understood that the memory 1302 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), ferromagnetic random access memory, flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory 1302 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

本发明实施例中的存储器1302用于存储各种类型的数据以支持电子设备1300的操作。这些数据的示例包括:用于在电子设备1300上操作的任何计算机程序,如应用程序13021。实现本发明实施例方法的程序可以包含在应用程序13021中。The memory 1302 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device 1300. Examples of such data include: any computer program used to operate on the electronic device 1300, such as the application 13021. The program implementing the method of the embodiment of the present invention may be included in the application 13021.

上述本发明实施例揭示的方法可以应用于处理器1301中,或者由处理器1301实现。处理器1301可能是一种集成电路芯片,具有信号的处理能力。在实现过程中,上述方法的各步骤可以通过处理器1301中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器1301可以是通用处理器、数字信号处理器(DSP,Digital Signal Processor),或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。处理器1301可以实现或者执行本发明实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本发明实施例所公开的方法的步骤,可以直接体现为硬件译码处理器执行完成,或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于存储介质中,该存储介质位于存储器1302,处理器1301读取存储器1302中的信息,结合其硬件完成前述方法的步骤。The method disclosed in the above embodiment of the present invention can be applied to the processor 1301, or implemented by the processor 1301. The processor 1301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1301 or the instruction in the form of software. The above processor 1301 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1301 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 1302. The processor 1301 reads the information in the memory 1302 and completes the steps of the above method in combination with its hardware.

在示例性实施例中,电子设备1300可以被一个或多个应用专用集成电路(ASIC,Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD,ProgrammableLogic Device)、复杂可编程逻辑器件(CPLD,Complex Programmable Logic Device)、FPGA、通用处理器、控制器、MCU、MPU、或其他电子元件实现,用于执行前述方法。In an exemplary embodiment, the electronic device 1300 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general purpose processors, controllers, MCUs, MPUs, or other electronic components to execute the aforementioned method.

本发明实施例还提供了一种存储介质,用于存储计算机程序。An embodiment of the present invention further provides a storage medium for storing a computer program.

可选的,该存储介质可应用于本发明实施例中的终端设备,并且该计算机程序使得计算机执行本发明实施例的各个方法中的相应流程,为了简洁,在此不再赘述。Optionally, the storage medium can be applied to the terminal device in the embodiment of the present invention, and the computer program enables the computer to execute the corresponding processes in the various methods in the embodiment of the present invention, which will not be described in detail here for the sake of brevity.

本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器,使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生配置为实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present invention is described with reference to the flowchart and/or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device configured to implement the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中,使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品,该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上,使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理,从而在计算机或其他可编程设备上执行的指令提供配置为实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps configured to implement the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

以上所述,仅为本发明的较佳实施例而已,并非配置为限定本发明的保护范围,凡在本发明的精神和原则之内所作的任何修改、等同替换和改进等,均应包含在本发明的保护范围之内。The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (27)

1. An information processing method, the method comprising:
acquiring original depth information corresponding to the depth information under the condition that the depth information of a target object is acquired through a depth information sensor, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information; the original depth information is used for improving the video quality of the video image;
acquiring video image data of the target object through an image sensor;
and carrying out merging coding on the original depth information and the video image data to obtain a video image code stream, and outputting the video image code stream.
2. The method of claim 1, wherein the merging encoding the original depth information and the video image data to obtain a video image code stream comprises:
And merging and encoding the original depth information corresponding to the appointed image frame in the image frames corresponding to the video image data and the video image data to obtain the video image code stream.
3. The method of claim 1, wherein the merging encoding the original depth information and the video image data to obtain a video image code stream comprises:
and merging and encoding the original depth information corresponding to the designated image position and the video image data to obtain the video image code stream.
4. A method according to any one of claims 1 to 3, wherein said combining encoding said original depth information and said video image data to obtain a video image bitstream comprises:
And carrying out mixed coding on the original depth information and the video image data according to the correlation between the original depth information and the video image data to obtain the video image code stream.
5. The method of claim 4, wherein the merging encoding of the original depth information and the video image data results in a video image bitstream, further comprising:
and writing the first coding information corresponding to the original depth information into the appointed position of the second coding information corresponding to the video image data.
6. A method according to any one of claims 1 to 3, wherein said jointly encoding said original depth information and said video image data comprises:
and respectively and independently encoding the original depth information and the video image data to obtain an image video code stream comprising a first code stream and a second code stream, wherein the first code stream is the code stream obtained by encoding the original depth information, and the second code stream is the code stream obtained by encoding the image video data.
7. The method of any of claims 1 to 6, wherein prior to combining encoding the original depth information and the video image data to obtain a video image bitstream, the method further comprises:
preprocessing the original depth information;
the step of carrying out merging coding on the original depth information and the video image data to obtain a video image code stream comprises the following steps:
and merging and encoding the preprocessed original depth information and the video image data to obtain a video image code stream.
8. The method of any of claims 1 to 7, wherein prior to jointly encoding the original depth information and the video image data to obtain a video image bitstream, the method further comprises:
And performing redundancy elimination processing on the original depth information to eliminate redundant information in the original depth information.
9. The method of claim 8, wherein the performing redundancy elimination processing on the original depth information includes at least one of:
redundancy elimination processing is carried out on the original depth information based on phase correlation;
performing redundancy elimination processing on the original depth information based on spatial correlation;
Performing redundancy elimination processing on the original depth information based on time correlation;
Performing redundancy elimination processing on the original depth information based on a specified depth;
performing redundancy elimination processing on the original depth information based on the frequency domain correlation;
redundancy elimination processing is performed on the coded bits of the original depth information based on correlation between the coded binary data.
10. The method of any of claims 1 to 9, wherein the original depth information comprises at least one of: charge information, phase information, and attribute parameters of the depth information sensor.
11. An information processing method, the method comprising:
Receiving a video image code stream, wherein the video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained under the condition that depth information of a target object is obtained through a depth information sensor, and the video image data is obtained through an image sensor; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensor or information beyond the acquired depth information; the original depth information is used for improving the video quality of the video image;
decoding the video image code stream to obtain the original depth information and a video image corresponding to the video image data;
and carrying out image processing on the original depth information and the video image to obtain a target video image.
12. The method of claim 11, wherein,
Decoding the video image code stream through a video image decoder to obtain the original depth information and a video image corresponding to the video image data;
and carrying out image processing on the original depth information and the video image by an image processor to obtain a target video image.
13. The method of claim 12, wherein the video image decoder and the image processor are independent of each other or the image processor is integrated within the video image decoder.
14. The method of any of claims 11 to 13, wherein the original depth information comprises at least one of: charge information, phase information, and attribute parameters of the depth information sensor.
15. The method of claim 14, wherein when the original depth information is charge information, the performing image processing on the original depth information and the video image to obtain a target video image includes:
and denoising or adjusting white balance of the video image according to the charge information to obtain the target video image.
16. The method of claim 14, wherein when the original depth information is phase information, the performing image processing on the original depth information and the video image to obtain a target video image includes:
and performing deblurring processing on the video image according to the phase information to obtain the target video image.
17. The method of any one of claims 11 to 16, wherein the method further comprises:
and recovering the original depth information to obtain a depth image.
18. The method of claim 17, wherein the original depth information is restored by a depth image generator resulting in the depth image.
19. A terminal device, the terminal device comprising:
The first acquisition unit is configured to acquire original depth information corresponding to the depth information under the condition that the depth information of the target object is acquired through the depth information sensing unit, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensing unit or information other than the acquired depth information; the original depth information is used for improving the video quality of the video image;
A second acquisition unit configured to acquire video image data of the target object through an image sensing unit;
the coding unit is configured to perform combined coding on the original depth information and the video image data to obtain a video image code stream;
and the output unit is configured to output the video image code stream.
20. The terminal device of claim 19, wherein the encoding unit is further configured to:
And carrying out mixed coding on the original depth information and the video image data according to the correlation between the original depth information and the video image data to obtain the video image code stream.
21. The terminal device of claim 19 or 20, wherein the encoding unit is further configured to:
Encoding the original depth information to obtain first encoded information;
encoding the video image data to obtain second encoded information;
And combining the first coding information and the second coding information to obtain the video image code stream.
22. The terminal device of any of claims 19 to 21, wherein the terminal device further comprises:
a preprocessing unit configured to:
And preprocessing the original depth information before carrying out combined coding on the original depth information and the video image data to obtain a video image code stream.
23. The terminal device of any of claims 19 to 22, wherein the terminal device further comprises:
An elimination unit configured to:
And before the original depth information and the video image data are combined and encoded to obtain a video image code stream, carrying out redundancy elimination processing on the original depth information so as to eliminate redundancy information in the original depth information.
24. A terminal device, the terminal device comprising:
the receiving unit is configured to receive a video image code stream, wherein the video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained when the depth information of a target object is obtained through the depth information sensing unit, and the video image data is obtained through the image sensing unit; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensing unit or information beyond the acquired depth information; the original depth information is used for improving the video quality of the video image;
The decoding unit is configured to decode the video image code stream to obtain the original depth information and a video image corresponding to the video image data;
and the processing unit is configured to perform image processing on the original depth information and the video image to obtain a target video image.
25. The terminal device of claim 24, wherein the terminal device further comprises:
and the generating unit is configured to restore the depth information to obtain a depth image.
26. A terminal device comprising a processor and a memory configured to store a computer program capable of running on the processor, wherein the processor is configured to perform the steps of the information processing method of any of the preceding claims 1 to 10 or the steps of the information processing method of any of the preceding claims 11 to 18 when running the computer program.
27. A storage medium storing an executable program which, when executed by a processor, implements the information processing method of any one of the above claims 1 to 10, or implements the information processing method of any one of the above claims 11 to 18.
CN201980100362.9A 2019-11-06 2019-11-06 Information processing method, terminal device and storage medium Active CN114391259B (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/116055 WO2021087819A1 (en) 2019-11-06 2019-11-06 Information processing method, terminal device and storage medium

Publications (2)

Publication Number Publication Date
CN114391259A CN114391259A (en) 2022-04-22
CN114391259B true CN114391259B (en) 2024-05-31

Family

ID=75849423

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201980100362.9A Active CN114391259B (en) 2019-11-06 2019-11-06 Information processing method, terminal device and storage medium

Country Status (2)

Country Link
CN (1) CN114391259B (en)
WO (1) WO2021087819A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110189371B (en) * 2019-05-20 2023-06-30 东南大学 Mouse balance state discriminating device and method based on TOF depth camera
CN116170581B (en) * 2023-02-17 2024-01-23 厦门瑞为信息技术有限公司 Video information encoding and decoding method based on target perception and electronic equipment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102630026A (en) * 2011-02-03 2012-08-08 美国博通公司 Method and system for processing video
WO2019142163A1 (en) * 2018-01-19 2019-07-25 Interdigital Vc Holdings, Inc. Processing a point cloud
CN110336973A (en) * 2019-07-29 2019-10-15 联想(北京)有限公司 Information processing method and its device, electronic equipment and medium
CN110335211A (en) * 2019-06-24 2019-10-15 Oppo广东移动通信有限公司 Method for correcting depth image, terminal device and computer storage medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130021446A1 (en) * 2011-07-20 2013-01-24 GM Global Technology Operations LLC System and method for enhanced sense of depth video
CN103440662B (en) * 2013-09-04 2016-03-09 清华大学深圳研究生院 Kinect depth image acquisition method and device
CN110355758B (en) * 2019-07-05 2021-02-09 北京史河科技有限公司 Machine following method and equipment and following robot system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102630026A (en) * 2011-02-03 2012-08-08 美国博通公司 Method and system for processing video
WO2019142163A1 (en) * 2018-01-19 2019-07-25 Interdigital Vc Holdings, Inc. Processing a point cloud
CN110335211A (en) * 2019-06-24 2019-10-15 Oppo广东移动通信有限公司 Method for correcting depth image, terminal device and computer storage medium
CN110336973A (en) * 2019-07-29 2019-10-15 联想(北京)有限公司 Information processing method and its device, electronic equipment and medium

Also Published As

Publication number Publication date
WO2021087819A1 (en) 2021-05-14
CN114391259A (en) 2022-04-22

Similar Documents

Publication Publication Date Title
US11064110B2 (en) Warp processing for image capture
US8374444B2 (en) Method and apparatus for providing higher resolution images in an embedded device
US10003768B2 (en) Apparatus and methods for frame interpolation based on spatial considerations
JP6837056B2 (en) Methods and equipment for encoding and decoding light field-based images and corresponding computer program products
JP2017163587A (en) System and method for encoding and decoding light field image file
KR20180049028A (en) A method and device for encoding and decoding an optical field-based image, and a corresponding computer program product
WO2021057689A1 (en) Video decoding method and apparatus, video encoding method and apparatus, storage medium, and electronic device
US11783447B2 (en) Methods and apparatus for optimized stitching of overcapture content
KR20110052933A (en) Filming device and filming method
CN114391259B (en) Information processing method, terminal device and storage medium
CN112565603A (en) Image processing method and device and electronic equipment
CN113409209B (en) Image deblurring method, device, electronic device and storage medium
KR102505130B1 (en) A method and a device for encoding a signal representative of a light-field content
CN110784722B (en) Encoding and decoding method, encoding and decoding device, encoding and decoding system and storage medium
WO2024222387A1 (en) Coding method, decoding method, and apparatus
JP3859989B2 (en) Image matching method and image processing method and apparatus capable of using the method
CN116033155A (en) Binocular image compression method, device and readable storage medium
WO2021087810A1 (en) Information processing methods and systems, and encoding apparatus, decoding apparatus and storage medium
CN110809152A (en) Information processing method, encoding device, decoding device, system, and storage medium
CN113747145A (en) Image processing circuit, electronic device, and image processing method
JP7303930B1 (en) Image processing method, device, electronic device and readable storage medium
CN113038267B (en) Video processing method and device, computer readable storage medium and electronic device
US20200311902A1 (en) Data processing apparatus, data analysis system, data analysis method and storage medium
WO2024078403A1 (en) Image processing method and apparatus, and device
CN115776570A (en) Video stream encoding and decoding method, device, processing system and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant