CN114391259B - Information processing method, terminal device and storage medium - Google Patents

Abstract

The invention discloses an information processing method, which comprises the following steps: acquiring original depth information corresponding to the depth information under the condition that the depth information of a target object is acquired through a depth information sensor, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information; acquiring video image data of the target object through an image sensor; and carrying out merging coding on the original depth information and the video image data to obtain a video image code stream, and outputting the video image code stream. The invention also discloses another information processing method, terminal equipment and a storage medium.

Description

Information processing method, terminal device and storage medium

Technical Field

The present invention relates to computer technology, and in particular to an information processing method, terminal equipment and storage medium.

Background technique

In today's society, more and more terminals are equipped with cameras, so that users can take photos or videos anytime and anywhere. In actual use, the encoding end uses the existing camera device to use a time of flight technology (TOF) camera, a binocular camera and other depth information sensors to obtain the depth information of the target object, and the decoding end uses the depth information to restore the depth image of the target object. However, the depth image only provides the depth information of the target object and cannot improve the image quality of the video image of the target object.

Summary of the invention

The embodiments of the present invention provide an information processing method, a terminal device and a storage medium, which can improve the image quality of a video image of a target object.

In a first aspect, an embodiment of the present invention provides an information processing method, including:

When the depth information of the target object is acquired through the depth information sensor, original depth information corresponding to the depth information is acquired, where the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information;

Acquiring video image data of the target object through an image sensor;

The original depth information and the video image data are combined and encoded to obtain a video image code stream, and the video image code stream is output.

In a second aspect, an embodiment of the present invention provides an information processing method, including:

receiving a video image code stream, the video image code stream being obtained by merging and encoding original depth information and video image data, the original depth information being obtained when a depth information sensor acquires depth information of a target object, the video image data being acquired by an image sensor of the target object, the original depth information representing an acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information;

Decoding the video image code stream to obtain the original depth information and the video image corresponding to the video image data;

Image processing is performed on the original depth information and the video image to obtain a target video image.

In a third aspect, an embodiment of the present invention provides a terminal device, including:

A first acquisition unit is configured to acquire original depth information corresponding to the depth information when the depth information of the target object is acquired through the depth information sensing unit, wherein the original depth information represents an acquisition state of the depth information acquired by the depth information sensing unit or information other than the acquired depth information;

A second acquisition unit, configured to acquire video image data of the target object through an image sensing unit;

An encoding unit configured to combine and encode the original depth information and the video image data to obtain a video image code stream;

The output unit is configured to output the video image code stream.

In a fourth aspect, an embodiment of the present invention provides a terminal device, including:

a receiving unit configured to receive a video image code stream, wherein the video image code stream is obtained by merging and encoding original depth information and video image data, wherein the original depth information is obtained when a depth information sensing unit obtains depth information of a target object, and the video image data is obtained by an image sensing unit of the target object; the original depth information represents a state of the depth information sensing unit collecting the depth information or information other than the collected depth information;

A decoding unit, configured to decode the video image code stream to obtain the original depth information and the video image corresponding to the video image data;

The image processing unit is configured to perform image processing on the original depth information and the video image to obtain a target video image.

In a fifth aspect, an embodiment of the present invention provides a terminal device, comprising a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the information processing method executed by the above-mentioned terminal device when running the computer program.

In a sixth aspect, an embodiment of the present invention provides a storage medium storing an executable program, which, when executed by a processor, implements the information processing method executed by the above-mentioned terminal device.

The information processing method provided by the embodiment of the present invention includes: at the encoding end, when the depth information of the target object is obtained through the depth information sensor, the original depth information corresponding to the depth information is obtained; the video image data of the target object is obtained through the image sensor; the original depth information and the video image data are combined and encoded to obtain a video image code stream, and the video image code stream is output. At the decoding end, the video image code stream is received; the video image code stream is decoded to obtain the original depth information and the video image corresponding to the video image data; the original depth information and the video image are image processed to obtain a target video image. Thus, the original depth information obtained by the depth sensor is directly written into the video image code stream at the encoding end, and parsed at the decoding end, and the original depth information is parsed to obtain the video image obtained by the image data collected by the image sensor, and the target video image is obtained, thereby improving the quality of the video image and bringing a more realistic image and video experience to the user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1A is a schematic diagram of an optional structure of an information processing system according to an embodiment of the present invention;

FIG1B is a schematic diagram of an optional structure of an encoding end according to an embodiment of the present invention;

FIG. 1C is a schematic diagram of an optional structure of a decoding end according to an embodiment of the present invention.

FIG2 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

FIG3 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

FIG4 is a schematic diagram of an optional processing flow of an information processing method according to an embodiment of the present invention;

FIG5 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

FIG6 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

FIG7 is a schematic diagram of an optional processing flow of the information processing method according to an embodiment of the present invention;

FIG8A is a schematic diagram of an optional framework of an information processing system according to an embodiment of the present invention;

FIG8B is a schematic diagram of an optional framework of an information processing system according to an embodiment of the present invention;

FIG9A is a schematic diagram of an optional framework of a decoding end according to an embodiment of the present invention;

FIG9B is a schematic diagram of an optional framework of a decoding end according to an embodiment of the present invention;

FIG9C is a schematic diagram of an optional framework of a decoding end according to an embodiment of the present invention;

FIG9D is a schematic diagram of an optional framework of a decoding end according to an embodiment of the present invention;

FIG10 is a schematic diagram of sampling of original depth information according to an embodiment of the present invention;

FIG11 is a schematic diagram of an optional structure of a terminal device implemented in the present invention;

12 is a schematic diagram of an optional structure of a terminal device according to an embodiment of the present invention;

FIG. 13 is a schematic diagram of an optional structure of an electronic device provided in an embodiment of the present invention.

Detailed ways

In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present invention, the implementation of the embodiments of the present invention is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not intended to limit the embodiments of the present invention.

Before describing in detail the information processing method provided by the embodiment of the present invention, the depth image process is first introduced.

A depth image, also known as a range image, is an image that uses the distance (depth) from the image sensor to each point in the scene as a pixel value, and can directly reflect the geometry of the visible surface of the target object. The depth image can be calculated as point cloud data after coordinate transformation, and point cloud data with rules and necessary information can also be inversely calculated into depth image data.

Here, the encoder performs video encoding on the depth image captured by the depth information sensor to obtain the encoded depth image information, and the decoder can only restore the depth image based on the encoded depth image information. However, the amount of information received by the depth information sensor far exceeds the amount of information in the depth image. After the depth image is generated, this massive amount of information is discarded as redundancy. Therefore, in the above scheme, other functions of this redundant information, such as image enhancement at the decoding end, are not taken into account.

Based on the above problems, an embodiment of the present invention provides an information processing method. The information processing method of the embodiment of the present invention can be applied to an information processing system.

Exemplarily, the information processing system 100 applied in the embodiment of the present invention may be as shown in FIG1A. The information processing 100 may include an encoding end 101 and a decoding end 102. The encoding end 101 is used to collect video image data and original depth information, and encode the video image data and the original depth information to form a video image code stream. The decoding end 120 is used to decode the image video code stream to obtain the video image data and the original depth information, and perform image processing on the video image data and the original depth information to obtain the target video image.

The encoding end 101 and the decoding end 102 may include a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a handheld device such as a smart phone, a television, a camera, a display device, a digital media player, a video game console, a car computer, or the like.

As shown in FIG1A , the decoding end 102 may receive the encoded video image code stream from the encoding end 101 via the link 103. The link 103 may include one or more media and/or devices capable of moving the video image code stream from the encoding end 101 to the decoding end 102.

In one example, the link 103 may include one or more communication media that enable the encoder 101 to directly send the encoded video data to the decoder 102 in real time. In this instance, the encoder 101 may modulate the video image code stream according to a communication standard (e.g., a wireless communication protocol), and may send the modulated video image code stream to the decoder 102.

In one example, link 103 may include a storage medium storing a video image code stream formed by encoding end 101. In this example, decoding end 102 may access the storage medium via disk access or card access. The storage medium may include a variety of local access data storage media, such as a Blu-ray disc, DVD, CD-ROM, flash memory, or other suitable digital storage media for storing video image code streams.

In another example, link 103 may include a file server or another intermediate storage device storing the video image code stream formed by encoding end 101. In this example, decoding end 102 may access the video image code stream stored at the file server or other intermediate storage device via streaming or downloading. The file server may be a server type capable of storing the video image code stream and sending the video image code stream to decoding end 102. The file server includes a web server (for example, for a website), a file transfer protocol server, a network attached storage device, and a local disk drive, etc.

The decoding end 102 can access the video image code stream via a standard data connection (e.g., an Internet connection). Example types of data connections include a wireless link (e.g., a Wi-Fi connection) suitable for accessing the video image code stream stored on a file server, a wired connection (e.g., DSL, cable modem, etc.), or a combination of the two.

As shown in FIG. 1B , the encoding end 101 includes: a depth information sensor 1011, an image sensor 1012 and a video image encoder 1013. The depth information sensor 1011 is used to obtain original depth information, the image sensor 1012 is used to obtain video image data, and the video image encoder 1013 is used to encode the original depth information and video image data to form a video image code stream.

As shown in FIG1C , the decoding end 102 includes: a video image decoder 1021 and an image processor 1022. The video image decoder 1021 is used to decode the video image code stream to obtain the original depth information and the video image corresponding to the video image data. The image processor 1022 is used to process the original depth information and the video image to obtain the target video image. Here, the original depth information acts on the video image to obtain a high-quality video image with high definition and low noise.

In an example, as shown in FIG. 1C , the decoding end 102 further includes: a depth image generator 1023 for generating a depth image based on original depth information.

An optional processing flow of the information processing method provided in an embodiment of the present invention is applied to an encoding end, as shown in FIG2 , and includes the following steps:

S201 : When acquiring depth information of a target object through a depth information sensor, acquire original depth information corresponding to the depth information.

The original depth information represents a state in which the depth information sensor collects the depth information or information other than the collected depth information.

The depth information sensor is a sensor capable of collecting depth information of a target object. In one example, the depth information sensor is a TOF module using a TOF ranging method. In one example, the depth information sensor is a binocular camera.

In the embodiment of the present invention, when the depth information sensor collects depth information, the encoder obtains original depth information through the depth information sensor, and the original depth information includes at least one of the following: charge information, phase information, and attribute parameters of the depth information sensor. Among them, the charge information and phase information are information other than the depth information collected by the depth information sensor, and the attribute parameters of the depth information sensor represent the collection state of the depth information sensor in collecting the depth information.

Taking the original depth information as charge information as an example, the charge information at a time point can be reflected as a charge image. Here, the depth information sensor receives the light signal when collecting the depth information, and converts the light signal into an electrical signal through photoelectric conversion, and the electrical signal is quantified to generate a charge image.

Taking the original depth information as phase information as an example, the phase information at a time point can be reflected as a phase image.

Taking the original depth information as the attribute parameter of the depth information sensor as an example, the original depth information may include attribute parameters such as temperature and posture.

S202: Acquire video image data of the target object through an image sensor.

The encoding end obtains video image data of the target object through an image sensor in an image preview or video shooting scenario. Here, the video image data includes at least one image frame.

In the embodiment of the present invention, the original depth information corresponds to the video frame one by one. In one example, different charge images or phase images correspond to different image frames.

S203: Combine and encode the original depth information and the video image data to obtain a video image code stream, and output the video image code stream.

The encoding end combines and encodes the original depth information and the video image data through a video image encoder, and the video image encoder outputs a video image code stream, and outputs the video image code stream output by the video image encoder to the decoding end, so that the decoding end performs image processing on the video image corresponding to the video image data based on the original depth information.

Optionally, the video image encoder adopts a video image coding and decoding protocol to encode the video image frame or the original depth information to obtain video image code stream information; the video coding and decoding protocol may be H.264, H.265, H.266, VP9 or AV1, etc.

Optionally, a video image coding and decoding protocol is used to encode the original depth information and the image video data. In this case, the data carried by the video image information does not include the depth information.

In an embodiment of the present invention, when the data carried by the video image information does not include depth information, the encoding end may only obtain the original depth information when the depth information sensor obtains the depth information, and may not obtain the depth information collected by the depth information sensor, or may discard the collected depth information.

Optionally, the original depth information and image video data are encoded using a video image encoding and decoding protocol, and the depth information collected by the depth information sensor is encoded using a video image encoding and decoding protocol. At this time, the data carried by the video image information includes: original depth information, depth information and video image data.

In the embodiment of the present invention, no limitation is imposed on the processing of the depth information collected by the depth information sensor.

Optionally, the video image encoder adopts an industry standard or a specific standard of a specific organization to encode the video image frame or the original depth information to obtain a video image code stream.

The encoding end may input all the original depth information into the video image encoder to encode all the original depth information, or may input only part of the original depth information into the video image encoder to encode part of the original depth information. Optionally, the part of the original depth information is the original depth information corresponding to the specified image frame. Optionally, the part of the original depth information is the original depth information corresponding to the specified image position.

Taking part of the original depth information as the original depth information corresponding to the specified image video as an example, the original depth information and the video image data are merged and encoded to obtain a video image code stream, including: merging the original depth information corresponding to a specified image frame in the image frame corresponding to the video image data and the video image data to obtain a video image code stream.

Optionally, the designated image frame is one of the image frames corresponding to the video image data. Optionally, the designated image frame includes a plurality of image frames corresponding to the video image data.

The embodiment of the present invention does not impose any restriction on the number of designated image frames.

The encoding end only performs combined encoding on the original depth information corresponding to the designated image frame and the video image data, and does not encode the original depth information corresponding to non-designated video frames other than the designated image frame in the image frames corresponding to the video image data.

Taking part of the original depth information as the original depth information corresponding to the specified image position as an example, the original depth information and the video image data are merged and encoded to obtain a video image code stream, including: the original depth information corresponding to the specified image position and the video image data are merged and encoded to obtain a video image code stream.

The designated image position is the position of the designated point within the image acquisition range. Optionally, the designated image position is the position of the designated area within the image acquisition range. The embodiment of the present invention does not impose any limitation on the range size or position of the designated image position.

The encoding end only performs combined encoding on the original depth information corresponding to the designated image position and the video image data, and does not encode the original depth information corresponding to the non-designated video position other than the designated image position in the image frame.

In the embodiment of the present invention, the encoding mode in which the encoding end performs combined encoding on the original depth information and the video image data includes one of the following:

Coding method 1: performing mixed coding on the original depth information and the video image data according to the correlation between the original depth information and the video image data to obtain a video image code stream;

Coding method 2: independently encode the original depth information and the video image data respectively to obtain an image video code stream including a first code stream and a second code stream, wherein the first code stream is a code stream obtained after encoding the original depth information, and the second code stream is a code stream obtained after encoding the image video data.

In the first encoding method, the original depth information and the video image data are encoded using the same encoding and decoding protocol.

Optionally, in encoding mode 1, the encoding information in the video image code stream is mixed encoding information obtained by jointly encoding the original depth information and the video image data. The video image encoder may jointly encode the original depth information and the video image data by utilizing the spatial correlation or temporal correlation of the original depth information and the video image data.

Optionally, in encoding mode 1, the first encoding information corresponding to the original depth information is written into a designated position of the second encoding information corresponding to the video image data. Optionally, the designated position may be an image information header, a sequence information header, an additional parameter set or any other position.

Optionally, in encoding method one, the original depth information is encoded using spatial correlation or temporal correlation between the original depth information and the image video data to obtain first encoding information, and the video image data is encoded to obtain second encoding information, and the first encoding information is written into a specified position of the first encoding information to obtain a video image code stream.

In the second encoding method, the codec protocol used for the original depth information is independent of the codec protocol used for encoding the video image data. Optionally, the codec protocol used for the original depth information is the same as the codec protocol used for encoding the video image data. Optionally, the codec protocol used for the original depth information is different from the codec protocol used for encoding the video image data.

In one embodiment, as shown in FIG3 , before S203 , the following steps are included:

204A, pre-process the original depth information.

The original depth information and the video image data are combined and encoded in S203, which can be executed as S203A: the pre-processed original depth information and the video image data are combined and encoded to obtain a video image code stream.

In the embodiments of the present invention, the preprocessing may be one or two processing methods such as filtering, denoising, signal amplification, phase calibration, etc., or other processing methods. The specific preprocessing may be determined according to actual conditions, and the embodiments of the present invention do not limit this.

Optionally, the encoding end preprocesses the original depth information through a depth information sensor.

In one embodiment, as shown in FIG4 , before S203 , the following steps are included:

204B: Perform redundancy elimination processing on the original depth information to eliminate redundant information in the original depth information.

The original depth information and the video image data are combined and encoded in S203, which can be executed as S203B: the original depth information after redundancy elimination processing and the video image data are combined and encoded to obtain a video image code stream.

The encoding end can eliminate redundant information in the original depth information by performing redundancy elimination processing on the original depth information, thereby compressing the information amount of the original depth information, thereby reducing the size of the video data code stream.

In the embodiment of the present invention, performing redundancy elimination processing on the original depth information includes at least one of the following:

Eliminate redundancy of the original depth information based on phase correlation;

Performing redundancy elimination processing on the original depth information based on spatial correlation;

Performing redundancy elimination processing on the original depth information based on time correlation;

Performing redundancy elimination processing on the original depth information based on the specified depth;

Performing redundancy elimination processing on the original depth information based on frequency domain correlation;

Redundancy elimination processing is performed on the coded bits of the original depth information based on the correlation between the coded binary data.

Optionally, the original depth information is converted into a frequency domain, and redundancy elimination processing is performed on the original depth information converted into the frequency domain based on frequency domain correlation.

Optionally, the designated depth is a scene-sensitive depth range where the target object is located, and redundancy elimination processing is performed on the original depth information based on the designated depth, and original depth information corresponding to depths outside the scene-sensitive depth range is eliminated as redundancy.

Optionally, entropy coding is performed on the original depth information, and redundancy elimination processing is performed on coded bits of the entropy coding result of the original depth information based on the correlation between the coded binary data.

Taking the redundancy elimination processing of the original depth information based on spatial correlation as an example, the original depth information at the encoding end corresponds to at least one viewpoint; an interval viewpoint is determined from the at least one viewpoint, and the original depth information corresponding to the interval viewpoint is used as the interval original depth information; the original depth information other than the interval original depth in the original depth information is eliminated as redundancy, and the interval original depth information and the video image data are merged and encoded to obtain a video data code stream.

Taking the redundancy elimination processing of the original depth information based on time correlation as an example, the encoding end obtains the original depth information within one end time, and samples the obtained original depth information based on the sampling interval, and retains the sampled original depth information, and eliminates the original depth information other than the sampled original depth information in the obtained original depth information as redundancy, and merges and encodes the sampled original depth information and the video image data to obtain a video data code stream.

An optional processing flow of the information processing method provided in an embodiment of the present invention is applied to a decoding end, as shown in FIG5 , and includes the following steps:

S501, receiving a video image code stream.

The decoding end receives the video image code stream sent by the encoding end through the link. The video image code stream is obtained by merging and encoding the original depth information and the video image data, the original depth information is obtained when the depth information sensor obtains the depth information of the target object, and the video image data is obtained by the image sensor of the target object; the original depth information represents the acquisition state of the depth information collected by the depth information sensor or information other than the acquired depth information.

S502: Decode the video image code stream to obtain the original depth information and the video image corresponding to the video image data.

Here, the video image code stream is decoded by a video image decoder to obtain the original depth information and the video image corresponding to the video image data.

The decoding end sends the received video image code stream to the video image decoder, and the video image decoder decodes the video image code stream.

Optionally, the video image decoder and the video encoder at the encoding end support the same video image encoding and decoding protocol.

Optionally, when the video image encoder performs hybrid encoding on the original depth information and the video image data,

The video image decoder performs hybrid decoding on the video image code stream to obtain the original depth information and the video image corresponding to the video image data.

Optionally, when the video image encoder encodes the original depth information and the video image data independently, the video image decoder independently decodes the first code stream and the second code stream in the video image code stream, decodes the first code stream of the video image data to obtain the original depth information, and decodes the second code stream to obtain the video image corresponding to the video image data. Here, the video image corresponding to the video image data may also be referred to as the original video image. The original video image obtained by decoding the video code stream may include one or more frames of original video images.

S503: Perform image processing on the original depth information and the video image to obtain a target video image.

The original depth information and the video image are processed by an image processor to obtain a target video image.

After the decoding end obtains the original depth information and video image data through decoding, the original depth information is applied to the video image through the image processor, and the video image is processed to obtain the target video image. The image quality of the target video image is higher than that of the original video image.

Optionally, the decoding end may perform redundant recovery on the original depth information obtained by decoding based on phase correlation, spatial correlation, temporal correlation, specified depth, frequency domain correlation, and correlation between encoded binary data to obtain the redundantly recovered original depth information, and perform image processing on the video image based on the redundantly recovered original depth information to obtain the target video image.

Taking the redundant recovery of the original depth information obtained by decoding based on spatial correlation to obtain the redundantly restored original depth information as an example, the decoding end independently decodes or hybrid decodes the video image bitstream to obtain the original depth information of the interval viewpoints and the video image of at least one viewpoint; the original depth information of the interval viewpoints is differentiated to obtain the original depth information of other viewpoints in at least one viewpoint except the interval viewpoint; the video image is processed using the original depth information of the interval viewpoints and the original depth information of other viewpoints to obtain the target video image.

Taking the redundant recovery of the decoded original depth information based on time correlation to obtain the redundantly restored original depth information as an example, the decoding end independently decodes or hybrid decodes the video image bitstream to obtain the sampled original depth information, and restores the original depth information between adjacent sampled original depth information based on the temporally adjacent sampled original depth information. The video image is processed using the decoded original depth information and the restored original depth information to obtain the target video image.

Optionally, the video image decoder and the image processor are independent of each other. Optionally, the image processor is integrated into the video image decoder.

In one example, taking the original depth information as charge information, performing image processing on the original depth information and the video image to obtain a target video image includes: performing denoising or white balance adjustment on the video image according to the original depth information to obtain the target video image.

In one example, taking the original depth information as phase information, performing image processing on the original depth information and the video image to obtain a target video image includes: performing deblurring processing on the video image according to the original depth information to obtain the target video image.

The image processor in the decoding end analyzes each phase information to obtain an analysis result, and uses the analysis result to deblur the corresponding video frame to obtain a target video image.

In one example, in a high dynamic range (HDR) video, each frame of HDR image is obtained by fusing a long exposure image and a short exposure image. At the current moment, for the same scene, the image sensor is controlled to capture the long exposure image and the short exposure image, and the depth information sensor is controlled to capture the phase image, and the phase image is used as the original depth information; the phase image and the long exposure image are mixed or independently encoded, and the phase image and the short exposure image are mixed or independently encoded to obtain a video image stream; and the video image stream is output to a decoding end; the decoding end decodes the long exposure image, the short exposure image and the phase image from the video image stream; the phase image is then used to deblur the long exposure image and the short exposure image, respectively, to obtain a deblurred long exposure image and a deblurred short exposure image; the deblurred long exposure image and the deblurred short exposure image are fused to obtain a clearer HDR image.

As shown in FIG. 6 , after S502 , the method further includes:

S504: Restore the original depth information to obtain a depth image.

Optionally, the original depth information is restored by a depth image generator to obtain the depth image.

It should be noted that, in the embodiment of the present invention, FIG. 6 takes S504 after S503 as an example to illustrate the order of obtaining the target video image and obtaining the depth image. In actual applications, S504 and S503 are executed in any order.

Optionally, the depth image generator is independent of the video image decoder. Optionally, the depth image generator is integrated into the video image decoder.

In one example, the video image decoder, the depth image generator and the image processor are independent of each other. In this case, the video image code stream is input into the video image decoder, the video image decoder outputs the original depth information and the video image, and the original depth information and the video image are input into the image processor, the original depth information is input into the depth image generator, the image processor outputs the target video image, and the depth image generator outputs the depth image.

In one example, the depth image generator and the image processor are integrated in a video image decoder. In this case, the video image code stream is input into the video image decoder, and the video image decoder outputs a target video image and a depth image.

In one example, the depth image generator is integrated in the video image decoder, and the image processor and the video image decoder are independent of each other. At this time, the video image code stream is input into the video image decoder, the video image decoder outputs the original depth information and the target video image, and the original depth information is input into the depth image generator, and the depth image generator outputs the depth image.

In one example, the image processor is integrated in the video image decoder, and the depth image generator and the video image decoder are independent of each other. At this time, the video image code stream is input into the video image decoder, the video image decoder outputs the original depth information, the video image and the depth image, and the original depth information and the video image are input into the image processor, and the image processor outputs the target video image.

The embodiment of the present invention further provides an information processing method, which is applied to an information processing system including an encoding end and a decoding end, as shown in FIG7 , including:

S701: When a depth information sensor collects depth information of a target object, the encoding end obtains original depth information corresponding to the depth information.

The original depth information represents a state in which the depth information sensor collects the depth information or information other than the collected depth information;

S702, the encoding end obtains video image data of the target object through an image sensor;

S703, the encoding end combines and encodes the original depth information and the video image data to obtain a video image code stream, and outputs the video image code stream.

S704: The decoding end receives the video image code stream.

S705: The decoding end decodes the video image code stream to obtain the original depth information and the video image corresponding to the video image data.

S706: The decoding end performs image processing on the original depth information and the video image to obtain a target video image.

In an embodiment of the present invention, a decoding end receives a video image code stream including encoding information of original depth information and encoding information of image video information. In this way, the decoding end can decode the original depth information and the video image from the video image code stream. Furthermore, the decoding end can not only use the original depth information to restore the depth image, but also use the original depth information to perform optimization processing such as denoising, white balance adjustment and deblurring on the video image, thereby improving information utilization. The target video image obtained after the optimization processing has higher image quality than the original video image.

The information processing method provided by the embodiment of the present invention is described below by taking examples of scenarios.

The framework of the information system of the present invention is shown in FIG8A and FIG8B. The video image encoder 1013 combines and encodes the original depth information 801 obtained by the depth information sensor 1011 and the video image data 802 collected by the image sensor 1012 to form a video image code stream 803; after the video image decoder 1021 obtains the video image code stream 803, it parses the video image code stream 803 to obtain the original depth information 804 and the video image 805, the depth image generator 1023 restores the original depth information 804 to obtain the depth image 806, and the image processor 1022 processes the video image 805 obtained by the video image decoder 1021 through the original depth information 804 to obtain the target video image 807. Among them, the depth image generator 1023, the image processor 1022 and the video image decoder 1021 can be independent, as shown in FIG8A; the depth image generator 1023 and the image processor 1022 can also be used as a component of the video image decoder 1021, as shown in FIG8B.

The original depth information output by the depth information sensor can be the initial data information acquired by the depth information sensor, that is, the original depth information without preprocessing, or it can be the intermediate data information obtained after preprocessing the initial data information, that is, the preprocessed original depth information; when the output information is the initial data information, the output information can be an electrical signal after photoelectric conversion such as charge information or phase information; when the output information is the intermediate data information, the output information can be intermediate video image data that can generate a depth image after phase calibration or other processing of the initial data information.

The video image encoder encodes the input original depth information to form a video image code stream. The encoding method includes:

Coding method 1: Utilize the correlation between video image data and original depth information and mix the two for encoding;

Coding method 2: independently encode the video image data and the original depth information.

In encoding method 1, the encoding information of the original depth information is in any position such as an information header, a sequence information header, an additional parameter set or other positions of the encoding information of the video image data.

In encoding method 2, the original depth information itself is encoded separately by using other correlations such as spatial correlation or temporal correlation of the original depth information.

In the video image encoder, the original depth information corresponding to each video image may be encoded, or only the original depth information corresponding to a specified image or a specified image position may be encoded, and the original depth information corresponding to other non-specified images or non-specified image positions may not be encoded.

For the image processor, in the photo-taking or previewing scene, to generate the depth of field, the original depth information can be directly used on the video image to form a target video image with depth of field, without superimposing the depth image and the video image to generate the target video image with depth of field.

In the process of encoding the original depth information at the encoder, in order to compress the data volume, the following correlations can be used but are not limited to eliminate redundancy:

1. If the original depth information includes phase information of multiple video images, the correlation between the phases is used to eliminate phase data redundancy; if the original depth information is other data, the spatial correlation between these data and other correlations are used to eliminate data redundancy;

2. Eliminate data redundancy by using the temporal correlation of the original depth information;

3. Eliminate scenario-based data redundancy using specified depth;

4. Convert the original depth information into the frequency domain and use the frequency domain correlation to eliminate the data redundancy in the frequency domain;

5. Utilize the correlation between the encoded binary data to eliminate the bit redundancy of the encoding; wherein the encoding here can be entropy coding.

In an embodiment of the present invention, in a video image bitstream containing original depth information formed by a video image encoder, the original depth information and video image data can be independently decoded, that is, the video image bitstream has decoupling or independence, so that a video image decoder using various video image standard encoding and decoding protocols can extract only the video image from the video image bitstream without extracting the original depth information, or can extract only the original depth information without extracting the video image.

As shown in Figures 9A to 9D, the video image decoder, the depth image generator and the image processor cooperate with each other to decode the video image code stream according to the video image standard codec protocol and generate the processed image and original depth information; the video image standard codec protocol can be a private standard customized by the manufacturer or an industry standard. The composition of the video image decoder, the depth image generator and the image processor includes:

Composition 1, as shown in FIG9A , the video image decoder 1021, the depth image generator 1023 and the image processor 1022 are independent of each other. After parsing the video image code stream 803 to obtain the video image 805 and the original depth information 804, the video image decoder 1021 sends the original depth information 804 to the depth image generator 1023 to generate the depth image 806, and sends the video image 805 and the original depth information 804 to the image processor 1022 to generate the processed target video image 807;

Composition method 2, as shown in Figure 9B, the depth image generator 1023 and the image processor 1022 are embedded in the video image decoder 1021, and the video image code stream 803 is processed inside the video image decoder 1021 to directly output the depth image 806 and the processed target video image 807.

Composition mode 3, as shown in FIG9C , the depth image generator 1023 is embedded in the video image decoder 1021, and the video image code stream 803 is first processed in the video image decoder 1021, and a depth image 806 and a video image 805 are output, and then the video image 805 and the original depth information 804 are sent to the image processor 1022, and the processed target video image 807 is output;

Composition method 4, as shown in Figure 9D, the image processor 1022 is embedded in the video image decoder 1021, and the image video code stream 803 is first processed in the video image decoder 1021 to output the original depth information 804 and the processed target video image 807, and then the original depth information 804 is sent to the depth image generator 1023 to output the depth image 806.

The information processing method provided by the embodiment of the present invention, at the encoding end, obtains the original depth information through the depth information sensor, encodes the original depth information into a video image, and forms a video image code stream for transmission; at the decoding end, not only can the depth image be restored through the video image code stream, but also the original video image can be processed by parsing the original depth information to obtain a target video image with higher image quality.

In one example, the original depth information is phase information, and the depth image can be restored by multiple phase images sampled at different time points. When the original video image is blurred due to motion, since the multiple phase images can carry more information at different time points, the blurred original video image can be restored by motion estimation based on the phase information to obtain a clearer target video image.

In another example, the depth information sensor is a TOF architecture or module, and the original depth information is charge information, which can not only generate a depth image, but also judge the noise and external visible light of the shooting scene based on the charge information. The charge information is used to denoise and adjust the white balance of the original video image to obtain a video image with better image quality, giving users a more beautiful and realistic image and video experience.

In the embodiment of the present invention, the original depth information is obtained in the following ways, including but not limited to:

method one

Using the continuously modulated TOF method, under two different emission signal frequencies, by controlling the integration time, a total of 8 groups of light signals with different phases are obtained through TOF sensor sampling, and the 8 groups of light signals are photoelectrically converted to obtain 8 groups of charge signals, and then the 8 groups of charge signals are quantized by 10 bits to generate 8 original charge images; the decoding end encodes the 8 original charge images together with the temperature and other attribute parameters of the TOF sensor as the original depth information; or the 8 original charge images are pre-processed to generate 2 process depth data and one background data, and the 2 process depth data and one background data are encoded as the original depth information.

Method 2

The principle of binocular imaging is adopted, and two video images are taken with a binocular camera. The parallax and other information are calculated according to the postures of the two video images, and the parallax information and camera parameters are encoded as the original depth information.

In an embodiment of the present invention, taking the codec protocol 3D High Efficiency Video Coding (3D HEVC) as an example, when encoding the original depth information, as a possible implementation method, each viewpoint and the corresponding original depth information are encoded; as another possible implementation method, the original depth information can be interpolated based on the viewpoint, that is, since there is a strong correlation between the original depth information such as the phase map or the charge image between different viewpoints at the same time, the correlation can be used to reduce the amount of video image code stream data transmitted. In one example, for video encoding of three viewpoints, at the encoding end, only the original depth data of the left and right viewpoints need to be retained in the video image code stream, and at the decoding end, the original depth information of the middle viewpoint can be obtained by interpolating the original depth information of the left and right viewpoints.

In an embodiment of the present invention, taking the redundancy elimination of original depth information based on time correlation as an example, as a possible implementation method, it is not necessary to encode all the original depth information, but only needs to adopt a sampling method to sample the original depth information collected by the depth information sensor with a fixed step size, and encode these sampled signals through a video image encoder; after the decoding end recovers these sampled signals, the unsampled original depth information is recovered through interpolation and other methods.

In an example, as shown in FIG10 , the original depth information includes: signal 1, signal 2, signal 3, signal 4…signal N, and the original depth information is sampled using a fixed step size of 3. The sampled original depth information is encoded and decoded, and the decoded non-sampled signal is restored based on its adjacent sampled signal; for example, signal 1 and signal 4 are interpolated and restored to obtain signal 2, and signal 2 and signal 4 are interpolated and restored to obtain signal 3, and so on.

In an embodiment of the present invention, in an AR scenario, as a possible implementation method, it is not necessary to encode the original depth information corresponding to the entire depth image, but only to encode part of the picture, so as to achieve the encoding and transmission of the specified local original depth information.

To implement the above information processing method, an embodiment of the present invention further provides a terminal device. The component structure of the terminal device is shown in FIG11 . The terminal device 1100 includes:

A first acquisition unit 1101 is configured to acquire original depth information corresponding to the depth information when the depth information of the target object is acquired through the depth information sensing unit, wherein the original depth information represents an acquisition state of the depth information acquired by the depth information sensing unit or information other than the acquired depth information;

A second acquisition unit 1102 is configured to acquire video image data of the target object through an image sensing unit;

The encoding unit 1103 is configured to combine and encode the original depth information and the video image data to obtain a video image code stream;

The output unit 1104 is configured to output the video image code stream.

In the embodiment of the present invention, the encoding unit 1103 is further configured as:

The original depth information corresponding to a designated image frame view in an image frame corresponding to the video image data and the video image data are combined and encoded to obtain the video image code stream.

In the embodiment of the present invention, the encoding unit 1103 is further configured as:

The original depth information corresponding to the designated image position and the video image data are combined and encoded to obtain the video image code stream.

In the embodiment of the present invention, the encoding unit 1103 is further configured as:

According to the correlation between the original depth information and the video image data, the original depth information and the video image data are mixed encoded to obtain the video image code stream.

In the embodiment of the present invention, the encoding unit 1103 is further configured as:

Encoding the original depth information to obtain first encoded information;

Writing the first encoding information into a specified position of the video image data;

The video image data written into the first coding information is encoded to obtain the video image code stream.

In the embodiment of the present invention, the encoding unit 1103 is further configured as:

Encoding the original depth information to obtain first encoded information;

Encoding the video image data to obtain second encoding information;

The first coding information and the second coding information are combined to obtain the video image code stream.

In the embodiment of the present invention, the terminal device further includes:

Preprocessing unit, configured as:

Before the original depth information and the video image data are combined and encoded to obtain a video image code stream, the original depth information is preprocessed.

In the embodiment of the present invention, the terminal device further includes:

Elimination unit, configured as:

Before the original depth information and the video image data are combined and encoded to obtain a video image code stream, redundancy elimination processing is performed on the original depth information to eliminate redundant information in the original depth information.

In the embodiment of the present invention, the elimination unit is further configured as at least one of the following:

Eliminate redundancy of the original depth information based on phase correlation;

Performing redundancy elimination processing on the original depth information based on spatial correlation;

Performing redundancy elimination processing on the original depth information based on time correlation;

Performing redundancy elimination processing on the original depth information based on the specified depth;

Performing redundancy elimination processing on the original depth information based on frequency domain correlation;

Redundancy elimination processing is performed on the coded bits of the original depth information based on the correlation between the coded binary data.

In the embodiment of the present invention, the original depth information includes at least one of the following: charge information, phase information, and attribute parameters of the depth information sensing unit.

An embodiment of the present invention also provides a terminal device, comprising a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the information processing method executed by the above-mentioned terminal device when running the computer program.

It should be noted that the depth information sensing unit, the image sensing unit and the video image encoding unit in the embodiment of the present invention may be a depth information sensor, an image sensor and a video image encoder, respectively.

To implement the above information processing method, an embodiment of the present invention further provides a terminal device. The component structure of the terminal device is shown in FIG12. The terminal device 1200 includes:

The receiving unit 1201 is configured to receive a video image code stream, where the video image code stream is obtained by merging and encoding original depth information and video image data, where the original depth information is obtained when a depth information sensing unit obtains depth information of a target object, and the video image data is obtained by an image sensing unit of the target object; the original depth information represents an acquisition state of the depth information sensing unit acquiring the depth information or information other than the acquired depth information;

A decoding unit 1202 is configured to decode the video image code stream to obtain the original depth information and the video image corresponding to the video image data;

The processing unit 1203 is configured to perform image processing on the original depth information and the video image to obtain a target video image.

In the embodiment of the present invention, the decoding unit 1202 is further configured to decode the video image code stream through a video image decoding unit to obtain the original depth information and the video image corresponding to the video image data;

The processing unit 1203 is further configured to perform image processing on the original depth information and the video image through the video image decoding unit to obtain a target video image.

In the embodiment of the present invention, the video image decoding unit and the image processing unit are independent of each other, or the image processing unit is integrated into the video image decoding unit.

In the embodiment of the present invention, the original depth information includes at least one of the following: charge information, phase information, and attribute parameters of the depth information sensing unit.

In the embodiment of the present invention, the processing unit 1203 is further configured as follows:

When the original depth information is charge information, denoising or white balance adjustment is performed on the video image according to the charge information to obtain the target video image.

In the embodiment of the present invention, the processing unit 1203 is further configured as follows:

When the original depth information is phase information, the video image is deblurred according to the phase information to obtain the target video image.

In the embodiment of the present invention, the terminal device further includes:

The generating unit is configured to restore the original depth information to obtain a depth image.

In the embodiment of the present invention, the generating unit is further configured to restore the original depth information through a depth image generating unit to obtain a depth image, thereby obtaining the depth image.

An embodiment of the present invention also provides a terminal device, comprising a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the information processing method executed by the above-mentioned terminal device when running the computer program.

It should be noted that the video image decoding unit, the image processing unit and the depth image generating unit in the embodiment of the present invention may be a video image decoder, an image processor and a depth image generator, respectively.

FIG13 is a schematic diagram of the hardware composition structure of an electronic device (terminal device) according to an embodiment of the present invention. The electronic device 1300 includes: at least one processor 1301, a memory 1302, and at least one network interface 1304. The various components in the electronic device 1300 are coupled together via a bus system 1305. It is understood that the bus system 1305 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1305 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus systems 1305 in FIG13.

It can be understood that the memory 1302 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), ferromagnetic random access memory, flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory 1302 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

The memory 1302 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device 1300. Examples of such data include: any computer program used to operate on the electronic device 1300, such as the application 13021. The program implementing the method of the embodiment of the present invention may be included in the application 13021.

The method disclosed in the above embodiment of the present invention can be applied to the processor 1301, or implemented by the processor 1301. The processor 1301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1301 or the instruction in the form of software. The above processor 1301 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1301 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 1302. The processor 1301 reads the information in the memory 1302 and completes the steps of the above method in combination with its hardware.

In an exemplary embodiment, the electronic device 1300 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general purpose processors, controllers, MCUs, MPUs, or other electronic components to execute the aforementioned method.

An embodiment of the present invention further provides a storage medium for storing a computer program.

Optionally, the storage medium can be applied to the terminal device in the embodiment of the present invention, and the computer program enables the computer to execute the corresponding processes in the various methods in the embodiment of the present invention, which will not be described in detail here for the sake of brevity.

The present invention is described with reference to the flowchart and/or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device configured to implement the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps configured to implement the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (27)

1. An information processing method, the method comprising:

acquiring original depth information corresponding to the depth information under the condition that the depth information of a target object is acquired through a depth information sensor, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensor or information other than the acquired depth information; the original depth information is used for improving the video quality of the video image;

acquiring video image data of the target object through an image sensor;

and carrying out merging coding on the original depth information and the video image data to obtain a video image code stream, and outputting the video image code stream.

2. The method of claim 1, wherein the merging encoding the original depth information and the video image data to obtain a video image code stream comprises:

And merging and encoding the original depth information corresponding to the appointed image frame in the image frames corresponding to the video image data and the video image data to obtain the video image code stream.

3. The method of claim 1, wherein the merging encoding the original depth information and the video image data to obtain a video image code stream comprises:

and merging and encoding the original depth information corresponding to the designated image position and the video image data to obtain the video image code stream.

4. A method according to any one of claims 1 to 3, wherein said combining encoding said original depth information and said video image data to obtain a video image bitstream comprises:

And carrying out mixed coding on the original depth information and the video image data according to the correlation between the original depth information and the video image data to obtain the video image code stream.

5. The method of claim 4, wherein the merging encoding of the original depth information and the video image data results in a video image bitstream, further comprising:

and writing the first coding information corresponding to the original depth information into the appointed position of the second coding information corresponding to the video image data.

6. A method according to any one of claims 1 to 3, wherein said jointly encoding said original depth information and said video image data comprises:

and respectively and independently encoding the original depth information and the video image data to obtain an image video code stream comprising a first code stream and a second code stream, wherein the first code stream is the code stream obtained by encoding the original depth information, and the second code stream is the code stream obtained by encoding the image video data.

7. The method of any of claims 1 to 6, wherein prior to combining encoding the original depth information and the video image data to obtain a video image bitstream, the method further comprises:

preprocessing the original depth information;

the step of carrying out merging coding on the original depth information and the video image data to obtain a video image code stream comprises the following steps:

and merging and encoding the preprocessed original depth information and the video image data to obtain a video image code stream.

8. The method of any of claims 1 to 7, wherein prior to jointly encoding the original depth information and the video image data to obtain a video image bitstream, the method further comprises:

And performing redundancy elimination processing on the original depth information to eliminate redundant information in the original depth information.

9. The method of claim 8, wherein the performing redundancy elimination processing on the original depth information includes at least one of:

redundancy elimination processing is carried out on the original depth information based on phase correlation;

performing redundancy elimination processing on the original depth information based on spatial correlation;

Performing redundancy elimination processing on the original depth information based on time correlation;

Performing redundancy elimination processing on the original depth information based on a specified depth;

performing redundancy elimination processing on the original depth information based on the frequency domain correlation;

redundancy elimination processing is performed on the coded bits of the original depth information based on correlation between the coded binary data.

10. The method of any of claims 1 to 9, wherein the original depth information comprises at least one of: charge information, phase information, and attribute parameters of the depth information sensor.

11. An information processing method, the method comprising:

Receiving a video image code stream, wherein the video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained under the condition that depth information of a target object is obtained through a depth information sensor, and the video image data is obtained through an image sensor; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensor or information beyond the acquired depth information; the original depth information is used for improving the video quality of the video image;

decoding the video image code stream to obtain the original depth information and a video image corresponding to the video image data;

and carrying out image processing on the original depth information and the video image to obtain a target video image.

12. The method of claim 11, wherein,

Decoding the video image code stream through a video image decoder to obtain the original depth information and a video image corresponding to the video image data;

and carrying out image processing on the original depth information and the video image by an image processor to obtain a target video image.

13. The method of claim 12, wherein the video image decoder and the image processor are independent of each other or the image processor is integrated within the video image decoder.

14. The method of any of claims 11 to 13, wherein the original depth information comprises at least one of: charge information, phase information, and attribute parameters of the depth information sensor.

15. The method of claim 14, wherein when the original depth information is charge information, the performing image processing on the original depth information and the video image to obtain a target video image includes:

and denoising or adjusting white balance of the video image according to the charge information to obtain the target video image.

16. The method of claim 14, wherein when the original depth information is phase information, the performing image processing on the original depth information and the video image to obtain a target video image includes:

and performing deblurring processing on the video image according to the phase information to obtain the target video image.

17. The method of any one of claims 11 to 16, wherein the method further comprises:

and recovering the original depth information to obtain a depth image.

18. The method of claim 17, wherein the original depth information is restored by a depth image generator resulting in the depth image.

19. A terminal device, the terminal device comprising:

The first acquisition unit is configured to acquire original depth information corresponding to the depth information under the condition that the depth information of the target object is acquired through the depth information sensing unit, wherein the original depth information represents the acquisition state of the depth information acquired by the depth information sensing unit or information other than the acquired depth information; the original depth information is used for improving the video quality of the video image;

A second acquisition unit configured to acquire video image data of the target object through an image sensing unit;

the coding unit is configured to perform combined coding on the original depth information and the video image data to obtain a video image code stream;

and the output unit is configured to output the video image code stream.

20. The terminal device of claim 19, wherein the encoding unit is further configured to:

And carrying out mixed coding on the original depth information and the video image data according to the correlation between the original depth information and the video image data to obtain the video image code stream.

21. The terminal device of claim 19 or 20, wherein the encoding unit is further configured to:

Encoding the original depth information to obtain first encoded information;

encoding the video image data to obtain second encoded information;

And combining the first coding information and the second coding information to obtain the video image code stream.

22. The terminal device of any of claims 19 to 21, wherein the terminal device further comprises:

a preprocessing unit configured to:

And preprocessing the original depth information before carrying out combined coding on the original depth information and the video image data to obtain a video image code stream.

23. The terminal device of any of claims 19 to 22, wherein the terminal device further comprises:

An elimination unit configured to:

And before the original depth information and the video image data are combined and encoded to obtain a video image code stream, carrying out redundancy elimination processing on the original depth information so as to eliminate redundancy information in the original depth information.

24. A terminal device, the terminal device comprising:

the receiving unit is configured to receive a video image code stream, wherein the video image code stream is obtained by combining and encoding original depth information and video image data, the original depth information is obtained when the depth information of a target object is obtained through the depth information sensing unit, and the video image data is obtained through the image sensing unit; the original depth information characterizes the acquisition state of the depth information acquired by the depth information sensing unit or information beyond the acquired depth information; the original depth information is used for improving the video quality of the video image;

The decoding unit is configured to decode the video image code stream to obtain the original depth information and a video image corresponding to the video image data;

and the processing unit is configured to perform image processing on the original depth information and the video image to obtain a target video image.

25. The terminal device of claim 24, wherein the terminal device further comprises:

and the generating unit is configured to restore the depth information to obtain a depth image.

26. A terminal device comprising a processor and a memory configured to store a computer program capable of running on the processor, wherein the processor is configured to perform the steps of the information processing method of any of the preceding claims 1 to 10 or the steps of the information processing method of any of the preceding claims 11 to 18 when running the computer program.

27. A storage medium storing an executable program which, when executed by a processor, implements the information processing method of any one of the above claims 1 to 10, or implements the information processing method of any one of the above claims 11 to 18.