WO2018032841A1 - Method, device and system for drawing three-dimensional image - Google Patents

Abstract

Disclosed are a method, device and system for drawing a three-dimensional image. The method comprises: separately obtaining an invisible light image obtained by capturing a target from a first viewpoint and a first color image obtained by capturing the target from a second viewpoint; calculating a parallax between the first viewpoint and the second viewpoint by using the invisible light image; moving pixel coordinates of the first color image according to the parallax to obtain a second color image of the first viewpoint; and forming a three-dimensional image by using the first color image and the second color image. The method can improve a three-dimensional display effect.

Description

Method for drawing three-dimensional image, device and system thereof [Technical Field]

The present invention relates to the field of three-dimensional display technology, and in particular, to a method for drawing a three-dimensional image, and an apparatus and system thereof.

【Background technique】

Human eyes have visual differences when viewing objects with a certain distance due to their different positions. It is this parallax that gives people a three-dimensional sensory effect. According to this principle, the three-dimensional display technology generates a three-dimensional effect by respectively receiving the simultaneously acquired binocular images by the corresponding eyes. Since this technology has brought people a new stereoscopic viewing experience, the demand for 3D image resources has increased in recent years.

One of the methods for obtaining a three-dimensional image at present is to convert a two-dimensional image into a three-dimensional image by image processing technology. Specifically, the image depth information of the existing two-dimensional image is calculated by using image processing technology, and then other virtual viewpoint images are drawn, and the three-dimensional image is formed by using the existing two-dimensional image and the virtual other viewpoint image.

Since the depth information of the existing two-dimensional image used to draw the other viewpoint images is calculated, this process leads to the loss of the image detail information and affects the effect of the three-dimensional display.

[Summary of the Invention]

The technical problem mainly solved by the present invention is to provide a method for drawing a three-dimensional image, a device and a system thereof, and capable of improving a three-dimensional display effect.

In order to solve the above technical problem, a technical solution adopted by the present invention is to provide a method for drawing a three-dimensional image, comprising: separately acquiring an invisible light image obtained by acquiring a target with a first viewpoint and respectively aiming at the target with a second viewpoint; Performing the first color image obtained by the acquisition; calculating a parallax between the first viewpoint and the second viewpoint from the invisible image; and moving the pixel coordinates of the first color image according to the parallax to obtain the first a second color image of the viewpoint; a three-dimensional image is formed by the first color image and the second color image.

The invisible light image is obtained by projecting a structured light pattern to the target in a projection module, and acquiring the target by an invisible light image collector disposed at the first viewpoint, where the first color image is obtained by A color camera disposed at the second viewpoint acquires the target.

The calculating the disparity between the first view point and the second view point by the invisible light image comprises: calculating the invisible light including the structured light pattern according to a matching algorithm of digital image processing a displacement between the image and each pixel of the preset reference structured light image; a disparity between the first viewpoint and the second viewpoint is calculated from the displacement, wherein the displacement has a linear relationship with the parallax.

Wherein the disparity between the first view point and the second view point is calculated by the displacement, including: calculating a disparity d between the first view point and the second view point by using Equation 1 below,

Wherein B ₁ is the distance between the invisible image collector and the projection module, B ₂ is the distance between the invisible image collector and the color camera; Z ₀ is the reference structured light image The depth of the plane relative to the invisible image collector; f is the focal length of the invisible image collector and the color camera, and Δu is the displacement between the invisible image and the pixels of the preset reference structured light image.

The second color image of the first view is obtained by moving the pixel coordinates of the first color image according to the parallax, and the first pixel coordinate I _{ir of the} invisible image is established according to the disparity d (u) The correspondence between ^ir , v ^ir ) and the second pixel coordinate I _r (u ^r , v ^r ) of the first color image is: I _ir (u ^ir , v ^ir )=I _r (u ^r +d And v ^r ) setting a pixel value of the first pixel coordinate of the invisible light image as a pixel value of a second pixel coordinate corresponding to the first pixel coordinate in the first color image to form a a second color image of the target at the first viewpoint; and smoothing and denoising the second color image.

The method further includes: calculating, by using the invisible light image, a depth image of the first viewpoint; and using a three-dimensional image transformation theory, calculating the target according to the depth image of the first viewpoint and the first color image a third color image of the first viewpoint;

Forming the three-dimensional image from the first color image and the second color image includes: averaging pixel values of corresponding pixels in the second color image and the third color image Or a weighted average to obtain a fourth color image of the first viewpoint; a three-dimensional image is formed by the first color image and the fourth color image.

The positional relationship between the first viewpoint and the second viewpoint is a positional relationship between the eyes of the human body; the color camera and the invisible image collector and the projection module are on the same straight line; The invisible light image is an infrared image, and the invisible light image collector is an infrared camera.

Wherein, the color camera and the invisible image collector have the same image acquisition target surface size, the same resolution and focal length, and the optical axes are parallel to each other.

In order to solve the above technical problem, the present invention adopts another technical solution to provide an image processing device, which includes an input interface, a processor, and a memory; the input interface is used to obtain an invisible image collector and a color camera. The memory is used to store a computer program; the processor executes the computer program, respectively, by acquiring, by the input interface, the target of the invisible image collector of the first viewpoint a non-visible light image and a first color image obtained by collecting the target with the color camera of the second viewpoint; calculating a parallax between the first viewpoint and the second viewpoint from the invisible image; The parallax moves pixel coordinates of the first color image to obtain a second color image of the first viewpoint; and forms a three-dimensional image from the first color image and the second color image.

In order to solve the above technical problem, the present invention adopts another technical solution to provide a three-dimensional image drawing system, including a projection module, an invisible image collector, a color camera, and the invisible image collector and the color camera. An image processing device, configured to: respectively acquire an invisible light image obtained by acquiring an object by a non-visible light image collector of a first viewpoint and acquiring the target by using a color camera of a second viewpoint a color image; calculating a disparity between the first view point and the second view point by the invisible light image; moving a pixel coordinate of the first color image according to the disparity to obtain a second color of the first view point An image; a three-dimensional image is formed from the first color image and the second color image.

The present invention obtains the parallax of the first viewpoint and the second viewpoint by using the acquired invisible light image of the first viewpoint, and obtains the second color image of the second viewpoint by using the first color image of the second viewpoint and the parallax, and further A color image and a second color image form a three-dimensional image, and since the parallax of the first viewpoint and the second viewpoint is obtained from the acquired image data without image processing, the loss of image detail information is reduced, so as to obtain more accurate Color map of two viewpoints The image, in turn, reduces the distortion of the synthesized three-dimensional image and improves the three-dimensional display effect based on the two-dimensional image generation. Moreover, compared with the existing DIBR technology, the embodiment does not need to calculate the depth information of the image, avoids the error introduced by repeated calculations, and further improves the three-dimensional display effect.

[Description of the Drawings]

1 is a flow chart of an embodiment of a method for drawing a three-dimensional image according to the present invention;

2 is a schematic diagram of an application scenario of a method for drawing a three-dimensional image according to the present invention;

3 is a partial flow chart of another embodiment of a method for drawing a three-dimensional image according to the present invention;

4 is a partial flow chart of still another embodiment of a method for drawing a three-dimensional image according to the present invention;

5 is a flow chart of still another embodiment of a method for drawing a three-dimensional image of the present invention;

6 is a schematic structural view of an embodiment of a three-dimensional image drawing apparatus according to the present invention;

7 is a schematic structural view of an embodiment of a three-dimensional image rendering system of the present invention;

Figure 8 is a block diagram showing another embodiment of the three-dimensional image rendering system of the present invention.

【Detailed ways】

For a better understanding of the technical solutions of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

The terms used in the embodiments of the present invention are for the purpose of describing particular embodiments only and are not intended to limit the invention. The singular forms "a", "the" and "the" It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

Please refer to FIG. 1. FIG. 1 is a flow chart of an embodiment of a method for drawing a three-dimensional image according to the present invention. In this embodiment, the method can be performed by a three-dimensional image rendering device, including the following steps:

S11: Acquire an invisible light image obtained by collecting the target with the first viewpoint and a first color image obtained by acquiring the target by the second viewpoint.

It should be noted that the invisible light image and the color image according to the present invention are both two-dimensional images. The invisible light image is an image formed by acquiring the intensity of invisible light on the target.

Wherein the first viewpoint and the second viewpoint are located at different positions of the target to obtain the target An image at two viewpoints. Generally, since the three-dimensional sensory is formed by superimposing different images viewed by both eyes, the first viewpoint and the second viewpoint are used as two viewpoints of the eyes of the human body, that is, the positional relationship between the first viewpoint and the second viewpoint is The positional relationship between the eyes of the human body. For example, if the distance between the eyes of the conventional human body is t, the distance between the first viewpoint and the second viewpoint is set to t, which is specifically 6.5 cm. Moreover, in order to ensure that the image depths of the first view and the second view are the same or similar, the first view and the second view are set to be the same distance as the target or the distance does not exceed a set threshold. In a specific application, the device The threshold can be set to a value of no more than 10 cm or 20 cm.

In a specific application, as shown in FIG. 2, the invisible light image is a projected light pattern projected onto the target 23 by the projection module 25, and the invisible light image collector 21 disposed at the first viewpoint The target 23 is acquired, and the first color image is acquired by the color camera 22 disposed at the second viewpoint. The invisible light image collector 21 and the color camera transmit the acquired images thereof to the three-dimensional image drawing device 24 to perform acquisition of the following three-dimensional images. Since the position of the color camera and the invisible image collector is different, the spatial three-dimensional points corresponding to the same pixel coordinates in the first color image and the invisible image are not the same. In FIG. 2, the color camera 22 and the invisible light image collector 21 and the projection module 25 are on the same line, so that the color camera 22 and the invisible light image collector 21 and the projection module 25 are The depth of the target is the same. Of course, FIG. 2 is only used as an embodiment. In other applications, the above three types may not be on the same line.

Specifically, the projection module 25 is generally composed of a laser and a diffractive optical element. The laser may be an edge-emitting laser or a vertical cavity laser, which is an invisible light that can be collected by the invisible image collector. The diffractive optical element may be configured to have functions such as collimation, splitting, diffusion, etc. according to different structural light patterns. The structured light pattern may be an irregularly distributed speckle pattern, and the speckle center level needs to meet the requirements for harmlessness to the human body. Therefore, it is necessary to comprehensively consider the power of the laser and the arrangement of the diffractive optical element.

The intensity of the speckle pattern affects the speed and accuracy of the depth value calculation. The more speckle particles, the slower the calculation speed, but the higher the accuracy. Therefore, the projection module 25 can select an appropriate speckle particle density according to the approximate depth of the target area of the captured image, and still has a high calculation precision while ensuring the calculation speed. Of course, the speckle particle density can also be determined by the three-dimensional image rendering device 24 according to its own calculation requirements, and the determined density information is sent to the projection module 25.

The projection module 25 is, but is not limited to, projecting the speckle particle pattern at a certain diffusion angle to the target area.

After the projection module 25 projects the structured light image to the target, the invisible light image collector 21 collects the invisible light image of the target. Specifically, the invisible light may be any invisible light. For example, the invisible light image collector 21 may be an infrared collector, such as an infrared camera, and the invisible image is an infrared image; or the invisible image collector 21 may be an ultraviolet collector. Such as an ultraviolet camera, the invisible image is an ultraviolet image.

In order to achieve a good acquisition effect and avoid unnecessary calculations, the color camera and the invisible image collector can be set to be synchronously acquired and the number of acquisition frames is the same, so that the obtained color image and the invisible image can ensure a one-to-one correspondence. Easy for subsequent calculations.

S12: Calculate a disparity between the first view point and the second view point from the invisible light image.

For example, a matching algorithm such as a digital image correlation (DIC) algorithm using digital image processing calculates a parallax between an image of the first viewpoint and an image of the second viewpoint, that is, an image of the first viewpoint and a pixel of the second viewpoint image. The relative positional relationship between the coordinates.

S13: Move pixel coordinates of the first color image according to the parallax to obtain a second color image of the first view.

For example, the pixel coordinates of the first color image are shifted by the image disparity value d corresponding to the respective pixels, wherein the pixel values (also referred to as RGB values) of the obtained pixel coordinates (u ₁ +d, v ₁ ) are The pixel value of the pixel coordinates (u ₁ , v ₁ ) in a color image.

S14: Forming a three-dimensional image from the first color image and the second color image.

For example, the first color image and the second color image are respectively used as a human body binocular image to synthesize a three-dimensional image, and specifically may be a three-dimensional image for 3D display in a top-bottom format, a left-right format, or a red-blue format. Further, after the three-dimensional image is synthesized, the three-dimensional image may also be displayed or output to a connected external display device for display.

In this embodiment, the disparity image of the first viewpoint and the second viewpoint are obtained by using the acquired invisible light image of the first viewpoint, and the second color image of the second viewpoint is obtained by using the first color image of the second viewpoint and the parallax. Further, the first color image and the second color image are formed into a three-dimensional image. Since the parallax of the first viewpoint and the second viewpoint is obtained from the acquired image data without image processing, the loss of image detail information is reduced. More accurate access to two viewpoints The color image, in turn, reduces the distortion of the synthesized three-dimensional image and improves the three-dimensional display effect generated based on the two-dimensional image. Moreover, compared with the existing depth-image-based rendering (DIBR) technology, the embodiment does not need to calculate the depth information of the image, avoids the error introduced by repeated calculations, and further improves the three-dimensional display effect.

Referring to FIG. 3, in another embodiment, the invisible light image is a projected light pattern projected onto the target by the projection module, and the target is collected by an invisible light image collector disposed at the first viewpoint. It is obtained that the first color image is obtained by collecting the target by a color camera disposed at the second viewpoint. The difference between this embodiment and the above embodiment is that the foregoing S12 includes the following sub-steps:

S121: Calculate a displacement between the invisible light image including the structured light pattern and each pixel of the preset reference structured light image according to a matching algorithm of the digital image processing.

The matching algorithm of the digital image processing is a digital image correlation algorithm. The reference structured light pattern is obtained by previously projecting a reference structured light pattern onto a plane of a set distance by using the set projection module, and acquiring the reference structured light pattern of the plane by using the set invisible light image collector, The "set up" should be understood as that the image collector and the projection module are not moved when the invisible image is subsequently acquired after being set.

For example, a digital image correlation algorithm is used to obtain a displacement value Δu of each corresponding pixel between the invisible light image and the reference structured light pattern such as the reference speckle image. At present, the measurement accuracy of the digital image correlation algorithm can reach sub-pixel level, such as 1/8 pixel, that is, the value of Δu will be a multiple of 1/8, and the unit is pixel.

S122: Calculate a disparity between the first viewpoint and the second viewpoint by the displacement.

The displacement between the invisible image and each pixel of the reference structured light image has a linear relationship with the parallax. Therefore, the disparity between the first viewpoint and the second viewpoint can be calculated by the displacement and its linear relationship.

For example, the parallax d between the first viewpoint and the second viewpoint is calculated by the following formula 11,

Wherein B ₁ is the distance between the invisible image collector and the projection module, B ₂ is the distance between the invisible image collector and the color camera; Z ₀ is the reference structured light image The depth of the plane relative to the invisible image collector; f is the focal length of the invisible image collector and the color camera, and Δu is the displacement between the invisible image and the pixels of the preset reference structured light image. The plane of the reference structured light image is the plane on which the reference structured light pattern is projected, and the Z _{0 is} used to indicate the distance between the plane and the image collector, which can be used when the reference structured light image is previously tested. Distance information is obtained. In this embodiment, the unit of f is a pixel, and the value of f can be obtained by calibration in advance.

When the calculated value of the parallax d is not an integer, it may be rounded or rounded.

Referring to FIG. 4, in still another embodiment, it differs from the above embodiment in that the above 13 includes the following sub-steps:

S131: Establish a correspondence between a first pixel coordinate of the invisible light image and a second pixel coordinate of the first color image according to a parallax.

For example, based on the disparity d, establishing a first pixel coordinate I _ir (u ^ir , v ^ir ) of the invisible light image and a second pixel coordinate I _r (u ^r , v ^r ) of the first color image The correspondence is: I _ir (u ^ir , v ^ir )=I _r (u ^r +d, v ^r ).

S132: setting a pixel value of the first pixel coordinate of the invisible light image as a pixel value of a second pixel coordinate corresponding to the first pixel coordinate in the first color image to form the target A second color image of the first viewpoint.

For example, a pixel value (also referred to as an RGB value) of the first color image is assigned to the invisible light image according to the correspondence relationship to generate a second color image. Taking one of the pixel coordinates of the image as an example, if d is 1, the pixel coordinates (1, 1) of the invisible light image correspond to the pixel coordinates (2, 1) of the first color image. Then, the pixel value of the pixel coordinate (1, 1) of the invisible light image is set as the pixel value (r, g, b) of the pixel coordinate (2, 1) in the first color image.

S133: Perform smoothing and denoising processing on the second color image.

Since the data of the displacement value Δu often has some bad points, some voids and other problems appear in the final color image, and the data will be amplified when further processing is performed in the subsequent steps, thereby seriously affecting the three-dimensional display effect, in order to avoid the depth image. The effect of the bad point or area data on the three-dimensional display, the sub-step performs denoising and smoothing on the obtained second color image.

Of course, in other embodiments, the foregoing step S13 may include only the foregoing S131 and S132. Substeps.

Referring to FIG. 5, in still another embodiment, after the foregoing S11, the following steps are further included:

S15: Calculate a depth image of the first viewpoint by using the invisible light image.

For example, the depth image of the first viewpoint is calculated by using an infrared image, and the specific calculation manner may adopt an existing correlation algorithm.

S16: Calculate a third color image of the target at the first viewpoint according to the depth image of the first viewpoint and the first color image by using a three-dimensional image transformation theory.

According to the theory of 3D Image Wrapping, any three-dimensional coordinate point in space and two-dimensional coordinate points on the image acquisition plane can be related by the theory of transmission transformation, so the theory can be the first viewpoint and the second viewpoint. The pixel coordinates of the image are associated, and the pixel value of the corresponding pixel coordinate in the first color image of the second viewpoint is set for the image pixel coordinates of the first viewpoint according to the correspondence relationship and the pixel value of the first color image of the second viewpoint.

For example, the S16 includes the following substeps:

a: using the following formula 12 to calculate the first pixel coordinate (u _D , v _D ) of the depth image of the first viewpoint and the second pixel coordinate (u _R , v _R ) of the first color image Correspondence relationship,

Wherein, the Z _D is depth information in the first depth image, indicating a depth value of the target distance from the depth camera; and Z _R represents a depth value of the target distance from the color camera;

a pixel homogeneous coordinate on an image coordinate system of the color camera;

The homogeneous coordinates for the pixel in the depth image coordinate system of the camera; M _g is the internal reference matrix color camera, M _D is the matrix of intrinsic depth camera; R & lt depth camera is a color camera with respect to the external reference matrix In the rotation matrix, T is the translation matrix in the outer parameter matrix of the depth camera relative to the color camera.

The internal reference matrix and the external parameter matrix of the camera and the collector may be preset, and the internal reference matrix may be calculated according to setting parameters of the camera and the collector, and the external reference matrix may be between the invisible image collector and the color camera. The positional relationship is determined. In one embodiment, the internal parameter matrix formed by the pixel focal length of the image capture lens of the camera and the collector and the central position coordinates of the image acquisition target surface. Since the positional relationship between the first viewpoint and the second viewpoint is set to the positional relationship of the eyes of the human eye, there is no relative rotation between the eyes of the human body and only the distance of the set value t, so the rotation of the color camera with respect to the invisible image collector The matrix R is an identity matrix, and the translation matrix T = [t, 0, 0] ^-1 .

Further, the set value t can be adjusted according to the distance between the invisible light image collector and the color camera and the target. In still another embodiment, before the foregoing S11, the method further includes: acquiring a distance between the target and the invisible image collector and the color camera; and determining a distance between the target and the invisible image collector and the color camera When the value is greater than the first distance value, the set value t is increased; when it is determined that the distance between the target and the invisible image collector and the color camera is less than the second distance value, the setting is The setting value t is small.

The first distance value is greater than or equal to the second distance value. For example, when the distance between the target and the invisible image collector is 100 cm, the distance between the target and the color camera is also 100 cm, and since the 100 cm is smaller than the second distance value of 200 cm, the set value is reduced by one step value, or according to the current target. The distance between the invisible image collector and the color camera is calculated and adjusted. When the distance between the target and the invisible image collector and the color camera is 300 cm, since the 300 cm is larger than the second distance value 200 and smaller than the first distance value of 500 cm, the set value is not adjusted.

b: setting a pixel value of the first pixel coordinate of the invisible light image to a pixel value of a second pixel coordinate corresponding to the first pixel coordinate in the first color image to form the target The third color image of the first viewpoint.

For example, after substituting the depth information Z _D of the invisible light image of the first viewpoint into the above formula 12, the depth information of the second viewpoint on the left side of the formula 12, that is, the depth information Z _{R of the} first color image, and the first color can be obtained. Pixel homogeneous coordinates of the image coordinate system of the image

In this embodiment, the invisible light image collector and the color camera are at the same distance from the target, that is, the obtained Z _R and Z _D are equal. Homogeneous coordinates by pixel

Obtaining a second pixel coordinate (u _R , v _R ) of the first color image in one-to-one correspondence with the first pixel coordinates (u _D , v _D ) of the invisible light image, for example, the corresponding relationship is (u _R , v _R )=( u _D +d, v _D ). Then, according to the correspondence relationship, the pixel value of the first color image is assigned to the invisible light image to generate a third color image.

In still another embodiment, the foregoing S14 includes the following steps:

S141: Average or weight average the pixel values of the corresponding pixels in the second color image and the third color image to obtain a fourth color image of the first view.

Taking one pixel coordinate in the color image as an example, the pixel values of the pixel coordinates (Ur, Vr) in the second color image and the third color image are (r1, g1, b1) and (r2, g2, b2), respectively. Setting the pixel value of the pixel coordinates (Ur, Vr) in the fourth color image of the first viewpoint to

S142: Form a three-dimensional image from the first color image and the fourth color image.

For example, the first color image and the fourth color image are respectively used as a human binocular image to synthesize a three-dimensional image.

It can be understood that, in the foregoing embodiment, the image acquisition target surface of the invisible light image collector and the color camera may be set to be equal in size, the resolution is the same, and the focal length is the same. Alternatively, the color camera and the invisible image collector have different image acquisition target sizes, resolutions, and focal lengths, for example, the color camera has a larger target size and resolution than the invisible image collector. After the above S13, the obtaining method further comprises: performing interpolation and segmentation processing on the first color image and/or the second color image, so that the first color image and the second color image correspond to The target area is the same and the image size and resolution are the same. Since the color camera and the invisible image collector have errors in assembly, the image acquisition target surface of the invisible image collector and the color camera is equal in size, the resolution is the same, and the focal length is the same: the invisible image collector The image acquisition target size, resolution, and focal length of the color camera are the same within the tolerance range.

Moreover, the image includes a photo or a video, and when the image is a video, the acquisition frequency of the invisible image collector and the color camera is synchronized, or if the acquisition frequency of the invisible image collector and the color camera are not synchronized, the image is passed. Interpolation is used to obtain video images of consistent frequency.

Please refer to FIG. 6. FIG. 6 is a schematic structural diagram of an embodiment of a three-dimensional image drawing apparatus according to the present invention. In this embodiment, the drawing device 60 includes an obtaining module 61, a calculating module 62, a forming module 63, and a getting module 64. among them,

The acquiring module 61 is configured to respectively acquire an invisible light image obtained by acquiring the target by the first viewpoint and a first color image obtained by collecting the target by the second viewpoint;

The calculating module 62 is configured to calculate a disparity between the first view point and the second view point from the invisible light image;

The obtaining module 64 is configured to move the pixel coordinates of the first color image according to the parallax to obtain a second color image of the first view point;

The forming module 63 is configured to form a three-dimensional image from the first color image and the second color image.

Optionally, the invisible light image is obtained by projecting a structured light pattern to the target in a projection module, and acquiring the target by an invisible light image collector disposed at the first viewpoint, the first color The image is acquired by the color camera disposed at the second viewpoint.

Optionally, the calculating module 62 is specifically configured to calculate, according to a matching algorithm of the digital image processing, a displacement between the invisible light image including the structured light pattern and each pixel of the preset reference structured light image; The displacement calculation calculates a disparity between the first viewpoint and the second viewpoint, wherein the displacement has a linear relationship with the parallax.

Further optionally, the calculating module 62 performs the disparity calculation between the first viewpoint and the second viewpoint by the displacement calculation, including: calculating, between the first viewpoint and the second viewpoint, by using the above formula 11 Parallax d.

Optionally, the obtaining module 64 is specifically configured to establish, according to the disparity d, a first pixel coordinate I _ir (u ^ir , v ^ir ) of the invisible light image and a second pixel coordinate I _r (u of the first color image) The correspondence between ^r , v ^r ) is: I _ir (u ^ir , v ^ir )=I _r (u ^r +d, v ^r ); setting the pixel value of the first pixel coordinate of the invisible light image to a pixel value of a second pixel coordinate corresponding to the first pixel coordinate in the first color image to form a second color image of the target at a first viewpoint; smoothing the second color image Denoising processing.

Optionally, the calculating module 62 is further configured to calculate, by using the invisible light image, a depth image of the first view; and use a three-dimensional image transform theory to calculate, according to the depth image of the first view and the first color image. Obtaining a third color image of the target at a first viewpoint; the forming module 63 is configured to average or weight average the pixel values of the corresponding pixels in the second color image and the third color image to obtain a a fourth color image of the first viewpoint; a three-dimensional image is formed by the first color image and the fourth color image.

Optionally, the positional relationship between the first viewpoint and the second viewpoint is a positional relationship between the eyes of the human body; the color camera and the invisible image collector and the projection module are on the same straight line; The invisible light image is an infrared image, and the invisible light image collector For infrared cameras.

Optionally, the color camera and the invisible image collector have the same image acquisition target surface size, the same resolution and focal length, and the optical axes are parallel to each other.

Optionally, the invisible light image and the first color image are photos or videos, and the invisible light image collector and the color camera are collected when the invisible light image and the first color image are video. Frequency synchronization, or if the acquisition frequency of the invisible image collector and the color camera are not synchronized, a video image of the same frequency is obtained by image interpolation.

The above-mentioned modules of the drawing device are respectively used to perform the corresponding steps in the foregoing method embodiments, and the specific execution process is described in the foregoing method embodiment, and details are not described herein.

Please refer to FIG. 7. FIG. 7 is a schematic structural diagram of an embodiment of a three-dimensional image rendering system according to the present invention. In this embodiment, the system 70 includes a projection module 74, an invisible image collector 71, a color camera 72, and an image processing device 73 connected to the invisible image collector 71 and the color camera 72. The image processing device 73 includes an input interface 731, a processor 732, and a memory 733. Further, the image processing device 73 can also be connected to the projection module 74.

The input interface 731 is used to obtain images acquired by the invisible image collector 71 and the color camera 72.

The memory 733 is used to store a computer program and provide the computer program to the processor 732, and can store data used by the processor 732 for processing such as the internal parameter matrix and the external parameter matrix of the invisible light image collector 71 and the color camera 72. And the image obtained by the input interface 731.

The processor 732 is used to:

Obtaining, by the input interface 731, the invisible light image obtained by collecting the target by the invisible light image collector 71 of the first viewpoint and the first color image obtained by collecting the target by the color camera 72 of the second viewpoint;

Calculating a parallax between the first viewpoint and the second viewpoint from the invisible image;

Moving the pixel coordinates of the first color image according to the parallax to obtain a second color image of the first viewpoint;

A three-dimensional image is formed from the first color image and the second color image.

In this embodiment, the image processing device 73 may further include a display screen 734 for displaying the three-dimensional image to implement three-dimensional display. Of course, in another embodiment, the image processing device 73 is not used to display the three-dimensional image. As shown in FIG. 8, the three-dimensional image rendering system 70 further A display device 75 connected to the image processing device 73 for receiving a three-dimensional image output by the image processing device 73 and displaying the three-dimensional image is included.

Optionally, the processor 732 is specifically configured to calculate, according to a matching algorithm of the digital image processing, a displacement between the invisible light image including the structured light pattern and each pixel of the preset reference structured light image; The displacement calculation calculates a disparity between the first viewpoint and the second viewpoint, wherein the displacement has a linear relationship with the parallax.

Further optionally, the processor 732 performs the disparity calculation between the first view point and the second view point by the displacement calculation, including: calculating the first view point and the second view point by using Equation 1 below Parallax d between.

Optionally, the processor 732 performs the moving the pixel coordinates of the first color image according to the parallax to obtain a second color image of the first view, including: establishing a first image of the invisible image according to the disparity d The correspondence relationship between the pixel coordinates I _ir (u ^ir , v ^ir ) and the second pixel coordinates I _r (u ^r , v ^r ) of the first color image is: I _ir (u ^ir , v ^ir )=I _r (u ^r +d, v ^r ); setting a pixel value of the first pixel coordinate of the invisible light image to a second pixel coordinate of the first color image having a corresponding relationship with the first pixel coordinate a pixel value to form a second color image of the target at a first viewpoint; and smoothing, denoising the second color image.

Optionally, the processor 732 is further configured to calculate, by using the invisible light image, a depth image of the first view; and use a three-dimensional image transform theory to calculate, according to the depth image of the first view and the first color image. Obtaining a third color image of the target at a first viewpoint; the processor 732 performing the forming of the three-dimensional image by the first color image and the second color image, comprising: the second color image and the The pixel values of the corresponding pixels in the third color image are averaged or weighted averaged to obtain a fourth color image of the first viewpoint; and the three-dimensional image is formed by the first color image and the fourth color image.

Optionally, the positional relationship between the first viewpoint and the second viewpoint is a positional relationship between the eyes of the human body; the color camera 72 and the invisible image collector 71 and the projection module 74 are in the same On the straight line; the invisible light image is an infrared image, and the invisible light image collector 71 is an infrared camera.

Optionally, the color camera 72 and the invisible light image collector 71 have the same image acquisition target surface size, the same resolution and focal length, and the optical axes are parallel to each other.

Optionally, the invisible light image and the first color image are photos or videos, when When the invisible light image and the first color image are video, the acquisition frequency of the invisible image collector and the color camera is synchronized, or if the acquisition frequency of the invisible image collector and the color camera are not synchronized, the image is passed Interpolation is used to obtain video images of consistent frequency.

The image processing device 73 can be used as the above-described three-dimensional image drawing device for executing the method described in the above embodiments. For example, the method disclosed in the above embodiments of the present invention may also be applied to the processor 732 or implemented by the processor 732. Processor 732 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 732 or an instruction in a form of software. The processor 732 described above may be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or discrete hardware. Component. The methods, steps, and logical block diagrams disclosed in the embodiments of the present invention may be implemented or carried out. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present invention may be directly implemented by the hardware decoding processor, or may be performed by a combination of hardware and software modules in the decoding processor. The software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like. The storage medium is located in the memory 733, and the processor 732 reads the information in the corresponding memory and completes the steps of the above method in combination with the hardware thereof.

In the above solution, the disparity image of the first viewpoint and the second viewpoint are obtained by using the acquired invisible image of the first viewpoint, and the second color image of the second viewpoint is obtained by using the first color image of the second viewpoint and the parallax, and further Forming a three-dimensional image from the first color image and the second color image, since the parallax of the first viewpoint and the second viewpoint is obtained from the acquired image data without image processing, thereby reducing the loss of image detail information, thereby Accurately obtaining color images of two viewpoints, thereby reducing the distortion of the synthesized three-dimensional image and improving the three-dimensional display effect generated based on the two-dimensional image. Moreover, compared with the existing DIBR technology, it is not necessary to calculate the depth information of the image, and the error introduced by the repeated calculation is avoided, thereby further improving the three-dimensional display effect.

The above is only the embodiment of the present invention, and is not intended to limit the scope of the invention, and the equivalent structure or equivalent process transformations made by the description of the invention and the drawings are directly or indirectly applied to other related technologies. The fields are all included in the scope of patent protection of the present invention.

Claims (20)

1. A method of drawing a three-dimensional image, comprising:

   Acquiring the invisible light image obtained by acquiring the target with the first viewpoint and the first color image obtained by collecting the target by the second viewpoint respectively;

   Calculating a parallax between the first viewpoint and the second viewpoint from the invisible image;

   Moving the pixel coordinates of the first color image according to the parallax to obtain a second color image of the first viewpoint;

   A three-dimensional image is formed from the first color image and the second color image.
2. The method according to claim 1, wherein the invisible light image is a projected light pattern projected onto the target by the projection module, and the target is collected by an invisible light image collector disposed at the first viewpoint. It is obtained that the first color image is obtained by collecting the target by a color camera disposed at the second viewpoint.
3. The method of claim 2, wherein the calculating the disparity between the first viewpoint and the second viewpoint from the invisible light image comprises:

   Calculating a displacement between the invisible light image including the structured light pattern and each pixel of the preset reference structured light image according to a matching algorithm of the digital image processing;

   A parallax between the first viewpoint and the second viewpoint is calculated from the displacement, wherein the displacement has a linear relationship with the parallax.
4. The method of claim 3, wherein the calculating the disparity between the first viewpoint and the second viewpoint by the displacement comprises:

   Calculating the parallax d between the first viewpoint and the second viewpoint using Equation 1 below,

   

   Wherein B ₁ is the distance between the invisible image collector and the projection module, B ₂ is the distance between the invisible image collector and the color camera; Z ₀ is the reference structured light image The depth of the plane relative to the invisible image collector; f is the focal length of the invisible image collector and the color camera, and Δu is the displacement between the invisible image and the pixels of the preset reference structured light image.
5. The method according to claim 1, wherein the moving the pixel coordinates of the first color image according to the parallax to obtain a second color image of the first viewpoint comprises:

   Corresponding relationship between the first pixel coordinate L _ir (u ^ir , v ^ir ) of the invisible light image and the second pixel coordinate I _r (u ^r , v ^r ) of the first color image is established according to the parallax d for:

   I _ir (u ^ir ,v ^ir )=I _R (u ^r +d,v ^r );

   Setting a pixel value of the first pixel coordinate of the invisible light image as a pixel value of a second pixel coordinate corresponding to the first pixel coordinate in the first color image to form the target at the first a second color image of the viewpoint;

   Smoothing and denoising the second color image.
6. The method of claim 2, further comprising:

   Calculating a depth image of the first viewpoint by using the invisible light image;

   Calculating, by using a three-dimensional image transformation theory, a third color image of the target at a first viewpoint according to the depth image of the first viewpoint and the first color image;

   Forming the three-dimensional image by the first color image and the second color image includes:

   And averaging or weighting the pixel values of the corresponding pixels in the second color image and the third color image to obtain a fourth color image of the first view;

   A three-dimensional image is formed by the first color image and the fourth color image.
7. The method according to claim 2, wherein the positional relationship between the first viewpoint and the second viewpoint is a positional relationship between the eyes of the human body; the color camera and the invisible image collector and the projection The modules are on the same line; the invisible image is an infrared image, and the invisible image collector is an infrared camera.
8. The method according to claim 1, wherein the color camera and the invisible image collector have the same image acquisition target surface size, the same resolution and focal length, and the optical axes are parallel to each other.
9. An image processing device including an input interface, a processor, and a memory;

   The input interface is configured to obtain an image acquired by an invisible image collector and a color camera;

   The memory is for storing a computer program;

   The processor executes the computer program for:

   Obtaining, by the input interface, the invisible light image obtained by acquiring the target by the invisible light image collector of the first viewpoint and the first color image obtained by acquiring the target by the color camera of the second viewpoint ;

   Calculating a parallax between the first viewpoint and the second viewpoint from the invisible image;

   Moving the pixel coordinates of the first color image according to the parallax to obtain a second color image of the first viewpoint;

   A three-dimensional image is formed from the first color image and the second color image.
10. The image processing device according to claim 9, wherein the invisible light image is a projected light pattern projected onto the target by the projection module, and the target is set by the invisible light image collector disposed at the first viewpoint The acquisition is performed, and the first color image is obtained by collecting the target by a color camera disposed at the second viewpoint.
11. The image processing device according to claim 10, wherein the processor is specifically configured to:

    Calculating a displacement between the invisible light image including the structured light pattern and each pixel of the preset reference structured light image according to a matching algorithm of the digital image processing;

    A parallax between the first viewpoint and the second viewpoint is calculated from the displacement, wherein the displacement has a linear relationship with the parallax.
12. The image processing device according to claim 11, wherein said processor performs said disparity calculation between said first viewpoint and said second viewpoint by said displacement calculation, comprising:

    Calculating the parallax d between the first viewpoint and the second viewpoint using Equation 1 below,

    

    Wherein B ₁ is the distance between the invisible image collector and the projection module, B ₂ is the distance between the invisible image collector and the color camera; Z ₀ is the reference structured light image The depth of the plane relative to the invisible image collector; f is the focal length of the invisible image collector and the color camera, and Δu is the displacement between the invisible image and the pixels of the preset reference structured light image.
13. The image processing device according to claim 9, wherein the processor performs the moving the pixel coordinates of the first color image according to the parallax to obtain a second color image of the first viewpoint, comprising:

    Corresponding relationship between the first pixel coordinate I _ir (u ^ir , v ^ir ) of the invisible light image and the second pixel coordinate I _r (u ^r , v ^r ) of the first color image is established according to the parallax d for:

    I _ir (u ^ir ,v ^ir )=I _r (u ^r +d,v ^r );

    Setting a pixel value of the first pixel coordinate of the invisible light image as a pixel value of a second pixel coordinate corresponding to the first pixel coordinate in the first color image to form the target at the first a second color image of the viewpoint;

    Smoothing and denoising the second color image.
14. The image processing device according to claim 10, wherein the processor is further configured to:

    Calculating a depth image of the first viewpoint by using the invisible light image;

    Calculating, by using a three-dimensional image transformation theory, a third color image of the target at a first viewpoint according to the depth image of the first viewpoint and the first color image;

    Forming the three-dimensional image by the first color image and the second color image includes:

    And averaging or weighting the pixel values of the corresponding pixels in the second color image and the third color image to obtain a fourth color image of the first view;

    A three-dimensional image is formed by the first color image and the fourth color image.
15. The image processing device according to claim 10, further comprising a display screen for displaying said three-dimensional image.
16. A three-dimensional image drawing system, comprising: a projection module, an invisible image collector, a color camera, an image processing device connected to the invisible image collector and a color camera;

    The image processing device is used to:

    Obtaining, respectively, an invisible light image obtained by acquiring the target by the invisible light image collector of the first viewpoint and a first color image obtained by collecting the target by the color camera of the second viewpoint;

    Calculating a parallax between the first viewpoint and the second viewpoint from the invisible image;

    Moving the pixel coordinates of the first color image according to the parallax to obtain a second color image of the first viewpoint;

    A three-dimensional image is formed from the first color image and the second color image.
17. The three-dimensional image rendering system according to claim 16, wherein the invisible light image is a projected light pattern to the target in the projection module, and the invisible light image collector pair disposed at the first viewpoint The target is acquired, and the first color image is obtained by collecting the target by a color camera disposed at the second viewpoint.
18. A three-dimensional image rendering system according to claim 17, wherein said image processing The device is also used to:

    Calculating a depth image of the first viewpoint by using the invisible light image;

    Calculating, by using a three-dimensional image transformation theory, a third color image of the target at a first viewpoint according to the depth image of the first viewpoint and the first color image;

    Forming the three-dimensional image by the first color image and the second color image includes:

    And averaging or weighting the pixel values of the corresponding pixels in the second color image and the third color image to obtain a fourth color image of the first view;

    A three-dimensional image is formed by the first color image and the fourth color image.
19. The three-dimensional image rendering system according to claim 17, wherein a positional relationship between the first viewpoint and the second viewpoint is a positional relationship between the eyes of the human body; the color camera and the invisible image collector and The projection modules are on the same line; the invisible image is an infrared image, and the invisible image collector is an infrared camera; and/or

    The color camera and the invisible image collector have the same image acquisition target surface size, the same resolution and focal length, and the optical axes are parallel to each other.
20. The three-dimensional image drawing system according to claim 16, further comprising a display device connected to said image processing device, said display device for displaying said three-dimensional image output by said image processing device.

    

    

    

    

Images