WO2021253308A1

WO2021253308A1 - Image acquisition apparatus

Info

Publication number: WO2021253308A1
Application number: PCT/CN2020/096727
Authority: WO
Inventors: 吕萌
Original assignee: 深圳市汇顶科技股份有限公司
Priority date: 2020-06-18
Filing date: 2020-06-18
Publication date: 2021-12-23

Abstract

The embodiments of the present application disclose an image acquisition apparatus, which can improve the quality of acquired three-dimensional data. The image acquisition apparatus comprises: a micro lens array, comprising a plurality of micro lenses; a plurality of pixel units, corresponding to the plurality of micro lenses on a one-to-one basis, each pixel unit among the plurality of pixel units comprises a first pixel and a second pixel, and each micro lens being used for converging an optical signal returned by an object to the corresponding pixel unit, enabling the same angles of the optical signals received by the plurality of first pixels in the plurality of pixel units, and the same angles of the optical signals received by the plurality of second pixels; and a processing circuit for obtaining a first-viewing-angle image according to an output of the plurality of first pixels, obtaining a second-viewing-angle image according to an output of the plurality of second pixels, obtaining a TOF image of the object according to an output of the plurality of pixel units, and generating target three-dimensional data of the object at least according to the TOF image, the first-viewing-angle image, and the second-viewing-angle image.

Description

Image acquisition device

Technical field

The embodiments of the present application relate to the field of imaging, and more specifically, to an image acquisition device.

Background technique

In-depth measurement or three-dimensional data are increasingly being used in consumer-level electronic products. Typical technologies include binocular vision solutions, structured light solutions, and time of flight (TOF) solutions. The TOF solution has become more and more widely used due to its advantages such as no baseline requirement, compact structure, fast speed, and simple algorithm. It has been applied to more and more smart phones. However, consumer TOF generally requires low price, small size, and low power consumption. However, the quality of 3D data obtained by TOF under these conditions is poor, which obviously limits its use in 3D face recognition, photo background blur, somatosensory games, etc. Application effects and user experience.

Summary of the invention

The embodiment of the present application provides an image acquisition device, which can improve the quality of the acquired three-dimensional data.

In a first aspect, an image acquisition device is provided, including: a microlens array including a plurality of microlenses; a plurality of pixel units corresponding to the plurality of microlenses, each of the plurality of pixel units The pixel unit includes a first pixel and a second pixel, and each microlens is used to converge the light signal returned by the object to a corresponding pixel unit, and make the light received by a plurality of first pixels in the plurality of pixel units The angles of the signals are the same, and the angles of the light signals received by the plurality of second pixels are the same; the processing circuit is used to obtain a first viewing angle image according to the output of the plurality of first pixels, and according to the output of the plurality of second pixels Obtain a second perspective image by outputting, and obtain a time-of-flight TOF image of the object according to the output of the plurality of pixel units, and generate the object according to at least the TOF image, the first-perspective image, and the second-perspective image The target three-dimensional data.

The structure of a pixel unit in this application that includes multiple pixels is similar to the principle of multi-shot. Because its baseline is small, about a few millimeters, it can mainly measure three-dimensional data of nearby objects, and because the imaging method is Passive imaging, there is no multipath problem in TOF imaging mode. However, traditional TOF can measure 3D data at medium and long distances because it does not require a baseline, but the error in the 3D data measured at close distances will be larger. In this way, the two measurement methods can be combined to obtain a larger range of three-dimensional data, and the two data can be complementary, which can alleviate the multipath problem of TOF, and can obtain data quality than either method. All good three-dimensional data. Therefore, the solution of the embodiment of the present application can improve the quality of the obtained three-dimensional data.

In addition, compared with the structure of the traditional TOF, the embodiment of the present application only changes the correspondence between the microlens array and the pixels, and does not significantly increase new component materials, and does not cause a significant increase in cost.

In a possible implementation manner, the angles of the light signals received by the first pixel and the second pixel are different.

In a possible implementation manner, the first pixel and the second pixel in each pixel unit are arranged in a 1×2 matrix, or the first pixel in each pixel unit The pixels and the second pixels are arranged in a 2×1 matrix form.

In a possible implementation manner, the processing circuit is configured to: generate first three-dimensional data of the object according to the TOF image; generate the first three-dimensional data of the object according to the first perspective image and the second perspective image Second three-dimensional data of the object; and generating the target three-dimensional data based on the first three-dimensional data and the second three-dimensional data.

In a possible implementation, each pixel unit further includes a third pixel and a fourth pixel, and each microlens is used to converge the light signal returned by the object to a corresponding pixel unit, and The angles of the light signals received by the plurality of third pixels in the plurality of pixel units are the same, and the angles of the light signals received by the plurality of fourth pixels are the same.

In a possible implementation manner, the first pixel, the second pixel, the third pixel, and the fourth pixel in each pixel unit are arranged in a 2×2 matrix form.

In a possible implementation manner, the angles of the light signals received by the third pixel and the fourth pixel are different.

In a possible implementation manner, the processing circuit is configured to obtain a third-view image according to the output of the plurality of third pixels, obtain a fourth-view image according to the output of the plurality of fourth pixels, and according to the output of the plurality of fourth pixels. The TOF image, the first perspective image, the second perspective image, the third perspective image, and the fourth perspective image generate target three-dimensional data of the object.

In a possible implementation manner, the processing circuit is configured to generate first three-dimensional data of the object according to the TOF image; according to the first perspective image, the second perspective image, and the third The perspective image and the fourth perspective image are used to generate second three-dimensional data of the object; and the target three-dimensional data is generated based on the first three-dimensional data and the second three-dimensional data.

In a possible implementation manner, the processing circuit is used to fuse the first three-dimensional data and the second three-dimensional data to obtain the target three-dimensional data.

In a possible implementation manner, the processing circuit is used to fuse the first three-dimensional data and the second three-dimensional data by using an iterative closest point ICP algorithm.

In a possible implementation manner, the plurality of pixel units are pulsed TOF sensing units, and the TOF image is generated according to the light signals sensed by the plurality of pixel units at least once.

In a possible implementation manner, the plurality of pixel units are phase TOF sensing units, and the TOF image is generated according to the light signals sensed by the plurality of pixel units at least three times.

In a possible implementation manner, a main lens is further included, and the main lens is arranged above the microlens array for imaging the object.

In a possible implementation manner, the microlens array is arranged on the focal plane of the main lens.

In a possible implementation manner, a light source is further included, the light source is configured to emit a light signal of a specified wavelength to the object, and the plurality of pixel units are configured to sense the light signal of a specified wavelength returned by the object.

In a possible implementation manner, the optical signal of the specified wavelength is an infrared optical signal.

In a possible implementation manner, a control unit is further included, and the control unit is configured to control the light source to synchronize clocks with the plurality of pixel units.

In a possible implementation manner, a filter structure is further included, and the filter structure is disposed above the microlens array for filtering optical signals of non-designated wavelengths.

In a possible implementation manner, the filter structure is a filter film, and the filter film is provided on the upper surface of the microlens array by coating.

In a possible implementation manner, the filter structure is a filter.

In a second aspect, an image acquisition device is provided, including: a microlens array including a plurality of microlenses; a plurality of pixel units corresponding to the plurality of microlenses, each of the plurality of pixel units The pixel unit includes a first pixel and a second pixel, and each microlens is used to converge the light signal returned by the object to a corresponding pixel unit, and make the light received by a plurality of first pixels in the plurality of pixel units The angles of the signals are the same, and the angles of the light signals received by the plurality of second pixels are the same; wherein the output of the plurality of first pixels is used to generate a first perspective image, and the output of the plurality of second pixels is used to A second perspective image is generated, the output of the plurality of pixel units is used to generate a TOF image of the time of flight of the object, and the TOF image, the first perspective image, and the second perspective image are used to generate the The target three-dimensional data of the object.

Description of the drawings

Figure 1 is a schematic structural diagram of a conventional TOF system.

Fig. 2 is a schematic structural diagram of an image acquisition device provided by an embodiment of the present application.

3 and 4 are schematic structural diagrams of a pixel unit provided by an embodiment of the present application.

FIG. 5 is a schematic diagram of the propagation path of optical signals reflected from different points on an object provided by an embodiment of the present application.

Fig. 6 is a schematic structural diagram of another image acquisition device provided by an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

In-depth measurement or three-dimensional data is increasingly being applied to consumer-level electronic products. Typical technologies include dual (multi-) vision solutions, structured light solutions, and time of flight (TOF) solutions.

The principle of binocular vision is similar to that of human eyes. Two cameras are used to capture images of objects from different directions, and the captured graphics are merged to establish the correspondence between features and place the same physical point in different images. Corresponding to the image points to obtain the three-dimensional data of the object.

Binocular stereo vision acquires three-dimensional information based on the principle of triangulation, that is, a triangle is formed between the image plane of the two cameras and the measured object. Knowing the positional relationship between the two cameras and the coordinates of the object in the left and right images, the three-dimensional size of the object in the common field of view of the two cameras and the three-dimensional coordinates of the feature points of the space object can be obtained. Therefore, the binocular vision system is generally composed of two cameras.

Similarly, a multi-eye vision system is generally composed of multiple cameras that capture images of an object from different directions, and merge the captured images to obtain three-dimensional data of the object.

The structured light scheme uses an infrared laser to project light with certain structural characteristics onto the object to be photographed, and then a special infrared camera collects the reflected structured light pattern, and calculates the depth information according to the principle of triangulation.

The basic principle of ToF is to continuously emit light pulses (usually invisible light) to the observed object, and then use the sensor to receive the light returned from the object, and obtain the distance between the object and the camera by detecting the flight (round trip) time of the light pulse distance.

The TOF method can generally be divided into two types according to different modulation methods: pulsed modulation and continuous wave modulation.

The principle of the pulse modulation scheme is relatively simple. It directly calculates the distance based on the time difference t between pulse transmission and reception. This method can also be called the pulse TOF method. The specific calculation formula can be: distance L=c×t/2, and c is the speed of light.

In practical applications, continuous wave modulation usually uses sine wave modulation or other periodic signal modulation. Since the phase offset of the sine wave at the receiving end and the transmitting end is proportional to the distance between the object and the camera, the phase offset can be used to measure the distance. This method can also be called the phase TOF method or the indirect TOF method. The phase delay of light from emission to reception calculated through multiple frames

To calculate the distance. The specific calculation formula can be: distance

c is the speed of light, f is the operating frequency of the light source,

Is the phase delay.

The current consumer TOF depth cameras mainly include: Microsoft’s Kinect 2, MESA’s SR4000, Google Project Tango’s PMD Tech’s TOF depth camera, etc. These products have achieved many applications in somatosensory recognition, gesture recognition, environment modeling, etc. The most typical one is Microsoft Kinect2.

TOF depth camera can change the measurement distance of the camera by adjusting the frequency of the emitted pulse; TOF depth camera is different from the depth camera based on the feature matching principle, its measurement accuracy will not decrease with the increase of the measurement distance, and its measurement error is in the entire measurement range The interior is basically fixed; TOF depth camera has strong anti-interference ability. Therefore, in the occasions where the measurement distance is required to be relatively far (such as unmanned driving), the TOF depth camera has a very obvious advantage.

TOF depth cameras have high requirements for time measurement accuracy. Even if the most high-precision electronic components are used, it is difficult to achieve millimeter-level accuracy. Therefore, in the field of close-range measurement, especially in the range of 1m, the accuracy of TOF depth camera still has a large gap compared with other depth cameras, which limits its application in the field of close-range high-precision, such as face recognition Application scenarios such as unlocking and face recognition payment.

At present, the TOF solution is more and more widely used due to its advantages of no baseline requirements, compact structure, fast speed, and simple algorithm. It has been applied to smartphones of brands such as Huawei (Huawei P30 Pro, Samsung Note 10+). However, consumer TOF generally requires low price, small size, and low power consumption. This leads to poor quality of the 3D data obtained by TOF. For example, the depth information of the measured object is inaccurate, which obviously limits its use in 3D face recognition, and the photo background is blurred. The application effects of chemistry, somatosensory games, etc.

The structure of a conventional TOF image acquisition device may be as shown in FIG. 1, and the device includes a main lens 120 and a pixel array 140. The main lens 120 is used to converge the optical signals returned from the object 110, and the pixel array 140 is used to sense the optical signals passing through the main lens 120 to generate a TOF image of the object 110.

The main lens 120 may also be referred to as an imaging lens.

The pixels in the pixel array 140 in FIG. 1 are TOF sensing units.

In order to improve the light sensitivity, a layer of microlens array 130 can be covered on the pixel array 140, where one pixel corresponds to one microlens, that is, the pixel 141 corresponds to the microlens 131, the pixel 142 corresponds to the microlens 132, and the pixel 143 corresponds to the microlens. 133 corresponds. A microlens can converge the light signal reflected by the object 110 to a corresponding pixel. Through the converging effect of the microlens, the pixel can receive more light energy.

FIG. 1 shows a schematic diagram of the propagation path of an optical signal reflected by a point 111 on an object 110. The optical signal reflected by the point 111 is condensed by the main lens 120 and then reaches the microlens 131, and the pixel 141 can receive the optical signal condensed by the microlens 131.

Different pixels in the pixel array 140 can receive light signals reflected from different points on the object 110. Therefore, the embodiment of the present application can generate a TOF image of the object according to the light signals sensed by the pixel array to obtain depth information of the object. For example, the TOF image can be used to generate three-dimensional data of the object through a depth reconstruction algorithm.

The TOF image in the embodiments of the present application may represent the depth information of the object, or the TOF image may represent the depth information and image information of the object.

This application is improved on the basis of this solution to obtain an image acquisition device that can improve the quality of the acquired three-dimensional data.

As shown in FIG. 2, the image capture device may include a microlens array 230 and a plurality of pixel units 240. The microlens array 230 includes a plurality of microlenses, and the plurality of pixel units 240 can correspond to the plurality of microlenses one-to-one. Each pixel unit of the plurality of pixel units 240 may include a first pixel and a second pixel, and each microlens is used to converge the light signal returned by the object to a corresponding pixel unit, and make the plurality of pixel units The angles of the light signals received by the plurality of first pixels in 240 are the same, and the angles of the light signals received by the plurality of second pixels are the same.

Wherein, the output of the plurality of first pixels is used to generate a first perspective image, the output of the plurality of second pixels is used to generate a second perspective image, and the output of the plurality of pixel units is used to generate the object The TOF image of the time of flight, and the TOF image, the first perspective image, and the second perspective image are used to generate target three-dimensional data of the object.

The image acquisition device also includes a processing circuit for obtaining a first-view image according to the output of the plurality of first pixels, obtaining a second-view image according to the output of the plurality of second pixels, and according to the The output of a plurality of pixel units obtains the TOF image of the time of flight of the object, and generates the target three-dimensional data of the object according to at least the TOF image, the first view image and the second view image.

The light signal sensed by one pixel unit can be used to generate the depth information at the position of the pixel unit, and the light signals sensed by multiple pixel units can be used to generate the depth information at multiple pixel units respectively, so as to obtain the depth information of the object to generate TOF image of the object.

The first pixel in the embodiment of the present application may refer to pixels located at the same relative position among multiple pixel units, and the second pixel may refer to pixels located at the same relative position among multiple pixel units, and the first pixel and the second pixel may be The two pixels are different.

The pixels located at the same relative position may refer to the pixel a in one pixel unit and the pixel b corresponding to the pixel a in another pixel unit. For example, suppose that each pixel unit contains two pixels, and the relative positions of the two pixels in this pixel unit are the left and right sides, respectively. All pixels located on the left in the plurality of pixel units are referred to as pixels located at the same relative position, and all pixels located on the right in the multiple pixel units are referred to as pixels located at the same relative position.

Taking FIG. 2 as an example, one pixel unit includes two pixels, where the

pixels

241a, 242a, and 243a may be the first pixels, and the

pixels

241b, 242b, and 243b may be the second pixels.

The

pixels

241a, 242a, and 243a can be used to generate a first-view image, and the

pixels

241b, 242b, and 243b can be used to generate a second-view image.

Only the first-view image and the second-view image are described above. Of course, if a pixel unit includes x pixels, and x is an integer greater than 2, the embodiment of the present application may only generate images of the object in 2 views, or Generate x-view images of the object.

For example, each pixel unit in the embodiment of the present application may further include a third pixel and a fourth pixel, and each microlens is used to converge the light signal returned by the object to a corresponding pixel unit, and make all The angles of the light signals received by the plurality of third pixels in the plurality of pixel units are the same, and the angles of the light signals received by the plurality of fourth pixels are the same.

Due to the converging effect of the microlenses, the angles of the light signals received by the plurality of third pixels and the plurality of fourth pixels are different. Specifically, the angles of the light signals received by the plurality of first pixels, the plurality of second pixels, the plurality of third pixels, and the plurality of fourth pixels are all different.

The three-dimensional data in the embodiments of the present application refers to data that can describe the shape and spatial position of an object in a three-dimensional space. The three-dimensional data may include, but is not limited to, a three-dimensional point cloud (point cloud), a depth image (depth image/range image), or a three-dimensional network (mesh).

In the embodiment of the present application, the one-to-one correspondence between multiple pixel units and the multiple microlenses means that each pixel unit has a corresponding microlens, and the optical signal passing through one microlens can be received by all pixels in the corresponding pixel unit arrive.

For example, referring to FIG. 2, the light signal passing through the microlens 231 can be received by the pixels 241a and 241b, the light signal passing through the microlens 232 can be received by the

pixels

242a and 242b, and the light signal passing through the microlens 233 can be received by the pixels. 243a and pixel 243b are received.

The embodiment of the present application does not specifically limit the number of pixels included in a pixel unit. For example, one pixel unit may include 2 pixels or 4 pixels or even more pixels. Of course, one pixel unit may also include 3 pixels or other numbers of pixels.

Taking FIG. 2 as an example, one pixel unit includes 2 pixels. The plurality of pixel units 240 includes a pixel unit 241, a pixel unit 242, and a pixel unit 243. The pixel unit 241 includes a pixel 241a and a pixel 241b, the pixel unit 242 includes a pixel 242a and a pixel 242b, and the pixel unit 243 includes a pixel 243a and a pixel 243b.

The embodiment of the present application does not specifically limit the arrangement of pixels in a pixel unit. For example, if a pixel unit includes an even number of pixels, the even number of pixels can be arranged in a matrix.

For example, when a pixel unit includes 2 pixels, that is, when it includes a first pixel and a second pixel, the first pixel and the second pixel may be arranged in a matrix of 1×2 or 2×1. When a pixel unit includes 4 pixels, that is, when it includes the first pixel, the second pixel, the third pixel, and the fourth pixel, the first pixel, the second pixel, the third pixel, and the fourth pixel can be in the order of 2×2. Arrange in matrix form.

When each pixel unit contains an even number of pixels, the arrangement is relatively neat, which is beneficial to save space.

When a pixel unit includes two pixels, the two pixels can be arranged in an up-and-down structure, as shown in the left figure in Figure 3, or in a left-right structure, as shown in the left figure in Figure 3. Shown.

When a pixel unit includes 4 pixels, the 4 pixels can be arranged in a 2×2 matrix, as shown in FIG. 4.

The embodiment of the present application does not specifically limit the shape of a single pixel. For example, the pixel may be square or circular.

Combining multiple pixels into one pixel unit becomes the traditional TOF mode. That is, a TOF image of the object can be generated based on the light signal sensed by a pixel unit.

For example, the pixel values of multiple pixel units in a pixel unit can be combined, that is, the pixel values of multiple pixel units can be summed, and the depth value at each pixel unit can be calculated according to the summed pixel values to generate TOF image of the object. For another example, the pixel values of multiple pixel units in one pixel unit can be averaged, and the depth value at each pixel unit can be calculated according to the averaged pixel value to obtain the TOF image of the object. For another example, it is also possible to calculate a depth value for each pixel in a pixel unit, and then take the average of these depth values to obtain a TOF image of the object. No matter which calculation method is used, as long as the calculation method of each pixel unit is consistent.

For phase TOF, the pixel value of one pixel can be used to represent the integral value of the phase of the optical signal; for pulsed TOF, the pixel value of one pixel can be used to represent the flight time of the optical signal.

When using TOF mode, the pixel values of pixels 241a and 241b can be averaged to obtain the pixel value of pixel unit 241; the pixel values of 242a and pixel 242b can be averaged to obtain the pixel value of pixel unit 242; The pixel values of the 243a and the pixel 243b are averaged to obtain the pixel value of the pixel unit 243. Then, based on the pixel values of the pixel unit 241, the pixel unit 242, and the pixel unit 243, a TOF image of the object is generated.

The number of pixels included in each pixel unit is the same, and the arrangement manner is also the same. In this way, the angles of the light signals that can be received by the pixels at the same relative position in the multiple pixel units can be kept the same, while the angles of the light signals that can be received by the pixels at different relative positions within a pixel are different, that is, the first pixel and the The angle of the light signal received by the second pixel is different. In this way, the light signals sensed by the pixels at the same relative position can generate an image of one viewing angle. If a pixel unit includes n pixels, an image of n viewing angles can be generated, and n is an integer greater than or equal to 2.

According to the imaging principle of the microlens, the pixel 241a can receive the light signal within the angle ②, the pixel 241b can receive the light signal within the angle ①, the pixel 242a can receive the light signal within the angle ④, and the pixel 242b can receive the light signal within the angle ③. The pixel 243a can receive the light signal within the range of angle ⑥, and the pixel 243b can receive the light signal within the range of angle ⑤. Among them, angle ②, angle ④, angle ⑥ have the same size and direction, angle ①, angle ③, angle ⑤ have the same size and direction, and angle ②, angle ④, angle ⑥ and angle ①, angle ③, angle ⑤ direction different. That is, the pixel 241a, the pixel 242a, and the pixel 243a are light signals reflected from an object received from the same viewing angle, and the pixels 241b, 242b, and the pixel 243b are light signals reflected from an object received from the same viewing angle. Therefore, the light signals received by the pixel 241a, the pixel 242a, and the pixel 243a can be used to generate an image of one view of the object, and the pixels 241b, 242b, and the pixel 243b can be used to generate an image of another view of the object.

The structure of a pixel unit including multiple pixels in the embodiment of this application is similar to the principle of multi-shot. Because its baseline is small, about several millimeters, it can mainly measure the three-dimensional data of nearby objects, and because of the imaging The method is passive imaging, which is not affected by the multipath effect existing in the TOF method. However, traditional TOF can measure 3D data at medium and long distances because it does not require a baseline, but the error in the 3D data measured at close distances will be larger. In this way, the combination of the two measurement methods can obtain a larger range of three-dimensional data, and the data of the two complement each other, alleviate the multipath effect of TOF, and can obtain better data quality than either method. Three-dimensional data. Therefore, the solution of the embodiment of the present application can improve the quality of the obtained three-dimensional data.

Multipath effect can mean that the light signal emitted by the light source will be reflected multiple times in the target scene. An object will not only reflect the light signal emitted by the light source, but also light signals from other indirect paths. There will be interference between reflected light from multiple sources, which will cause data errors in the TOF method.

The pixel in the embodiment of the present application is a sensing unit that is the smallest unit for sensing light signals.

The image acquisition device in the embodiment of the present application may further include a main lens 220, which is disposed above the microlens array 230, and is used to converge the light signals returned from the object. The microlens array 230 can be used to converge the light signals passing through the main lens 220 to a plurality of pixel units 240.

The micro lens array 230 can be arranged on the focal plane of the main lens 220, that is, the distance between the micro lens array 230 and the main lens 220 is the focal length of the main lens 220, which can simplify the process of generating three-dimensional data of the object. Of course, the setting position of the microlens array 230 may also be offset from the imaging plane of the main lens 220 to a certain extent, so that the offset needs to be corrected when performing three-dimensional data processing.

The distance between the plurality of pixel units 240 and the microlens array 230 may be determined according to the focal length of a single microlens diameter and the size of the pixel unit. For example, the distance between the plurality of pixel units 240 and the microlens array 230 may be greater than one microlens array. The focal length of the lens.

FIG. 5 shows a schematic diagram of the propagation path of the optical signal reflected by the point 211 in the object, and the distance between the point 211 and the main lens 220 is G. According to the design of the main lens 220, the optical signal reflected by the point 211 passes through the main lens 220 and is received by a microlens 232. The optical signal passing through the microlens 232 is sensed by the pixel unit below it, and part of the optical signal is Pixel a is sensed, and another part of the light signal is sensed by pixel b.

In FIG. 5, white pixels in a plurality of pixel units 240 are used to generate an image of one view of the object, and shaded pixels are used to generate an image of another view of the object.

As shown in FIG. 6, the image acquisition device in the embodiment of the present application may further include a light source 520, the light source 520 may be used to emit a light signal of a specified wavelength to the object 510, and a plurality of pixel units 540 may be used to sense the light signal returned by the object 510 .

The light source may be a vertical-cavity surface-emitting laser (VCSEL) or a light emitting diode (LED).

The optical signal of the specified wavelength may be an invisible light signal, and the invisible light signal is not easily observed by the user's eyes, which is beneficial to improve the user experience. Of course, in some occasions with low requirements, visible light signals can also be used as the light source, which is not specifically limited in the embodiment of the present application.

Preferably, the optical signal of the designated wavelength may be an infrared optical signal.

The image acquisition device in the embodiment of the present application may further include a control unit that can be used to control the light source 520 to be time-synchronized with the multiple pixel units 540, so that the time difference or phase of the light signal from emission to reception by the pixel can be accurately calculated. Difference.

Optionally, the image acquisition device may further include a filter structure 530, which is disposed above the microlens array 560, and is used to filter optical signals of non-specified wavelengths, so that only the optical signals of specified wavelengths are multiplied. Each pixel unit 540 receives it, which can reduce the interference of ambient light on the TOF imaging process and improve the signal-to-noise ratio of the light signal collected by the TOF pixel unit.

Of course, the filter structure 530 can also be arranged between the micro lens array 560 and the plurality of pixel units 540, as long as it is ensured that only the optical signals of the specified wavelength can reach the plurality of pixel units 540.

The filter structure 530 may be a filter film, and the filter film may be provided on the upper surface of the microlens array by coating. Alternatively, the filter structure 530 may be a filter, and the filter is disposed on the upper surface of the micro lens array.

The image capture device in the embodiment of the present application may be set on a base 550, and the base 550 is used to fix the image capture device.

The multiple pixel units 540 in the embodiment of the present application may be TOF sensing units. The TOF induction unit can be a pulse type TOF induction unit or a phase type TOF induction unit.

If the multiple pixel units are pulsed TOF sensing units, the TOF image may be generated based on the light signals sensed by the multiple pixel units at least once. The pulsed TOF sensor unit only needs to transmit and collect at least once to obtain the TOF image.

If the plurality of pixel units are phase-type TOF sensing units, the TOF image may be generated based on the light signals sensed by the plurality of pixel units at least 3 times. The phased TOF sensing unit requires at least three transmission and acquisition processes to obtain the TOF image.

For example, take the phase-type TOF sensing unit as an example. Assuming that a pixel unit includes 2 pixels, the TOF image is generated based on the light signal sensed 4 times, and the light signal sensed by 2 pixels in a pixel unit is calculated each time. Average value, get 4 average values, respectively denoted as x1, x2, x3, x4, and then the depth information of the object can be obtained through the arctangent formula.

In addition, the TOF image in the embodiments of the present application may include not only depth information, but also image information of an object. In other words, the pixel values of the multiple pixel units in the embodiments of the present application can be used to generate image information of an object in addition to being used to generate depth information. For example, the pixel value of the pixel unit can be used to obtain the depth information and the image information through different algorithms.

The image acquisition process of the embodiment of the present application will be described below with reference to FIG. 6.

As shown in FIG. 6, the light source 520 emits a light signal of a specified wavelength λ to the object 510. The light signal is reflected by the object 510 and then reaches the filter 530. The filter can be used to block light signals of other wavelengths, and only make the wavelength λ The light signal of λ is transmitted through, so the plurality of pixel units 540 will only receive the light signal of the wavelength λ. After one emission collection process, multiple pixel units can collect a frame of image, and each pixel in the multiple pixel units has a corresponding gray value.

If it is a pulsed TOF, multiple pixel units only need to collect at least one image to calculate the three-dimensional data. If it is a phase TOF, multiple pixel units need to collect at least three images to calculate the three-dimensional data.

The TOF image, the first-view image, and the second-view image may be obtained based on the same one or multiple frames of images, or may be obtained based on different frames of images.

Taking pulsed TOF as an example, the light source emits a primary light signal, and multiple pixel units receive the corresponding primary light signal. The TOF image, the first viewing angle image, and the second viewing angle image are all based on the light received by multiple pixel units at the same time. Signal generated. In this way, TOF images, first-view images, and second-view images can be obtained through a single transmission and acquisition process, which can improve the processing speed of three-dimensional data.

For another example, the light source emits light signals twice, and multiple pixel units receive the corresponding light signals twice. The TOF image is generated based on the light signals received by the multiple pixel units for the first time. The viewing angle image is generated based on the light signal received by the multiple pixel units for the second time.

With reference to the above description, a pixel unit may include multiple pixels, and the pixels located in the same relative position in each pixel unit can be extracted to form an image with a viewing angle. In this way, if a pixel unit includes n pixels, images with n viewing angles can be obtained.

The embodiment of the present application does not specifically limit the manner in which the processing circuit generates the target three-dimensional data. For example, the processing circuit can directly generate the target three-dimensional data of the object according to the TOF image, the first-perspective image, and the second-perspective image through a preset algorithm. For another example, the processing circuit may generate first three-dimensional data of the object based on the TOF image, and generate second three-dimensional data of the object based on the first and second perspective images, and then based on the first three-dimensional data and the second three-dimensional data , Generate target three-dimensional data.

If a pixel unit includes a first pixel, a second pixel, a third pixel, and a fourth pixel, the output of a plurality of third pixels in the plurality of pixel units is used to generate a third view image, and the output of the plurality of fourth pixels The output is used to generate a fourth perspective image, the outputs of the multiple pixel units are used to generate a TOF image of the time of flight of the object, and the TOF image, the first perspective image, the second perspective image, and the The third-perspective image and the fourth-perspective image are used to generate target three-dimensional data of the object.

If the image acquisition device includes a processing circuit, the processing circuit is further configured to obtain a third-view image according to the output of the plurality of third pixels, obtain a fourth-view image according to the output of the plurality of fourth pixels, and according to the The TOF image, the first-perspective image, the second-perspective image, the third-perspective image, and the fourth-perspective image generate target three-dimensional data of the object.

Optionally, the processing circuit is configured to generate first three-dimensional data of the object according to the TOF image; according to the first perspective image, the second perspective image, the third perspective image, and the A fourth perspective image, generating second three-dimensional data of the object; and generating the target three-dimensional data based on the first three-dimensional data and the second three-dimensional data.

The first three-dimensional data may be generated according to the TOF algorithm; the second three-dimensional data may be generated according to a dual (multiple) vision algorithm, or a light field camera algorithm, or a machine learning algorithm.

The processing circuit can fuse the first three-dimensional data and the second three-dimensional data to obtain the target three-dimensional data.

This application does not specifically limit the integration method. For example, the processing circuit may use an iterative closest point (ICP) algorithm to fuse the first three-dimensional data and the second three-dimensional data.

The process of solving the three-dimensional data is described below.

Each time the light source 520 actively emits a light signal, the plurality of pixel units 540 collect an image of the returned signal light once. If it is a pulsed TOF, at least one emission collection process is used to obtain a grayscale image, which is the final grayscale image. If it is a phase TOF, it needs m (m≥3) times of emission collection process to obtain m gray-scale images, and the m gray-scale images are averaged, and the average gray-scale image obtained is the final gray-scale image.

Multi-pixel solution to three-dimensional data: Use multi-vision algorithms or deep learning algorithms to solve the final grayscale image obtained in the previous step to obtain the first three-dimensional data. For example, the ICP algorithm and/or various evolution algorithms of the ICP algorithm can be used to solve the first three-dimensional data.

TOF solves three-dimensional data: According to the data collected by multiple pixel units, the pixel values of multiple pixels in each pixel unit are averaged to obtain the average pixel value, so as to obtain the original data of the traditional TOF structure, and then obtain it according to the TOF algorithm The second three-dimensional data.

Three-dimensional data fusion: The first three-dimensional data calculated by multi-pixel and the second three-dimensional data calculated by TOF are fused to obtain better quality three-dimensional data.

Post-processing: processing the burr points in the fused 3D data, filling the holes in the data, and smoothing and filtering the data to obtain optimized 3D data.

The following describes a fusion method applicable to the embodiment of the present application, but the embodiment of the present application is not limited to this.

In the process of calculating the first three-dimensional data and the second three-dimensional data, the corresponding confidence maps can be generated respectively:

(1) Set two empirical thresholds, threshold one and threshold two. When the confidence map value corresponding to the data point in the first three-dimensional data is greater than threshold one, the three-dimensional information of the point is considered to be highly reliable, otherwise the credibility is low . In the same way, the confidence map value of the second three-dimensional data is compared with the threshold value 2, and the reliability of each data point in the second three-dimensional data can also be judged.

(2) When the credibility of the data point in the first three-dimensional data is high, and the credibility of the data point corresponding to the second three-dimensional data is low, take the data of the first three-dimensional data point as the value of the corresponding point of the target three-dimensional data ；

(3) When the credibility of the data point in the first three-dimensional data is low, and the credibility of the data point corresponding to the second three-dimensional data is high, the data of the second three-dimensional data point is taken as the value of the corresponding point of the target three-dimensional data;

(4) When the credibility of the data point in the first three-dimensional data is high, and the credibility of the data point corresponding to the second three-dimensional data is high, the average value of the data of the first and second three-dimensional data points is taken as the target three-dimensional data. The value of the corresponding point of the data;

(5) When the credibility of the first three-dimensional data point is low, and when the credibility of the second three-dimensional data point is low, the average value of the surrounding values of the corresponding point of the target three-dimensional data is taken as the value of the corresponding point of the target three-dimensional data.

In the actual application process of the embodiment of the present application, the fused three-dimensional data can be directly used. Or you can judge the distance between the object and multiple pixel units first, and then determine how to use the three-dimensional data. For example, if the distance between the object and multiple pixel units is greater than a preset threshold, the first three-dimensional data can be used. In this case, the target three-dimensional data is the first three-dimensional data; if the distance between the object and multiple pixel units is If the distance of is less than or equal to the preset threshold, the second three-dimensional data can be used. In this case, the target three-dimensional data is the second three-dimensional data.

The image acquisition device of the embodiment of the present application can be applied to various occasions, for example, scenes such as three-dimensional face recognition, image background blurring, and somatosensory games. Three-dimensional face recognition can be applied to scenes or devices related to three-dimensional data such as mobile phones, door locks, access control, and payment devices.

An embodiment of the present application also provides an electronic device, which includes any one of the image acquisition devices described above. The electronic device can be, for example, a mobile phone, a computer, or other devices.

It can be understood that the structure in the drawings of the present application only represents a schematic diagram, and does not represent the actual size and ratio, which does not limit the embodiments of the present application.

It should be noted that the terms used in the embodiments of the present application and the appended claims are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application.

For example, the singular forms of "a", "said", "above" and "the" used in the embodiments of this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other forms. meaning.

Those skilled in the art may be aware that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered as going beyond the scope of the embodiments of the present application.

If implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium , Including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the equipment, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed electronic equipment, apparatus, and method may be implemented in other ways.

For example, the division of units or modules or components in the device embodiments described above is only a logical function division, and there may be other divisions in actual implementation. For example, multiple units or modules or components can be combined or integrated. To another system, or some units or modules or components can be ignored or not executed.

For another example, the units/modules/components described above as separate/display components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units/modules/components can be selected according to actual needs to achieve the objectives of the embodiments of the present application.

Finally, it should be noted that the mutual coupling or direct coupling or communication connection shown or discussed above may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms. .

The above content is only the specific implementation manners of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art can easily think of within the technical scope disclosed in the embodiments of the present application. The change or replacement shall be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

An image acquisition device, characterized in that it comprises:

Micro lens array, including multiple micro lenses;

A plurality of pixel units corresponding to the plurality of microlenses, each pixel unit of the plurality of pixel units includes a first pixel and a second pixel, and each microlens is used for converging the light signal returned from the object To a corresponding pixel unit, so that the angles of the light signals received by the plurality of first pixels in the plurality of pixel units are the same, and the angles of the light signals received by the plurality of second pixels are the same;

A processing circuit for obtaining a first-view image according to the output of the plurality of first pixels, obtaining a second-view image according to the output of the plurality of second pixels, and obtaining the image according to the output of the plurality of pixel units The time-of-flight TOF image of the object, and the target three-dimensional data of the object is generated according to at least the TOF image, the first view image, and the second view image.
The image acquisition device according to claim 1, wherein the angles of the light signals received by the first pixel and the second pixel are different.
The image acquisition device according to claim 1 or 2, wherein the first pixel and the second pixel in each pixel unit are arranged in a 1×2 matrix form, or each The first pixels and the second pixels in the pixel unit are arranged in a 2×1 matrix form.
The image acquisition device according to any one of claims 1-3, wherein the processing circuit is used for:

According to the TOF image, generate first three-dimensional data of the object; generate second three-dimensional data of the object based on the first perspective image and the second perspective image; and generate second three-dimensional data of the object based on the first three-dimensional data and The second three-dimensional data generates the target three-dimensional data.
The image acquisition device according to claim 1 or 2, wherein each pixel unit further comprises a third pixel and a fourth pixel, and each microlens is used to converge the light signal returned by the object To a corresponding pixel unit, so that the angles of the light signals received by the plurality of third pixels in the plurality of pixel units are the same, and the angles of the light signals received by the plurality of fourth pixels are the same.
The image acquisition device according to claim 5, wherein the first pixel, the second pixel, the third pixel, and the fourth pixel in each pixel unit are in the order of 2×2 Arranged in a matrix form.
The image acquisition device according to claim 5 or 6, wherein the angles of the light signals received by the third pixel and the fourth pixel are different.
The image acquisition device according to any one of claims 5-7, wherein the processing circuit is configured to obtain a third perspective image according to the output of the plurality of third pixels, and according to the plurality of fourth pixels. The output of the pixels obtains a fourth perspective image, and the target three-dimensional image of the object is generated according to the TOF image, the first perspective image, the second perspective image, the third perspective image, and the fourth perspective image. data.
8. The image acquisition device according to claim 8, wherein the processing circuit is used to generate first three-dimensional data of the object according to the TOF image; according to the first perspective image and the second perspective Image, the third perspective image, and the fourth perspective image to generate second three-dimensional data of the object; and the target three-dimensional data is generated based on the first three-dimensional data and the second three-dimensional data.
The image acquisition device according to claim 4 or 9, wherein the processing circuit is used to fuse the first three-dimensional data and the second three-dimensional data to obtain the target three-dimensional data.
The image acquisition device according to claim 10, wherein the processing circuit is used to fuse the first three-dimensional data and the second three-dimensional data by using an iterative closest point ICP algorithm.
The image acquisition device according to any one of claims 1-11, wherein the plurality of pixel units are pulsed TOF sensing units, and the TOF image is sensed at least once according to the plurality of pixel units. The light signal is generated.
The image acquisition device according to any one of claims 1-11, wherein the plurality of pixel units are phase-type TOF sensing units, and the TOF image is sensed at least 3 times according to the plurality of pixel units. The light signal is generated.
15. The image acquisition device according to any one of claims 1-13, further comprising a main lens, the main lens being arranged above the microlens array for imaging the object.
The image capture device according to claim 14, wherein the micro lens array is arranged on the focal plane of the main lens.
The image acquisition device according to any one of claims 1-15, further comprising a light source, the light source is used to emit a light signal of a specified wavelength to the object, and the plurality of pixel units are used to sense The optical signal of the specified wavelength returned by the object.
The image acquisition device according to claim 16, wherein the optical signal of the specified wavelength is an infrared optical signal.
The image acquisition device according to claim 16 or 17, further comprising a control unit configured to control the light source to synchronize clocks with the plurality of pixel units.
The image acquisition device according to any one of claims 16-18, further comprising a filter structure, the filter structure is arranged above the microlens array for filtering light of non-designated wavelengths Signal.
18. The image acquisition device according to claim 19, wherein the filter structure is a filter film, and the filter film is provided on the upper surface of the microlens array by coating.
The image acquisition device according to claim 19, wherein the filter structure is a filter.
An image acquisition device, characterized in that it comprises:

Micro lens array, including multiple micro lenses;

A plurality of pixel units corresponding to the plurality of microlenses, each pixel unit of the plurality of pixel units includes a first pixel and a second pixel, and each microlens is used for converging the light signal returned from the object To a corresponding pixel unit, so that the angles of the light signals received by the plurality of first pixels in the plurality of pixel units are the same, and the angles of the light signals received by the plurality of second pixels are the same;

Wherein, the output of the plurality of first pixels is used to generate a first perspective image, the output of the plurality of second pixels is used to generate a second perspective image, and the output of the plurality of pixel units is used to generate the object The TOF image of the time of flight, and the TOF image, the first perspective image, and the second perspective image are used to generate target three-dimensional data of the object.