CN113489858A - Imaging system and related imaging method - Google Patents
Imaging system and related imaging method Download PDFInfo
- Publication number
- CN113489858A CN113489858A CN202110766336.9A CN202110766336A CN113489858A CN 113489858 A CN113489858 A CN 113489858A CN 202110766336 A CN202110766336 A CN 202110766336A CN 113489858 A CN113489858 A CN 113489858A
- Authority
- CN
- China
- Prior art keywords
- time
- video frame
- optical flow
- depth
- flow estimation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 70
- 230000003287 optical effect Effects 0.000 claims abstract description 102
- 238000010586 diagram Methods 0.000 description 11
- 238000006073 displacement reaction Methods 0.000 description 9
- 239000013598 vector Substances 0.000 description 8
- 238000000034 method Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 239000000284 extract Substances 0.000 description 3
- 239000012141 concentrate Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000011017 operating method Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000007493 shaping process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/00002—Diagnosis, testing or measuring; Detecting, analysing or monitoring not otherwise provided for
- H04N1/00007—Diagnosis, testing or measuring; Detecting, analysing or monitoring not otherwise provided for relating to particular apparatus or devices
- H04N1/00021—Picture signal circuits
Abstract
The application discloses an imaging system and a related imaging method. The imaging system includes: a time-of-flight sensor, comprising: an array of pixels; a depth information estimation unit for reading the pixel array and obtaining the flight time of the plurality of reflection light spots according to the pixel array to estimate the depth information; a two-dimensional image sensor; an optical flow estimation unit; and a depth information establishing unit; wherein the optical flow estimation unit performs optical flow estimation on a first two-dimensional video frame obtained by the two-dimensional image sensor at a first time and a second two-dimensional video frame obtained at a second time after the first time to generate an optical flow estimation result, and the depth information establishing unit generates a second depth video frame corresponding to the second time according to the optical flow estimation result and a first depth video frame obtained by the time-of-flight sensor at the first time.
Description
Technical Field
The present application relates to sensing systems, and more particularly, to an imaging system and related imaging method.
Background
The time of flight (TOF) ranging technique continuously transmits an optical signal from a transmitting end to a target object and receives the optical signal returned from the target object at a receiving end, thereby calculating the time of flight of the optical signal from the transmitting end to the receiving end and obtaining the distance between the target object and the transmitting end/receiving end. The time-of-flight ranging technology can be roughly divided into two different schemes of a point light source and a surface light source, wherein the scheme of the point light source can concentrate energy on a limited number of light spots, and is suitable for long-distance application. The scheme of this application to the pointolite improves, under the prerequisite that does not influence the degree of accuracy of time of flight range finding technique, satisfies the demand of low power consumption.
Disclosure of Invention
An objective of the present application is to disclose an imaging system, and an electronic device and an operating method of the imaging system, which solve the above problems.
An embodiment of the present application discloses an imaging system, including: a time-of-flight sensor, comprising: a pixel array for sensing a light reflection signal reflected by an object to the pixel array, the light reflection signal comprising a plurality of reflected light spots that hit a plurality of pixels in the pixel array; a depth information estimation unit for reading the pixel array and obtaining the flight time of the plurality of reflection light spots according to the pixel array to estimate the depth information; the two-dimensional image sensor is used for sensing two-dimensional video frames; the optical flow estimation unit is used for carrying out optical flow estimation on different two-dimensional video frames obtained by the two-dimensional image sensor at different times; and a depth information establishing unit coupled to the optical flow estimating unit and the depth information estimating unit; wherein the optical flow estimation unit performs optical flow estimation on a first two-dimensional video frame obtained by the two-dimensional image sensor at a first time and a second two-dimensional video frame obtained at a second time after the first time to generate an optical flow estimation result, and the depth information establishing unit generates a second depth video frame corresponding to the second time according to the optical flow estimation result and a first depth video frame obtained by the time-of-flight sensor at the first time.
An embodiment of the present application discloses an imaging method, including: sensing a two-dimensional video frame at a first time to obtain a first two-dimensional video frame; sensing light reflection signals at the first time to perform depth information estimation so as to obtain a first depth video frame; sensing a two-dimensional video frame at a second time after the first time to obtain a second two-dimensional video frame; performing optical flow estimation on the first two-dimensional video frame and the second two-dimensional video frame to generate optical flow estimation results; and generating a second depth video frame corresponding to the second time according to the optical flow estimation result and the first depth video frame.
The imaging system and the related imaging method disclosed by the application can reduce power consumption and/or improve the efficiency of three-dimensional estimation.
Drawings
Fig. 1 is a schematic view of a first embodiment of an imaging system of the present application.
Fig. 2a is a schematic diagram of a first two-dimensional video frame obtained at a first time by using the imaging system of the present application and its extracted features.
Fig. 2b is a schematic diagram of a second two-dimensional video frame obtained at a second time using the imaging system of the present application and its extracted features.
Fig. 3a is a schematic diagram of a first depth video frame obtained at a first time using the imaging system of the present application.
FIG. 3b is a schematic diagram of a first embodiment of a second depth video frame generated at a second time using the imaging system of the present application.
Fig. 4a is a schematic diagram of a depth video frame without depth compensation.
Fig. 4b is a schematic diagram of a depth video frame obtained by performing depth compensation using the imaging system of the present application.
Fig. 5 is a schematic view of a second embodiment of the imaging system of the present application.
Detailed Description
The following disclosure provides various embodiments or illustrations that can be used to implement various features of the disclosure. The embodiments of components and arrangements described below serve to simplify the present disclosure. It is to be understood that such descriptions are merely illustrative and are not intended to limit the present disclosure. For example, in the description that follows, forming a first feature on or over a second feature may include certain embodiments in which the first and second features are in direct contact with each other; and may also include embodiments in which additional elements are formed between the first and second features described above, such that the first and second features may not be in direct contact. In addition, the present disclosure may repeat reference numerals and/or characters in the various embodiments. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed.
Moreover, spatially relative terms, such as "under," "below," "over," "above," and the like, may be used herein to facilitate describing a relationship between one element or feature relative to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass a variety of different orientations of the device in use or operation in addition to the orientation depicted in the figures. The device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
Although numerical ranges and parameters setting forth the broad scope of the application are approximations, the numerical values set forth in the specific examples are reported as precisely as possible. Any numerical value, however, inherently contains certain standard deviations found in their respective testing measurements. As used herein, "the same" generally means that the actual value is within plus or minus 10%, 5%, 1%, or 0.5% of a particular value or range. Alternatively, the term "the same" means that the actual value falls within the acceptable standard error of the mean, subject to consideration by those of ordinary skill in the art to which this application pertains. It is understood that all ranges, amounts, values and percentages used herein (e.g., to describe amounts of materials, length of time, temperature, operating conditions, quantitative ratios, and the like) are "the same" unless otherwise specifically indicated or indicated. Accordingly, unless indicated to the contrary, the numerical parameters set forth in the specification and attached claims are approximations that may vary depending upon the desired properties sought to be obtained. At the very least, these numerical parameters are to be understood as meaning the number of significant digits recited and the number resulting from applying ordinary carry notation. Herein, numerical ranges are expressed from one end to the other or between the two ends; unless otherwise indicated, all numerical ranges set forth herein are inclusive of the endpoints.
The time-of-flight ranging technique can be roughly divided into two different schemes, namely a point light source scheme and a surface light source scheme, wherein in the point light source scheme, a time-of-flight ranging device emits a limited number of light spots (which may also be referred to as light spots or speckles) to a target object, and the depth of a position irradiated by the light spots is calculated according to reflected light spots reflected by the light spots. In the technical scheme of the surface light source, the flight time distance measuring device utilizes the surface light source to emit uniform light to a target object, and calculates the depth according to reflected light reflected by the uniform light. The point source arrangement is suitable for long-range applications because it concentrates energy in a limited number of spots. However, in order to ensure that the reflected light spot still has sufficient energy to be reflected back from a distance, the time-of-flight ranging device must consume a relatively high amount of energy when emitting the plurality of light spots, thereby resulting in increased power consumption.
The embodiment that this application provided can reduce time of flight range unit's consumption under the prerequisite that does not influence the degree of accuracy, can improve the precision even simultaneously.
Fig. 1 is a functional block schematic diagram of a first embodiment of an imaging system of the present disclosure. The imaging system 100 may be implemented by a three-dimensional imaging system for obtaining depth information (or depth video frames, as shown in fig. 4a and 4b) of surrounding objects. By way of example, but not limiting of the present disclosure, the imaging system 100 may be a time-of-flight imaging system qw that may obtain depth information of the target object 102 by measuring a distance between the target object 102 and the imaging system 100. It is noted that in some embodiments, the imaging system 100 may be a three-dimensional imaging system, which can determine the depth information of the target object 102 according to the pattern deformation of the light reflection signal received by the receiving end. For the sake of brevity, the imaging scheme of the present disclosure is described below in terms of an embodiment in which the imaging system 100 is implemented as a time-of-flight imaging system. However, those skilled in the art should appreciate that the imaging scheme of the present disclosure can be applied to other three-dimensional imaging systems that obtain depth video frames from optical signals at the transmitting end and the receiving end.
The imaging system 100 employs a point light source scheme, which includes (but is not limited to) a light emitting module 110, a time-of-flight sensor 120, a two-dimensional image sensor 126, an optical flow estimating unit 128, a depth information establishing unit 130, and a light source control unit 132. The light emitting module 110 is configured to generate an optical signal LS, wherein the optical signal LS may have a predetermined pattern (pattern) such that energy is concentrated in the predetermined pattern, for example, the predetermined pattern may be a speckle array, and the optical energy is concentrated in each scattered spot of the speckle array. The light emitting module 110 may include a light source 112 and an optical element 114. The optical element 114 may be used to change a traveling path, an illumination range, and the like of the light source signal LI output by the light source 112, thereby generating the light signal LS having the predetermined pattern. The projection of the light signal LS on the object 102 may form a plurality of light spots (light spots) separated from each other to reduce the influence of background noise on the measurement result. In this embodiment, the light source signal LI output by the light source 112 is invisible light, such as infrared light.
By way of example, and not limitation, the optical source 112 may include a Vertical-Cavity Surface-Emitting Laser (VCSEL) array, and the optical element 114 may include a Diffractive Optical Element (DOE) or a Refractive Optical Element (ROE) for cone-diffracting (or cone-refracting) the optical source signal LI to generate the optical signal LS, such that the optical signal LS may be projected on the target 102 to form a plurality of spots separated from each other. In some embodiments, a collimating lens is further included between the optical source 112 and the optical element 114 for shaping the light source signal LI into parallel light.
The time-of-flight sensor 120 is configured to sense the light reflection signal LR returned from the target 102 to generate a depth video frame, which is used to form a depth video, that is, a plurality of depth video frames continuously captured by the time-of-flight sensor 120 at a depth video frame rate can form a depth video, for example, the imaging system 100 is installed on a mobile phone, and a user can capture and watch a three-dimensional video through mobile phone software. Wherein the light reflection signal LR is generated by the object 102 reflecting the light signal LS. In this embodiment, time-of-flight sensor 120 includes, but is not limited to, a pixel array 122 and a depth information estimation unit 124. The optical signal LS is projected onto the surface of the target 102 to form a plurality of light spots correspondingly and reflected by the target to the pixel array 122, and the plurality of light spots can hit a plurality of pixels in the pixel array 122, thereby forming a plurality of reflected light spots separated from each other on the pixel array 122.
The depth information estimation unit 124 is coupled to the pixel array 122 for reading data of pixel cells in the pixel array 122 and obtaining position information of a plurality of reflection light spots on the pixel array 122, i.e. which pixel cells of the pixel array 122 are irradiated by the plurality of reflection light spots. Since the imaging system 100 adopts a point light source scheme, the depth information estimating unit 124 can determine which of the light spots of the optical signal LS emitted by the light emitting module 110 each reflected light spot should correspond to on the premise that the distance between the target 102 and the imaging system 100 is within an allowable range. Therefore, the depth information estimation unit 124 can obtain the flight times of the plurality of reflection light spots and perform depth information estimation to obtain the depth information of the target object 102.
Generally, the depth video frame can be created only by using the light emitting module 110 to emit the light signal LS and then using the time-of-flight sensor 120 to sense the light reflection signal LR. However, when depth information estimation is performed continuously on the time axis, the contents of two consecutive frames of depth video frames are relatively close to each other, for example, the contents of two frames of depth video frames are the same scene but the picture slightly moves in a certain direction. That is, the content of a depth video frame of a certain frame can be said to be a slight change of the content of the depth video frame of the previous frame. Since the scene will not change too fast on the time axis in general, the content of a depth video frame of a certain frame can be even quite similar to the content of the depth video frames of the previous frames. Through the above analysis, the present application assists in establishing a later depth video frame on the time axis based on an earlier depth video frame on the time axis, thereby changing the operation mode of the light emitting module 110 to save power consumption. The details thereof are explained below.
Specifically, the imaging system 100 of the present application is added with a two-dimensional image sensor 126 compared to a general imaging system for three-dimensional use, and the two-dimensional image sensor 126 can be used to sense the light signal L reflected to the two-dimensional image sensor 126 after the ambient light irradiates the object 102, in this application, the two-dimensional image sensor 126 can be a color (RGB) two-dimensional image sensor or a monochrome (BW) two-dimensional image sensor for sensing the visible light portion in the light signal L.
Since the two-dimensional image sensor 126 can sense under normal ambient light without an additional light emitting module 110 like the time-of-flight sensor 120, the power consumption of the two-dimensional image sensor 126 is relatively much lower. Therefore, in the present application, the two-dimensional image sensor 126 senses two consecutive frames of two-dimensional video frames on the time axis for the same target object, and the optical flow estimation unit 128 estimates the optical flow between the two consecutive two-dimensional video frames, that is, the optical flow can be represented as information of the moving speed and direction of the object corresponding to each image point in the two-dimensional video frames, and the optical flow can also be represented as a displacement vector (including the direction and the distance) between two consecutive two-dimensional video frames of the object corresponding to each image point in the two-dimensional video frames when viewed from the two consecutive two-dimensional video frames. The depth information building unit 130 may adjust the content of the depth video frame according to the optical flow and compensate the subsequent depth video frame accordingly. Optical flow estimation is an underlying vision technique, often used as an auxiliary information for some high-level vision tasks. Optical flow takes into account the correlation between frames by providing motion information between video frames.
A general three-dimensional imaging system performs three-dimensional sensing using uniform power to generate each depth video frame, and each depth video frame is independently generated when generating the depth video frame, and does not generate a depth video frame later on a time axis with a depth video frame earlier on the time axis. In order to further reduce the power consumption of the three-dimensional imaging system, the information in the earlier depth video frame on the time axis is adjusted and merged into the later depth video frame on the time axis through the optical flow estimation, so that the information which is not utilized originally can be effectively recovered, and therefore, the average power consumption of the actual light-emitting module 110 and the time-of-flight sensor 120 can be reduced on the premise of not reducing the three-dimensional sensing precision at least, and the details thereof are described later. In this application, the objective of optical flow estimation is to derive a displacement vector of an object corresponding to some or all pixels in a given two-dimensional video frame of two consecutive frames. For example, the optical flow estimate may be a sparse pixel match or a dense pixel match. Taking sparse pixel matching as an example, the optical flow estimation unit 128 extracts feature points for a first two-dimensional video frame obtained by the two-dimensional image sensor 126 at a first time T1, as shown in fig. 2a, where the points are marked as the extracted feature points. The feature extraction may be achieved by an image processing method, for example, down-sampling and filtering the two-dimensional video frame in different degrees, and observing the change of the two-dimensional video frame after the image processing in a two-dimensional space to find out feature points, including features such as edges and corners, where information of the features is more abundant and better, for example, the edges are curved in a clockwise or counterclockwise manner.
In the same manner, the optical flow estimation unit 128 extracts features from the second two-dimensional video frame obtained by the two-dimensional image sensor 126 at the second time T2, as shown in fig. 2b, where the points with the dotted marks are the extracted feature points. With two-dimensional video frames, the optical flow estimation unit 128 can associate feature points in the second two-dimensional video frame of fig. 2b with feature points in the first two-dimensional video frame of fig. 2a according to the information of each feature. For example, the feature information of a feature point in the first two-dimensional video frame of fig. 2a and the feature information of a feature point in the second two-dimensional video frame of fig. 2b are the same or highly similar, and the difference in position of the two in the video frames is within a certain range, which are considered to be associated with each other, i.e. represent the same position in the scene, e.g. the front right corner of the sofa in the first two-dimensional video frame of fig. 2a and the front right corner of the sofa in the second two-dimensional video frame of fig. 2b are highly similar to each other, so that the optical flow estimation unit 128 determines that the two represent the same position in the scene, and the optical flow estimation unit 128 then calculates the displacement vector of the front right corner of the sofa in the second two-dimensional video frame of fig. 2b compared to the first two-dimensional video frame of fig. 2 a.
It should be noted that the number and positions of the feature points are strongly related to the content of the scene in the two-dimensional video frame, and taking the case of fig. 2a and 2b as an example, the optical flow estimation unit 128 may not necessarily extract the feature points at all positions, such as the floor, the table, and the sofa cushion without complex changes. Therefore, in the case where the feature points are limited, the displacement vector of the limited feature points can be used to interpolate the displacement at the non-feature point. The vector complete optical flow of the first two-dimensional video frame, as an example, in fig. 2a and the second two-dimensional video frame, as an example, in fig. 2b is obtained by filling up the displacement of all the positions (e.g., all the pixel points) in the scene in the two-dimensional video frame as much as possible.
In the case of dense pixel matching, the optical flow estimation unit 128 performs optical flow estimation based on a learning method. For example, a deep learning framework is adopted, and the optical flow of the two-dimensional video frame is obtained in a mode of establishing a model without extracting feature points, so that the information of the moving speed and the moving direction of an object corresponding to each image point in the two-dimensional video frame is obtained. It should be noted that the present application does not specifically limit the implementation manner of the optical flow estimation unit 128, as long as optical flow estimation is possible, and the present application falls within the scope of the present application.
In the present embodiment, the two-dimensional image sensor 126 performs the same two-dimensional video frame sensing at both the first time T1 and the second time T2; however, the light emitting module 110 and the time of flight sensor 120 do not necessarily operate with full power at the first time T1 and the second time T2. The following description is made separately in three cases, that is, the present application includes at least the following three embodiments.
< case one >
The light source control unit 132 controls the light emitting module 110 to generate the light signal LS with full power at the first time T1, and does not generate the light signal LS at the second time T2, i.e., the power consumption of the light emitting module 110 at the second time T2 is zero. It should be noted that since the light emitting module 110 emits the light signal LS and returns the light reflection signal LR to the time-of-flight sensor 120 through the object 102, the total elapsed time of the depth video frame calculated by the depth information estimating unit 124 is very short, and compared with the difference between the first time T1 and the second time T2, which is not the same order of magnitude, it is ignored. That is, the light emitting module 110 emits the light signal LS at the first time T1, and the depth information estimation unit 124 also calculates the first depth video frame at the first time T1.
Please refer to fig. 3a, which is a schematic diagram of the first depth video frame obtained by the depth information estimation unit 124 at the first time T1, wherein 9 black dots (actually, the number may be more or more dense) represent that 9 light spots in the light reflection signal LR irradiate on the pixel array 122, and thus the first depth video frame includes the depth information at the 9 black dots. Since the light emitting module 110 does not generate the light signal LS at the second time T2, the depth information estimation unit 124 does not obtain the corresponding depth video frame at the second time T2. The present application may generate a second depth video frame at a second time T2 based on the first depth video frame obtained at the first time T1, and added with the optical-flow information generated by the optical-flow estimation unit 128, as shown in fig. 3 b.
In fig. 3b, the white dots indicate the positions of the 9 light spots in the light reflection signal LR illuminating the pixel array 122 if the light signal LS is normally emitted at the second time T2, and the black dots in fig. 3b indicate the corresponding objects in the first depth video frame and the corresponding positions in the second depth video frame of the 9 black spots at the first time T1. That is, the black point position of fig. 3b can be obtained by adding the optical flow information corresponding to each black point of fig. 3a to the position of each black point of fig. 3 a. Specifically, in the case where the light signal LS is not generated at the second time T2, the depth information at the 9 white points is completely lacking. However, the depth information creating unit 130 obtains the displacement vectors of the 9 black points of the first depth video frame at the first time T1 in fig. 3a compared to the first time T1 at the second time T2 according to the optical flow information estimated by the optical flow estimating unit 128 (please note that the displacement vectors of the 9 black points may be different), and places the depth information corresponding to the 9 black points of the first depth video frame at the first time T1 in the second depth video frame, i.e., the 9 black points in fig. 3b, according to the displacement vectors to reconstruct the second depth video frame. Therefore, the second depth video frame can also obtain the depth information of 9 positions to approximately compensate the depth information detection loss caused by not generating the light signal LS at the second time T2, so that the system can keep the same level with the original scheme of continuously emitting light for measuring the depth at the depth information output frequency of T1 and T2. Since the power consumption of the light source 112 dominates the power consumption of the imaging system 100, in general, this case (case one) can reduce the power consumption of the imaging system 100 by approximately half, but does not reduce the output frequency of the depth information, and in the approximate case where the optical flow information is accurate, the accuracy of this three-dimensional sensing is approximately unchanged.
< case two >
The light source control unit 132 controls the light emitting module 110 to generate the light signal LS with full power (higher power) at the first time T1 and generate the light signal LS with lower power at the second time T2, in the embodiment, the power of the light emitting module 110 at the second time T2 is half of the full power, that is, the intensity of the light signal LS at the second time T2 is half of the intensity of the light signal LS at the first time T1. Referring again to fig. 3b, in this case, the white dots in fig. 3b indicate that the depth information of the 9 positions obtained by the optical signal LS with only half the energy at the second time T2 is less accurate than the depth information of the 9 positions obtained by the optical signal LS with full energy (i.e., the situation of not saving power) in fig. 3 a.
However, as described in the first case, the depth information creating unit 130 reconstructs the depth information of the 9 black points at the second time T2, i.e. the 9 black points in fig. 3b, according to the 9 black points of the first depth video frame and the optical flow information obtained at the first time T1. In summary, since the second depth video frame has the depth information of the 9 white dot positions with poor accuracy in addition to obtaining the complete depth information of the 9 black dot positions as the additional compensation, in the case that the 9 black dot positions and the 9 white dot positions are not overlapped (i.e. in the case that the object corresponding to the 9 black dots of the first depth video frame obtained at the first time T1 moves in the second depth video frame by the second time T2), the number of light spots included in the optical signal LS becomes dense, the number of light spots included in the second depth video frame is equal to 9 and the number of black dot positions added to 9 black dot positions, and the number of light spots included in the optical signal LS is equal to 18, so that the total included information is not decreased or increased, the accuracy of the three-dimensional sensing can be improved, and the power consumption can be reduced by about 25%. In the most extreme case, where all objects in the first depth video frame obtained at the first time T1 and all objects in the second depth video frame obtained at the second time T2 have no relative movement, the 9 black point positions and the 9 white point positions in the second depth video frame completely overlap, which is not as great an aid to improving the accuracy of three-dimensional sensing as when the 9 black point positions and the 9 white point positions are completely or partially non-overlapping.
< case three >
The light source control unit 132 controls the light emitting module 110 to generate the light signal LS with full power at the first time T1 and the second time T2. Referring again to fig. 3b, in this case, the white dots in fig. 3b represent the depth information of 9 positions obtained by the full-energy optical signal LS at the second time T2, and the accuracy of the depth information of the white dot position of fig. 3b is unchanged compared to the depth information of 9 black dot positions obtained by the full-energy optical signal LS of fig. 3 a.
As described in the first and second cases, the depth information creating unit 130 reconstructs the depth information of the 9 black points at the second time T2, i.e. the 9 black points in fig. 3b, according to the 9 black points of the first depth video frame obtained at the first time T1 and the optical flow information. In summary, when the 9 black point positions and the 9 white point positions do not overlap (that is, when the object corresponding to the 9 black points of the first depth video frame obtained at the first time T1 moves in the second depth video frame at the second time T2), the second depth video frame has the complete depth information of the 9 white point positions in addition to the complete depth information of the 9 black point positions, and the total information included in the second depth video frame is twice without increasing the power consumption, so that the accuracy of three-dimensional sensing can be increased by at most twice as much as the original accuracy. Similarly, if the 9 black point positions and 9 white point positions in the second depth video frame completely overlap, the assistance to improve the accuracy of three-dimensional sensing is not as great as when the 9 black point positions and 9 white point positions do not completely or partially overlap.
Fig. 4a is a schematic diagram of a first depth video frame obtained at a first time T1, that is, a depth video frame obtained without depth reconstruction. Fig. 4b is a diagram illustrating a second depth video frame reconstructed according to the second embodiment of the second case two or the third case three at the second time T2. Note that, as can be seen at the circles of fig. 4a and 4b, the potted leaves in the second depth video frame of fig. 4b appear richer in detail and closer to reality than fig. 4 a.
In this application, the light source control unit 132 may adjust the power of the light source 112 in a fixed pattern, i.e. vary the intensity of the emitted light source signal LI. For example, the light source control unit 132 controls the intensity of the light source signal LI emitted by the light source 112 to be full intensity, no light emission, full intensity, no light emission at the first time T1, the second time T2, the third time T3, and the fourth time T4 ] of the fixed time interval; or full strength, half strength, full strength, half strength; or full strength, full strength.
In some embodiments, when the depth information creating unit 130 creates a depth video frame at time TN, in addition to the depth video frame at time TN being compensated by using the depth information of the depth video frame of the previous frame (time TN-1), one or more frames of depth video frames before the previous frame may also be included to compensate the depth video frame at time TN, that is, the depth video frame at time TN is compensated by using the depth video frame at time TN-2 even earlier, which may further increase the amount of depth information in the depth video frame at time TN.
Fig. 5 is a functional block diagram schematic of a second embodiment of the imaging system of the present disclosure. The difference between the imaging system 500 and the imaging system 100 is that the light source control unit 132 of the imaging system 500 determines how to adjust the power of the light source 112 by referring to the optical flow estimation result of the optical flow estimation unit 128, for example, when the complexity of the scene is too low, which results in the optical flow estimation unit 128 determining that the information amount of the estimated optical flow information is too low or the degree of confidence is too low, which results in the depth information establishing unit 130 not being able to properly utilize the earlier depth video frame to compensate the later depth video frame. In this embodiment, the optical flow estimation unit 128 notifies the light source control unit 132 that the situation is occurring, and the light source control unit 132 prevents the power of the light source 112 from being reduced in the situation, so as to avoid a great reduction in accuracy. Conversely, when the complexity of the scene is high, the optical flow estimation unit 128 notifies the light source control unit 132 that the power of the light source 112 can be reduced to save power consumption.
The imaging system 100/500 of fig. 1 and 5 may be implemented in an electronic device. In certain embodiments, the electronic device may be any electronic device such as a smart phone, a personal digital assistant, a handheld computer system, or a tablet computer.
The foregoing description has set forth briefly the features of certain embodiments of the present application so that those skilled in the art may more fully appreciate the various aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and/or achieving the same advantages of the embodiments introduced herein. Those skilled in the art should understand that they can still make various changes, substitutions and alterations herein without departing from the spirit and scope of the present disclosure.
Claims (20)
1. An imaging system, comprising:
a time-of-flight sensor, comprising:
a pixel array for sensing a light reflection signal reflected by an object to the pixel array, the light reflection signal comprising a plurality of reflected light spots that hit a plurality of pixels in the pixel array;
a depth information estimation unit for reading the pixel array and obtaining the flight time of the plurality of reflection light spots according to the pixel array to estimate the depth information;
the two-dimensional image sensor is used for sensing two-dimensional video frames;
the optical flow estimation unit is used for carrying out optical flow estimation on different two-dimensional video frames obtained by the two-dimensional image sensor at different times; and
a depth information establishing unit coupled to the optical flow estimating unit and the depth information estimating unit;
wherein the optical flow estimation unit performs optical flow estimation on a first two-dimensional video frame obtained by the two-dimensional image sensor at a first time and a second two-dimensional video frame obtained at a second time after the first time to generate an optical flow estimation result, and the depth information establishing unit generates a second depth video frame corresponding to the second time according to the optical flow estimation result and a first depth video frame obtained by the time-of-flight sensor at the first time.
2. The imaging system of claim 1, wherein the optical flow estimation unit performs the optical flow estimation includes performing sparse pixel matching.
3. The imaging system of claim 1, wherein the optical flow estimation unit to perform the optical flow estimation includes performing dense pixel matching.
4. The imaging system of claim 1, further comprising:
a light emitting module for sending an optical signal to the target to generate the optical reflection signal, wherein the light emitting module comprises:
a light source for outputting a light source signal; and
an optical element for changing the traveling route of the light source signal to generate the light signal.
5. The imaging system of claim 4, wherein the light source generates the light source signal at a first power at the first time and the light source generates the light source signal at a second power at the second time.
6. The imaging system of claim 5, wherein the second power is zero.
7. The imaging system of claim 5, wherein the second power is not equal to zero, and the depth information establishing unit further generates the second depth video frame from a second depth video frame obtained by the time-of-flight sensor at the second time.
8. The imaging system of claim 7, wherein the first power is greater than the second power.
9. The imaging system of claim 7, wherein the first power is equal to the second power.
10. The imaging system of claim 4, further comprising:
and the light source control unit is used for determining the power of the light source according to the result of the optical flow estimation performed by the optical flow estimation unit.
11. An imaging method, comprising:
sensing a two-dimensional video frame at a first time to obtain a first two-dimensional video frame;
sensing light reflection signals at the first time to perform depth information estimation so as to obtain a first depth video frame;
sensing a two-dimensional video frame at a second time after the first time to obtain a second two-dimensional video frame;
performing optical flow estimation on the first two-dimensional video frame and the second two-dimensional video frame to generate optical flow estimation results; and
generating a second depth video frame corresponding to the second time according to the optical flow estimation result and the first depth video frame.
12. The imaging method of claim 11, wherein performing the optical flow estimation comprises performing sparse pixel matching.
13. The imaging method of claim 11, wherein performing the optical flow estimation comprises performing dense pixel matching.
14. The imaging method of claim 11, further comprising:
transmitting an optical signal to generate the optical reflection signal, comprising:
outputting a light source signal; and
changing a travel route of the light source signal to generate the light signal.
15. The imaging method of claim 14, wherein outputting the light source signal comprises: generating the light source signal at a first power at the first time; and
generating the light source signal at a second power at the second time.
16. The imaging method of claim 15, wherein the second power is zero.
17. The imaging method of claim 15, wherein the second power is not equal to zero, and generating the second depth video frame corresponding to the second time from the optical flow estimation and the first depth video frame comprises:
generating a second depth video frame corresponding to the second time according to the optical flow estimation result, the first depth video frame and a second depth video frame obtained by the time-of-flight sensor at the second time.
18. The imaging method of claim 17, wherein the first power is greater than the second power.
19. The imaging method of claim 17, wherein the first power is equal to the second power.
20. The imaging method of claim 14, further comprising:
and determining the power for outputting the light source signal according to the optical flow estimation result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110766336.9A CN113489858A (en) | 2021-07-07 | 2021-07-07 | Imaging system and related imaging method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110766336.9A CN113489858A (en) | 2021-07-07 | 2021-07-07 | Imaging system and related imaging method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113489858A true CN113489858A (en) | 2021-10-08 |
Family
ID=77941624
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110766336.9A Pending CN113489858A (en) | 2021-07-07 | 2021-07-07 | Imaging system and related imaging method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113489858A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130286017A1 (en) * | 2010-05-07 | 2013-10-31 | David MARIMÓN SANJUAN | Method for generating depth maps for converting moving 2d images to 3d |
| CN107204012A (en) * | 2016-03-16 | 2017-09-26 | 美国亚德诺半导体公司 | Reduce the power consumption of time-of-flight depth imaging |
| CN111432144A (en) * | 2020-06-12 | 2020-07-17 | 深圳市汇顶科技股份有限公司 | Imaging system and related electronic device and operating method of imaging system |
| CN112740065A (en) * | 2018-09-25 | 2021-04-30 | 苹果公司 | Enhanced depth mapping using visual inertial ranging |
| CN113009494A (en) * | 2021-02-22 | 2021-06-22 | 曜芯科技有限公司 | Depth detection device and electronic equipment |
-
2021
- 2021-07-07 CN CN202110766336.9A patent/CN113489858A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130286017A1 (en) * | 2010-05-07 | 2013-10-31 | David MARIMÓN SANJUAN | Method for generating depth maps for converting moving 2d images to 3d |
| CN107204012A (en) * | 2016-03-16 | 2017-09-26 | 美国亚德诺半导体公司 | Reduce the power consumption of time-of-flight depth imaging |
| CN112740065A (en) * | 2018-09-25 | 2021-04-30 | 苹果公司 | Enhanced depth mapping using visual inertial ranging |
| CN111432144A (en) * | 2020-06-12 | 2020-07-17 | 深圳市汇顶科技股份有限公司 | Imaging system and related electronic device and operating method of imaging system |
| CN113009494A (en) * | 2021-02-22 | 2021-06-22 | 曜芯科技有限公司 | Depth detection device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9826216B1 (en) | Systems and methods for compact space-time stereo three-dimensional depth sensing | |
| US20210185190A1 (en) | Compact, low cost vcsel projector for high performance stereodepth camera | |
| US10839539B2 (en) | System and method for active stereo depth sensing | |
| Kadambi et al. | 3d depth cameras in vision: Benefits and limitations of the hardware: With an emphasis on the first-and second-generation kinect models | |
| Yang et al. | Fusion of active and passive sensors for fast 3D capture | |
| Dal Mutto et al. | Time-of-flight cameras and Microsoft KinectTM | |
| Stoykova et al. | 3-D time-varying scene capture technologies—A survey | |
| US8491135B2 (en) | Interactive projection with gesture recognition | |
| Kowdle et al. | The need 4 speed in real-time dense visual tracking | |
| US20060221250A1 (en) | Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing | |
| CN110325879A (en) | System and method for compress three-dimensional depth sense | |
| US20110164191A1 (en) | Interactive Projection Method, Apparatus and System | |
| CN111083453B (en) | Projection device, method and computer readable storage medium | |
| US20170285146A1 (en) | Apparatus, method for distance measurement, and non-transitory computer-readable storage medium | |
| Tsalakanidou et al. | Real-time acquisition of depth and color images using structured light and its application to 3D face recognition | |
| WO2006130734A2 (en) | Method and system to increase x-y resolution in a depth (z) camera using red, blue, green (rgb) sensing | |
| WO2013120041A1 (en) | Method and apparatus for 3d spatial localization and tracking of objects using active optical illumination and sensing | |
| WO2022054422A1 (en) | Obstacle detection device, obstacle detection system, and obstacle detection method | |
| JP2022045947A5 (en) | ||
| US20220012905A1 (en) | Image processing device and three-dimensional measuring system | |
| CN113489858A (en) | Imaging system and related imaging method | |
| KR20220039101A (en) | Robot and controlling method thereof | |
| US20230273357A1 (en) | Device and method for image processing | |
| WO2022195954A1 (en) | Sensing system | |
| US20230003894A1 (en) | Time-of-flight imaging circuitry, time-of-flight imaging system, time-of-flight imaging method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220117 Address after: 17-00, No. 16, Goliath wharf, Raffles square, Singapore Applicant after: Dick Innovation Technology Co.,Ltd. Address before: 35th floor, No.66, Section 1, Zhongxiao West Road, Zhongzheng District, Taipei, China Applicant before: Yaoxin Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230727 Address after: 51 Science Park Road, Aries Tower # 04-22, Singapore Applicant after: Huiding Technology Private Ltd. Address before: 16 # 17-00, Godfrey Pier, Raffles Place Applicant before: Dick Innovation Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right |