US20210344892A1 - Method and system for determining optimal exposure time and number of exposures in structured light-based 3d camera - Google Patents
Method and system for determining optimal exposure time and number of exposures in structured light-based 3d camera Download PDFInfo
- Publication number
- US20210344892A1 US20210344892A1 US17/365,985 US202117365985A US2021344892A1 US 20210344892 A1 US20210344892 A1 US 20210344892A1 US 202117365985 A US202117365985 A US 202117365985A US 2021344892 A1 US2021344892 A1 US 2021344892A1
- Authority
- US
- United States
- Prior art keywords
- pattern
- exposure time
- reference image
- structured light
- slope
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000009825 accumulation Methods 0.000 claims abstract description 36
- 238000012545 processing Methods 0.000 claims description 8
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/71—Circuitry for evaluating the brightness variation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/743—Bracketing, i.e. taking a series of images with varying exposure conditions
-
- H04N5/2351—
-
- H04N5/2353—
-
- H04N5/2356—
Definitions
- the present invention relates to a method for determining an exposure of a structured light-based 3D camera.
- technologies for reconstructing a 3D image of an object using one or more cameras may be largely classified as either being active or passive techniques.
- active techniques include laser triangulation and structured light.
- An example of a passive technique is stereovision.
- Active techniques are typically preferred for research or industrial purposes because they may achieve higher precision scanning than passive techniques.
- a 3D camera system for use with structured light can be considered as a modification of a stereo camera system.
- a 3D camera system used for structured light is configured such that one camera is replaced by a projection device such, as a beam projector.
- a projection device such as a beam projector.
- one or more patterns are projected onto an object using a projection device, an image of the object onto which the pattern is projected is captured using an image-capturing device, such as a camera, and 3D information about the object is acquired by analyzing the captured image.
- a structured light-based camera system has a fast processing speed and high spatial resolution. Thanks to these advantages, the structured light-based camera system is widely used in various fields, such as object modeling and recognition, 3D measurement, industrial inspection, reverse engineering, and the like.
- objects have various reflection coefficients. Accordingly, when patterns are projected from a projection device onto an object having a low reflection coefficient (for example, a black ball), it is difficult to correctly acquire the projected patterns using a capturing device. Conversely, when patterns are projected onto an object having a high reflection coefficient (for example, a white ball having a glossy surface), because the gloss causes saturation of patterns (seen as blurred patterns) in the image acquired using a capturing device, it is difficult to acquire accurate patterns. Also, light conditions of a real environment may affect the image acquisition of patterns. Additionally, it may be difficult to acquire accurate patterns in a real environment that is too bright or too dark.
- the present invention provides a method and system for determining the optimal exposure time and the number of exposures by which, when an object having a low reflection coefficient and an object having a high reflection coefficient are mixed together in an environment, the optimal exposure time of a structured light-based 3D camera is determined, and by which, when it is difficult to reconstruct 3D data using only a single exposure, two exposure times having the optimal lengths and the optimal number of the exposures are determined.
- An object of the present invention is to improve reliability and to increase the extent to which 3D data may be reconstructed using such a method and system.
- a method for determining an optimal exposure time and a number of exposures in a structured light-based 3D camera system includes capturing a reference image onto which a first pattern and a second pattern are projected, calculating a slope related to brightness and an exposure time for a pixel of the reference image and categorizing a state of the slope, creating an upper loss pixel distribution chart and an accumulation pixel distribution chart by calculating a number of upper loss pixels and a number of accumulation pixels for an exposure time of the structured light-based 3D camera system, and determining the optimal exposure time and the number of exposures of the structured light-based 3D camera system using the upper loss pixel distribution chart and the accumulation pixel distribution chart.
- the first pattern and the second pattern may include at least one of an all-black pattern and an all-white pattern.
- the capturing the reference image may include determining a first exposure time so as to prevent a brightness value of an object having a high reflection coefficient from being greater than 250, and determining a second exposure time so as to prevent a brightness value of an object having a low reflection coefficient from being less than 10.
- the calculating the slope and categorizing the state of the slope may include defining a state of brightness depending on a brightness value of a pixel in the reference image that is captured with the first exposure time and the second exposure time.
- the calculating the slope and categorizing the state of the slope may further include defining the state of brightness as a case in which the brightness value of the pixel in the reference image captured with the first exposure time and the second exposure time is less than 10, a case in which the brightness value of the pixel in the reference image captured with the first exposure time and the second exposure time is greater than 250, and a case in which the brightness value is not less than 10 and is not greater than 250.
- the calculating the slope and categorizing the state of the slope may further include calculating the slope related to the brightness and the exposure time for each of the first pattern and the second pattern depending on the state of brightness, and categorizing the state of the slope.
- the creating the upper loss pixel distribution chart and the accumulation pixel distribution chart may include excluding states in which the brightness value of the pixel is less than a predetermined threshold brightness value depending on the state of the slope for each of the first pattern and the second pattern when predicting a pixel to be reconstructed.
- the determining the optimal exposure time and the number of exposures may include searching for an exposure time that first satisfies a predetermined first criterion in the accumulation pixel distribution chart, and determining whether a percentage of upper loss pixels at the found exposure time satisfies a predetermined second criterion.
- the determining the optimal exposure time and the number of exposures may further include a first step in which, when the percentage of the upper loss pixels does not satisfy the predetermined second criterion, whether the corresponding exposure time is a maximum exposure time of the structured light-based 3D camera system is determined; a second step in which, when the corresponding exposure time is not the maximum exposure time of the structured light-based 3D camera system, the corresponding exposure time is increased by a predetermined amount; a third step in which whether the percentage of the upper loss pixels at the increased exposure time satisfies the predetermined second criterion is determined again; and repeating the first, second, and third steps until the percentage of the upper loss pixels satisfies the predetermined second criterion or until the corresponding exposure time becomes the maximum exposure time.
- the determining the optimal exposure time and the number of exposures may further include setting the number of exposures to a single exposure when the percentage of the upper loss pixels satisfies the predetermined second criterion.
- the determining the optimal exposure time and the number of exposures may further include, when the corresponding exposure time is the maximum exposure time of the structured light-based 3D camera system, setting the exposure time that first satisfies the predetermined first criterion in the accumulation pixel distribution chart as a long exposure time; and setting the exposure time at which the percentage of the upper loss pixels first satisfies the predetermined second criterion in the upper loss pixel distribution chart as a short exposure time.
- a structured light-based 3D camera system includes a projection unit for projecting a pattern onto a target object, a capturing unit for capturing an image that includes the pattern projected from the projection unit, and a processing unit for determining an optimal exposure time and a number of exposures using a reference image acquired by the capturing unit, wherein the processing unit includes an image acquisition unit for acquiring an image onto which a pattern is projected; a slope calculation and state categorization unit for calculating a slope related to brightness and an exposure time for a pixel of the reference image acquired from the image acquisition unit and for categorizing a state of the slope; a distribution chart creation unit for creating an upper loss pixel distribution chart and an accumulation pixel distribution chart by calculating a number of upper loss pixels and a number of accumulation pixels for the exposure time of the structured light-based 3D camera system; and an optimal exposure time and frequency determination unit for determining the optimal exposure time and the number of exposures of the structured light-based 3D camera system using the
- FIG. 1 is a flowchart of a method for determining the optimal exposure time
- FIG. 2A and FIG. 2B are detailed flowcharts of the process of calculating a slope having high reliability in a captured reference image and of categorizing the state of the slope;
- FIG. 3 is an example of a slope1/slope0 distribution chart
- FIG. 4A is an example of an accumulation pixel distribution chart
- FIG. 4B is an example of an upper loss pixel distribution chart
- FIG. 5 is a flowchart of the process of determining an exposure time and the number of exposures
- FIG. 6 is a schematic block diagram of a structured light-based 3D camera system in which a method for determining the optimal exposure according to the present invention is used;
- FIG. 7 shows an experimental environment for comparing the case in which a single exposure is set according to an existing method for determining the optimal exposure with the case in which two exposures are set according to the method proposed by the present invention
- FIG. 8 is a table that shows an exposure time and the number of reconstructed pixels when the scene of FIG. 7 is captured using a structured light-based 3D camera system.
- FIGS. 9A, 9B and 9C show the result of reconstruction of 3D data when the scene of FIG. 7 is captured using a structured light-based 3D camera system.
- the present invention relates to methods and apparatus for determining an optimal number of exposures and a duration thereof in a structured light-based 3D camera system and a structured light-based 3D camera system using such methods and apparatus.
- an exposure of a structured light-based 3D camera means the amount of light that is introduced when an image is captured using the camera.
- the exposure of a camera is adjusted using an aperture and/or shutter speed of the camera.
- FIG. 1 is a flowchart of a method for determining the optimal exposure time.
- the method for determining the optimal exposure of a structured light-based 3D camera includes capturing a reference image at step S 100 , calculating a slope and categorizing the state of the slope at step S 200 , creating a distribution chart at step S 300 , and determining the optimal exposure time and the number of exposures at step S 400 .
- pattern 0 means an all-black pattern (when a projection device is turned off)
- pattern 1 means an all-white pattern.
- two reference images are captured for each of pattern 0 and pattern 1.
- reliability may be reduced because the brightness value of a pixel becomes 255 even when the actual brightness value of a pixel is greater than 255 (because the maximum brightness value of a pixel in an 8-bit image is 255, the brightness beyond that is also represented as 255 ).
- the exposure time for capturing a reference image is set as described below.
- the shutter speed is set fast (that is, a short exposure time is set).
- the shutter speed is set slow (that is, a long exposure time is set).
- a reference image for calculating a reliable slope is selected from among the captured reference images, a slope is calculated for the selected reference image, and the state thereof is categorized.
- a slope for pattern 0 and a slope for pattern 1 may be calculated for all the pixels in the reference image according to the processes of the flowchart in FIG. 2A and FIG. 2B .
- E 1 means a short exposure time, which is set in a camera when a reference image is captured
- E 2 means a long exposure time, which is set in the camera when the reference image is captured
- I 1 means the brightness value of a pixel in the reference image, which is captured when E 1 is set
- I 2 means the brightness value of a pixel in the reference image, which is captured when E 2 is set. Because there is a linear relationship between I and E, the value of E corresponding to the desired exposure time may be estimated using the slope of I/E.
- the state of brightness of a pixel is defined as follows.
- the state When I ⁇ 10, the state is defined as ‘under’, and when I>250, the state is defined as ‘over’. When the state is neither ‘under’ nor ‘over’, the state is defined as ‘normal’. Depending on the values of I 1 and I 2 , there may be seven cases, as shown in FIG. 2A and FIG. 2B .
- S 201 is the case in which both the value of I 1 and the value of I 2 are in the ‘normal’ state, in which case the mean of the two slopes for the brightness and the exposure time in the ‘normal’ state is set as the value of a slope.
- S 202 and S 203 are the cases in which either the value of I 1 or the value of I 2 is in the ‘normal’ state, in which case the slope for the brightness and the exposure time in the ‘normal’ state is set as the value of a slope.
- S 204 and S 205 are the cases in which the value of I 1 and the value of I 2 are respectively in the ‘under’ and ‘over’ states or in the ‘over’ and ‘under’ states.
- the value in the ‘over’ state indicates the case in which the actual brightness value cannot be estimated
- the value in the ‘under’ state indicates the case in which the percentage of noise is high but the value is more reliable than the value in the ‘over’ state. Accordingly, the slope for the brightness and the exposure time in the ‘under’ state is set as the value of a slope.
- S 206 is the case in which both the value of I 1 and the value of I 2 are in the ‘under’ state.
- the greater brightness value is selected, and the slope for the brightness and the exposure time corresponding to the selected brightness value is set as the value of a slope.
- S 207 is the case in which both the value of I 1 and the value of I 2 are in the ‘over’ state, in which case the slope for the brightness and the exposure time corresponding to the lower brightness value is set as the value of a slope.
- pixels that can be reconstructed and pixels that cannot be reconstructed at each exposure time in the structured light-based camera system are predicted (that is, the number of upper loss pixels and the number of accumulation pixels are calculated), whereby an upper loss pixel distribution chart and an accumulation pixel distribution chart are acquired.
- the structured light-based 3D camera may calculate 3D data only when a patterned region, which includes patterns projected from a projection device, is distinguishable from a non-patterned region. Therefore, the difference between the brightness in the patterned region and the brightness in the non-patterned region must be greater than a threshold value I th .
- the Slope 1 /Slope 0 distribution chart shown in FIG. 3 is used.
- Slope 0 means the slope that is calculated when pattern 0 is projected
- Slope 1 is the slope that is calculated when pattern 1 is projected.
- Equation (1) Equation (1)
- Equation (3) Equation (3)
- a pixel at which Slope 0 is less than p 0 and Slope 1 is less than p 1 and greater than p 2 is estimated as a pixel from which 3D data may be reconstructed.
- a pixel at which Slope 0 is greater than p 0 and Slope 1 is greater than p 1 is referred to as an upper loss pixel
- a pixel at which Slope 1 is greater than p 2 is referred to as an accumulation pixel.
- the maximum number of accumulation pixels is referred to as N M .
- the number of accumulation pixels at each exposure time may be calculated using Equation (3).
- the number of accumulation pixels at each exposure time E n is referred to as N n . Accordingly, the percentage of accumulation pixels to the maximum number of accumulation pixels is calculated as
- the number of pixels that satisfy the condition of Slope 0 ⁇ (255 ⁇ I th )/E and Slope 1 >255/E is the number of upper loss pixels.
- the number of upper loss pixels calculated at each of all exposure times (from E 1 to E M ) is referred to as L n . Accordingly, the percentage of upper loss pixels to the maximum number of accumulation pixels is calculated as
- the value of (Slope 1 ⁇ Slope 0 )E may be greater than the actual value thereof.
- the corresponding pixel may be assumed to be a pixel that can be reconstructed.
- the calculated value of (Slope 1 ⁇ Slope 0 )E may be less than the actual value thereof, and the value may be less than I th .
- the optimal exposure time and the number of exposures are determined at step S 400 , as shown in the flowchart of FIG. 5 .
- S 501 is the step in which the number of exposures is set to a single exposure and the optimal duration thereof is the exposure time at which the percentage of pixels lost in the upper and lower regions is less than 1%.
- the percentage of lower loss pixels is
- the exposure time at which the percentage of accumulation pixels is 99.5% is searched for in the accumulation pixel distribution chart at step S 51 , whether the percentage of upper loss pixels is equal to or less than 0.5% is determined at step S 52 , and whether the number of exposures is set to a single exposure is determined at step S 53 .
- step S 502 is the step in which the number of exposures is set to two exposures.
- step S 502 includes steps S 54 , S 55 , and S 56 .
- step S 54 when it has been determined at step S 52 that the percentage of upper loss pixels is greater than 0.5%, whether the exposure time found at step S 51 is the maximum exposure time of the camera is determined.
- step S 55 the exposure time at which the percentage of accumulation pixels first becomes 99.5% is searched for in the accumulation pixel distribution chart, and the found exposure time is set as a long exposure time.
- step S 56 the exposure time at which the percentage of upper loss pixels is equal to or less than 0.5% is searched for, and the found exposure time and the long exposure time are set as the duration of the two exposures.
- the exposure time found at step S 51 is not the maximum exposure time of the camera, the exposure time is increased by 1 ms and the process returns to step S 52 . If the exposure time found at step S 51 is the maximum exposure time of the camera, the exposure time found at step S 51 and the newly found exposure time at step S 56 are set as the exposure times of the two exposures.
- the number of exposures is set to a single exposure, but if not, the number of exposures is set to two exposures in order to enable an object having a low reflection coefficient to be reconstructed at a long exposure time and an object having a high reflection coefficient to be reconstructed at a short exposure time, whereby 99% of pixels can be reconstructed.
- a structured light-based 3D camera system 10 that uses the above-described method for determining the optimal number of exposures and the duration thereof may be provided.
- the structured light-based 3D camera system 10 includes a projection unit 11 , a capturing unit 12 , and a processing unit 13 .
- the projection unit 11 may project various patterns onto a target object.
- the capturing unit 12 may capture an image that includes the pattern projected from the projection unit 11 .
- the processing unit 13 includes an image acquisition unit 131 , a slope calculation and state categorization unit 132 , a distribution chart creation unit 133 and an optimal exposure time and frequency determination unit 134 .
- the image acquisition unit 131 may acquire an image onto which patterns, such as pattern 0 (an all-black pattern), pattern 1 (an all-white pattern) or the like, are projected.
- the slope calculation and state categorization unit 132 calculates a slope related to brightness and an exposure time for all the pixels in the reference image, onto which pattern 0 and pattern 1 are projected, and categorizes the state of the slope, the reference image being acquired from the image acquisition unit.
- the distribution chart creation unit 133 may acquire a distribution chart by predicting pixels that can be reconstructed and pixels that cannot be reconstructed (that is, by calculating the number of upper loss pixels and the number of accumulation pixels) at each exposure time using the difference between the slope calculated when pattern 1 is projected and the slope calculated when pattern 0 is projected.
- the optimal exposure time and frequency determination unit 134 may determine the optimal exposure time and the optimal number of exposures using the above-described method that uses the upper loss pixel distribution chart and the accumulation pixel distribution chart.
- FIG. 7 shows an experimental example for comparing the case in which a single exposure is set according to an existing method for determining the optimal exposure with the case in which two exposures are set according to the method proposed by the present invention.
- This example is configured to include an object having a high reflection coefficient (the white object at the left side) and an object having a low reflection coefficient (the black object in the box).
- FIG. 8 is a table that shows the exposure time and the number of reconstructed pixels when the scene of FIG. 7 is captured using a structured light-based 3D camera system.
- the number of exposures is set to a single exposure, the duration thereof is set to a range from 1 ms to 100 ms, having an interval of 3 ms, and the number of pixels that are actually reconstructed into 3D data is measured.
- the exposure duration is set to 28 ms, the maximum number of pixels are reconstructed, in which case the number of reconstructed pixels is 237,536.
- FIGS. 9A, 9B and 9C show the result of reconstruction of 3D data when the scene of FIG. 7 is captured using a structured light-based 3D camera system.
- the necessary number of exposures is set to two exposures, and the durations thereof are set to 3 ms and 84 ms, respectively.
- FIG. 9A is the result of capturing an image when the exposure time is set to 3 ms, in which case only a region that has a high reflection coefficient is reconstructed.
- FIG. 9B is the result of capturing an image when the exposure time is set to 84 ms, in which case only a region that has a low reflection coefficient is reconstructed.
- FIG. 9A is the result of capturing an image when the exposure time is set to 3 ms, in which case only a region that has a high reflection coefficient is reconstructed.
- FIG. 9B is the result of capturing an image when the exposure time is set to 84 ms, in which case only a region that has a low reflection coefficient
- 9C is the result of a combination of the reconstructed pixels of FIG. 9A and the reconstructed pixels of FIG. 9B , with a total of 251,900 reconstructed pixels. Consequently, it is confirmed that the percentage of reconstructed pixels is increased by 6% when the number of exposures is set to two exposures compared to when the number of exposures is set to one exposure.
- the structured light-based 3D camera system of the present invention may more accurately reconstruct 3D data of an object having a low or high reflection coefficient in industrial fields.
- the structured light-based 3D camera system of the present invention sets two exposures having the optimal duration when it is difficult to reconstruct 3D data only through a single exposure in an environment in which an object having a low reflection coefficient and an object having a high reflection coefficient are both present, whereby reliability may be improved and the extent to which 3D data are reconstructed may be increased.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Electromagnetism (AREA)
- Studio Devices (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
Disclosed is a method for determining the optimal exposure time and the number of exposures in a structured light-based 3D camera system. The method includes capturing a reference image onto which first and second patterns are projected, calculating a slope related to brightness and an exposure time for a pixel of the reference image and categorizing a state of the slope, creating an upper loss pixel distribution chart and an accumulation pixel distribution chart by calculating the number of upper loss pixels and the number of accumulation pixels for the exposure time of the structured light-based 3D camera system, and determining the optimal exposure time and the number of exposures of the structured light-based 3D camera system using the upper loss pixel distribution chart and the accumulation pixel distribution chart.
Description
- The present application is a continuation of U.S. patent application Ser. No. 16/558,151, entitled “Method and System for Determining Optimal Exposure Time and Number of Exposures in Structured Light-Based 3D Camera”, filed on Sep. 1, 2019, which is a continuation of U.S. patent application Ser. No. 15/334,314, entitled “Method and System for Determining Optimal Exposure Time and Number of Exposures in Structured Light-Based 3D Camera”, filed on Oct. 26, 2016, now U.S. Pat. No. 10,404,968, which claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2015-0149156 (Filed on Oct. 27, 2015), the entirety of all two of which are expressly incorporated herein by reference.
- The present invention relates to a method for determining an exposure of a structured light-based 3D camera.
- Generally, technologies for reconstructing a 3D image of an object using one or more cameras may be largely classified as either being active or passive techniques. Examples of active techniques include laser triangulation and structured light. An example of a passive technique is stereovision. Active techniques are typically preferred for research or industrial purposes because they may achieve higher precision scanning than passive techniques.
- A 3D camera system for use with structured light can be considered as a modification of a stereo camera system. In particular, unlike a stereo camera system, in which two identical cameras are used, a 3D camera system used for structured light is configured such that one camera is replaced by a projection device such, as a beam projector. Accordingly, in a 3D camera system used for structured light scanning, one or more patterns are projected onto an object using a projection device, an image of the object onto which the pattern is projected is captured using an image-capturing device, such as a camera, and 3D information about the object is acquired by analyzing the captured image.
- In other words, because a stereo camera system passively uses features extracted from an image but a structured light-based camera system actively projects one or more patterns onto an object using a projection device and uses the projected pattern to distinguish features of the object, a structured light-based camera system has a fast processing speed and high spatial resolution. Thanks to these advantages, the structured light-based camera system is widely used in various fields, such as object modeling and recognition, 3D measurement, industrial inspection, reverse engineering, and the like.
- However, structured light-based camera systems have the following general problems.
- Generally, in the real world, objects have various reflection coefficients. Accordingly, when patterns are projected from a projection device onto an object having a low reflection coefficient (for example, a black ball), it is difficult to correctly acquire the projected patterns using a capturing device. Conversely, when patterns are projected onto an object having a high reflection coefficient (for example, a white ball having a glossy surface), because the gloss causes saturation of patterns (seen as blurred patterns) in the image acquired using a capturing device, it is difficult to acquire accurate patterns. Also, light conditions of a real environment may affect the image acquisition of patterns. Additionally, it may be difficult to acquire accurate patterns in a real environment that is too bright or too dark.
- In order to solve the above problems, it is necessary to adjust the exposure of a camera by controlling the aperture of a camera depending on the environment.
- With the introduction of 3D printers, there is an increase in the use of structured light-based cameras by laypersons. However, because it is not easy to adjust the parameters of such cameras to adapt in real time to changing environmental conditions, it is difficult for laypersons, who are not experts, to use such cameras. Also, with the development of structured light-based 3D cameras, there is an increase in the demand for acquiring accurate 3D data in industrial fields. Here, in order to construct an automated system for use in industry, a structured light-based camera is fixed to an industrial robot, whereby accurate 3D data may be acquired. However, because most objects to be recognized in industrial fields are made of metal, which has a low reflection coefficient, the construction of an automated system for use in industrial fields is impeded.
- Therefore, it is necessary to automatically adjust the number of exposures and the exposure time of a 3D camera depending on the environment.
- The present invention provides a method and system for determining the optimal exposure time and the number of exposures by which, when an object having a low reflection coefficient and an object having a high reflection coefficient are mixed together in an environment, the optimal exposure time of a structured light-based 3D camera is determined, and by which, when it is difficult to reconstruct 3D data using only a single exposure, two exposure times having the optimal lengths and the optimal number of the exposures are determined. An object of the present invention is to improve reliability and to increase the extent to which 3D data may be reconstructed using such a method and system.
- According to one aspect of the present invention, there is provided a method for determining an optimal exposure time and a number of exposures in a structured light-based 3D camera system. The method includes capturing a reference image onto which a first pattern and a second pattern are projected, calculating a slope related to brightness and an exposure time for a pixel of the reference image and categorizing a state of the slope, creating an upper loss pixel distribution chart and an accumulation pixel distribution chart by calculating a number of upper loss pixels and a number of accumulation pixels for an exposure time of the structured light-based 3D camera system, and determining the optimal exposure time and the number of exposures of the structured light-based 3D camera system using the upper loss pixel distribution chart and the accumulation pixel distribution chart.
- The first pattern and the second pattern may include at least one of an all-black pattern and an all-white pattern.
- The capturing the reference image may include determining a first exposure time so as to prevent a brightness value of an object having a high reflection coefficient from being greater than 250, and determining a second exposure time so as to prevent a brightness value of an object having a low reflection coefficient from being less than 10.
- The calculating the slope and categorizing the state of the slope may include defining a state of brightness depending on a brightness value of a pixel in the reference image that is captured with the first exposure time and the second exposure time.
- The calculating the slope and categorizing the state of the slope may further include defining the state of brightness as a case in which the brightness value of the pixel in the reference image captured with the first exposure time and the second exposure time is less than 10, a case in which the brightness value of the pixel in the reference image captured with the first exposure time and the second exposure time is greater than 250, and a case in which the brightness value is not less than 10 and is not greater than 250.
- The calculating the slope and categorizing the state of the slope may further include calculating the slope related to the brightness and the exposure time for each of the first pattern and the second pattern depending on the state of brightness, and categorizing the state of the slope.
- The creating the upper loss pixel distribution chart and the accumulation pixel distribution chart may include excluding states in which the brightness value of the pixel is less than a predetermined threshold brightness value depending on the state of the slope for each of the first pattern and the second pattern when predicting a pixel to be reconstructed.
- The determining the optimal exposure time and the number of exposures may include searching for an exposure time that first satisfies a predetermined first criterion in the accumulation pixel distribution chart, and determining whether a percentage of upper loss pixels at the found exposure time satisfies a predetermined second criterion.
- The determining the optimal exposure time and the number of exposures may further include a first step in which, when the percentage of the upper loss pixels does not satisfy the predetermined second criterion, whether the corresponding exposure time is a maximum exposure time of the structured light-based 3D camera system is determined; a second step in which, when the corresponding exposure time is not the maximum exposure time of the structured light-based 3D camera system, the corresponding exposure time is increased by a predetermined amount; a third step in which whether the percentage of the upper loss pixels at the increased exposure time satisfies the predetermined second criterion is determined again; and repeating the first, second, and third steps until the percentage of the upper loss pixels satisfies the predetermined second criterion or until the corresponding exposure time becomes the maximum exposure time.
- The determining the optimal exposure time and the number of exposures may further include setting the number of exposures to a single exposure when the percentage of the upper loss pixels satisfies the predetermined second criterion.
- The determining the optimal exposure time and the number of exposures may further include, when the corresponding exposure time is the maximum exposure time of the structured light-based 3D camera system, setting the exposure time that first satisfies the predetermined first criterion in the accumulation pixel distribution chart as a long exposure time; and setting the exposure time at which the percentage of the upper loss pixels first satisfies the predetermined second criterion in the upper loss pixel distribution chart as a short exposure time.
- According to another aspect of the present invention, there is provided a structured light-based 3D camera system. The system includes a projection unit for projecting a pattern onto a target object, a capturing unit for capturing an image that includes the pattern projected from the projection unit, and a processing unit for determining an optimal exposure time and a number of exposures using a reference image acquired by the capturing unit, wherein the processing unit includes an image acquisition unit for acquiring an image onto which a pattern is projected; a slope calculation and state categorization unit for calculating a slope related to brightness and an exposure time for a pixel of the reference image acquired from the image acquisition unit and for categorizing a state of the slope; a distribution chart creation unit for creating an upper loss pixel distribution chart and an accumulation pixel distribution chart by calculating a number of upper loss pixels and a number of accumulation pixels for the exposure time of the structured light-based 3D camera system; and an optimal exposure time and frequency determination unit for determining the optimal exposure time and the number of exposures of the structured light-based 3D camera system using the upper loss pixel distribution chart and the accumulation pixel distribution chart.
- The above and other objects, features and other advantages of the present invention will be more clearly understood from the following detailed description when taken in conjunction with the accompanying drawings, in which:
-
FIG. 1 is a flowchart of a method for determining the optimal exposure time; -
FIG. 2A andFIG. 2B are detailed flowcharts of the process of calculating a slope having high reliability in a captured reference image and of categorizing the state of the slope; -
FIG. 3 is an example of a slope1/slope0 distribution chart; -
FIG. 4A is an example of an accumulation pixel distribution chart; -
FIG. 4B is an example of an upper loss pixel distribution chart; -
FIG. 5 is a flowchart of the process of determining an exposure time and the number of exposures; -
FIG. 6 is a schematic block diagram of a structured light-based 3D camera system in which a method for determining the optimal exposure according to the present invention is used; -
FIG. 7 shows an experimental environment for comparing the case in which a single exposure is set according to an existing method for determining the optimal exposure with the case in which two exposures are set according to the method proposed by the present invention; -
FIG. 8 is a table that shows an exposure time and the number of reconstructed pixels when the scene ofFIG. 7 is captured using a structured light-based 3D camera system; and -
FIGS. 9A, 9B and 9C show the result of reconstruction of 3D data when the scene ofFIG. 7 is captured using a structured light-based 3D camera system. - The present invention relates to methods and apparatus for determining an optimal number of exposures and a duration thereof in a structured light-based 3D camera system and a structured light-based 3D camera system using such methods and apparatus.
- The present invention may be variously changed, and may have various embodiments, and specific embodiments will be described in detail below with reference to the attached drawings. However, it should be understood that those embodiments are not intended to limit the present invention to specific disclosure forms and that they include all changes, equivalents or modifications included in the spirit and scope of the present invention.
- In the present specification, a singular expression includes a plural expression unless a description to the contrary is specifically pointed out in context. In the present specification, it should be understood that terms such as “include” or “have” are merely intended to indicate that features, numbers, steps, operations, components, parts, or combinations thereof are present, and are not intended to exclude the possibility that one or more other features, numbers, steps, operations, components, parts, or combinations thereof will be present or added.
- Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, the same reference numerals are used to designate the same or similar elements throughout the drawings, and repeated descriptions of the same components will be omitted.
- In the present invention, an exposure of a structured light-based 3D camera means the amount of light that is introduced when an image is captured using the camera. Generally, the exposure of a camera is adjusted using an aperture and/or shutter speed of the camera. This specification describes a method and system for determining the optimal exposure (by adjusting the exposure time and/or the number of exposures) of a structured light-based 3D camera.
-
FIG. 1 is a flowchart of a method for determining the optimal exposure time. Referring toFIG. 1 , the method for determining the optimal exposure of a structured light-based 3D camera includes capturing a reference image at step S100, calculating a slope and categorizing the state of the slope at step S200, creating a distribution chart at step S300, and determining the optimal exposure time and the number of exposures at step S400. - In the capturing the reference image at step S100, images of an object onto which
pattern 0 andpattern 1 are projected, respectively, are captured. Here,pattern 0 means an all-black pattern (when a projection device is turned off), andpattern 1 means an all-white pattern. Here, in order to improve reliability when a slope related to brightness and an exposure time is calculated at step S200, two reference images are captured for each ofpattern 0 andpattern 1. Here, if only a single reference image is used, reliability may be reduced because the brightness value of a pixel becomes 255 even when the actual brightness value of a pixel is greater than 255 (because the maximum brightness value of a pixel in an 8-bit image is 255, the brightness beyond that is also represented as 255). Also, if the brightness value of a pixel is less than 10, because the percentage of noise in a Signal-to-Noise Ratio (SNR) may be equal to or greater than 10%, reliability may be reduced when a slope is calculated. Accordingly, in order to calculate a reliable slope, the exposure time for capturing a reference image is set as described below. First, in order to prevent the brightness value of an object having a high reflection coefficient from being greater than 250, the shutter speed is set fast (that is, a short exposure time is set). Conversely, in order to prevent the brightness value of an object having a low reflection coefficient from being less than 10, the shutter speed is set slow (that is, a long exposure time is set). - In the calculating a slope and categorizing the state of the slope at step S200, a reference image for calculating a reliable slope is selected from among the captured reference images, a slope is calculated for the selected reference image, and the state thereof is categorized. In other words, a slope for
pattern 0 and a slope forpattern 1 may be calculated for all the pixels in the reference image according to the processes of the flowchart inFIG. 2A andFIG. 2B . - In
FIG. 2A andFIG. 2B , E1 means a short exposure time, which is set in a camera when a reference image is captured, and E2 means a long exposure time, which is set in the camera when the reference image is captured. Also, I1 means the brightness value of a pixel in the reference image, which is captured when E1 is set, and I2 means the brightness value of a pixel in the reference image, which is captured when E2 is set. Because there is a linear relationship between I and E, the value of E corresponding to the desired exposure time may be estimated using the slope of I/E. Here, in order to acquire a reliable slope, the state of brightness of a pixel is defined as follows. When I<10, the state is defined as ‘under’, and when I>250, the state is defined as ‘over’. When the state is neither ‘under’ nor ‘over’, the state is defined as ‘normal’. Depending on the values of I1 and I2, there may be seven cases, as shown inFIG. 2A andFIG. 2B . - S201 is the case in which both the value of I1 and the value of I2 are in the ‘normal’ state, in which case the mean of the two slopes for the brightness and the exposure time in the ‘normal’ state is set as the value of a slope.
- S202 and S203 are the cases in which either the value of I1 or the value of I2 is in the ‘normal’ state, in which case the slope for the brightness and the exposure time in the ‘normal’ state is set as the value of a slope.
- S204 and S205 are the cases in which the value of I1 and the value of I2 are respectively in the ‘under’ and ‘over’ states or in the ‘over’ and ‘under’ states. Here, the value in the ‘over’ state indicates the case in which the actual brightness value cannot be estimated, and the value in the ‘under’ state indicates the case in which the percentage of noise is high but the value is more reliable than the value in the ‘over’ state. Accordingly, the slope for the brightness and the exposure time in the ‘under’ state is set as the value of a slope.
- S206 is the case in which both the value of I1 and the value of I2 are in the ‘under’ state. Here, because there is less noise as the brightness value of a pixel is greater, the greater brightness value is selected, and the slope for the brightness and the exposure time corresponding to the selected brightness value is set as the value of a slope.
- S207 is the case in which both the value of I1 and the value of I2 are in the ‘over’ state, in which case the slope for the brightness and the exposure time corresponding to the lower brightness value is set as the value of a slope.
- Next, in the creating a distribution chart at step S300, using the difference between the slope that is calculated when
pattern 1 is projected and the slope that is calculated whenpattern 0 is projected, pixels that can be reconstructed and pixels that cannot be reconstructed at each exposure time in the structured light-based camera system are predicted (that is, the number of upper loss pixels and the number of accumulation pixels are calculated), whereby an upper loss pixel distribution chart and an accumulation pixel distribution chart are acquired. - The structured light-based 3D camera may calculate 3D data only when a patterned region, which includes patterns projected from a projection device, is distinguishable from a non-patterned region. Therefore, the difference between the brightness in the patterned region and the brightness in the non-patterned region must be greater than a threshold value Ith.
- In order to predict pixels that can be reconstructed and pixels that cannot be reconstructed in the structured light-based 3D camera system, the Slope1/Slope0 distribution chart shown in
FIG. 3 is used. - In
FIG. 3 , Slope0 means the slope that is calculated whenpattern 0 is projected, and Slope1 is the slope that is calculated whenpattern 1 is projected. - p0 is determined using the following Equation (1):
-
p 0=(255−I th)/E (1) - p1 is determined using the following Equation (2):
-
p 1=255/E (2) - p2 is determined using the following Equation (3):
-
p 2=(I th /E)+Slope0 (3) - In the Slope1/Slope0 distribution chart illustrated in
FIG. 3 , a pixel at which Slope0 is less than p0 and Slope1 is less than p1 and greater than p2 is estimated as a pixel from which 3D data may be reconstructed. InFIG. 3 , a pixel at which Slope0 is greater than p0 and Slope1 is greater than p1 is referred to as an upper loss pixel, and a pixel at which Slope1 is greater than p2 is referred to as an accumulation pixel. - When the maximum exposure time allowable in the structured light-based 3D camera system is referred to as EM, the number of pixels that satisfy the condition in which Slope1 is greater than p2, which is calculated after EM is substituted for E in Equation (3), that is, the number of pixels that satisfy the condition of Slope1>p2=(Ith/EM)+Slope0, is the maximum number of accumulation pixels. Here, the maximum number of accumulation pixels is referred to as NM. Also, for all exposure times (from E1 to EM) allowable in the structured light-based 3D camera system, the number of accumulation pixels at each exposure time may be calculated using Equation (3). Here, the number of accumulation pixels at each exposure time En is referred to as Nn. Accordingly, the percentage of accumulation pixels to the maximum number of accumulation pixels is calculated as
-
- whereby the distribution chart shown in
FIG. 4A may be acquired. - Meanwhile, the number of pixels that satisfy the condition of Slope0≥(255−Ith)/E and Slope1>255/E is the number of upper loss pixels. Here, the number of upper loss pixels calculated at each of all exposure times (from E1 to EM) is referred to as Ln. Accordingly, the percentage of upper loss pixels to the maximum number of accumulation pixels is calculated as
-
- whereby the distribution chart shown in
FIG. 4B may be acquired. - Here, when the calculated Slope1 is in the ‘over’ state and the calculated Slope0 is in the ‘normal’ state, the value of (Slope1−Slope0)E may be greater than the actual value thereof. In this case, because the condition of (Slope1−Slope0)E>Ith is satisfied, the corresponding pixel may be assumed to be a pixel that can be reconstructed. However, when Slope1 is in the ‘under’ state and Slope0 is in the ‘over’ state, the calculated value of (Slope1−Slope0)E may be less than the actual value thereof, and the value may be less than Ith. In this case, an erroneous determination that a pixel can be reconstructed may be made. Therefore, the case in which the value of (Slope1−Slope0)E may be less than Ith because the value (Slope1−Slope0)E is less than the actual value thereof may be excluded when a pixel that can be reconstructed is predicted. Such a case is as follows.
- (1) the case in which Slope1 is in the ‘over’ state and Slope0 is in the ‘over’ state
- (2) the case in which Slope1 is in the ‘normal’ state and Slope0 is in the ‘over’ state
- (3) the case in which Slope1 is in the ‘under’ state and Slope0 is in the ‘over’ state
- After the accumulation pixel distribution chart and the upper loss pixel distribution chart, as shown in
FIG. 4A andFIG. 4B , are acquired, the optimal exposure time and the number of exposures are determined at step S400, as shown in the flowchart ofFIG. 5 . - Referring to
FIG. 5 , S501 is the step in which the number of exposures is set to a single exposure and the optimal duration thereof is the exposure time at which the percentage of pixels lost in the upper and lower regions is less than 1%. Here, the percentage of lower loss pixels is -
- Specifically, the exposure time at which the percentage of accumulation pixels is 99.5% is searched for in the accumulation pixel distribution chart at step S51, whether the percentage of upper loss pixels is equal to or less than 0.5% is determined at step S52, and whether the number of exposures is set to a single exposure is determined at step S53.
- Also, S502 is the step in which the number of exposures is set to two exposures. Here, step S502 includes steps S54, S55, and S56. At step S54, when it has been determined at step S52 that the percentage of upper loss pixels is greater than 0.5%, whether the exposure time found at step S51 is the maximum exposure time of the camera is determined. At step S55, the exposure time at which the percentage of accumulation pixels first becomes 99.5% is searched for in the accumulation pixel distribution chart, and the found exposure time is set as a long exposure time. Also, at step S56, the exposure time at which the percentage of upper loss pixels is equal to or less than 0.5% is searched for, and the found exposure time and the long exposure time are set as the duration of the two exposures. Specifically, if the exposure time found at step S51 is not the maximum exposure time of the camera, the exposure time is increased by 1 ms and the process returns to step S52. If the exposure time found at step S51 is the maximum exposure time of the camera, the exposure time found at step S51 and the newly found exposure time at step S56 are set as the exposure times of the two exposures.
- In the flowchart of
FIG. 5 , if 99% of pixels can be reconstructed through a single exposure, the number of exposures is set to a single exposure, but if not, the number of exposures is set to two exposures in order to enable an object having a low reflection coefficient to be reconstructed at a long exposure time and an object having a high reflection coefficient to be reconstructed at a short exposure time, whereby 99% of pixels can be reconstructed. - 3D Camera System Based on Structured Light
- As an embodiment of the present invention, a structured light-based
3D camera system 10 that uses the above-described method for determining the optimal number of exposures and the duration thereof may be provided. - Referring to
FIG. 6 , the structured light-based3D camera system 10 includes aprojection unit 11, a capturingunit 12, and aprocessing unit 13. - The
projection unit 11 may project various patterns onto a target object. - The capturing
unit 12 may capture an image that includes the pattern projected from theprojection unit 11. - The
processing unit 13 includes animage acquisition unit 131, a slope calculation andstate categorization unit 132, a distributionchart creation unit 133 and an optimal exposure time andfrequency determination unit 134. - The
image acquisition unit 131 may acquire an image onto which patterns, such as pattern 0 (an all-black pattern), pattern 1 (an all-white pattern) or the like, are projected. - The slope calculation and
state categorization unit 132 calculates a slope related to brightness and an exposure time for all the pixels in the reference image, onto whichpattern 0 andpattern 1 are projected, and categorizes the state of the slope, the reference image being acquired from the image acquisition unit. - The distribution
chart creation unit 133 may acquire a distribution chart by predicting pixels that can be reconstructed and pixels that cannot be reconstructed (that is, by calculating the number of upper loss pixels and the number of accumulation pixels) at each exposure time using the difference between the slope calculated whenpattern 1 is projected and the slope calculated whenpattern 0 is projected. - The optimal exposure time and
frequency determination unit 134 may determine the optimal exposure time and the optimal number of exposures using the above-described method that uses the upper loss pixel distribution chart and the accumulation pixel distribution chart. -
FIG. 7 shows an experimental example for comparing the case in which a single exposure is set according to an existing method for determining the optimal exposure with the case in which two exposures are set according to the method proposed by the present invention. This example is configured to include an object having a high reflection coefficient (the white object at the left side) and an object having a low reflection coefficient (the black object in the box). -
FIG. 8 is a table that shows the exposure time and the number of reconstructed pixels when the scene ofFIG. 7 is captured using a structured light-based 3D camera system. Here, the number of exposures is set to a single exposure, the duration thereof is set to a range from 1 ms to 100 ms, having an interval of 3 ms, and the number of pixels that are actually reconstructed into 3D data is measured. When the exposure duration is set to 28 ms, the maximum number of pixels are reconstructed, in which case the number of reconstructed pixels is 237,536. -
FIGS. 9A, 9B and 9C show the result of reconstruction of 3D data when the scene ofFIG. 7 is captured using a structured light-based 3D camera system. When the method proposed in the present invention is applied to the scene ofFIG. 7 , the necessary number of exposures is set to two exposures, and the durations thereof are set to 3 ms and 84 ms, respectively.FIG. 9A is the result of capturing an image when the exposure time is set to 3 ms, in which case only a region that has a high reflection coefficient is reconstructed.FIG. 9B is the result of capturing an image when the exposure time is set to 84 ms, in which case only a region that has a low reflection coefficient is reconstructed.FIG. 9C is the result of a combination of the reconstructed pixels ofFIG. 9A and the reconstructed pixels ofFIG. 9B , with a total of 251,900 reconstructed pixels. Consequently, it is confirmed that the percentage of reconstructed pixels is increased by 6% when the number of exposures is set to two exposures compared to when the number of exposures is set to one exposure. - In a general 3D camera system based on structured light, it is difficult to accurately detect patterns when the patterns are projected onto an object having a reflection coefficient that is too high or too low. This problem may be solved by adjusting the number of exposures and/or the exposure time of a camera, whereby reliable 3D data may be reconstructed. Therefore, the structured light-based 3D camera system of the present invention may more accurately reconstruct 3D data of an object having a low or high reflection coefficient in industrial fields.
- Also, the structured light-based 3D camera system of the present invention sets two exposures having the optimal duration when it is difficult to reconstruct 3D data only through a single exposure in an environment in which an object having a low reflection coefficient and an object having a high reflection coefficient are both present, whereby reliability may be improved and the extent to which 3D data are reconstructed may be increased.
- In the case of existing structured light-based 3D camera systems, it is difficult for laypersons to adjust the aperture of the camera because the adjustment thereof is not simple. However, in the structured light-based 3D camera system of the present invention, because the exposure time of the camera is automatically determined, laypersons may easily use the camera.
- The above embodiments have been described for illustrative purposes and should not be construed as limiting the scope of the present invention, which is defined by the accompanying claims. Accordingly, other embodiments may fall into the range of technical spirit defined by the accompanying claims. For example, those skilled in the art may make various modifications without departing from the scope of the present invention. Additionally, because some of the above-described steps are independent from the order in which the steps are performed, they may be performed in a different order from the order in the above description. Also, some of the steps may be omitted or other steps may be further included.
Claims (16)
1. A method for determining an exposure parameter for a structured light-based 3D camera system, the method comprising:
capturing a first reference image and a second reference image, the first reference image including a reflection of a first pattern from a target object and the second reference image including a refection of a second pattern from the target object;
calculating a pixel distribution from the first reference image and the second reference image; and
determining the exposure parameter using the pixel distribution.
2. The method of claim 1 , wherein the calculating the pixel distribution includes calculating a plurality of slopes related to a brightness and an exposure time for selected pixels of the first reference image and the second reference image, and categorizing the plurality of slopes.
3. The method of claim 1 , wherein the calculating the pixel distribution further includes calculating an upper loss pixel distribution and an accumulation pixel distribution from a number of upper loss pixels and a number of accumulation pixels.
4. The method of claim 1 , wherein the exposure parameter includes at least one of an exposure time and a number of exposures.
5. The method of claim 1 , wherein each of the first pattern and the second pattern includes at least one of an all-black pattern and an all-white pattern.
6. The method of claim 3 , wherein the calculating the upper loss pixel distribution and the accumulation pixel distribution includes excluding states in which a brightness value of a pixel associated with the first reference image and a brightness value of a pixel associated with the second reference image are less than a predetermined threshold brightness value.
7. The method of claim 1 , wherein the first pattern and the second pattern are the same.
8. The method of claim 1 , wherein the first pattern and the second pattern are different.
9. A structured light-based 3D camera system, comprising:
a projection unit for projecting a first pattern and a second pattern onto a target object;
a capturing unit for capturing a first reference image and a second reference image, the first reference image including a reflection of the first pattern from the target object and the second reference image including a reflection of the second pattern from the target object; and
a processing unit for calculating a pixel distribution from the first reference image and the second reference image, and determining an exposure parameter for the structured light-based 3D camera system using the pixel distribution.
10. The structured light-based 3D camera system of claim 9 , wherein each of the first pattern and the second pattern includes at least one of an all-black pattern and an all-white pattern.
11. The structured light-based 3D camera system of claim 9 , wherein the exposure parameter includes at least one of an exposure time and a number of exposures.
12. The structured light-based 3D camera system of claim 9 , wherein the processing unit calculates the pixel distribution by calculating a plurality of slopes related to a brightness and an exposure time for selected pixels of the first reference image and the second reference image, and categorizing the plurality of slopes.
13. The structured light-based 3D camera system of claim 9 , wherein the processing unit calculates the pixel distribution by calculating an upper loss pixel distribution and an accumulation pixel distribution from a number of upper loss pixels and a number of accumulation pixels.
14. The structured light-based 3D camera system of claim 13 , wherein the calculating the upper loss pixel distribution and the accumulation pixel distribution comprise excluding states in which a brightness value of a pixel associated with the first reference image and a brightness value of a pixel associated with the second reference image are less than a predetermined threshold brightness value.
15. The structured light-based 3D camera system of claim 9 , wherein the first pattern and the second pattern are the same.
16. The structured light-based 3D camera system of claim 9 , wherein the first pattern and the second pattern are different.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/365,985 US20210344892A1 (en) | 2015-10-27 | 2021-07-01 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3d camera |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020150149156A KR101737518B1 (en) | 2015-10-27 | 2015-10-27 | Method and system for determining optimal exposure time and frequency of structured light based 3d camera |
KR10-2015-0149156 | 2015-10-27 | ||
US15/334,314 US10404968B2 (en) | 2015-10-27 | 2016-10-26 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3D camera |
US16/558,151 US11070788B2 (en) | 2015-10-27 | 2019-09-01 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3D camera |
US17/365,985 US20210344892A1 (en) | 2015-10-27 | 2021-07-01 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3d camera |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/558,151 Continuation US11070788B2 (en) | 2015-10-27 | 2019-09-01 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3D camera |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210344892A1 true US20210344892A1 (en) | 2021-11-04 |
Family
ID=58559388
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/334,314 Expired - Fee Related US10404968B2 (en) | 2015-10-27 | 2016-10-26 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3D camera |
US16/558,151 Active US11070788B2 (en) | 2015-10-27 | 2019-09-01 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3D camera |
US17/365,985 Abandoned US20210344892A1 (en) | 2015-10-27 | 2021-07-01 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3d camera |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/334,314 Expired - Fee Related US10404968B2 (en) | 2015-10-27 | 2016-10-26 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3D camera |
US16/558,151 Active US11070788B2 (en) | 2015-10-27 | 2019-09-01 | Method and system for determining optimal exposure time and number of exposures in structured light-based 3D camera |
Country Status (2)
Country | Link |
---|---|
US (3) | US10404968B2 (en) |
KR (1) | KR101737518B1 (en) |
Families Citing this family (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101887945B1 (en) * | 2017-08-10 | 2018-08-13 | 성균관대학교산학협력단 | Method and apparatus for determining of exposure time of 3D camera |
CN108234896B (en) * | 2018-01-16 | 2020-12-01 | 昆山晔芯电子科技有限公司 | High dynamic recovery method and system for segmented exposure imaging |
CN110049257B (en) * | 2019-05-31 | 2021-09-21 | 影石创新科技股份有限公司 | Method for determining semi-synchronous exposure parameters and electronic device |
GB2584907A (en) * | 2019-06-21 | 2020-12-23 | Zivid As | Method for determining one or more groups of exposure settings to use in a 3D image acquisition process |
CN112729246B (en) * | 2020-12-08 | 2022-12-16 | 广东省科学院智能制造研究所 | Black surface object depth image measuring method based on binocular structured light |
US11889195B2 (en) * | 2020-12-29 | 2024-01-30 | SK Hynix Inc. | Image sensing system and operating method thereof |
WO2022147670A1 (en) * | 2021-01-06 | 2022-07-14 | Jiangsu Jitri Micro-Nano Automation Institute Co., Ltd | Automatic exposure selection method for high dynamic range 3d optical measurements |
CN114363525B (en) * | 2022-03-17 | 2022-06-28 | 杭州灵西机器人智能科技有限公司 | HDR fast automatic imaging method, structured light camera, electronic device, and storage medium |
CN115187743B (en) * | 2022-07-29 | 2024-07-05 | 江西科骏实业有限公司 | Subway station internal environment arrangement prediction and white mode acquisition method and system |
CN115761458B (en) * | 2022-11-24 | 2023-09-01 | 北京的卢铭视科技有限公司 | Indoor and outdoor environment judging method, electronic equipment and storage medium |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080074517A1 (en) * | 2006-09-21 | 2008-03-27 | Samsung Electronics Co., Ltd. | System, method and medium correcting brightness of an image |
US20100302399A1 (en) * | 2007-09-14 | 2010-12-02 | The Trustee Of The University Of Pennsylvania | High linear dynamic range imaging |
US20130332866A1 (en) * | 2012-06-10 | 2013-12-12 | Apple Inc. | Tempered auto-adjusting, image-editing operation |
US20150116582A1 (en) * | 2013-10-30 | 2015-04-30 | Canon Kabushiki Kaisha | Image processing apparatus and image processing method |
US20150244916A1 (en) * | 2014-02-21 | 2015-08-27 | Samsung Electronics Co., Ltd. | Electronic device and control method of the same |
-
2015
- 2015-10-27 KR KR1020150149156A patent/KR101737518B1/en active IP Right Grant
-
2016
- 2016-10-26 US US15/334,314 patent/US10404968B2/en not_active Expired - Fee Related
-
2019
- 2019-09-01 US US16/558,151 patent/US11070788B2/en active Active
-
2021
- 2021-07-01 US US17/365,985 patent/US20210344892A1/en not_active Abandoned
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080074517A1 (en) * | 2006-09-21 | 2008-03-27 | Samsung Electronics Co., Ltd. | System, method and medium correcting brightness of an image |
US20100302399A1 (en) * | 2007-09-14 | 2010-12-02 | The Trustee Of The University Of Pennsylvania | High linear dynamic range imaging |
US20130332866A1 (en) * | 2012-06-10 | 2013-12-12 | Apple Inc. | Tempered auto-adjusting, image-editing operation |
US20150116582A1 (en) * | 2013-10-30 | 2015-04-30 | Canon Kabushiki Kaisha | Image processing apparatus and image processing method |
US20150244916A1 (en) * | 2014-02-21 | 2015-08-27 | Samsung Electronics Co., Ltd. | Electronic device and control method of the same |
Also Published As
Publication number | Publication date |
---|---|
KR101737518B1 (en) | 2017-05-18 |
KR20170048720A (en) | 2017-05-10 |
US20170118456A1 (en) | 2017-04-27 |
US10404968B2 (en) | 2019-09-03 |
US20190387215A1 (en) | 2019-12-19 |
KR101737518B9 (en) | 2022-03-15 |
US11070788B2 (en) | 2021-07-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11070788B2 (en) | Method and system for determining optimal exposure time and number of exposures in structured light-based 3D camera | |
USRE48595E1 (en) | Method and system for determining optimal exposure of structured light based 3D camera | |
US10257428B2 (en) | Image processing apparatus and image processing method that adjust, based on a target object distance, at least one of brightness of emitted pattern light and an exposure amount | |
JP6795993B2 (en) | Shape measurement system, shape measurement device and shape measurement method | |
US9998728B2 (en) | Information processing apparatus, control method for the same and storage medium | |
US9501834B2 (en) | Image capture for later refocusing or focus-manipulation | |
US10755417B2 (en) | Detection system | |
US10698308B2 (en) | Ranging method, automatic focusing method and device | |
US9007602B2 (en) | Three-dimensional measurement apparatus, three-dimensional measurement method, and computer-readable medium storing control program | |
US10430962B2 (en) | Three-dimensional shape measuring apparatus, three-dimensional shape measuring method, and storage medium that calculate a three-dimensional shape of an object by capturing images of the object from a plurality of directions | |
JP6161276B2 (en) | Measuring apparatus, measuring method, and program | |
US20140037212A1 (en) | Image processing method and device | |
CN103096117B (en) | Video noise detection method and device | |
CN111970500A (en) | Automatic distance step calibration method and system for projection equipment | |
JP2021092887A (en) | Visual inspection device, visual inspection system, feature quantity estimation device, and visual inspection program | |
US20210356259A1 (en) | Three-dimensional measurement apparatus, three-dimensional measurement method, and three-dimensional measurement non-transitory computer readable medium | |
KR101887945B1 (en) | Method and apparatus for determining of exposure time of 3D camera | |
US11631183B2 (en) | Method and system for motion segmentation | |
EP3070432B1 (en) | Measurement apparatus | |
US10958899B2 (en) | Evaluation of dynamic ranges of imaging devices | |
JP2009210470A (en) | Shape measurement method and device | |
KR101087863B1 (en) | Method of deciding image boundaries using structured light, record medium for performing the same and apparatus for image boundary recognition system using structured light | |
KR20130087991A (en) | Method and system for an adaptive auto-focus algorithm | |
JP2010133735A (en) | Shape measurement method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: RESEARCH & BUSINESS FOUNDATION SUNGKYUNKWAN UNIVERSITY, KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LEE, SUK-HAN;CHOI, YUN-SU;SHIN, IN-SUB;REEL/FRAME:056752/0113 Effective date: 20161118 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |