TWI842465B - Establishment system for 3d point cloud information and method thereof - Google Patents

Abstract

An establishment system for 3D point cloud information and method thereof is applied to establish the 3D point cloud information for an object. The establishment system includes a stage, a 2D imaging unit, a first processor, a depth sensor, and a second processor. The stage is configured to bear an object. Wherein, the stage is provided with a light emitting unit, and the light emitting is configured to illuminate the object. The 2D imaging unit is configured to capture a plurality of images of the object. The first processor is coupled to the 2D imaging unit, and configured to analyze each of the plurality of images to obtain the edge information of each of the plurality of images. Wherein, each of the plurality of images includes the X information and the Y information of a plurality of points. The depth sensor is coupled to the first processor, and configured to capture a plurality of Z information of each of the plurality of points according to the edge information of each of the plurality of images. The second processor is coupled to the first processor, and configured to map the X information, the Y information, and the plurality of Z information of each of the plurality of points to generate 3D point cloud information of the object.

Description

Three-dimensional point cloud information construction system and construction method

This case is about three-dimensional point cloud information, and more particularly about a system and method for constructing three-dimensional point cloud information.

Compared to traditional flat screens, curved screens have gradually become mainstream products in high-end markets (such as automotive electronics, e-sports, and televisions). However, various process technologies used in traditional flat liquid crystal display (LCD) panels cannot be directly applied to curved LCD panels, such as protective film (PET)/explosion-proof film (ASF)/adhesive film (PSA) laser cutting technology and automated optical inspection (AOI) technology. If you want to apply the laser cutting technology and the automated optical inspection technology to curved LCD panels, you must first establish the three-dimensional point cloud information of the curved LCD panel in order to execute such high-precision process technology.

In view of this, the inventor proposes a system and method for constructing three-dimensional point cloud information to establish three-dimensional point cloud information of the object to be measured, thereby providing high-precision process technology application.

In some embodiments, a system for constructing three-dimensional point cloud information includes: a carrier, a two-dimensional imaging unit, a first processor, a depth sensor, and a second processor. The carrier is used to carry an object to be measured, wherein the carrier is provided with a light-emitting unit, and the light-emitting unit is used to emit light to illuminate the object to be measured. The two-dimensional imaging unit is used to capture multiple two-dimensional images of the object to be measured, wherein the optical axis of the two-dimensional imaging unit points to the carrier. The first processor is coupled to the two-dimensional imaging unit, and is used to analyze each two-dimensional image to obtain edge information of each two-dimensional image. The edge information of each two-dimensional image includes X information and Y information of multiple points. The depth sensor is coupled to the first processor, and is used to capture multiple Z information of each point based on the edge information of each two-dimensional image, wherein the optical axis of the depth sensor points to the carrier. The second processor is coupled to the first processor and is used to map the X information, Y information and multiple Z information of each point to generate three-dimensional point cloud information of the object to be measured.

In some embodiments, the two-dimensional imaging unit includes a charge coupled device (CCD) sensor and a telecentric lens, the optical axis of the CCD sensor and the optical axis of the telecentric lens are coaxial, and the CCD sensor is coupled to the first processor.

In some embodiments, the working distance of the two-dimensional imaging unit is not less than 65 mm.

In some embodiments, the depth sensor is a chromatic confocal sensor.

In some embodiments, the optical axis of the two-dimensional imaging unit and the optical axis of the depth sensor are parallel.

In some embodiments, the construction system further includes a beam splitter, which is located between the stage and the two-dimensional imaging unit and between the stage and the depth sensor, and the optical axis of the two-dimensional imaging unit and the optical axis of the depth sensor are coaxial through the beam splitter.

In some embodiments, the construction system further includes a moving device coupled to the first processor, wherein the two-dimensional imaging unit and the depth sensor are disposed on the moving device, and under the control of the first processor, the moving device is used to move the two-dimensional imaging unit according to an initial imaging path and to move the depth sensor according to a depth imaging path.

In some embodiments, the system further includes a moving device coupled to the first processor, wherein the stage is disposed on the moving device, and under the control of the first processor, the moving device is used to move the stage according to an initial imaging path and a depth imaging path.

In some embodiments, a method for constructing three-dimensional point cloud information includes: capturing a plurality of two-dimensional images of an object to be measured; analyzing each two-dimensional image to obtain edge information of each two-dimensional image, wherein the edge information of each two-dimensional image includes X information and Y information of a plurality of points; capturing a plurality of Z information of each point based on the edge information of each two-dimensional image; and mapping the X information, Y information and the plurality of Z information of each point to generate a three-dimensional point cloud information of the object to be measured.

In some embodiments, the construction method further includes: establishing an initial imaging path for a plurality of two-dimensional images according to a specification diagram of the object to be tested, wherein the step of capturing a plurality of two-dimensional images of the object to be tested is performed based on the initial imaging path.

In some embodiments, the step of analyzing each two-dimensional image to obtain edge information of each two-dimensional image includes: binarizing each two-dimensional image to obtain an edge image of each two-dimensional image; highlighting the edge image of each two-dimensional image; connecting multiple points on the edge image of each two-dimensional image to obtain edge information; and performing anti-aliasing on the edge information of each two-dimensional image.

In some embodiments, the step of capturing multiple Z information of each point based on the edge information of each two-dimensional image includes: calculating a depth acquisition path based on the edge information of each two-dimensional image; moving a depth sensor based on the depth acquisition path so that the optical axis of the depth sensor is aligned with the multiple points in sequence; and when the depth sensor is aligned with each point, sequentially detecting multiple film surfaces at different depths of the object to be measured through the depth sensor to obtain Z information of each film surface.

In summary, according to any embodiment, the system or method for constructing three-dimensional point cloud information is suitable for obtaining high-precision three-dimensional coordinates (such as contour size and film thickness) of the object to be measured, and constructing three-dimensional point cloud information of the object to be measured accordingly. In this way, the three-dimensional point cloud information of the object to be measured can be provided for application in high-precision processes such as laser cutting and automated optics of the object to be measured. In some embodiments, the system or method for constructing three-dimensional point cloud information is also particularly suitable for constructing high-precision three-dimensional point cloud information of an object to be measured having a multi-layer structure with a transparent film.

Please refer to FIG. 1 , a three-dimensional point cloud information construction system (hereinafter referred to as construction system 10 ) is suitable for constructing the three-dimensional point cloud information of the object to be tested 20 . In some embodiments, the object to be tested 20 may be an object with a transparent film layer. The object with a transparent film layer may be, for example, but not limited to, a protective film, a lens, a glass panel, a touch panel, or a liquid crystal display (LCD) panel. In some embodiments, the object with a transparent film layer is a non-planar object of a multi-layer structure in which at least one layer is a transparent film layer. The non-planar object may be, for example, but not limited to, a large-sized object such as a curved liquid crystal display panel or a special-shaped liquid crystal display panel. For example, the non-planar object is a large-sized curved panel. The long side of the curved panel is not less than 750 millimeters (mm), the short side of the curved panel is not less than 150 mm, the depth of the curved panel is not less than 120 mm, and the radius of curvature of the curved panel is 2000 mm. In other words, the construction system 10 is particularly suitable for constructing three-dimensional point cloud information of non-planar objects with transparent film layers.

The construction system 10 includes a stage 100, a two-dimensional imaging unit 110, a first processor 120, a depth sensor 130, and a second processor 140. The first processor 120 is coupled to the two-dimensional imaging unit 110 and the depth sensor 130, and the second processor 140 is coupled to the first processor 120.

The stage 100 is used to carry an object to be tested 20. The stage 100 is provided with a light-emitting unit 101. For example, during the construction process, the object to be tested 20 can be fixed on the surface of the stage 100 (hereinafter referred to as the carrying surface) to obtain the three-dimensional coordinates of the object to be tested 20. In addition, the light-emitting unit 101 emits light to illuminate the object to be tested 20, so that the two-dimensional imaging unit 110 can capture a two-dimensional image of the object to be tested 20. In some embodiments, the stage 100 can fix the carried object to be tested 20 through a fixture such as a clamp or a vacuum adsorption device (not shown), so that the object to be tested 20 can be stably set on the stage 100 without deviation or shaking. In some embodiments, the light-emitting unit 101 may be an active light-emitting light source device, such as but not limited to an incandescent light panel, an LED light panel, a fluorescent light tube, or a light-emitting panel.

1 and 2 , the optical axis Ax1 of the two-dimensional imaging unit 110 and the optical axis Ax2 of the depth sensor 130 point to the stage 100 . In other words, the lens of the two-dimensional imaging unit 110 and the lens of the depth sensor 130 face the supporting surface of the stage 100 .

1 to 3 , when the system 10 starts to operate, the two-dimensional imaging unit 110 captures a plurality of two-dimensional images of the object 20 (step S110 ). Specifically, the two-dimensional imaging unit 110 continuously captures different regions of the object 20 to generate a plurality of two-dimensional images.

In some embodiments, the sensing light of the two-dimensional imaging unit 110 is collimated beam. In some embodiments, the two-dimensional imaging unit 110 uses a larger working distance WD to capture a two-dimensional image of the object to be tested 20. Here, the larger working distance WD refers to a distance sufficient to make the sensing light of the two-dimensional imaging unit 110 approach the collimated light. Among them, the working distance WD is the distance between the front end of the lens of the two-dimensional imaging unit 110 (i.e., one side adjacent to the stage 100) and the upper surface of the object to be tested 20 (i.e., the other side surface relative to the stage 100). In this way, the two-dimensional imaging unit 110 can obtain a two-dimensional image with sharper edges and clearer. In some embodiments, the working distance WD of the two-dimensional imaging unit 110 is not less than 65 millimeters (mm). The working distance WD of the two-dimensional imaging unit 110 is, for example, 65 mm to 305 mm, but is not limited thereto.

In some embodiments, referring to FIG. 1 , FIG. 2 and FIG. 4 , the two-dimensional imaging unit 110 includes a charge-coupled device (CCD) sensor 111 and a telecentric lens 112 . The telecentric lens 112 is located between the CCD sensor 111 and the stage 100 , and the CCD sensor 111 is coupled to the first processor 120 .

In some embodiments, when the two-dimensional imaging unit 110 captures a two-dimensional image of the object to be tested 20, the telecentric lens 112 changes a light reflected by the object to be tested 20 into a parallel light path, and the charge-coupled device sensor 111 generates a corresponding two-dimensional image by sensing the parallel light path. In some embodiments, the optical axis Ax11 of the charge-coupled device sensor 111 and the optical axis Ax12 of the telecentric lens 112 are coaxial (i.e., the optical axis Ax1). Therefore, when performing high-precision measurement, the charge-coupled device sensor 111 captures the two-dimensional image of the object to be tested 20 through the telecentric lens 112, thereby obtaining a high-resolution two-dimensional image and reducing errors, while avoiding distortion of the captured two-dimensional image.

Please refer to FIG5 , taking a curved panel as an example, the long side Lth of the object to be tested 20 corresponds to the X-axis coordinate in the three-dimensional coordinate system, the short side Wth of the object to be tested 20 corresponds to the Y-axis coordinate in the three-dimensional coordinate system, and the depth Dth of the object to be tested 20 corresponds to the Z-axis coordinate in the three-dimensional coordinate system. Please refer to FIG1 , FIG2 , FIG5 , FIG6 and FIG7 , the two-dimensional imaging unit 110 is facing the front view direction of the object to be tested 20 (i.e., facing the XY plane, in other words, the optical axis Ax1 of the two-dimensional imaging unit 110 is substantially vertical), so that the visible range (Field of View, FOV, also known as the field of view) Rf11~Rfst of the two-dimensional imaging unit 110 falls on the object to be tested 20, and the picture within the visible range Rf11~Rfst is captured to obtain a two-dimensional image of the object to be tested 20. In other words, the object image Obt presented in the two-dimensional image Img1 captured by the two-dimensional imaging unit 110 is the image of the area where the object 20 to be tested falls within the visible range Rf1t of the two-dimensional imaging unit 110 (as shown in FIG. 7 ).

As shown in FIG. 6 , in some embodiments, the size of the visible range Rf11 -Rfst of the two-dimensional imaging unit 110 is much smaller than the size of the object to be measured 20 (long side Lth×short side Wth).

In some embodiments, the two-dimensional imaging unit 110 continuously captures multiple two-dimensional images of the object 20 along the edge of the object 20 with a fixed visible range Rf11/Rf12/…/Rf1t/Rf2t/…/Rf(s-1)t/Rfst. Wherein, s and t can be the same or different positive integers greater than 1. For example, continuing the previous example, when the two-dimensional imaging unit 110 captures a two-dimensional image along the long side Lth of the object 20, the short sides of any consecutive two-time captured visible ranges Rf1t, Rf2t/Rf2t, Rf3t/…/Rf(s-1)t, Rfst are spliced together. When the two-dimensional imaging unit 110 captures a two-dimensional image along the short side Wth of the object to be tested 20, the long sides of any two consecutive captured visible ranges Rf11, Rf12/Rf12, Rf13/…/Rf1(t-1), and Rf1t are spliced together. In other words, the two-dimensional images captured by the two-dimensional imaging unit 110 can be spliced to form a spliced image showing the edge of the complete object to be tested 20.

In some embodiments, any two-dimensional images captured consecutively twice are spliced in a partially overlapping manner. In other words, any two-dimensional images captured consecutively twice have a part of the same image. In two opposite sides of each two-dimensional image, the edge area of one side is the same as the edge area of one side of the next captured two-dimensional image, and the edge area of the other side is the same as the edge area of one side of the last captured two-dimensional image.

After obtaining the two-dimensional image Img1, the first processor 120 analyzes each two-dimensional image to obtain edge information of each two-dimensional image Img1 (step S120). Please refer to FIG8 , the edge information of each two-dimensional image Img1 includes X information x1~xn and Y information y1~yn of a plurality of points P11~P1n. Among them, the X information x1/X2/.../xn is the X coordinate value of the position of the object 20 represented by this point P11/P12/.../P1n on the coordinate system of the object 20, and the Y information y1/y2/.../yn is the Y coordinate value of the position of the object 20 represented by this point P11/P12/.../P1n on the coordinate system of the object 20. And, n is a positive integer.

In some embodiments of step S120, please refer to FIG. 1, FIG. 3 and FIG. 7. After the two-dimensional imaging unit 110 captures each two-dimensional image Img1 of the object to be tested 20, the first processor 120 continues to analyze the two-dimensional image Img1 to obtain edge information Edi1 of the two-dimensional image Img1. Specifically, the first processor 120 performs image processing on each two-dimensional image Img1 to find an image showing an edge in the two-dimensional image Img1 (hereinafter referred to as edge image Edp), and then obtains the two-dimensional coordinate value (i.e., X information x1~xn and Y information y1~yn) of the relative position of each pixel (i.e., point P11~P1n) forming the edge image Edp on the coordinate system of the object to be tested 20, which is the edge information Edi1.

In some embodiments, the first processor 120 may be a computer device having an image processing function, such as but not limited to a central processing unit (CPU), a graphics processing unit (GPU), or an image signal processor (ISP), etc. The image processing function includes, but is not limited to, the ability to perform image processing such as de-mosaicing, noise reduction, white balance, exposure correction, sharpening, color conversion, encoding, or any combination thereof.

Please refer to FIG. 1 , FIG. 3 , FIG. 8 and FIG. 9 . After step S120 , the depth sensor 130 will capture the multiple Z information z10-z15 of each point P11/…/P1n according to the edge information of each two-dimensional image Img1 (step S130 ). Among them, each Z information z10/z11/…/z15 is the position of the film surface Sf0/Sf1/…/Sf5 of a film layer 21 of the object to be tested 20 at the corresponding point P11/…/P1n at the depth Dth of the object to be tested 20 . In some embodiments, the depth sensor 130 is a color confocal sensor to sense transparent materials and further measure the distance (i.e., depth value) of the film layers of multiple transparent materials.

In some embodiments, when the object 20 to be tested is a structure with multiple film layers 21, the construction system 10 needs to obtain the thickness information of each film layer 21 in the object 20 to establish the three-dimensional point cloud information of the object 20. Therefore, the construction system 10 captures the multiple Z information z10-z15 of all the film layers 21 at each point P11/…/P1n of the edge information Edi1 of the two-dimensional image Img1 through the color confocal sensor (i.e., the depth sensor 130) to represent the thickness information of each film layer 21 in the object 20 to be tested.

Specifically, the optical axis Ax2 of the depth sensor 130 is aligned with the point P11/…/P1n to be measured, and then the Z information z10/z11/…/z15 of each film surface Sf0/Sf1/…/Sf5 of the object to be measured is detected and captured. Here, each Z information z10/z11/…/z15 corresponds to the film surface Sf0~Sf5 of the different film layer 21 at the corresponding point P11/…/P1n, so the difference between the Z information z10, z11/z11, z12/…/z14, z15 of two adjacent film surfaces Sf0, Sf1/Sf1, Sf2/…/Sf4, Sf5 is the thickness information of the corresponding film layer 21 at the point P11. In other words, the first processor 120 can obtain the thickness information of the corresponding film layer 21 by calculating the difference between two different Z information (two of z10-z15) of the same point P11/.../P1n.

For example, the object to be tested 20 is a curved liquid crystal panel. In some embodiments, the film layer 21 of the object to be tested 20 includes a protective film (Polyethylene Terephthalate, PET) 201, an explosion-proof film (Anti-Scattering Film, ASF) 202, an adhesive film (Pressure Sensitive Adhesive, PSA) 203, a glass layer 204, and a light shielding layer (Black Matrix, BM) 205. The glass layer 204 is stacked on the light shielding layer 205, the adhesive film 203 is stacked on the glass layer 204, the explosion-proof film 202 is stacked on the adhesive film 203, and the protective film 201 is stacked on the explosion-proof film 202. Among the Z information z10~z15 captured by the depth sensor 130 at point P11, Z information z10 corresponds to the film surface Sf0 of the protective film 201, Z information z11 corresponds to the film surface Sf1 of the explosion-proof film 202, Z information z12 corresponds to the film surface Sf2 of the adhesive film 203, Z information z13 corresponds to the film surface Sf3 of the glass layer 204, and Z information z14 corresponds to the film surface Sf4 of the shading layer 205. Therefore, the difference between Z information z10 and Z information z11 is the thickness information of the protective film 201 at point P11, the difference between Z information z11 and Z information z12 is the thickness information of the explosion-proof film 202 at point P11, the difference between Z information z12 and Z information z13 is the thickness information of the adhesive film 203 at point P11, the difference between Z information z13 and Z information z14 is the thickness information of the glass layer 204 at point P11, and the difference between Z information z14 and Z information z15 is the thickness information of the shading layer 205 at point P11.

As shown in FIG. 2 , in some embodiments, the two-dimensional imaging unit 110 and the depth sensor 130 may be arranged side by side. In other words, the optical axis Ax1 of the two-dimensional imaging unit 110 and the optical axis Ax2 of the depth sensor 130 are parallel. Here, the two-dimensional imaging unit 110 and the depth sensor 130 are both facing the front view direction of the object to be measured 20 (i.e., facing the XY plane, in other words, their optical axes Ax1 and Ax2 are substantially parallel to the Z axis), and there is a fixed distance CD between the optical axis Ax1 of the two-dimensional imaging unit 110 and the optical axis Ax2 of the depth sensor 130. Among them, when the value of the fixed distance CD is smaller, the accuracy of the depth sensor 130 in sensing the object to be measured 20 is higher. Therefore, in some embodiments, the smaller the value of the fixed distance CD is, the better, so that the depth sensor 130 can more accurately sense the depth Dth of the object 20. In some embodiments of step S130, after the first processor 120 analyzes and obtains the X information (x1) and Y information (y1) of each point P11 in the edge information of the two-dimensional image Img1 (step S120), the first processor 120 calculates the movement vector of the depth sensor 130 from the fixed position to the point P11, and moves the depth sensor 130 to the position of the optical axis Ax2 aligned with the point P11 according to the movement vector. Then, the depth sensor 130 captures the multiple Z information z10~z15 of all points P11~P1n in the edge information of the two-dimensional image Img1.

Please refer to FIG. 10. In some embodiments, the two-dimensional imaging unit 110 and the depth sensor 130 may form a coaxial axis through a beam splitter 150. In other words, the construction system 10 may further include a beam splitter 150. The beam splitter 150 is located between the stage 100 and the two-dimensional imaging unit 110 and between the stage 100 and the depth sensor 130. The two-dimensional imaging unit 110 and the depth sensor 130 are arranged in different directions. In other words, one of the two-dimensional imaging unit 110 and the depth sensor 130 is facing the front direction of the object 20 to be measured (i.e., facing the XY plane, in other words, the optical axis Ax1/Ax2 is substantially parallel to the Z axis), and a beam splitter 150 is arranged between it and the stage 100. The other of the two-dimensional imaging unit 110 and the depth sensor 130 is facing the front view direction of the spectroscope 150 (i.e., facing the YZ plane, in other words, the optical axis Ax2/optical axis Ax1 is substantially parallel to the X axis), and the optical axis Ax2/Ax1 parallel to the X axis is turned toward the front view direction of the object to be measured 20 by the spectroscope 150 and overlapped with the other optical axis Ax1/Ax2 to form a single optical axis Ax3. Specifically, the optical axis Ax1 and the optical axis Ax3 of the two-dimensional imaging unit 110 are substantially parallel to each other and overlap together, and the optical axis Ax2 and the optical axis Ax3 of the depth sensor 130 are substantially perpendicular to each other.

After step S130, the first processor 120 transmits the X information, Y information and multiple Z information of each point on the edge information of all two-dimensional images to the second processor 140, and then the second processor 140 maps the X information, Y information and multiple Z information of each point to generate three-dimensional point cloud information of the object to be tested 20 (step S140). The three-dimensional point cloud information of the object to be tested 20 is a data set of three-dimensional coordinates (also known as Cartesian coordinates) of each position (i.e., point P11/…/P1n) on the contour of the object to be tested 20. In other words, when all the points P11~P1n presenting three-dimensional coordinates in the three-dimensional point cloud information are connected together, the appearance of the object to be tested 20 can be presented.

In some embodiments of step S140, the second processor 140 maps the X information, Y information, and multiple Z information of each point P11/…/P1n on the edge information of all two-dimensional images Img1 of the object to be tested 20 to a mapping table to record the three-dimensional coordinates of each point P11/…/P1n on the edge information of all two-dimensional images Img1 of the object to be tested 20, and then establish the three-dimensional point cloud information of the object to be tested 20.

In some embodiments, the first processor 120 is coupled to the second processor 140 via a wired communication, and transmits the X information, Y information, and multiple Z information of each point on the edge information of all two-dimensional images of the object under test 20 to the second processor 140 via the wired communication. The wired line is, for example, but not limited to, a transmission line, a bus, or a network line. In some embodiments, the second processor 140 can be a device with a wired communication function, such as, but not limited to, a laptop or a desktop computer.

In some embodiments, the first processor 120 may also be coupled to the second processor 140 via a wireless communication method, and transmit the X information, Y information, and the plurality of Z information of each point on the edge information of all two-dimensional images of the object to be measured 20 to the second processor 140 via wireless communication. The wireless line may be, for example, but not limited to, a wireless network (Wifi) or Bluetooth. In some embodiments, the second processor 140 may be a device with a wireless communication function, such as, but not limited to, a cloud system, a server system, a laptop computer, or a desktop computer with a wireless communication function.

Please refer to FIG. 1 and FIG. 3 , in some embodiments, before step S110, the first processor 120 of the system 10 will first establish an initial imaging path for all two-dimensional images according to the specification of the object 20 to be tested (step S100), and then drive the two-dimensional imaging unit 110 to capture the two-dimensional image of the object 20 to be tested based on the established initial imaging path (step S110). In some embodiments of step S100, the first processor 120 can know the size of the object 20 to be tested from the specification of the object 20 to be tested in advance, and establish an initial imaging path moving along the edge of the object 20 to be tested according to the size of the object 20 to be tested. Then, in step S110 , the two-dimensional imaging unit 110 moves its position according to the initial imaging path to obtain a two-dimensional image of the object 20 to be tested, thereby ensuring that the edge of the object 20 to be tested falls into the visible range of the two-dimensional imaging unit 110 section by section.

Please refer to Figures 1 to 3, 5, 6 and 11 to 14. In some embodiments of step S120, for each two-dimensional image Img1', the first processor 120 first binarizes the captured two-dimensional image Img1' to obtain the edge image Edp of the two-dimensional image Img1' (step S121). In some embodiments, the first processor 120 first performs grayscale division of the two-dimensional image Img1' to convert all pixels in the two-dimensional image Img1' into "0" or "1" or "grayscale minimum value" or "grayscale maximum value" according to the grayscale value of each pixel and a grayscale threshold. In other words, the first processor 120 converts the two-dimensional image Img1' from a grayscale image to a binary image (as shown in FIG12). When the grayscale value of a pixel is higher than the grayscale threshold, the first processor 120 sets the grayscale value of the pixel to black, that is, the data thereof is 1 (or 255). When the grayscale value of a pixel is lower than the grayscale threshold, the first processor 120 sets the grayscale value of the pixel to white, that is, the data thereof is 0. The adjacent portions of two adjacent pixels having different values in the binary two-dimensional image Img1' constitute the edge image Edp.

After step S121, the first processor 120 continues to highlight the edge image Edp of the two-dimensional image Img1' (step S122). In some embodiments, the first processor 120 forms a connected area with pixels having similar colors, textures, and other features in the binarized two-dimensional image Img1' to highlight the edge image Edp of the two-dimensional image Img1' (as shown in FIG. 13). The first processor 120 forms a connected block Rg1 with pixels converted to black in the two-dimensional image Img1', and forms another connected block Rg2 with pixels converted to white in the two-dimensional image Img1', so that the boundary line (i.e., the edge image Edp) between the two connected blocks Rg1 and Rg2 is highlighted.

After step S122, the first processor 120 connects each point on the edge image Edp to obtain edge information Edi1 (step S123). In some embodiments, after the edge image Edp is highlighted, the first processor 120 connects all points on the boundary between the connected area Rg1 composed of black pixels and the connected block Rg2 composed of white pixels (as shown by gray pixels) to form edge information Edi1' (as shown in FIG. 14).

After step S123, the first processor 120 performs anti-aliasing processing on the edge information Edi1' of the two-dimensional image Img1' (step S124). In some embodiments, the edge information Edi1' formed by connecting a plurality of pixels together presents a jagged connection due to the matrix arrangement of the pixels. Therefore, the first processor 120 performs smoothing processing on the connection (i.e., the edge information Edi1') to eliminate all jagged edges on the edge information Edi1', thereby forming a smoother and softer edge information Edi1 (as shown in FIG8).

Please refer to Figures 1 to 3, 8, 9 and 15. In some embodiments of step S130, for each two-dimensional image Img1, after obtaining its edge information Edi1, the first processor 120 will calculate a depth acquisition path according to the edge information Edi1 of the two-dimensional image Img1 (step S131), and move the depth sensor 130 based on the depth acquisition path so that the optical axis Ax2 of the depth sensor 130 is aligned with all points P11~P1n on the edge information of the two-dimensional image Img1 in sequence (step S132). Furthermore, when the optical axis Ax2 of the depth sensor 130 is aligned with each point P11/P12/…/P1n, the depth sensor 130 sequentially detects multiple film surfaces Sf0~Sf5 of different depths of the object 20 to obtain Z information z10/z11/…/z15 of each film surface Sf0/Sf1/…/Sf5 (step S133).

Please refer to Figures 1 to 3, 8, 15 and 16. In some embodiments, the depth imaging path includes a motion vector Vt and edge information Edi1. Specifically, whenever the two-dimensional imaging unit 110 captures a two-dimensional image Img1, the first processor 120 analyzes and obtains the edge information Edi1 of the two-dimensional image Img1 (step S120), and then calculates the motion vector Vt from the fixed point position Da1 to the position of the first point P11 in the edge information Edi1 (hereinafter referred to as the detection position Dp11) according to the edge information Edi1 of the two-dimensional image Img1 (step S131).

Here, the fixed point position Da1 refers to the position of the optical axis Ax2 of the depth sensor 130 on the XY plane when the two-dimensional imaging unit 110 performs two-dimensional image capture. The detection position Dp11 is the position of the first point P11 in the edge information Edi1 on the object to be measured 20, which is equivalent to the X information x1 and Y information y1 of the first point P11. In other words, when the X information and the Y information are expressed in the three-dimensional coordinate system where the object to be measured 20 is located, the detection position Dp11 is the X information x1 and the Y information y1 of the first point P11 in the edge information Edi1. Subsequently, the depth sensor 130 moves according to the movement vector Vt, so that its optical axis Ax2 moves from the alignment fixed point position Da1 to the first point P11 in the alignment edge information Edi1 (step S132), and then measures the distance of all film surfaces Sf0~Sf5 of the object 20 to be tested at the first point P11 (i.e., Z information z10~z15) (step S133).

After obtaining the Z information z10~z15 of the first point P11, the depth sensor 130 moves according to the edge information Edi1, so that its optical axis Ax2 moves from the detection position Dp11 aligned with the current detection point P11 to the detection position Dp12 aligned with the next point P12 in the edge information Edi1 (step S132), and obtains the Z information of the next point P12 (step S133), and then repeats this process until it moves to the detection position Dp1n of the last point P1n in the edge information Edi1 (step S132) and obtains the Z information of point P1n (step S133). When the two-dimensional imaging unit 110 captures the next two-dimensional image, the same process from step S120 to step S133 is performed on the two-dimensional image to obtain the Z information of all points of the edge information Edi1 on the two-dimensional image. And the same is true for all two-dimensional images of the object 20 to be tested.

That is, during the construction process, the depth sensor 130 moves with the two-dimensional imaging unit 110, and the optical axis Ax2 of the depth sensor 130 is coaxial with the optical axis Ax1 of the two-dimensional imaging unit 110 or is parallelly pointed to the object 20 at a fixed distance CD, so that the optical axis Ax2 of the depth sensor 130 is fixed at a fixed point (i.e., the fixed point position Da1) in the visible range Rf11/…/Rfst of the two-dimensional imaging unit 110. Although all points of the edge information Edi1 fall within the visible range Rf11/…/Rfst of the two-dimensional imaging unit 110, they are not all aligned with the optical axis Ax1 of the two-dimensional imaging unit 110.

For example, please further refer to Figures 17 to 20, assuming that the two-dimensional imaging unit 110 and the depth sensor 130 are coaxially pointing to the stage 100. The two-dimensional imaging unit 110 captures a two-dimensional image Img2 (as shown in Figure 18) at a capture position of the edge Edo of the object to be measured 20 with a visible range Rf1 (t-1) (as shown in Figures 6 and 17). At this time, the fixed point position Da2 where the optical axis Ax2 of the depth sensor 130 falls within the visible range Rf1 (t-1) is at a distance Dis1 from the detection position Dp21 of the first point P21 in the edge information Edi2 of the two-dimensional image Img2. After the first processor 120 calculates the depth acquisition path corresponding to the edge information Edi2 according to the edge information Edi2 of the two-dimensional image Img2, the depth sensor 130 moves to the detection position Dp21 of the point P21 along the depth acquisition path corresponding to the edge information Edi2 and obtains the Z information of the point P21, and then moves to the detection positions of other points in the edge information Edi2 one by one and obtains their Z information.

In some embodiments, since the two-dimensional imaging unit 110 and the depth sensor 130 use a coaxial optical structure, the distance that the depth sensor 130 needs to move is very short (millimeter level). Therefore, the two-dimensional imaging unit 110 and the depth sensor 130 can obtain the X information, Y information and multiple Z information of each point on the edge information of the two-dimensional image almost simultaneously, thereby shortening the construction time of the three-dimensional point cloud information.

Then, the two-dimensional imaging unit 110 moves the edge Edo of the object 20 to the next capture position according to the initial image capture path, that is, moves to the position of the visible range Rf1t. At the same time, the depth sensor 130 moves with the two-dimensional imaging unit 110, so that the optical axis Ax2 of the depth sensor 130 moves to the fixed point position Da3 within the visible range Rf1t (as shown in FIG. 17 ). After the two-dimensional imaging unit 110 captures the two-dimensional image Img3 with the visible range Rf1t (as shown in FIG. 19 ), the first processor 120 calculates the depth acquisition path corresponding to the edge information Edi3 according to the edge information Edi3 of the two-dimensional image Img3, so that the depth sensor 130 moves to the detection position Dp31 of the first point P31 in the edge information Edi3 along the depth acquisition path corresponding to the edge information Edi3 and obtains the Z information z10~z15 of the point P31 (as shown in FIG. 21 ), and then moves to the detection positions of other points in the edge information Edi3 one by one and obtains their Z information.

Similarly, after obtaining the Z information of all points in the edge information Edi3, the two-dimensional imaging unit 110 moves to the next capture position of the edge Edo of the object 20 to be measured according to the initial imaging path, that is, moves to the position of the visible range Rf2t. At the same time, the depth sensor 130 moves with the two-dimensional imaging unit 110, so that the optical axis Ax2 of the depth sensor 130 moves to the fixed point position Da4 aligned with the visible range Rf2t (as shown in FIG. 17). After the two-dimensional imaging unit 110 captures the two-dimensional image Img4 with the visible range Rf2t (as shown in FIG. 20 ), the first processor 120 obtains the edge information Edi4 of the two-dimensional image Img4 and calculates the depth acquisition path corresponding to the edge information Edi4 according to the edge information Edi4 of the two-dimensional image Img4, so that the depth sensor 130 moves to the detection position Dp41 of the first point P41 in the edge information Edi4 along the depth acquisition path corresponding to the edge information Edi4 and obtains the Z information z10~z15 of the point P41 (as shown in FIG. 21 ), and then moves to the detection positions of other points in the edge information Edi4 one by one and obtains their Z information.

In FIG. 21 , a frame Rs1 is a side view of the object under test 20 within the visible range Rf1t under a section line passing through point P31, and a frame Rs2 is a side view of the object under test 20 within the visible range Rf2t under a section line passing through point P41.

Please refer to FIG. 1 and FIG. 22. In some embodiments, the construction system 10 may further include a moving device 160. The moving device 160 is coupled to the first processor 120 and is controlled by the first processor 120. Here, the two-dimensional imaging unit 110 and the depth sensor 130 are assembled on the moving device 160. In step S110, the moving device 160 drives the two-dimensional imaging unit 110 to perform two-dimensional movement on the XY plane in the three-dimensional coordinate system according to the initial imaging path, so that the visual range of the two-dimensional imaging unit 110 moves along the edge of the object 20 to capture the two-dimensional image of the object 20. In step S132, the moving device 160 drives the depth sensor 130 to move along the edge Edo of the object 20 according to the depth imaging path, so that the optical axis Ax2 of the depth sensor 130 can be displaced along the edge information of each two-dimensional image, thereby capturing all Z information of each point on the edge information. In some embodiments, the moving device 160 can be a device with a two-dimensional moving mechanism, such as but not limited to a robot arm.

Please refer to FIG. 1 and FIG. 23. In some embodiments, the construction system 10 further includes a moving device 161. The moving device 161 is coupled to the first processor 120 and is controlled by the first processor 120. Here, the carrier 100 is assembled on the moving device 161. In step S110, the moving device 161 drives the carrier 100 to move two-dimensionally on the XY plane in the three-dimensional coordinate system relative to the visible range of the two-dimensional imaging unit 110 according to the initial imaging path, so that the object 20 to be tested moves relative to the visible range of the two-dimensional imaging unit 110, causing the visible range of the two-dimensional imaging unit 110 to produce a relative displacement along the edge Edo of the object 20 to be tested. In step S132, the moving device 160 drives the stage 100 to move two-dimensionally relative to the position of the optical axis Ax2 of the depth sensor 130 on the XY plane in the three-dimensional coordinate system according to the depth imaging path, so that the object 20 moves relative to the optical axis Ax2 of the depth sensor 130, causing the optical axis Ax2 of the depth sensor 130 to be relatively displaced along the edge Edo of the object 20. In some embodiments, the moving device 161 can be a moving platform with a two-dimensional moving mechanism, and the two-dimensional moving mechanism is, for example but not limited to, a slide rail, a gear, a roller, a ball, a motor, or a conveyor belt.

In summary, according to any embodiment, the system or method for constructing three-dimensional point cloud information is suitable for obtaining high-precision three-dimensional coordinates (such as contour size and film thickness) of the object 20 to be tested, and then constructing the three-dimensional point cloud information of the object 20 to be tested. In this way, the three-dimensional point cloud information of the object 20 to be tested can be provided for application in high-precision processes such as laser cutting and automated optics of the object 20 to be tested. In some embodiments, the system or method for constructing three-dimensional point cloud information is particularly suitable for constructing high-precision three-dimensional point cloud information of the object 20 to be tested having a multi-layer structure with a transparent film.

Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the invention of the present invention. Anyone with ordinary knowledge in the relevant technical field may make some modifications and changes without departing from the spirit and scope of the present disclosure. However, such modifications and changes are still within the scope of the patent application of the present invention.

10: Build the system

100: Carrier

101: Light-emitting unit

110: Two-dimensional imaging unit

111: Charge-coupled device (CCD) sensor

112:Telecentric lens

120: first processor

130: Depth sensor

140: Second processor

150: Spectroscope

160,161: Mobile device

20: Object to be tested

21: Membrane layer

201: Protective film

202: Explosion-proof membrane

203: Adhesive film

204: Glass layer

205:Shading layer

Ax1, Ax2, Ax3: optical axis

Ax11, Ax12: optical axis

CD: Fixed distance

Da1~Da4: Fixed point position

Dis1: Distance

Dp11,Dp12,Dp21,Dp31,Dp41,Dp1n: Detection location

Dth: Depth

Edi1~Edi4: marginal information

Edi1’: marginal information

Edo:Edge

Edp:Edge Image

Img1~Img4: 2D images

Img1’: 2D image

Lth: Long side

Obt: Object image

P11~P1n: point

P21, P31, P41: point

Rf11~Rf1t: Visible range

Rf2t~Rfst: Visible range

Rg1, Rg2: connected blocks

Rs1, Rs2: Box

S100~S140: Steps

S121~S124: Steps

S131~S133: Steps

Sf0~Sf5: membrane surface

Vt:Transfer vector

WD: Working distance

Wth: Short side

x1~xn:X information

y1~yn:Y information

z10~z15:Z information

FIG. 1 is a functional block diagram of a system for constructing three-dimensional point cloud information according to some embodiments. FIG. 2 is a schematic diagram of a first embodiment of a two-dimensional imaging unit and a depth sensor in FIG. 1. FIG. 3 is a flow chart of a construction method according to some embodiments. FIG. 4 is a schematic diagram of an embodiment of a two-dimensional imaging unit in FIG. 2. FIG. 5 is a three-dimensional schematic diagram of an exemplary state of the object to be tested in FIG. 1. FIG. 6 is a front view plane schematic diagram of the object to be tested in FIG. 5. FIG. 7 is a schematic diagram of an exemplary state of a two-dimensional image captured in a visible range in FIG. 6. FIG. 8 is a schematic diagram of an exemplary state of edge information of the two-dimensional image in FIG. 7. FIG. 9 is a schematic diagram of a cross-section of the object to be tested corresponding to the two-dimensional image in FIG. 7. FIG. 10 is a schematic diagram of a second embodiment of the two-dimensional imaging unit and the depth sensor in FIG. 1. FIG. 11 is a detailed flowchart of an embodiment of step S120 in FIG. 3 . FIG. 12 is a schematic diagram of an exemplary state of the two-dimensional image in FIG. 7 after being processed by step S121 in FIG. 11 . FIG. 13 is a schematic diagram of an exemplary state of the two-dimensional image in FIG. 12 after being processed by step S122 in FIG. 11 . FIG. 14 is a schematic diagram of an exemplary state of the two-dimensional image in FIG. 13 after being processed by step S123 in FIG. 11 . FIG. 15 is a detailed flowchart of an embodiment of step S130 in FIG. 3 . FIG. 16 is a schematic diagram of an exemplary state of the motion vector in step S131 in FIG. 15 . FIG. 17 is a schematic diagram of an exemplary state of the fixed point position in step S131 in FIG. 15 . FIG. 18 is a schematic diagram of a two-dimensional image corresponding to the visible range on the left in FIG. 17. FIG. 19 is a schematic diagram of a two-dimensional image corresponding to the visible range in the middle in FIG. 17. FIG. 20 is a schematic diagram of a two-dimensional image corresponding to the visible range on the right in FIG. 17. FIG. 21 is a schematic diagram of a cross section of an object to be tested within the visible range in the middle and on the right in FIG. 17. FIG. 22 is a schematic diagram of a third embodiment of a two-dimensional imaging unit and a depth sensor in FIG. 1. FIG. 23 is a schematic diagram of a fourth embodiment of a two-dimensional imaging unit and a depth sensor in FIG. 1.

S110~S140: Steps

Claims (10)

A system for constructing three-dimensional point cloud information includes: a carrier for carrying an object to be measured, wherein the carrier is provided with a light-emitting unit, and the light-emitting unit is used to emit light to illuminate the object to be measured; a two-dimensional imaging unit, used to capture a plurality of two-dimensional images of the object to be measured, wherein the optical axis of the two-dimensional imaging unit points to the carrier; a first processor, coupled to the two-dimensional imaging unit, used to analyze each of the two-dimensional images to obtain edge information of each of the two-dimensional images, wherein the edge information of each of the two-dimensional images includes X information and Y information of a plurality of points; a depth sensor, coupled to the two-dimensional imaging unit, used to obtain edge information of each of the two-dimensional images; The first processor is used to capture the multiple Z information of each point according to the edge information of each two-dimensional image, wherein the optical axis of the depth sensor points to the stage; a second processor is coupled to the first processor, and is used to map the X information, the Y information and the Z information of each point to generate a three-dimensional point cloud information of the object to be measured; and a spectroscope, the spectroscope is located between the stage and the two-dimensional imaging unit and between the stage and the depth sensor, and the optical axis of the two-dimensional imaging unit and the optical axis of the depth sensor are coaxial through the spectroscope. A system as described in claim 1, wherein the two-dimensional imaging unit includes a charge coupled device (CCD) sensor and a telecentric lens, the optical axis of the CCD sensor and the optical axis of the telecentric lens are coaxial, and the CCD sensor is coupled to the first processor. A system as described in claim 1, wherein the working distance of the two-dimensional imaging unit is not less than 65 mm. A system as described in claim 1, wherein the depth sensor is a chromatic confocal sensor. A system as described in claim 1, wherein the optical axis of the two-dimensional imaging unit and the optical axis of the depth sensor are parallel. The system as described in claim 1 further comprises: a moving device coupled to the first processor, wherein the two-dimensional imaging unit and the depth sensor are arranged on the moving device, and under the control of the first processor, the moving device is used to move the two-dimensional imaging unit according to an initial imaging path and move the depth sensor according to a depth imaging path. The construction system as described in claim 1 further comprises: a moving device coupled to the first processor, wherein the carrier is disposed on the moving device, and under the control of the first processor, the moving device is used to move the carrier according to an initial imaging path and a depth imaging path. A method for constructing three-dimensional point cloud information includes: capturing multiple two-dimensional images of an object to be measured; analyzing each of the two-dimensional images to obtain edge information of each of the two-dimensional images, wherein the edge information of each of the two-dimensional images includes X information and Y information of multiple points; capturing multiple Z information of each of the points based on the edge information of each of the two-dimensional images; and mapping the X information, the Y information and the Z information of each of the points to generate a three-dimensional point cloud information of the object to be measured; , the step of acquiring the Z information of each point according to the edge information of each two-dimensional image includes: calculating a depth imaging path according to the edge information of each two-dimensional image; moving a depth sensor based on the depth imaging path so that the optical axis of the depth sensor is aligned with the points in sequence; and when the depth sensor is aligned with each point, sequentially detecting multiple film surfaces of the object to be tested at different depths through the depth sensor to obtain the Z information of each film surface. The construction method as described in claim 8 further includes: establishing an initial imaging path for the two-dimensional images according to a specification drawing of the object to be tested, wherein the step of capturing the two-dimensional images of the object to be tested is performed based on the initial imaging path. The construction method as described in claim 8, wherein the step of analyzing each of the two-dimensional images to obtain the edge information of each of the two-dimensional images includes: binarizing each of the two-dimensional images to obtain an edge image of each of the two-dimensional images; highlighting the edge image of each of the two-dimensional images; connecting the points on the edge image of each of the two-dimensional images to obtain the edge information; and performing anti-aliasing on the edge information of each of the two-dimensional images.