TW202238530A - Image processing method and electronic device using the same - Google Patents

Abstract

An image processing method includes the following steps. Firstly, a three-dimensional point cloud image is captured, wherein the three-dimensional point cloud image includes several pixels. Then, the pixels are distinguished into several reliable points and several unreliable points. Then, from these reliable points, several straight lines are determined, wherein each straight line passes through two of the reliable points. Then, selected one of the straight line is used as a wall line segment. Then, the unreliable points are moved to the wall line segment.

Description

Image processing method and electronic device using same

The present invention relates to a processing method and an electronic device using it, and in particular to an image processing method and an electronic device using it.

Multipath Interference (MPI) is an innate characteristic of Time of Flight (ToF) cameras. It refers to the interference caused by multiple reflections of obstacles on the detection signal sent by ToF cameras. This phenomenon causes the depth measured by the ToF camera to be incorrect. For example, under multi-path interference, the detection result of a curved wall will be obtained for an actually straight wall. Therefore, how to improve the prevention of multipath interference is an important issue for those in the technical field.

Therefore, the present invention proposes an image processing method and an electronic device using the same, which can improve the conventional problems.

An embodiment of the invention provides an image processing method. The image processing method includes the following steps. Extract a 3D point cloud image, the 3D point cloud image contains several pixel points; distinguish these pixel points into several reliable points and plural unreliable points; from these reliable points, determine several straight line segments, wherein each straight line segment passes through this Two of these reliable points; one of these straight line segments is selected as a wall line segment; and these unreliable points are moved to the wall line segment.

Another embodiment of the invention provides an electronic device. The electronic device includes a camera and a processor. The camera is used to capture a 3D point cloud image, and the 3D point cloud image includes several pixels. The processor is used to: distinguish these pixel points as several reliable points and plural unreliable points; from these reliable points, determine several straight line segments, wherein each straight line segment passes through these two reliable points; One of the selected ones is used as a wall line segment; and, these unreliable points are moved to the wall line segment.

In order to have a better understanding of the above-mentioned and other aspects of the present invention, the following specific examples are given in detail with the accompanying drawings as follows:

Please refer to Figures 1 and 2A~2E, Figure 1 shows a schematic diagram of an electronic device 100 according to an embodiment of the present invention, and Figures 2A~2E show a process diagram of the image processing method of the electronic device 100 in Figure 1 , wherein Figure 2A shows a schematic diagram of the three-dimensional point cloud image M1 captured by the camera 110 of the electronic device 100 in Figure 1, and Figures 2B-2E show the two-dimensional point cloud image of the three-dimensional point cloud image M1 in Figure 2A Schematic of M2.

The electronic device 100 is, for example, a cleaning robot. The electronic device 100 can detect the boundary of an obstacle (eg, a wall) ahead, so as to avoid the obstacle during the movement.

The electronic device 100 includes a camera 110 and a processor 120 . The camera 110 is used to: capture a 3D point cloud image M1 (as shown in FIG. 2A ), and the 3D point cloud image M1 includes several pixel points P1 (as shown in FIG. 2A ). The processor 120 is used to: (1). Distinguish these pixel points P1 into several reliable points P21 and several unreliable points P22 (reliable points P21 and unreliable points P22 are shown in FIG. 2C); (2). From then on Among these reliable points P21, determine a number of straight line segments L1 (as shown in Figure 2C), wherein each straight line segment L1 passes through both of these reliable points P21; (3). Select one of these straight line segments L1 Or as a wall line segment (L21~L23 shown in Figure 2C); (4). Move these unreliable points P22 to the wall line segment. In this way, the electronic device 100 can improve the problem caused by multipath interference, that is, the electronic device 100 can obtain a detection result close to an actual straight wall.

The camera 110 is, for example, a ToF camera. The 3D point cloud image M1 can be obtained using ToF detection technology. The camera 110 can continuously capture several 3D point cloud images M1 at different or consecutive time points, and the camera 110 can execute the image processing procedure described herein for each point cloud image M1. The processor 120 is, for example, a physical circuit (circuit) formed by semiconductor manufacturing process.

Please refer to FIG. 3 , which shows a flow chart of the image processing method of the electronic device 100 in FIG. 1 .

In step S110 , as shown in FIG. 2A , the camera 110 captures a 3D point cloud image M1 , and the 3D point cloud image M1 includes several pixel points P1 . Each pixel point P1 has a coordinate (not shown), and its coordinate value is, for example, defined relative to the illustrated x-y-z coordinate system. The z-axis is, for example, the front direction of the camera 110 , the x-y plane is, for example, perpendicular to the z-axis, and the x-z plane is, for example, the moving plane of the electronic device 100 .

In step S120 , as shown in FIG. 2B , the processor 120 converts the pixels P1 of the 3D point cloud image M1 into a 2D point cloud image M2 . The 2D point cloud image M2 includes several pixel points P2, wherein each pixel point P1 of the 3D point cloud image M1 is converted to the corresponding pixel point P2 of the 2D point cloud image M2. In addition, the number of pixels P2 in the 2D point cloud image M2 is substantially the same as the number of pixels P1 in the 3D point cloud image M1 . In the two-dimensional point cloud image M2, the y-coordinate value of each pixel point P1 is, for example, 0. In one embodiment, the processor 120 can convert the pixel points P1 of the 3D point cloud image M1 into a 2D point cloud image M2 through a transformation matrix RT. The conversion matrix RT can be obtained by any suitable mathematical method, which is not limited in the embodiment of the present invention.

In step S130 , as shown in FIG. 2C , the processor 120 distinguishes all pixel points P2 of the two-dimensional point cloud image M2 into several reliable points P21 and several unreliable points P22 . The so-called "reliable points" refer to pixels that are not affected by multipath interference or can be ignored (the probability of belonging to the wall is high), while "unreliable points" refer to pixels that are affected by multipath interference. The generated pixels (the probability of belonging to the wall is small). In one embodiment, the processor 120 may first determine unreliable points from all the pixel points P2, and the rest of the pixel points P2 may be classified as reliable points. In one embodiment, the camera 110 can emit two detection lights of different frequencies, and obtain two point cloud images according to the two reflection lights reflected by the two detection lights from obstacles (such as walls), and compare the two point cloud images. The corresponding pixel points of the cloud image are used to distinguish or judge whether the corresponding pixel points are reliable points or unreliable points. However, as long as it can determine and/or analyze the multipath interference degree of the pixel point P2 of the two-dimensional point cloud image M2, the embodiment of the present invention does not limit the specific technology for distinguishing reliable points and unreliable points. In addition to having coordinate values (such as coordinate values of x and z), each pixel point P2 also has a reliability attribute, and the recorded pixel point P2 belongs to "unreliability" or "reliability".

In step S140, as shown in FIG. 2C, from these reliable points P21, the processor 120 determines several straight line segments L1 of the two-dimensional point cloud image M2, wherein each straight line segment L1 passes through at least two of these reliable points P21 . The processor 120 expresses the straight line segment L1 by, for example, a straight line equation, so the straight line segment L1 in FIG. 2C is expressed in an infinitely extended form. In one embodiment, the processor 120 may use the hough transform technique to determine several straight line segments L1 from the reliable points P21.

In step S150 , as shown in FIG. 2C , the processor 120 uses the selected one of the straight line segments L1 as a wall line segment. In this embodiment, the processor 120 uses three selected ones of the straight line segments L1 as three wall line segments L21 , L22 and L23 . The embodiment of the present invention does not limit the number of wall line segments. Depending on the number of wall surfaces in the actual environment, the processor 120 may determine less than three or more than three wall line segments from the straight line segments L1.

The following is one of several methods for selecting (or determining) wall line segments.

In one embodiment, as shown in FIG. 2C , each straight line segment L1 has a length S1 . The processor 120 is further used to: (1). Determine whether the length S1 of each straight line segment L1 is greater than a preset length value; (2). If the length S1 is greater than the preset length value, the The straight line segment L1 is used as the wall line segment. For example, where the length S1 of the straight line segment L1 is greater than the preset length value, the straight line segment L1 is set as the wall line segment L21. The determination method of the other wall line segments L22-L23 is the same as the determination method of the wall line segment L21, which will not be repeated here. The length S1 of each straight line segment L1 is, for example, the length S1 between at least two reliable points P21 that it passes through, and the two reliable points P21 are, for example, the two most reliable points at the two ends of all the reliable points P21 that the straight line segment L1 passes through. Embodiments of the invention are not limited thereto. The embodiment of the present invention does not limit the value range of the "preset length value", which may depend on the frame content of the 3D point cloud image M1 captured by the electronic device 100 . In one embodiment, the "preset length value" is, for example, a critical value that can be judged as a wall surface, such as any integer between 10 cm and 2 meters, but it can also be less than 10 cm or greater than 2 meters.

In step S160 , as shown in FIG. 2D , the processor 120 moves the unreliable points P22 to the wall line segment. For example, the processor 120 rewrites the coordinate value of the unreliable point P22 so that the coordinate value of the unreliable point P22 conforms to the equation of the wall line segment.

In one embodiment, as shown in FIG. 2D, the processor 120 is further used to: (1). Obtain the distances H1-H3 between one of the unreliable points P22 and each wall line segment L21-L23; (2). Move this one of these unreliable points P22 to the wall line segment corresponding to the smallest one of these distances H1~H3. Taking the unreliable point P22' as an example, the distances between the unreliable point P22' and the three wall line segments L21-L23 are respectively H1-H3, wherein the distance H2 is the smallest, so the processor 120 will determine the unreliable point P22 'Move to the wall line segment L22 corresponding to the minimum (for example, distance H2). The moving path of the unreliable point is, for example, along the direction of the aforementioned distance. However, as long as the unreliable point can move to the wall line segment, the embodiment of the present invention does not limit the moving path of the unreliable point.

In addition, each of the distances H1 - H3 is, for example, the shortest distance between one of the unreliable points P22 and the corresponding wall line segment, such as the vertical distance. Taking the unreliable point P22' as an example, the distance H1 is the shortest distance between the unreliable point P22' and the wall line segment L21, the distance H2 is the shortest distance between the unreliable point P22' and the wall line segment L22, and the distance H3 is the unreliable The shortest distance between point P22' and wall line segment L23.

In addition, the processor 120 is used to: move at least one of all the unreliable points P22 to the corresponding wall line segment in the same manner as described above (the method of moving the unreliable point P22' to the corresponding wall line segment). In one embodiment, when the minimum of several distances between an unreliable point (for example, unreliable point P22'') and several wall line segments is still greater than a preset value, it means that this unreliable point does not belong to any For the wall line segment, the processor 120 may not move the unreliable point P22 ″ to the wall line segment. In another embodiment, if there is only one wall line segment, the processor 120 may move at least one of all unreliable points P22 to the one wall line segment without additional distance calculation. In addition, for the reliable point P21 that is not located on the wall line segment (that is, the reliable point P21 that the wall line segment does not pass through), the same method can also be used to move to the corresponding wall line segment, but it is not necessary to move to the corresponding wall line segment.

As shown in FIG. 2E, after the processor 120 moves the reliable point P21 and the unreliable point P22 to the corresponding wall line segments, it can obtain a two-dimensional point cloud image M2', which includes wall line segments L21~L23 and several points located at The pixel points of the wall line segment L21~L23. In addition, the two-dimensional point cloud map M2' may also only include (display) the wall line segments L21~L23, but not include (not display) at least one of all reliable points P21 and unreliable points P22.

Steps S130-S160 of the image processing method in the aforementioned embodiments are completed on the 2D point cloud image M2, but in another embodiment, step S160 can also be completed on the 3D point cloud image M1. The following is an example of Figures 4A~4B.

Please refer to Figures 4A~4B and 5. Figure 4A shows the schematic diagram of the wall line segment L21~L23 in Figure 2C converted into the wall plane W1~W3 of the 3D point cloud image M1, and Figure 4B shows the schematic diagram of Figure 4A. A schematic diagram of moving the unreliable point P22 to the corresponding wall planes W1 - W3 , and FIG. 5 shows a flow chart of another image processing method of the electronic device 100 in FIG. 1 .

The image processing method of the electronic device 100 in this embodiment includes steps S110-S150 and S260-S280, wherein the steps S110-S150 have been described above, and will not be repeated here, and the description starts from step S260.

In step S260 , as shown in FIG. 4A , the processor 120 converts the wall line segments L21 ˜ L23 from the 2D point cloud image M2 into several wall planes W1 ˜ W3 of the 3D point cloud image M1 . The wall plane is, for example, perpendicular to the plane where several pixels P2 of the two-dimensional point cloud image M2 are located, that is, the wall planes W1-W3 in FIG. 4A are perpendicular to the x-z plane of the two-dimensional point cloud image M2.

In step S270 , as shown in FIG. 4B , the processor 120 obtains the distances H1 - H3 between one of the unreliable points P22 and each wall plane W1 - W3 .

In step S280 , the processor 120 moves one of the unreliable points P22 to the wall plane corresponding to the smallest one of the distances H1 ˜ H3 . Taking the unreliable point P22' as an example, the distances between the unreliable point P22' and the three wall planes W1-W3 are respectively H1-H3, wherein the distance H2 is the smallest, so the processor 120 will determine the unreliable point P22 'Move to the wall plane W2 corresponding to the smallest one (for example, distance H2).

In addition, the processor 120 is used to: move at least one of all the unreliable points P22 to the corresponding wall plane in the same manner as described above (the method of moving the unreliable point P22' to the corresponding wall line segment). In one embodiment, when the minimum of several distances between an unreliable point (for example, unreliable point P22'') and several wall planes is still greater than a preset value, it means that this unreliable point does not belong to any The processor 120 may not move the unreliable point P22 ″ to the wall plane. In another embodiment, if there is only one wall plane, the processor 120 may move at least one of all the unreliable points P22 to the one wall plane without additional distance calculation. In addition, for the reliable point P21 that is not located on the wall plane (ie, the reliable point P21 that the wall plane does not pass through), the same method can be used to move to the corresponding wall plane, but it is not necessary to move to the corresponding wall plane. The embodiment of the present invention does not limit the numerical range of the “preset value”, which may depend on the frame content of the 3D point cloud image M1 captured by the electronic device 100 . In one embodiment, the "preset value" is, for example, a critical value that can be judged as a feature of the wall, such as any integer between 1 cm and 1 meter, but it can also be smaller than 1 cm or larger than 1 meter.

In addition, the distances H1-H3 in FIG. 4B are, for example, the shortest distances between the unreliable points P22 and the corresponding wall plane, such as vertical distances. Taking the unreliable point P22' as an example, the distance H1 is the shortest distance between the unreliable point P22' and the wall plane W1, the distance H2 is the shortest distance between the unreliable point P22' and the wall plane W2, and the distance H3 is the unreliable The shortest distance between point P22' and wall plane W3.

To sum up, the image processing method of the embodiment of the present invention and the electronic device using it select or determine the wall surface that can be classified as the wall surface from several straight line segments connected by several reliable pixel points of the three-dimensional point cloud image. segment, and move unreliable pixels to the selected wall segment. In this way, the problem of inaccurate detection caused by multipath interference can be improved.

To sum up, although the present invention has been disclosed by the above embodiments, it is not intended to limit the present invention. Those skilled in the art of the present invention can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be defined by the scope of the appended patent application.

100: Electronic device 110: camera 120: Processor H1~H3: Distance L1: straight line segment L21, L22, L23: wall line segment M1: 3D point cloud M2: 2D point cloud P1: pixel point P21: Reliable P22: Unreliable P2: pixel point RT: Transformation matrix S1: Length S110~S160, S260~S280: steps W1~W3: wall plane

FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. FIG. 2A is a schematic diagram of a three-dimensional point cloud image captured by the camera of the electronic device in FIG. 1 . Figures 2B-2E show schematic diagrams of the two-dimensional point cloud image of the three-dimensional point cloud image in Figure 2A. FIG. 3 shows a flow chart of the image processing method of the electronic device in FIG. 1 . FIG. 4A shows a schematic diagram of converting the wall line segment in FIG. 2C into a wall plane of a three-dimensional point cloud image. Fig. 4B shows a schematic diagram of moving the unreliable points in Fig. 4A to the corresponding wall plane. FIG. 5 shows a flow chart of another image processing method of the electronic device in FIG. 1 .

S110~S160: steps

Claims (10)

An image processing method, comprising: Retrieving a 3D point cloud image, the 3D point cloud image includes a plurality of pixel points; Distinguishing these pixel points into a plurality of reliable points and a plurality of unreliable points; From the reliable points, determine a plurality of straight line segments, each of which passes through two of the reliable points; Take one of the straight line segments as a wall line segment; and Move those unreliable points to the wall segment. The image processing method as described in claim 1, wherein each of the straight line segments has a length; the step of using the selected one of the line segments as the wall line segment includes: judging whether the length of each straight line segment is greater than a preset length value; and If the length is greater than the preset length value, the straight line segment whose length is greater than the preset length value is used as the wall line segment. The image processing method as described in claim 1, wherein the step of using the selected one of the straight line segments as the wall line segment includes: using the plurality of the selected ones of the straight line segments as the plurality of the walls Surface segment; Among them, the steps of moving these unreliable points to the wall line segment include: obtain a distance between one of the unreliable points and each of the wall line segments; and Move the one of the unreliable points to the wall line segment corresponding to the smallest one of the distances. The image processing method as described in claim 3, wherein each of the distances is the shortest distance between one of the unreliable points and the corresponding wall line segment. The image processing method as described in Claim 1, wherein in the step of capturing the 3D point cloud image, the image processing method further includes: converting the pixels of the 3D point cloud image into a 2D point cloud image; Among them, the step of distinguishing these pixel points, the step of determining these straight line segments, the step of using the selected one of these straight line segments as the wall line segment, and the step of moving these unreliable points to the wall line segment The steps are completed based on the two-dimensional point cloud image. The image processing method as described in claim 1, wherein the step of using the selected one of the straight line segments as the wall line segment includes: using the plurality of the selected ones of the straight line segments as the plurality of the walls Surface segment; The steps of moving these unreliable points to the corresponding wall line segment include: converting the wall line segments from the two-dimensional point cloud image into a plurality of wall planes of the three-dimensional point cloud image; a distance between one of the unreliable points and each of the wall planes; and Move the one of the unreliable points to the wall plane corresponding to the smallest one of the distances. The image processing method as described in Claim 6, wherein each of the distances is the shortest distance between one of the unreliable points and the corresponding wall plane. In the image processing method described in Claim 6, each wall plane is perpendicular to the plane where the pixels of the two-dimensional point cloud image are located. The image processing method as described in Claim 1, wherein the step of capturing the 3D point cloud image includes: using Time of Flight (ToF) technology to obtain the 3D point cloud image. An electronic device comprising: a camera for: Retrieving a 3D point cloud image, the 3D point cloud image includes a plurality of pixel points; and a processor for: Distinguish these pixel points as a plurality of reliable points and a plurality of unreliable points; From the reliable points, determine a plurality of straight line segments, each of which passes through two of the reliable points; take one of the straight line segments as a wall line segment; and Move those unreliable points to the wall segment.

Images