CN112819877B

CN112819877B - Laser line point cloud generation method and device and computer readable storage medium

Info

Publication number: CN112819877B
Application number: CN202110035554.5A
Authority: CN
Inventors: 李楚翘; 邓亮; 陈先开; 冯良炳
Original assignee: Shenzhen Cosmosvision Intelligent Technology Co ltd
Current assignee: Shenzhen Cosmosvision Intelligent Technology Co ltd
Filing date: 2021-01-12
Publication date: 2024-07-09
Anticipated expiration: 2041-01-12

Abstract

The application provides a laser line point cloud generation method and device and a computer readable storage medium, and belongs to the technical field of point cloud generation in industrial machine vision. According to the embodiment of the application, at least 2N+1 shot laser lines are sequentially exposed from high to low by using three exposure times of long, medium and short; extracting laser line pixel points of each acquired image; dividing the laser line into a dark area point set and a bright area point set; then classifying the laser sections into standard sections, over-wide sections and sparse sections; and restoring the laser line; and converting the laser line into point cloud through a coordinate conversion relation to generate point cloud information. Therefore, the application can cope with more complex illumination operating environment by exposing and imaging the laser line for a plurality of times; the laser lines are classified and individually corrected in a segmented and partitioned mode, so that more accurate laser center line coordinates can be obtained, and more accurate point clouds can be obtained.

Description

Laser line point cloud generation method and device and computer readable storage medium

Technical Field

The present invention relates to the field of point cloud generation in industrial machine vision, and in particular, to a method and apparatus for generating a laser line point cloud, and a computer readable storage medium.

Background

Laser guided based weld tracking technology has found numerous applications in industrial welding, including a set of imaging systems consisting of lasers and cameras. The working principle of the system is as follows: firstly scanning a workpiece by using a laser, then shooting a laser line by using a camera to obtain the coordinates of the laser line on an image, and finally converting the coordinates of the laser line image into point cloud by using the calibration relation of the camera and the robot.

However, the conventional method for generating the point cloud by using the laser line needs to adjust the exposure of the camera according to the irradiation of the scene (workpiece and background), so that the imaging of the laser line can be clear and accurate. However, the existing method faces the problems: when the field Jing Fuzhao spans a complex environment with a large span (i.e., a large light-dark gap), a single exposure cannot obtain clear laser line imaging in both bright and dark areas, and cannot obtain an accurate point cloud.

Disclosure of Invention

In view of the above, the present invention aims to provide a method and a device for generating laser line point cloud, and a method for generating laser line point cloud on a computer readable storage medium, which aims to solve the problems that laser line imaging is unclear and accurate point cloud cannot be obtained in a complex environment.

The technical scheme adopted by the invention for solving the technical problems is as follows:

the first aspect of the invention provides a laser line point cloud generating method, which comprises the following steps:

exposing at least 2N+1 shot laser lines sequentially from high to low by using three exposure times of long, medium and short; n is a natural number;

extracting laser line pixel points of each acquired image;

segmenting a laser line, and particularly dividing the laser line into a dark area point set and a bright area point set;

classifying the laser segments;

restoring the laser line;

And converting the laser line into point cloud through a coordinate conversion relation to generate point cloud information.

In some embodiments, the step of segmenting the laser line into a set of dark and a set of light spots comprises:

Traversing each point in the laser point set cell _i, adding the point into the bright area point set list1 if the current point p (x, y) is communicated with any point eight adjacent points in the point set, or adding the dark area point set list2 if the current point p (x, y) is not communicated with any point eight adjacent points in the point set;

Corroding the connected domain in the bright area point set list1, and if the number of pixels in the connected domain is too small after corrosion, moving the points in the connected domain into the dark area point set list2;

Performing expansion operation on the connected domain in the bright area point set list 1; the boundary of the connected domain expands outwards after expansion, and if the expanded pixel overlaps with a point in the dark region point set list2, the overlapped pixel is moved from the dark region point set list2 to the bright region point set list1.

In some embodiments, the classifying the laser segments includes:

Classifying the bright area point set list1, calculating the width of each light band in the list1, marking the segment as a standard segment if the width is smaller than a width threshold value, otherwise marking the segment as an oversized segment;

And classifying the dark area point sets list2, calculating the density of each segment point set in the list2, and marking the segment as a standard segment if the density is larger than a density threshold value, otherwise marking the segment as a sparse segment.

In some embodiments, the dark region point set list2 classification includes the steps of:

Dividing discrete points in the dark area point set list2 into a plurality of point sets seti, wherein one point set represents one light band;

Sorting discrete points in the dark area point set list2 from small to large according to x coordinates, and dividing points with the distance of less than n pixels in the x direction into the same point set;

Calculating the point set density of each section of light band;

optical tape volume = optical tape length × optical tape width

Optical tape length length_set= coordinate _xmax-coordinate _xmin

Optical tape width width_set= coordinate _ymax-coordinate _ymin

Wherein the maximum x-coordinate coordinate _xmax of each point set, the minimum x-coordinate coordinate _xmin; maximum y-coordinate coordinate _ymax and minimum y-coordinate coordinate _ymin;

Setting a density threshold value th_density, marking the current set as a standard segment if the density of the set is greater than the density threshold value, otherwise marking the set as a sparse segment;

Extracting a neutral line from a standard segment in an imaging image img _n+1 of a laser line to be repaired currently; and adding the midline to the final set of laser line pixels.

In some embodiments, the restoring laser line comprises:

Restoring sparse segments in the imaging image img _n+1 of the currently needed repair laser line;

The over-wide segment of the imaged image img _n+1 of the currently required repair laser line is adjusted.

In some embodiments, the restoring sparse segments in the imaged image img _n+1 of the currently-needed repair laser line includes:

extracting a standard segment central line in an img ₁～img_n dark region point set list2, and taking a point set with higher density as a standard segment if the dark region point sets of a plurality of pictures are all standard segments;

extracting a line in a light band;

lowering the gray threshold of img _n+1, re-acquiring a laser point set cell _n+1, determining a sparse segment and calculating the point density of the sparse segment;

Calculating the distance between the point of the laser point set cell _n+1 and each point on the central line, calculating the minimum distance, comparing and judging the minimum distance with a distance threshold value, and adding the point into a corresponding sparse segment set _n+1 if the minimum distance is smaller than the distance threshold value;

re-calculating the density of the sparse segment set _n+1, and adding set _n+1 into the standard point set if the standard is reached;

If the density does not reach the standard, the gray threshold is lowered again, and the steps of determining the sparse segment and calculating the dot density of the sparse segment are repeated.

In some embodiments, the adjusting the over-width segment of the imaging image img _n+1 of the currently required repair laser line includes:

Selecting a standard segment of a bright area in img _n+2～img_2n+1 as a reference segment, and selecting a thinner light band as the reference segment if the bright areas of the multiple images are all standard segments;

Extracting a midline _ref of the reference standard segment; correcting the midline of the oversized section by the midline of the standard section

Setting the gray value of the reference segment point to 255, and solving a neutral line _bias again;

Solving a distance bias _distance from a midline _ref to a line _bias of the reference segment;

The centerline _n+1 is corrected using the offset distance bias _distance to obtain a new centerline _n+1.

In some embodiments, the method of extracting an optical tape centerline comprises:

for a two-dimensional image, a black-filled Hessian matrix describes the two-dimensional derivative of each point in the principal direction, for any point p (x, y) in the set of light band points seti, its Hessian matrix can be expressed as Wherein rxrx represents the second order bias of the point along the X direction, ry represents the second order bias of the point along the Y direction;

The eigenvector (n _x,n_y) corresponding to the maximum eigenvalue lambda of the Hessian matrix is the normal direction of the light band;

With point (x 0, y 0) as the datum point, the subpixel (px, py) at the center of the band. Assuming that there is a coefficient t such that (px, py) = (x0+ tnx, y0+ tny)

In the formula

Wherein rx, ry is the x, y direction partial derivative, rxx, ryy is the second partial derivative, rxy is the mixed partial derivative;

when |t| is less than 0.5, (x 0, y 0) is a point on the midline.

The second aspect of the present invention also provides a laser line point cloud generating device, the device executing the above laser line point cloud generating method, the device comprising: the device comprises a laser line acquisition module, a pixel point acquisition module, a laser line segmentation module, a laser segment classification module, a laser line restoration module and a point cloud generation module;

the laser line acquisition module is used for exposing at least 2N+1 shot laser lines from high to low in sequence by using three exposure times of long, medium and short; n is a natural number;

The laser line segmentation module is used for segmenting the laser line into a dark area point set and a bright area point set;

The laser segment classification module is used for classifying the laser segments;

The laser line restoration module is used for restoring the laser line;

The point cloud generation module is used for converting the laser line into point cloud through a coordinate conversion relation to generate point cloud information.

In some embodiments, the laser line segmentation module comprises a point set classification unit, a new point set acquisition unit and an expansion operation unit;

the point set classification unit is used for classifying a bright area point set list1 and a dark area point set list2;

the new point set acquisition unit is used for carrying out morphological corrosion on the dark area point set, splitting the pixel blocks adhered in the dark area to obtain a new bright area point set and a new dark area point set;

The expansion operation unit is used for carrying out expansion operation on the connected domain in the bright area point set list1 and the dark area point set list2 obtained after updating; the boundary of the connected domain expands outwards after expansion, and if the newly added expanded pixel overlaps with a point in the dark region point set list2, the overlapped pixel is moved from the dark region point set list2 to the bright region point set list1.

In some embodiments, the laser line restoration module includes a sparse segment restoration unit, an oversized segment adjustment unit, and a midline extraction unit;

The sparse segment restoration unit is used for restoring the sparse segment in the imaging image img _n+1 of the laser line to be restored currently;

The over-width section adjusting unit is used for adjusting an over-width section of an imaging image img _n+1 of the laser line to be repaired currently;

And the midline extraction unit is used for extracting a midline _ref of the reference section and correcting the midline of the excessively wide section through the midline of the standard section.

The application also provides a computer readable storage medium comprising a processor, a computer readable storage medium and a computer program stored on the computer readable storage medium, which when executed by the processor, performs the steps of the method as described above.

The laser line point cloud generation method, the laser line point cloud generation device and the computer storage medium provided by the embodiment of the application sequentially expose at least 2N+1 shot laser lines from high to low by using three exposure times of long, medium and short; extracting laser line pixel points of each acquired image; dividing the laser line into a dark area point set and a bright area point set; then classifying the laser sections into standard sections, over-wide sections and sparse sections; and restoring the laser line; and converting the laser line into point cloud through a coordinate conversion relation to generate point cloud information. Therefore, the application can cope with more complex illumination operating environment by exposing and imaging the laser line for a plurality of times; the laser lines are classified and individually corrected in a segmented and partitioned mode, so that more accurate laser center line coordinates can be obtained, and more accurate point clouds can be obtained.

Drawings

FIG. 1 is a flow chart of a method for generating a laser line point cloud according to an embodiment of the present invention;

fig. 2 is a distribution diagram of a laser area of a laser line point cloud generating method according to an embodiment of the present invention;

FIG. 3 is a flow chart of a method for dividing a laser line into a dark area point set and a bright area point set according to the laser line point cloud generating method provided by the embodiment of the invention;

FIG. 4 is a flow chart of a method for classifying laser segments of a laser line point cloud generating method according to an embodiment of the present invention;

FIG. 5 is a flowchart of a method for classifying and estimating a dark area point set list2 of a laser line point cloud generating method according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of classification of a dark area point set list2 of a laser line point cloud generating method according to an embodiment of the present invention;

fig. 7 is a schematic diagram of laser points with gray threshold th_gray=128 according to an embodiment of the present invention;

fig. 8 is a schematic diagram of a laser spot that appears when the gray threshold th_gray=80 is lowered according to an embodiment of the present invention;

FIG. 9 is a flow chart of a method for restoring laser lines according to an embodiment of the present invention;

FIG. 10 is a flowchart of a method for restoring sparse segments in img3 according to one embodiment of the invention;

FIG. 11 is a schematic view of a line in an extracted optical tape according to an embodiment of the present invention;

FIG. 12 is a flowchart of a method for adjusting the over-width segment of the imaging image img3 of the currently required repair laser line according to an embodiment of the invention;

FIG. 13 is an excessively wide section schematic diagram of a laser line point cloud generating method according to an embodiment of the present invention;

fig. 14 is a standard segment schematic diagram of a laser line point cloud generating method according to an embodiment of the present invention;

Fig. 15 is a schematic diagram of a sparse segment of a laser line point cloud generating method according to an embodiment of the present invention;

FIG. 16 is a block diagram illustrating an embodiment of a laser line point cloud generating device according to an embodiment of the present invention;

Fig. 17 is a block diagram of another embodiment of a laser line point cloud generating device according to an embodiment of the present invention.

Detailed Description

In order to make the technical problems, technical schemes and beneficial effects to be solved more clear and obvious, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the particular embodiments described herein are illustrative only and are not limiting upon the invention.

In the following description, suffixes such as "module", "component", or "unit" for representing elements are used only for facilitating the description of the present invention, and have no specific meaning per se. Thus, "module," "component," or "unit" may be used in combination.

The terminal may be implemented in various forms. For example, the terminals described in the present invention may include mobile terminals such as a mobile phone, a tablet computer, a notebook computer, a palm computer, a Personal digital assistant (Personal DIGITAL ASSISTANT, PDA), a Portable media player (Portable MEDIA PLAYER, PMP), a navigation device, a wearable device, a smart bracelet, a pedometer, and the like, as well as fixed terminals such as a digital TV, a desktop computer, and the like.

Embodiment one:

The application provides a laser line point cloud generation method, referring to fig. 1, comprising the following steps:

s10, sequentially exposing at least 2N+1 shot laser lines from high to low by using three exposure times of long, medium and short according to a certain gradient;

Specifically, at least 2n+1 shot laser lines are sequentially exposed from high to low according to a certain gradient by using three exposure times of long, medium and short, and N belongs to a natural number. In this embodiment, n=2 will be taken as an example for explanation. And sequentially exposing at least 5 shot laser lines Q _n-2,Q_n-1,Q_n,Q_n+1,Q_n+2 from high to low by using three exposure times of long, medium and short, sequentially reducing the exposure time of the laser lines of the 5 exposure times Q _n-2,Q_n-1,Q_n,Q_n+1,Q_n+2 according to a certain gradient to respectively obtain corresponding images img _n-2,img_n-1,img_n,img_n+1,img_n+2, wherein img _n is an imaging image of the laser line which needs to be repaired currently, Q _n is the target shot laser line which needs to be repaired currently, and n is more than or equal to 3.

S12, extracting laser line pixel points of each acquired image;

In particular, the laser lines have a higher energy and higher gray values on the image relative to other light sources in the scene. Setting a gray threshold value th_gray, traversing pixels on the image, and adding pixels with gray values higher than the gray threshold value th_gray into the laser point set. In this embodiment, a laser spot set cell _n-2,cell_n-1,cell_n,cell_n+1,cell_n+2 corresponding to 5 images img _n-2,img_n-1,img_n,img_n+1,img_n+2 is obtained respectively.

S14, segmenting the laser line, and particularly dividing the laser line into a dark area point set and a bright area point set.

Specifically, as shown in fig. 2, the distribution diagram of the laser area in the embodiment of the application is shown, the laser points are more discrete in the dark area and more densely distributed in the bright area. Referring to fig. 3, the step of dividing the laser line into a dark area point set and a bright area point set specifically includes:

S141, taking a laser point set cell _i＝{p₀,p₁...p_n as an example for illustration, if the current point p (x, y) is communicated with any point eight adjacent to the point in the point set, adding the point into a bright area point set list1, otherwise, adding a dark area point set list2;

Taking a laser point set cell _i＝{p₀,p₁...p_n as an example to illustrate a specific method for segmenting a laser line, traversing each point in the laser point set, taking i=3 as an example to illustrate that if a laser point exists in the eight neighborhood of the current point p (x, y), adding the point into a bright area point set list1, otherwise adding into a dark area point set list2, wherein the cell _i is the laser point set; p _n is the laser point where the current position is located, and both i and n belong to natural numbers. The following table is a schematic representation of the current point p (x, y) image coordinates and eight neighborhood point connections.

(x-1,y-1)	(x,y-1)	(x+1,y-1)
			(x-1,y)	P(x,y)	(x+1,y)
(x-1,y+1)	(x,y+1)	(x+1,y+1)

The program code for dividing a laser line into a set of dark and a set of light spots is expressed as follows:

S142, splitting pixel blocks adhered to the dark area point set, corroding the connected domain in the bright area point set list1, and if the number of pixels in the connected domain is too small after corrosion, moving the points in the connected domain into the dark area point set list2.

Specifically, after classifying the points, the bright area point set list1 is composed of several connected domains, i.e., list1= { area0, area1.. areai }, where small connected domains of dark areas are also included, which now need to be found out and re-classified to the dark area point set list2. The morphological corrosion can reduce the range of the target area, reduce the number of pixel points, and eliminate the boundary points of the area at the same time so as to separate the adhered connected areas. Therefore, the number of pixels in the connected domain after etching can be used to distinguish the small connected domain in the dark region. And corroding the connected domain in the bright area point set list1, and splitting the pixel blocks adhered with the dark area to obtain a new bright area point set and a new dark area point set.

The corrosion of the region A by the structural element B is recorded as

And scanning each connected domain in the bright area point set list1 by taking the structural element B as a window template and step=1 as a step length, if the structural element B and the part of the connected domain at the current position completely belong to the bright area point set list1, reserving the point at the position, otherwise deleting. After corrosion of the bright area point set list1, a new bright area point set and a connected domain list 1_union= { area_ero_0, area_ero_1. Setting a pixel number threshold value th_ numb, and if the pixel number of the connected domain of list1_Erosion is smaller than th_ numb, moving the point in the original connected domain from the bright area point set list1 to the dark area point set list2.

The code procedure for morphologically corroding the bright and dark sets of points list1 and list2 is as follows:

s143, detecting burr pixels of the bright area point set, and performing expansion operation on the connected domain in the bright area point set list 1; after expansion, the boundary of the connected domain expands outwards, and if the expanded pixel overlaps with the point set in the dark area point set list2, the overlapped pixel is moved from the dark area point set list2 to the bright area point set list1.

Specifically, edge burr pixels of the bright area laser line are determined to be discrete points added to the dark area point set list2 during segmentation, and the burr pixels are found out and moved into corresponding connected domains of the bright area point set list 1. The updated bright area point set list1 and the updated dark area point set list2 are obtained in the last step, the connected domain in the bright area point set list1 is expanded, the boundary of the expanded connected domain is expanded outwards, and if newly added pixels are overlapped with the dark area point set list2, the overlapped pixels are moved into the connected domain corresponding to the bright area point set list1 from the dark area point set list 2. For example, if the point p is a point obtained by expanding area3 and belongs to the dark area point set list2 at the same time, the point p is added to area3.

Expansion of the A region using structure B is denoted as

And scanning each connected domain in the bright area point set list1 by taking the structural element B as a window template and step=1 as a step length, and adding the pixel at the current position into the expansion point set dilation _list if the intersection between the current position B and the connected domain is not empty. After the scanning is finished, the expansion point set is compared with the dark area point set list2, and if the points in the expansion point set belong to the dark area point set list2 at the same time, the points are moved from the dark area point set list2 to the bright area point set list1.

The code for expanding the updated bright area point set list1 and the updated dark area point set list2 is as follows:

s16, classifying laser segments;

Judging whether a bright area point set list1 and a dark area point set list2 of each image meet the standard, and directly adopting a standard segment in an imaging image img3 of a laser line to be repaired at present as the laser line, and carrying out next deviation correction on a non-standard segment.

Referring to fig. 4, the laser segment classification specifically includes the following method steps:

S161, classifying a bright area point set list1, calculating the width of each light band in the list1, marking the segment as a standard segment if the width is smaller than a width threshold value, otherwise marking the segment as an oversized segment;

Specifically, a width threshold value th_width is set, if the width of a certain section of light band of the bright area point set list1 is larger than the width threshold value, the section of connected domain is marked as an oversized section, and if the width is smaller than the threshold value, the section of light band is marked as a standard section.

Calculating the width of the light band:

The current band areai has a maximum y-coordinate coordinate _ymax and a minimum y-coordinate coordinate _ymin, then the band width width_area= coordinate _ymax-coordinate _ymin.

Setting a width threshold value, having area, if the width of the optical tape, having area, is greater than the width threshold value, having area, the optical tape is marked as an over-wide segment, otherwise the standard segment.

S162, classifying the dark area point set list2, calculating the density of each segment point set in the list2, marking the segment as a standard segment if the density is larger than a density threshold value, otherwise marking the segment as a sparse segment;

and setting a density threshold value th_density, if the density of a certain segment point set of the dark region point set list2 is smaller than the density threshold value th_density, marking the segment point set as a sparse segment, and if the density is higher than the density threshold value th_density, marking the segment point set as a standard segment.

Specifically, referring to fig. 5 and 6, the classification of the dark region list2 includes the steps of:

S1621, firstly, dividing the discrete points in the dark area point set list2 into a plurality of point sets seti, wherein one point set represents a light band, and i is a natural number.

S1622, sorting the discrete points in the dark area point set list2 from small to large according to the x coordinate, and dividing the points with the x direction distance smaller than n pixels into the same point set.

The maximum x-coordinate coordinate _xmax and the minimum x-coordinate coordinate _xmin, and the maximum y-coordinate coordinate _ymax and the minimum y-coordinate coordinate _ymin of each set of points set are simultaneously noted. Ordered points [ p0 (x 0, y 0), p1 (x 1, y 1), p2 (x 2, y 2) ] pn (xn, yn) ], the x-coordinate of the ordered points [x₀,x₁,x₂...x_n],x₀<x₁<x₂...x_n-1<x_n

Pstart with p0 as starting point pstart =p0

The dark region point set can be expressed as list2= { set0, set1, set2.. seti }

S1623, calculating the point set density of each section of the light band;

Point set Density

Optical tape volume = optical tape length × optical tape width

Optical tape length length_set= coordinate _xmax-coordinate _xmin

Optical tape width width_set= coordinate _ymax-coordinate _ymin

Wherein the maximum x-coordinate coordinate _xmax of each point set, the minimum x-coordinate coordinate _xmin; maximum y-coordinate coordinate _ymax and minimum y-coordinate coordinate _ymin.

S1624, setting a density threshold th_density, marking the current set as a standard segment if the density of the set is greater than the density threshold, otherwise marking the set as a sparse segment.

S1625, extracting a neutral line from the imaging image img _n+1 of the laser line to be repaired currently, namely a standard segment in the intermediate exposure image, and adding the final laser line pixel point set laser_pixel into the neutral line.

In this embodiment, img _n+1 is img3 when n=2, and the method for extracting the neutral line is described in detail in step S1812 below, which is not described here again.

Img3 is an example of an imaging image of a laser line currently in need of repair in an embodiment of the present application.

S18, restoring the laser line

Referring to fig. 9, the restoration laser line specifically includes the following steps:

S181, restoring sparse segments in an imaging image img _n+1 of the laser line to be restored currently;

If there is a sparse segment in img3, the gray threshold th_gray of img3 needs to be lowered so that more laser spots appear in the sparse segment, but at the same time, lowering the gray threshold brings about noise, at which time the newly appearing noise spots need to be constrained by the standard segments of other exposed images. As shown in fig. 7, which is a laser spot where th_gray=128 occurs. Now the gray threshold is th_gray=80, and noise occurs as shown in fig. 8.

Referring to fig. 10 to 15, the restoring the sparse segment in img _n+1 specifically includes the following steps:

s1811, extracting a standard segment central line in an img ₁～img_n dark region point set list2, and taking a higher point set density as a standard segment if dark region point sets of a plurality of images are standard segments;

in this embodiment, img _n+1 is img3 when n=2, specifically, firstly, extracting the standard segment center line in img2 or img1 dark region point set list2, and if the dark regions of img2 and img1 are both standard segments, adopting the standard segment with higher point set density.

S1812, extracting a light band centerline;

for a two-dimensional image, a black-filled Hessian matrix describes the two-dimensional derivative of each point in the principal direction, for any point p (x, y) in the set of light band points seti, its Hessian matrix can be expressed as Where rx rx represents the second order bias of the point along the X direction, ry represents the second order bias of the point along the Y direction, and the other terms have similar meaning.

The eigenvector (n _x,n_y) corresponding to the maximum eigenvalue λ of the Hessian matrix is the normal direction of the light band.

In the formula

when |t| is less than 0.5, (x 0, y 0) is a point on the midline.

S1813, reducing the gray threshold of img _n+1, re-acquiring the laser point set cell _n+1, determining the sparse segment and calculating the point density of the sparse segment

Specifically, lowering the gray threshold of img3, re-acquiring a laser point set cell3, determining a sparse segment, and calculating the point density of the sparse segment;

after the gray threshold of the imaging image img3 of the laser line to be repaired at present is reduced, adding points larger than the gray threshold into the laser point set cell3, wherein the laser point set cell3 comprises bright area points, dark area points and noise points. The point of cell3 close to the midline (midline of img2 or img 1) is considered to be the newly added point on the img3 sparse segment, and the point far from the midline is considered to be the invalid point.

S1814, calculating the distance between the point of the laser point set cell _n+1 and each point on the central line, solving the minimum distance, comparing and judging the minimum distance with a distance threshold, and adding the point into a corresponding sparse segment set _n+1 if the minimum distance is smaller than the distance threshold;

Specifically, in this embodiment, the distance between the point of the cell3 and the point on the central line is calculated, the minimum distance is compared with the distance threshold value, and if the minimum distance is smaller than the distance threshold value, the point is added into the corresponding sparse segment set3;

specifically, a distance threshold value th_distance is set, the distance between the point of the cell3 and the point on the central line is calculated, the minimum distance is obtained, and if the minimum distance is smaller than the distance threshold value th_distance, the point is added into the corresponding sparse segment set3.

S1815, recalculating the density of the sparse segment set _n+1, and adding set _n+1 into the standard point set if the standard is reached;

Re-calculating the density of the sparse segment set3, and adding the set3 into the standard point set cell_laser if the standard is reached; if the density does not reach the standard, the gray threshold is lowered again, and the steps S1813-S1815 are repeated.

And (3) calculating the dot density of the sparse segment again, and if the dot density reaches the standard, marking the segment as the standard segment and adding the standard segment into the standard dot set cell_laser. And extracting the center line of the new standard segment, and adding the final laser line pixel point set laser_pixel.

S182, adjusting an excessively wide section of an imaging image img _n+1 of the laser line to be repaired currently;

When n=2, img _n+1 is img3, and if img3 has an excessively wide segment, the bright standard segment of img4 or img5 is used for adjustment.

The pixels of the over-wide section are basically in an over-exposure state, and an accurate light band center line cannot be obtained through the change of gray values. The adjusting of the over-wide section of the imaging image img3 of the laser line currently required to be repaired specifically comprises the following steps:

S1821, selecting a standard segment of a bright area in img _n+2～img_2n+1 as a reference segment, and selecting a thinner light band as the reference segment if the bright areas of the multiple images are all standard segments;

specifically, in this embodiment, the standard segment of the bright area in img4 or img5 is selected as the reference segment, and if both standard segments are used, the standard segment with finer light band is selected as the reference standard segment.

S1822, extracting a midline _ref (such as a black line in the middle of the white laser line in FIG. 12) of the reference standard segment, correcting the midline of the over-wide segment by the midline of the standard segment, and the midline extraction method is the same as the midline extraction method in the step S1812, which is not described herein.

S1823, setting the gray value of the reference segment point to 255, and obtaining a neutral line _bias again.

S1824, solving the distance bias _distance from the line _ref to the line _bias of the middle line of the reference segment.

Assuming that the principal direction of the laser line on the image is lateral (i.e., the angle between the laser line and the x-axis of the image is less than the angle between the laser line and the y-axis of the image), the mapping point of line _ref on line _bias can be found. And (3) finding a point in line _bias, which is identical to the x coordinate of line _ref, according to the x-axis coordinate reference of the line _ref point, calculating the difference of the y coordinates of the two points as bias _distance, and if the point, which is identical to the x-axis coordinate, does not exist in line _bias, then corresponding the x coordinate in line _bias to the nearest point.

line_ref＝{p_ref0,p_ref1....p_refi}

line_bias＝{p_bias0,p_bias1....p_biasi}

Ifp_biasi.x＝p_refi.x

{

bias_distance＝p_refi.y-p_biasi.y

}

Else

{

If line _bias does not have the same point as the p _refi x coordinate

It is assumed that p _biasi-1. X is closest to p _refi. X

bias_distance＝p_refi.y-p_biasi-1.y

}

P _refi is the point on midline _ref and p _biasi is the point on midline _bias.

The extraction method for obtaining the midline 3 of the img3 oversized section is the same as the midline extraction method, and is not described herein.

S1825, correct the centerline line _n+1 to obtain a new centerline line _n+1 using the offset distance bias _distance.

Specifically, the center line is corrected by the offset distance bias _distance obtained in the previous step. When n=2, line _n+1 is line3, and a new centerline line3 is obtained by adding an offset corresponding to bias _distance to the point of line3. The new centerline line3 point is added to the final laser line pixel point set laser_pixel.

line3＝{p0,p1,p2...pn}

bias_distance＝{bias0,bias1,bias2.....biasn}

For(int i＝0；i<line3.size()；++i)

{

pi.x＝pi.x+biasi；

}

S20, converting the laser line into point cloud through a coordinate conversion relation to generate point cloud information;

let the image coordinates of the point on the laser mid-line obtained in the previous step be (u, v, z), where z is the distance from the laser plane to the camera known.

Assume that the conversion relation transform _C2R = [ R, T ] of the camera coordinate system to the robot coordinate system has been obtained by calibration. The laser line can be converted into a point cloud, i.e

Thus, the laser line can be converted into the point cloud through the formula conversion.

According to the method, the laser lines are shot by using three exposure times of long, medium and short, so that clear laser line imaging of a dark area and a bright area is respectively obtained; dividing a laser line into a dark area point set and a bright area point set through a connected domain; then dividing the laser band into a sparse section, a standard section and an oversized section through a density threshold value and a width threshold value; obtaining denser laser points by reducing the gray threshold value, and then using the neutral line constraint noise of the high exposure standard section; and correcting the midline of the oversized section through the midline of the standard section, and generating a point cloud through a coordinate conversion relation. Therefore, the application can cope with more complex illumination operation environment by exposing and imaging the laser line for a plurality of times; the laser lines are classified and individually corrected in a segmented and partitioned mode, so that more accurate laser center line coordinates can be obtained, and more accurate point clouds can be obtained.

Embodiment two:

The embodiment of the invention provides a laser line point cloud generating device, which completes the generation of laser line point cloud by adopting the laser line point cloud generating method in the first embodiment, and can cope with more complicated illumination operating environment by exposing and imaging laser lines for a plurality of times; the laser lines are classified and individually corrected in a segmented and partitioned mode, so that more accurate laser center line coordinates can be obtained, and more accurate point clouds can be obtained.

Referring to fig. 16 and 17, the laser line point cloud generating device according to the embodiment of the present invention includes a laser line obtaining module 301, a pixel point obtaining module 302, a laser line segmentation module 303, a laser segment classification module 304, a laser line restoration module 305, and a point cloud generating module 306.

The laser line acquiring module 301 is configured to sequentially expose at least 2n+1 shot laser lines from high to low with three exposure times, where N is a natural number.

The pixel point obtaining module 302 is configured to extract a laser line pixel point of the obtained image.

The laser line segmentation module 303 is configured to segment a laser line into a dark area point set and a bright area point set.

The laser line segmentation module 303 includes a point set classification unit 3031, a new point set acquisition unit 3032, and an expansion operation unit 3033.

The point set classification unit 3031 is configured to classify a bright area point set list1 and a dark area point set list2;

Specifically, if there is a laser spot within the eight neighbors of the current point p (x, y), then the point is added to the bright region point set list1, otherwise, the dark region point set list2 is added.

The new point set obtaining unit 3032 is configured to perform morphological corrosion on the dark area point set, and split the pixel blocks adhered to the dark area to obtain a new bright area point set and a new dark area point set.

Specifically, after classifying the points by the point set classifying unit 3031, the bright area point set list1 is composed of a plurality of connected domains, i.e., list1= { area0, area1.. areai }, wherein the small connected domain of the dark area is also included, and now the small connected domain of the dark area needs to be found out and re-classified to the dark area point set list2. The morphological corrosion can reduce the range of the target area, reduce the number of pixel points, and eliminate the boundary points of the area at the same time so as to separate the adhered connected areas. Therefore, the number of pixels in the connected domain after etching can be used to distinguish the small connected domain in the dark region.

The corrosion of the region A by the structural element B is recorded as

And scanning each connected domain in the list1 by taking the structural element B as a window template and step=1 as a step length, if the structural element B and the part of the connected domain at the current position completely belong to the list1, reserving the point at the position, otherwise deleting. After corrosion of the bright area point set list1, a new bright area point set and a connected domain list 1_union= { area_ero_0, area_ero_1. Setting a pixel number threshold value th_ numb, and if the pixel number of the connected domain of list1_Erosion is smaller than th_ numb, moving the point in the original connected domain from the bright area point set list1 to the dark area point set list2.

The expansion operation unit 3033 is configured to perform expansion operation on the connected domain in the list1 by using the updated bright area point set list1 and the updated dark area point set list 2; the boundary of the connected domain expands outwards after expansion, and if the newly added expanded pixel overlaps with a point in the list2, the overlapped pixel is moved from the dark area point set list2 to the bright area point set list1.

Specifically, edge burr pixels of the bright area laser line are determined to be discrete points added to list2 at the time of segmentation, and the burr pixels are found out and moved into corresponding connected domains of the bright area list1. And (3) obtaining updated bright area list1 and dark area list2 in the last step, expanding the connected domain in the list1, expanding the boundary of the expanded connected domain outwards, and if the newly added pixel is overlapped with the list2, moving the overlapped pixel from the list2 to the list1. For example, if the point p is the point obtained by expanding area3 and belongs to list2 at the same time, p point is added to area3.

Expansion of the A region using structure B is denoted as

The laser segment classification module 304 is configured to classify laser segments. The laser segment classification module 304 includes a bright region classification unit 3041 and a dark region classification unit 3042.

The bright region classifying unit 3041 is configured to classify the bright region point set list1, calculate the width of each light band in the list1, and mark the segment as a standard segment if the width is smaller than a width threshold, otherwise mark the segment as an oversized segment.

Specifically, a width threshold value th_width is set, if the width of a certain section of light band of the bright area point set list1 is larger than the width threshold value, the section of connected domain is marked as an oversized section, and if the width is smaller than the threshold value, the section of connected domain is marked as a standard section.

Calculating the width of the light band:

The dark region classification unit 3042 is configured to classify the dark region point set list2, calculate the density of each segment point set in the list2, and mark the segment as a standard segment if the density is greater than a density threshold, otherwise, mark the segment as a sparse segment.

Specifically, a density threshold value th_density is set, if the density of a certain segment point set of the dark region point set list2 is smaller than the density threshold value th_density, the segment point set is marked as a sparse segment, and if the density is higher than the density threshold value th_density, the segment point set is marked as a standard segment.

The laser line restoration module 305 is configured to restore a laser line.

The laser line restoration module 305 includes a sparse segment restoration unit 3051, an over-wide segment adjustment unit 3052, and a midline extraction unit 3053.

The sparse segment restoration unit 3051 is configured to restore a sparse segment in an imaging image img _n+1 of a laser line currently required to be restored;

The method specifically comprises the following steps:

In this embodiment, when n=2, the standard segment center line in the dark region point set list2 of img2 or img1 is first extracted, and if the dark regions of img2 and img1 are both standard segments, the standard segment with higher point set density is adopted.

Extracting a light band center line;

For a two-dimensional image, the Hessian matrix describes the two-dimensional derivative of each point in the principal direction, and for any point p (x, y) in the set of light band points seti, its Hessian matrix can be expressed as Where rx rx represents the second order bias of the point along the X direction, and the other terms are similar in meaning. The eigenvector (n _x,n_y) corresponding to the maximum eigenvalue λ of the Hessian matrix is the normal direction of the optical band.

In the formula

when |t| is less than 0.5, (x 0, y 0) is a point on the midline.

In this embodiment, n=2, that is, after the gray threshold of the imaging image img3 of the laser line to be repaired is reduced, the point greater than the gray threshold is added to the point set cell3, where the cell3 includes the bright area point, the dark area point and the noise point. The point of cell3 near the midline (midline of img2 or img 1) is considered to be the point of the new addition on the sparse segment of img3, and the point far from the midline is considered to be the null point.

specifically, a distance threshold value th_distance is set, the distance between the point of the cell3 and the point on the central line is calculated, the minimum distance is obtained, and if the minimum distance is smaller than the distance threshold value th_distance, the point is added into the corresponding sparse segment.

And calculating the dot density of the sparse segment again, and marking the segment as a standard segment if the standard criterion is met. Extracting the center line of the new standard segment, and adding the final laser line pixel point set laser_pixel.

The too-wide section adjusting unit 3052 is configured to adjust an image img3 too-wide section of the laser line to be repaired currently;

the method specifically comprises the following steps: selecting a standard segment of a bright area in img _n+2～img_2n+1 as a reference segment, and selecting a thinner light band as the reference segment if the bright areas of the multiple images are all standard segments;

In this embodiment, if img3 has an excessive segment, the adjustment is performed by using the bright standard segment of img4 or img5 as the reference segment.

The pixels of the over-wide section are basically in an over-exposure state, and an accurate light band center line cannot be obtained through the change of gray values. The standard segment of the bright area in img4 or img5 is selected as a reference segment, and if the standard segments are both standard segments, a thinner light band is selected as the reference segment.

And the midline extraction unit 3053 extracts a midline _ref of the reference segment, and corrects the midline of the over-width segment through the midline of the standard segment.

Specifically, the gray value of the reference segment point is set to 255, and the centerline line _bias (yellow dotted line in the lower diagram) is found again

The distance bias _distance (blue dashed line) from the midline _ref to line _bias of the reference segment is found. Assuming that the principal direction of the laser line on the image is lateral (i.e., the laser line is at a smaller angle to the x-axis of the image than the laser line is to the y-axis of the image), the mapping point of line _ref on line _bias can be found. The point of line _bias which is the same as the x-coordinate of the point is found by taking the x-axis coordinate reference of the line _ref point as a bias _distance, and if the point of line _bias which is the same as the x-axis coordinate is not present, the point of line _bias which is the closest to the x-coordinate is corresponding to the point.

And a point cloud generating module 306, configured to convert the laser line into a point cloud through a coordinate conversion relationship, and generate point cloud information.

According to the device provided by the embodiment of the application, the laser line acquisition module 301 is used for shooting the laser line by using three exposure times of long, medium and short, and the pixel point acquisition module 302 is used for respectively acquiring clear laser line imaging pixels of a dark area and a bright area; the laser line segmentation module 303 divides the laser line into a dark area point set and a bright area point set through the connected domain; then the laser segment classification module 304 is used for dividing the light band into a thin segment, a standard segment and an oversized segment through a density threshold value and a width threshold value; obtaining denser laser points by reducing the gray threshold value, and then using the central line constraint noise of the high exposure standard section; and correcting the midline of the oversized section through the midline of the standard section, and generating the point cloud through the coordinate conversion relation through the point cloud generating module 306. Therefore, the application can cope with more complex illumination operating environment by exposing and imaging the laser line for a plurality of times; the laser lines are classified and individually corrected in a segmented and partitioned mode, so that more accurate laser center line coordinates can be obtained, and more accurate point clouds can be obtained.

Embodiment III:

According to an embodiment of the present invention, a computer readable storage medium is provided, on which a computer program is stored, where the computer program when executed by a processor implements the steps in the electronic price tag communication method, and specific steps are described in the first embodiment and are not repeated herein.

The memory in this embodiment can be used to store software programs as well as various data. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the cellular phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

According to an example of the present embodiment, all or part of the flow of the method of the above embodiment may be implemented by a computer program to instruct related hardware, where the program may be stored in a computer readable storage medium, and in an embodiment of the present invention, the program may be stored in a storage medium of a computer system and executed by at least one processor in the computer system to implement the flow of the embodiment including the methods as described above. The storage medium includes, but is not limited to, magnetic disks, flash disks, optical disks, read-Only Memory (ROM), and the like.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

From the above description of the embodiments, it will be clear to those skilled in the art that the above embodiment method may be implemented by means of software plus necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and including several instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.

The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the protection of the claims, which fall within the protection of the present invention.

Claims

1. A method for generating a laser line point cloud, the method comprising the steps of:

extracting laser line pixel points of each acquired image;

classifying the laser segments;

restoring the laser line;

Converting the laser line into point cloud through a coordinate conversion relationship to generate point cloud information;

the step of segmenting the laser line into a dark area point set and a bright area point set specifically comprises the following steps:

Corroding the connected domain in the bright area point set list1, and if the number of pixels in the connected domain after corrosion is smaller than a threshold value th_ numb of the number of pixels, moving the points in the connected domain into the dark area point set list2;

Performing expansion operation on the connected domain in the bright area point set list1; after expansion, the boundary of the connected domain expands outwards, and if the expanded pixel overlaps with a point in the dark area point set list2, the overlapped pixel is moved into the bright area point set list1 from the dark area point set list 2;

the classifying of the laser segments includes:

Classifying the dark area point sets list2, calculating the density of each segment point set in the list2, marking the segment as a standard segment if the density is larger than a density threshold value, otherwise marking the segment as a sparse segment;

the restoring laser line includes:

2. The laser line point cloud generating method according to claim 1, wherein the dark area point set list2 classification includes the steps of:

Calculating the point set density of each section of light band;

optical tape volume = optical tape length × optical tape width

Optical tape length length_set= coordinate _xmax-coordinate _xmin

Optical tape width width_set= coordinate _ymax-coordinate _ymin

Extracting a midline from an imaging image img _n+1 of a laser line to be repaired currently, namely a standard segment in the intermediate exposure image; and adding the midline to the final set of laser line pixels.

3. The method of claim 1, wherein restoring the sparse segment in the imaged image img _n+1 of the currently needed repair laser line comprises:

Extracting a standard segment central line in an img ₁～img_n dark region point set list2, and if all the dark region point sets of the graph are standard segments, adopting the standard segment with the highest point set density;

extracting a line in a light band;

4. The method of claim 3, wherein said adjusting the over-width segment of the imaged image img _n+1 of the currently required repair laser line comprises:

selecting a standard segment of a bright area in img _n+2～img_2n+1 as a reference segment, and selecting the narrowest light band as the reference segment if the bright areas of all the images are standard segments;

extracting a midline _ref of the reference standard segment; correcting the midline of the oversized section through the midline of the standard section;

5. The method of claim 4, wherein the method of extracting an optical band centerline comprises:

For a two-dimensional image, a black-and-plug Hessian matrix describes the two-dimensional derivative of each point in the principal direction, which Hessian matrix can be expressed as Wherein rxrx represents the second order bias of the point along the X direction, ry represents the second order bias of the point along the Y direction;

taking the point (x 0, y 0) as a reference point, then the subpixel (px, py) at the center of the band; assuming that there is a coefficient t such that (px, py) = (x0+ tnx, y0+ tny)

In the formula

when |t| is less than 0.5, (x 0, y 0) is a point on the midline.

6. A laser line point cloud generating device, characterized in that the device comprises: the device comprises a laser line acquisition module, a pixel point acquisition module, a laser line segmentation module, a laser segment classification module, a laser line restoration module and a point cloud generation module;

The laser line acquisition module is used for exposing at least 2N+1 shot laser lines sequentially from high to low by using three exposure times of long, medium and short; n is a natural number;

the step of dividing the laser line segment into a dark area point set and a bright area point set comprises the following steps:

the classifying of the laser segments includes:

The laser line restoration module is used for restoring the laser line;

the restoring laser line includes:

adjusting an excessively wide section of an imaging image img _n+1 of the laser line to be repaired currently;

7. The laser line point cloud generating apparatus according to claim 6, wherein the laser line segmentation module includes a point set classification unit, a new point set acquisition unit, and an expansion operation unit;

The point set classification unit is used for classifying a bright area point set list1 and a dark area point set l ist2;

The expansion operation unit is used for carrying out expansion operation on the connected domain in the bright area point set list1 and the dark area point set list2 obtained after updating; the boundary of the connected domain expands outward after expansion, and if the newly added expanded pixel overlaps with a point in the dark region point set list2, the overlapped pixel is moved from the dark region point set list2 into the bright region point set list 1.

8. The laser line point cloud generating apparatus according to claim 6, wherein the laser line restoration module includes a sparse segment restoration unit, an oversized segment adjustment unit, and a midline extraction unit;

And the midline extraction unit is used for extracting a midline _ref of the reference section and correcting the midline of the over-wide section through the midline of the standard section.

9. A computer readable storage medium comprising a processor, a computer readable storage medium and a computer program stored on the computer readable storage medium, which when executed by the processor, performs the steps of the method according to any one of claims 1 to 5.