CN114283121A - Paving width detection method and device, readable storage medium and paver - Google Patents
Paving width detection method and device, readable storage medium and paver Download PDFInfo
- Publication number
- CN114283121A CN114283121A CN202111463709.1A CN202111463709A CN114283121A CN 114283121 A CN114283121 A CN 114283121A CN 202111463709 A CN202111463709 A CN 202111463709A CN 114283121 A CN114283121 A CN 114283121A
- Authority
- CN
- China
- Prior art keywords
- image
- point cloud
- area
- cloud data
- paving width
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Road Repair (AREA)
Abstract
The invention provides a paving width detection method and device, a readable storage medium and a paver. The paving width detection method comprises the following steps: acquiring image information of a paved road area through a calibrated vehicle-mounted image acquisition device; extracting an RGB image and a depth image of the paved road area from image information of the paved road area; determining edge position information of a loose material area in a paved road area according to the RGB image; extracting point cloud data corresponding to edge position information of the loose material area from the depth image; and determining the paving width according to the point cloud data. The paving width detection method provided by the invention can calculate the image of the paved road surface to obtain the paving width value, is a direct detection mode, and has more accurate detection result compared with the mode of indirectly measuring the paving width in the related technology, wherein the mode adopts the distance sensor to measure the width of the screed plate, thereby being capable of better helping to judge the paving quality and measure and predict the engineering quantity.
Description
Technical Field
The invention relates to the technical field of pavers, in particular to a paving width detection method and device, a readable storage medium and a paver.
Background
Because the paving width is an important index for measuring the paving construction quality of the road surface and predicting the paving engineering quantity, the real-time acquisition and recording of the paving width are very critical.
In the related art, a method for indirectly measuring the paving width is generally used, for example, a distance sensor such as a laser sensor is mounted on a screed, and the distance sensor is used to measure the width of the screed so as to indirectly measure the paving width, and the above-mentioned method mainly has the following disadvantages: firstly, the measurement accuracy can be influenced by various factors, such as the camber of the screed plate and the like, and secondly, the measurement accuracy is not flexible enough, and when the screed plate is assembled, the installation position of the distance sensor may need to be adjusted.
Disclosure of Invention
The present invention is directed to solving or improving at least one of the technical problems of the prior art or the related art.
Therefore, the invention provides a paving width detection method based on a vehicle-mounted image acquisition device in a first aspect.
The invention provides a paving width detection device based on a vehicle-mounted image acquisition device in a second aspect.
The invention provides a paving width detection device based on a vehicle-mounted image acquisition device.
A fourth aspect of the invention provides a readable storage medium.
A fifth aspect of the invention provides a paving machine.
In view of the above, according to a first aspect of the present invention, a paving width detection method based on an on-vehicle image capturing device is provided, including: acquiring image information of a paved road area through a calibrated vehicle-mounted image acquisition device; extracting an RGB image and a depth image of the paved road area from image information of the paved road area; determining edge position information of a loose material area in a paved road area according to the RGB image; extracting point cloud data corresponding to edge position information of the loose material area from the depth image; and determining the paving width according to the point cloud data.
The paving width detection method provided by the invention comprises the steps of firstly obtaining image information of a paved road area through a calibrated vehicle-mounted image acquisition device, and extracting RGB images and depth images of the paved road area from the image information. And then, the RGB image of the paved road is preprocessed, so that the edge position information of the loose material area in the paved road area can be positioned. And extracting point cloud data corresponding to the edge position information of the loose material area from the depth image. And calculating the paving width according to the point cloud data. The paving width detection method provided by the invention obtains the paving width by calculating the image of the paved road surface area, is a direct detection mode, is suitable for both straight plate pavers and telescopic pavers, and has more accurate detection result compared with a mode of indirectly measuring the paving width by adopting a distance sensor to measure the width of a screed plate in the related technology, thereby being capable of better helping to judge the paving quality and measure and predict the engineering quantity.
Among them, the RGB image is an image of an RGB color mode (Red Green Blue color mode), which is a color standard in the industry, and various colors are obtained by changing three color channels of Red, Green, and Blue and superimposing them on each other.
Depth Image (Depth Image), an Image storing three-dimensional Depth feature information, also called Range Image (Range Image), refers to an Image in which distance (Depth) values of points in a scene collected by an Image collector are taken as pixel values, and directly reflects the geometric shape of a visible surface of a scene. The depth image may be calculated as point cloud data through coordinate transformation.
Specifically, the vehicle-mounted image capturing device may be a binocular vision camera or a tof (time of light camera) camera.
The paving width detection method provided by the invention can also have the following technical characteristics:
in the above technical solution, the step of determining the edge position information of the loose material area in the paved road area according to the RGB image specifically includes: determining a loose material area in the RGB image according to the image characteristic value; determining corresponding edge position information according to the loose material area; wherein the image feature values comprise any one or a combination of: color values, luminance values, gray values, texture values.
In the technical scheme, when the edge position information of the loose material area in the paved road area is determined according to the RGB image of the paved road, the image characteristic value of the loose material area on the image is obviously different from the image characteristic value of the road area or the road boundary area (such as the boundary ground or the wall surface) without the loose material on the image, so that the loose material area can be determined according to the change of the image characteristic value. And then the corresponding edge position information can be positioned according to the loose paving material area.
The image feature value includes any one or a combination of a color value, a brightness value, a gray value, and a texture value, but is not limited thereto.
In any of the above technical solutions, the step of extracting the loose material region from the RGB image according to the image feature value specifically includes: and determining the loose material area by using a region growing algorithm by taking any point in the loose material area as a seed point and the change gradient of the image characteristic value as a growing condition, and stopping growing when the change gradient of the image characteristic value is larger than a preset value.
In the technical scheme, when the loose material area in the RGB image is determined according to the image characteristic value, the loose material area can be extracted from the RGB image by adopting an area growing algorithm. The seed points can be selected as any point in the loose material area, the change gradient of the image characteristic value is used as a growth condition, pixel points with similar properties are combined and generated in the image, and the growth is stopped until the change gradient of the image characteristic value is larger than a preset value. By the technical scheme, the paved road surface area on the paved road surface can be accurately determined, so that more accurate edge position information can be obtained, the paving width can be calculated, and the detection result is more accurate.
Therein, region growing refers to the process of developing groups of pixels or regions into larger regions. Starting from the set of seed points, the region from these points grows by merging into this region neighboring pixels with similar properties like intensity, gray level, texture color, etc. as each seed point.
In any of the above technical solutions, the step of determining the paving width according to the point cloud data specifically includes: the point cloud data comprises first point cloud data and second point cloud data, and a boundary straight line is fitted according to the first point cloud data; calculating the average distance from the second point cloud data to the boundary straight line, and recording the average distance as the paving width; the first point cloud data and the second point cloud data respectively correspond to first edge position information and second edge position information of the loose material area in the direction perpendicular to the extending direction of the road.
In the technical scheme, the point cloud data corresponding to the edge position information of the loose material area comprises first point cloud data and second point cloud data. The first point cloud data and the second point cloud data respectively correspond to first edge position information and second edge position information of the loose material area in the direction perpendicular to the extending direction of the road. When the paving width is determined according to the point cloud data, firstly, a boundary straight line is fitted according to the first point cloud data, secondly, the average distance from the second point cloud data to the boundary straight line is calculated, and the average distance is recorded as the paving width. The calculation method of the invention is simple and reliable, thereby obtaining more accurate and reliable paving width and being beneficial to improving paving quality.
Specifically, a first boundary straight line can be fitted according to the first point cloud data, a second boundary straight line can be fitted according to the second point cloud data, the average distance between the first boundary straight line and the second boundary straight line is calculated, and the average distance is recorded as the paving width.
In any of the above technical solutions, before the step of acquiring the image information of the paved road area by the calibrated vehicle-mounted image acquisition device, the method further includes: acquiring image information of a road surface area through a vehicle-mounted image acquisition device; extracting a depth image of the road surface region from the image information of the road surface region; extracting third point cloud data corresponding to a road surface area except a road boundary area and fourth point cloud data corresponding to the road boundary area from the depth image; calculating a point cloud normal vector according to the third point cloud data and the fourth point cloud data; calculating external parameters of the vehicle-mounted image acquisition device according to the point cloud normal vector and the reference normal vector so as to calibrate the depth image acquired by the vehicle-mounted image acquisition device; the pavement area is a paved pavement area or a pavement area to be paved.
In the technical scheme, before the image information of the paved road surface area is acquired through the calibrated vehicle-mounted image acquisition device, external reference calibration is carried out on the vehicle-mounted image acquisition device. Specifically, image information of a road surface area is obtained through a vehicle-mounted image acquisition device, a depth image of the road surface area is extracted from the image information, third point cloud data corresponding to the area of the road surface area except a road boundary area and fourth point cloud data corresponding to the road boundary area are extracted from the depth image, a point cloud normal vector can be calculated through the third point cloud data and the fourth point cloud data, an external parameter of the vehicle-mounted image acquisition device can be obtained through calculation of the point cloud normal vector and a reference normal vector, and the vehicle-mounted image acquisition device can be calibrated according to the external parameter. After the calibration is finished, the coordinate system of the vehicle-mounted image acquisition device can be associated and unified to a world coordinate system, namely, the coordinate system is in the same coordinate system with the paver and the screed plate. Therefore, the image data obtained by the vehicle-mounted image acquisition device is reliable and accords with the reality, and the calculated paving width is more real and reliable under the condition of reliable image data.
In addition, it should be noted that, in the above calibration process, the depth image used is not limited to the depth image of the paved road area, and the depth image of the paved road area is also applicable.
In any of the above technical solutions, the method further includes: acquiring position information of a paved road; drawing a paving width map according to the position information and the paving width of the paved road; and sending the paving width map to a display device for displaying.
In the technical scheme, the sampled paving width map of the paved road can be obtained by acquiring the position information of the paved road and combining the paving width, and the paving width map is sent to the display device to be displayed, so that the paving width under each section of road can be more visually checked, and the method can be used for helping to judge the paving quality and the measurement and prediction of the engineering quantity.
According to a second aspect of the present invention, a paving width detection device based on a vehicle-mounted image acquisition device is provided, comprising: a memory storing programs or instructions; and the processor is used for realizing the steps of the paving width detection method based on the vehicle-mounted image acquisition device according to any one of the technical schemes when executing the program or the instruction.
The paving width detection device based on the vehicle-mounted image acquisition device comprises a memory and a processor, wherein the memory stores programs or instructions, and the processor realizes the steps of the paving width detection method according to any one of the technical schemes when executing the programs or the instructions.
According to a third aspect of the present invention, a paving width detection device based on a vehicle-mounted image acquisition device is provided, comprising: the acquisition unit is used for acquiring image information of a paved road area through a calibrated vehicle-mounted image acquisition device; the extraction unit is used for extracting the RGB image and the depth image of the paved road surface area from the image information of the paved road surface area; the first processing unit is used for determining the edge position information of a loose material area in a paved road surface area according to the RGB image; the second processing unit is used for extracting point cloud data corresponding to the edge position information from the depth image; and the calculating unit is used for determining the paving width according to the point cloud data.
The paving width detection device provided by the invention comprises an acquisition unit, an extraction unit, a first processing unit, a second processing unit and a calculation unit. Firstly, an acquisition unit acquires image information of a paved road area through a calibrated vehicle-mounted image acquisition device, and an extraction unit extracts RGB images and depth images of the paved road area. And then the first processing unit can position the edge position information of the loose material area in the paved road area by preprocessing the RGB image of the paved road. The second processing unit extracts point cloud data corresponding to edge position information of the loose material area from the depth image. The calculation unit can calculate the paving width according to the point cloud data. The paving width detection device provided by the invention obtains the paving width by calculating the image of the paved road surface area, is a direct detection mode, is suitable for both a straight plate paver and a telescopic paver, and has more accurate detection result compared with a mode of indirectly measuring the paving width by adopting a distance sensor to measure the width of a screed plate in the related technology, thereby being capable of better helping to judge the paving quality and measure and predict the engineering quantity.
Among them, the RGB image is an image of an RGB color mode (Red Green Blue color mode), which is a color standard in the industry, and various colors are obtained by changing three color channels of Red, Green, and Blue and superimposing them on each other.
Depth Image (Depth Image), an Image storing three-dimensional Depth feature information, also called Range Image (Range Image), refers to an Image in which distance (Depth) values of points in a scene collected by an Image collector are taken as pixel values, and directly reflects the geometric shape of a visible surface of a scene. The depth image may be calculated as point cloud data through coordinate transformation.
Specifically, the on-vehicle image acquisition device may be a binocular vision camera or a TOF camera.
In the above technical solution, the first processing unit is specifically configured to determine a loose material area in the RGB image according to the image feature value; determining corresponding edge position information according to the loose material area; wherein the image feature values comprise any one or a combination of: color values, luminance values, gray values, texture values.
In the technical solution, when the first processing unit determines the edge position information of the loose material area in the paved road area according to the RGB image of the paved road, since the image characteristic value of the loose material area shown on the image is significantly different from the image characteristic value of the unpaved road area or road boundary area (such as boundary ground or wall) shown on the image, the loose material area can be determined according to the change of the image characteristic value. And then the corresponding edge position information can be positioned according to the loose paving material area.
The image feature value includes any one or a combination of a color value, a brightness value, a gray value, and a texture value, but is not limited thereto.
In any of the above technical solutions, the first processing unit is further specifically configured to determine the loose material area by using any point in the loose material area as a seed point and using a change gradient of the image characteristic value as a growth condition, and stop growing when the change gradient of the image characteristic value is greater than a preset value.
In the technical scheme, when the first processing unit determines the loose material area in the RGB image according to the image feature value, the loose material area can be extracted from the RGB image by using an area growing algorithm. The seed points can be selected as any point in the loose material area, the change gradient of the image characteristic value is used as a growth condition, pixel points with similar properties are combined and generated in the image, and the growth is stopped until the change gradient of the image characteristic value is larger than a preset value. By the technical scheme, the paved road surface area on the paved road surface can be accurately determined, so that more accurate edge position information can be obtained, the paving width can be calculated, and the detection result is more accurate.
Therein, region growing refers to the process of developing groups of pixels or regions into larger regions. Starting from the set of seed points, the region from these points grows by merging into this region neighboring pixels with similar properties like intensity, gray level, texture color, etc. as each seed point.
In any of the above technical solutions, the calculating unit is specifically configured to: the point cloud data comprises first point cloud data and second point cloud data, and a boundary straight line is fitted according to the first point cloud data; calculating the average distance from the second point cloud data to the boundary straight line, and recording the average distance as the paving width; the first point cloud data and the second point cloud data respectively correspond to first edge position information and second edge position information of the loose material area in the direction perpendicular to the extending direction of the road.
In the technical scheme, the point cloud data corresponding to the edge position information of the loose material area comprises first point cloud data and second point cloud data. The first point cloud data and the second point cloud data respectively correspond to first edge position information and second edge position information of the loose material area in the direction perpendicular to the extending direction of the road. When the calculation unit determines the paving width according to the point cloud data, firstly, a boundary straight line is fitted according to the first point cloud data, secondly, the average distance from the second point cloud data to the boundary straight line is calculated, and the average distance is recorded as the paving width. The calculation method of the invention is simple and reliable, thereby obtaining more accurate and reliable paving width and being beneficial to improving paving quality.
Specifically, a first boundary straight line can be fitted according to the first point cloud data, a second boundary straight line can be fitted according to the second point cloud data, the average distance between the first boundary straight line and the second boundary straight line is calculated, and the average distance is recorded as the paving width.
In any of the above technical solutions, the paving width detecting device further includes a calibration unit, and the calibration unit is configured to: before the step of extracting point cloud data corresponding to the edge position information of the loose material area from the depth image, extracting third point cloud data corresponding to the road surface area and fourth point cloud data corresponding to the road boundary area from the depth image; calculating a point cloud normal vector according to the third point cloud data and the fourth point cloud data; and calculating external parameters of the vehicle-mounted image acquisition device according to the point cloud normal vector and the reference normal vector so as to calibrate the depth image acquired by the vehicle-mounted image acquisition device.
In the technical scheme, before the depth image is processed to extract point cloud data corresponding to the paving material edge position information, external reference calibration needs to be carried out on the vehicle-mounted image acquisition device. Specifically, the depth image is obtained through the calibration unit, third point cloud data corresponding to the road surface area and fourth point cloud data corresponding to the road boundary area are extracted from the depth image, a point cloud normal vector can be calculated through the third point cloud data and the fourth point cloud data, an external parameter of the vehicle-mounted image acquisition device can be obtained through calculation of the point cloud normal vector and a reference normal vector, and the vehicle-mounted image acquisition device can be calibrated according to the external parameter. After the calibration is finished, the coordinate system of the vehicle-mounted image acquisition device can be associated and unified to a world coordinate system, namely, the coordinate system is in the same coordinate system with the paver and the screed plate. Therefore, the image data obtained by the vehicle-mounted image acquisition device is reliable and accords with the reality, and the calculated paving width is more real and reliable under the condition of reliable image data.
In addition, it should be noted that, in the above calibration process, the depth image used is not limited to the depth image of the paved road area, and the depth image of the paved road area is also applicable.
In any of the above technical solutions, the paving width detecting device further includes a display unit, and the display unit is configured to: acquiring position information of a paved road; drawing a paving width map according to the position information and the paving width of the paved road; and sending the paving width map to a display device for displaying.
In the technical scheme, the sampled paving width map of the paved road can be obtained by acquiring the position information of the paved road and combining the paving width, and the paving width map is sent to the display device to be displayed, so that the paving width under each section of road can be more visually checked, and the method can be used for helping to judge the paving quality and the measurement and prediction of the engineering quantity.
According to a fourth aspect of the present invention, there is provided a readable storage medium on which a program or instructions are stored, which program or instructions, when executed by a processor, implement the steps of the paving width detecting method according to any one of the above-mentioned technical solutions.
The readable storage medium provided by the present invention, when the stored program or instructions are executed, may implement the steps of the paving width detection method according to any one of the above technical solutions, so that all the beneficial effects of the paving width detection method are achieved, and are not discussed herein.
In a fifth aspect of the present invention, there is provided a paver comprising: the paving width detection device based on the vehicle-mounted image acquisition device in any technical scheme is described; and/or a readable storage medium as described in the previous claims.
The paver provided by the technical scheme comprises the paving width detection device based on the vehicle-mounted image acquisition device and/or the readable storage medium based on the technical scheme, so that the paver has all the beneficial effects of the paving width detection device and/or the readable storage medium, and the details are not repeated.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a schematic flow chart of a paving width detection method based on a vehicle-mounted image acquisition device according to an embodiment of the invention;
FIG. 2 is a second schematic flowchart of a paving width detection method based on a vehicle-mounted image capturing device according to an embodiment of the present invention;
FIG. 3 is a third schematic flowchart of a paving width detection method based on a vehicle-mounted image acquisition device according to an embodiment of the present invention;
FIG. 4 is a fourth schematic flowchart of a paving width detection method based on a vehicle-mounted image capturing device according to an embodiment of the present invention;
FIG. 5 is a calibration diagram of a paving width detection method based on a vehicle-mounted image acquisition device according to an embodiment of the invention;
FIG. 6 is one of schematic block diagrams of a paving width detection device based on an on-board image acquisition device according to an embodiment of the present invention;
FIG. 7 is a second schematic block diagram of a paving width detection device based on an on-board image acquisition device according to an embodiment of the present invention;
FIG. 8a is a schematic structural diagram of a paving width detection device based on a vehicle-mounted image acquisition device according to an embodiment of the invention;
fig. 8b is an RGB image of a paved road surface area in a paving width detection method based on an on-vehicle image capture device according to an embodiment of the present invention;
FIG. 9 is a fifth schematic flowchart of a paving width detection method based on a vehicle-mounted image capturing device according to an embodiment of the present invention;
fig. 10 is a logic diagram of a paving width detection method based on a vehicle-mounted image acquisition device according to an embodiment of the invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited to the specific embodiments disclosed below.
A paving width detection method (hereinafter, referred to as a paving width detection method) and apparatus of an on-vehicle image capture apparatus according to some embodiments of the present invention, a readable storage medium, and a paving machine will be described with reference to fig. 1 to 10.
Example one
FIG. 1 is one of the flow diagrams of a paving width detection method according to one embodiment of the present invention. The paving width detection method comprises the following steps:
102, acquiring image information of a paved road area through a calibrated vehicle-mounted image acquisition device;
104, extracting RGB images and depth images of the paved road area from the image information of the paved road area;
106, determining the edge position information of a loose material area in the paved road area according to the RGB image;
108, extracting point cloud data corresponding to the edge position information of the loose material area from the depth image;
and step 110, determining the paving width according to the point cloud data.
The paving width detection method provided by this embodiment first obtains image information of a paved road area through a vehicle-mounted image acquisition device, and extracts an RGB image and a depth image of the paved road area from the image information. And then, the RGB image of the paved road is preprocessed, so that the edge position information of the loose material area in the paved road area can be positioned. And extracting point cloud data corresponding to the edge position information of the loose material area from the depth image. And calculating the paving width according to the point cloud data. The paving width detection method provided by the embodiment of the invention obtains the paving width by calculating the image of the paved road surface area, is a direct detection mode, is suitable for both straight-plate pavers and telescopic pavers, and has more accurate detection result compared with a mode of measuring the width of an ironing plate by adopting a distance sensor and indirectly measuring the paving width in the related technology, thereby being capable of better helping to judge the paving quality and measure and predict the engineering quantity.
In particular, the on-board image acquisition device may be a binocular vision camera or a TOF camera or a lidar module.
Example two
FIG. 2 is a second schematic flow chart of a paving width detection method according to an embodiment of the present invention. The paving width detection method comprises the following steps:
and step 212, determining the paving width according to the point cloud data.
Wherein the image feature values comprise any one or a combination of: color values, luminance values, gray values, texture values.
In this embodiment, when determining the edge position information of the loose material area in the paved road area according to the RGB image of the paved road, since the image characteristic value of the loose material area displayed on the image is significantly different from the image characteristic value of the road area or the road boundary area (such as the boundary ground or the wall surface) without the loose material displayed on the image, the loose material area can be determined according to the change of the image characteristic value. And then the corresponding edge position information can be positioned according to the loose paving material area.
EXAMPLE III
In the second embodiment, the step of determining the loose material area in the RGB image according to the image feature value specifically includes: and determining the loose material area by using a region growing algorithm by taking any point in the loose material area as a seed point and the change gradient of the image characteristic value as a growing condition, and stopping growing when the change gradient of the image characteristic value is larger than a preset value.
In this embodiment, when determining the loose material region in the RGB image according to the image feature value, a region growing algorithm may be used to extract the loose material region from the RGB image. The seed points can be selected as any point in the loose material area, the change gradient of the image characteristic value is used as a growth condition, pixel points with similar properties are combined and generated in the image, and the growth is stopped until the change gradient of the image characteristic value is larger than a preset value. By the technical scheme, the paved road surface area on the paved road surface can be accurately determined, so that more accurate edge position information can be obtained, the paving width can be calculated, and the detection result is more accurate.
Example four
FIG. 3 is a third schematic flow chart of a paving width detection method according to an embodiment of the present invention. The paving width detection method comprises the following steps:
308, determining corresponding edge position information according to the loose material area;
and step 314, calculating the average distance from the second point cloud data to the boundary straight line, and recording the average distance as the paving width.
The image feature value includes any one or a combination of a color value, a brightness value, a gray value, and a texture value, but is not limited thereto. The first point cloud data and the second point cloud data correspond to first edge position information and second edge position information of the loose material area in a direction perpendicular to the extending direction of the road respectively.
In this embodiment, the point cloud data corresponding to the edge position information of the loose-material region includes first point cloud data and second point cloud data. The first point cloud data and the second point cloud data respectively correspond to first edge position information and second edge position information of the loose material area in the direction perpendicular to the extending direction of the road. When the paving width is determined according to the point cloud data, firstly, a boundary straight line is fitted according to the first point cloud data, secondly, the average distance from the second point cloud data to the boundary straight line is calculated, and the average distance is recorded as the paving width. The calculation method of the embodiment of the invention is simple and reliable, so that more accurate and reliable paving width can be obtained, and the improvement of paving quality is facilitated.
Specifically, a first boundary straight line can be fitted according to the first point cloud data, a second boundary straight line can be fitted according to the second point cloud data, the average distance between the first boundary straight line and the second boundary straight line is calculated, and the average distance is recorded as the paving width.
EXAMPLE five
FIG. 4 is a fourth flowchart of a paving width detection method according to an embodiment of the invention. The paving width detection method comprises the following steps:
and a step 414 of sending the paving width map to a display device for displaying.
In this embodiment, by acquiring the position information of the paved road and combining the paving width, a sampled paving width map of the paved road can be obtained, and the paving width map is sent to the display device to be displayed, so that the paving width under each section of road can be more visually checked, and the method can be used for helping to judge the paving quality and the measurement and prediction of the engineering quantity.
EXAMPLE six
Fig. 5 is a schematic diagram of a method for calibrating a depth image in a paving width detection method according to an embodiment of the present invention. The paving width detection method comprises the following steps:
502, acquiring image information of a road surface area through a vehicle-mounted image acquisition device;
and 510, calculating external parameters of the vehicle-mounted image acquisition device according to the point cloud normal vector and the reference normal vector so as to calibrate the depth image acquired by the vehicle-mounted image acquisition device.
The pavement area is a paved pavement area or a pavement area to be paved. In this embodiment, before the calibrated vehicle-mounted image capturing device obtains the image information of the paved road area, the vehicle-mounted image capturing device needs to be calibrated externally. Specifically, image information of a road surface area is obtained through a vehicle-mounted image acquisition device, a depth image of the road surface area is extracted from the image information, third point cloud data corresponding to the area of the road surface area except a road boundary area and fourth point cloud data corresponding to the road boundary area are extracted from the depth image, a point cloud normal vector can be calculated through the third point cloud data and the fourth point cloud data, an external parameter of the depth image can be obtained through calculation of the point cloud normal vector and a reference normal vector, the depth image can be calibrated according to the external parameter, and after calibration is finished, the depth image can be associated and unified to a world coordinate system, namely, the depth image and a paver are in the same coordinate system. The image data obtained in this way is reliable and conforms to the reality, and the calculated paving width is more real and reliable under the condition of reliable image data.
EXAMPLE seven
FIG. 6 is one of the schematic block diagrams of a paving width detection apparatus 600 in accordance with one embodiment of the present invention.
Wherein, this width detection device 600 paves includes:
a memory 602 storing programs or instructions;
The paving width detection device 600 provided by the present embodiment includes a memory 602 and a processor 604, where the memory 602 stores a program or instructions, and the processor 604 executes the program or instructions to implement the steps of the paving width detection method according to any one of the above embodiments, so that the paving width detection device 600 has all the advantages of the paving width detection method, and will not be discussed herein any more.
Example eight
Fig. 7 is a second schematic block diagram of a paving width detection apparatus 700 in accordance with an embodiment of the present invention.
Wherein, this width detection device 700 paves includes:
an acquisition unit 702, configured to acquire image information of a paved road area through a vehicle-mounted image acquisition device;
an extraction unit 704 for extracting an RGB image and a depth image of the paved road area from the image information of the paved road area;
the first processing unit 706 is used for determining the edge position information of a loose material area in the paved road surface area according to the RGB image;
a second processing unit 708 for extracting point cloud data corresponding to the edge position information from the depth image;
and the calculating unit 710 is used for determining the paving width according to the point cloud data.
The paving width detection apparatus 700 provided in this embodiment includes an acquisition unit 702, an extraction unit 704, a first processing unit 706, a second processing unit 708, and a calculation unit 710. Firstly, the obtaining unit 702 obtains image information of a paved road area through a calibrated vehicle-mounted image collecting device, and the extracting unit 704 extracts an RGB image and a depth image of the paved road area. The first processing unit 706 may then locate edge position information of a loose material region in the paved road region by preprocessing the RGB image of the paved road. The second processing unit 708 extracts point cloud data corresponding to edge position information of the loose material region from the depth image. The calculation unit 710 may calculate the paving width according to the point cloud data. The paving width detection device 700 provided by the invention obtains the paving width by calculating the image of the paved road surface area, is a direct detection mode, is suitable for both straight plate pavers and telescopic pavers, and has more accurate detection result compared with a mode of indirectly measuring the paving width by adopting a distance sensor to measure the width of a screed plate in the related technology, thereby being capable of better helping to judge the paving quality and measure and predict the engineering quantity.
Among them, the RGB image is an image of an RGB color mode (Red Green Blue color mode), which is a color standard in the industry, and various colors are obtained by changing three color channels of Red, Green, and Blue and superimposing them on each other.
Depth Image (Depth Image), an Image storing three-dimensional Depth feature information, also called Range Image (Range Image), refers to an Image in which distance (Depth) values of points in a scene collected by an Image collector are taken as pixel values, and directly reflects the geometric shape of a visible surface of a scene. The depth image may be calculated as point cloud data through coordinate transformation.
Specifically, the on-vehicle image acquisition device may be a binocular vision camera or a TOF camera.
Example nine
In the above embodiment, the first processing unit 706 is specifically configured to determine a loose material region in the RGB image according to the image feature value; determining corresponding edge position information according to the loose material area; wherein the image feature values comprise any one or a combination of: color values, luminance values, gray values, texture values.
In this embodiment, when determining the edge position information of the loose-material area in the paved road area according to the RGB image of the paved road, the first processing unit 706 may determine the loose-material area according to the change of the image characteristic value, because the image characteristic value of the loose-material area presented on the image is significantly different from the image characteristic value of the unpaved road area or the road boundary area (such as the boundary ground or the wall surface) presented on the image. And then the corresponding edge position information can be positioned according to the loose paving material area.
The image feature value includes any one or a combination of a color value, a brightness value, a gray value, and a texture value, but is not limited thereto.
Example ten
In any of the above embodiments, the first processing unit 706 is further specifically configured to determine the loose material region by using any point in the loose material region as a seed point and using the change gradient of the image characteristic value as a growth condition, and stop growing when the change gradient of the image characteristic value is greater than a preset value.
In this embodiment, the first processing unit 706 may extract the loose-material region from the RGB image by using a region growing algorithm when determining the loose-material region in the RGB image according to the image feature value. The seed points can be selected as any point in the loose material area, the change gradient of the image characteristic value is used as a growth condition, pixel points with similar properties are combined and generated in the image, and the growth is stopped until the change gradient of the image characteristic value is larger than a preset value. By the embodiment of the invention, the paved road surface area on the paved road surface can be accurately determined, so that more accurate edge position information can be obtained, the paving width can be calculated, and the detection result is more accurate.
Therein, region growing refers to the process of developing groups of pixels or regions into larger regions. Starting from the set of seed points, the region from these points grows by merging into this region neighboring pixels with similar properties like intensity, gray level, texture color, etc. as each seed point.
EXAMPLE eleven
In any of the embodiments above, the computing unit 710 is specifically configured to: the point cloud data comprises first point cloud data and second point cloud data, and a boundary straight line is fitted according to the first point cloud data; calculating the average distance from the second point cloud data to the boundary straight line, and recording the average distance as the paving width; the first point cloud data and the second point cloud data respectively correspond to first edge position information and second edge position information of the loose material area in the direction perpendicular to the extending direction of the road.
In this embodiment, the point cloud data corresponding to the edge position information of the loose-material region includes first point cloud data and second point cloud data. The first point cloud data and the second point cloud data respectively correspond to first edge position information and second edge position information of the loose material area in the direction perpendicular to the extending direction of the road. When the calculation unit 710 determines the paving width according to the point cloud data, a boundary straight line is fitted according to the first point cloud data, an average distance from the second point cloud data to the boundary straight line is calculated, and the average distance is recorded as the paving width. The calculation method of the invention is simple and reliable, thereby obtaining more accurate and reliable paving width and being beneficial to improving paving quality.
Specifically, a first boundary straight line can be fitted according to the first point cloud data, a second boundary straight line can be fitted according to the second point cloud data, the average distance between the first boundary straight line and the second boundary straight line is calculated, and the average distance is recorded as the paving width.
Example twelve
In any of the above embodiments, the paving width detecting apparatus 700 further comprises a calibration unit 712, the calibration unit 712 being configured to: before the step of acquiring the image information of the paved road area through a calibrated vehicle-mounted image acquisition device, acquiring the image information of the road area through the vehicle-mounted image acquisition device; extracting a depth image of the road surface region from the image information of the road surface region; extracting third point cloud data corresponding to a road surface area except a road boundary area and fourth point cloud data corresponding to the road boundary area from the depth image; calculating a point cloud normal vector according to the third point cloud data and the fourth point cloud data; calculating external parameters of the vehicle-mounted image acquisition device according to the point cloud normal vector and the reference normal vector so as to calibrate the depth image acquired by the vehicle-mounted image acquisition device; the pavement area is a paved pavement area or a pavement area to be paved.
In this embodiment, before the calibrated vehicle-mounted image capturing device obtains the image information of the paved road area, the vehicle-mounted image capturing device needs to be calibrated externally. Specifically, image information of a road surface area is obtained through a vehicle-mounted image acquisition device, a depth image of the road surface area is extracted from the image information, third point cloud data corresponding to the area of the road surface area except a road boundary area and fourth point cloud data corresponding to the road boundary area are extracted from the depth image, a point cloud normal vector can be calculated through the third point cloud data and the fourth point cloud data, an external parameter of the vehicle-mounted image acquisition device can be obtained through calculation of the point cloud normal vector and a reference normal vector, and the vehicle-mounted image acquisition device can be calibrated according to the external parameter. After the calibration is finished, the coordinate system of the vehicle-mounted image acquisition device can be associated and unified to a world coordinate system, namely, the coordinate system is in the same coordinate system with the paver and the screed plate. Therefore, the image data obtained by the vehicle-mounted image acquisition device is reliable and accords with the reality, and the calculated paving width is more real and reliable under the condition of reliable image data.
In addition, it should be noted that, in the above calibration process, the depth image used is not limited to the depth image of the paved road area, and the depth image of the paved road area is also applicable.
EXAMPLE thirteen
In any of the above embodiments, the paving width detecting device 700 further comprises a display unit 714, the display unit 714 being configured to: acquiring position information of a paved road; drawing a paving width map according to the position information and the paving width of the paved road; and sending the paving width map to a display device for displaying.
In this embodiment, by acquiring the position information of the paved road and combining the paving width, a sampled paving width map of the paved road can be obtained, and the paving width map is sent to the display device to be displayed, so that the paving width under each section of road can be more visually checked, and the method can be used for helping to judge the paving quality and the measurement and prediction of the engineering quantity.
Example fourteen
Because the paving width is an important index for measuring the paving construction quality of the road surface and predicting the paving engineering quantity, the real-time acquisition and recording of the paving width are very critical.
The embodiment provides a paving width detection method and device based on a vehicle-mounted image acquisition device aiming at the requirement of paving width detection in pavement construction.
FIG. 8a is a schematic structural diagram of a paving width detection device based on a vehicle-mounted image acquisition device according to the embodiment; fig. 8b is an RGB image of the paved road surface area in the paving width detection method based on the vehicle-mounted image capturing device according to the present embodiment; fig. 9 is a schematic flow chart of the paving width detection method of the embodiment. Fig. 10 is a schematic diagram of a paving width detection method of the present embodiment. The paving width detection method will be described in detail with reference to fig. 8a, 8b, 9 and 10.
As shown in fig. 10, in the paving width detection method provided in this embodiment, on one hand, a paving area (loose paving area) and further an image boundary line of the paving area may be determined according to an RGB image of a paved road, and on the other hand, external reference calibration may be performed according to a depth image of the paved road. And then, point cloud boundary scattered points are determined according to the calibrated depth image and the image boundary line, the point cloud boundary scattered points comprise left boundary scattered points (first point cloud data) and right boundary scattered points (second point cloud data), and the paving width can be calculated according to the left boundary scattered points and the right boundary scattered points.
As shown in fig. 8a, the paving width detection device includes an image acquisition module, a display module, a satellite differential positioning module, an industrial personal computer, and the like.
And the image acquisition module and the industrial personal computer are powered, connected through a network cable and tested, and data communication is ensured.
As shown in fig. 9, the paving width detection method includes:
904, transmitting the high-precision position information acquired by the satellite differential positioning system and the satellite time service time data and the image which are responded to the high-precision position information and the satellite time service time data to an industrial personal computer of a detection platform which is carried on a paver while acquiring the image of the paved road surface;
step 908: by calculating the size between the two edges and combining satellite positioning data, a sampling area width value can be obtained and stored;
step 910: the display module is installed on the paver, and width data combined with satellite positioning information is transmitted to the display screen to be displayed.
The image acquisition module can simultaneously output an RGB image and a depth image, and specifically can be a binocular camera or a TOF camera or a laser radar sensor.
In particular, the display module may be an LED display screen.
Specifically, in step 904, the satellite positioning module is subjected to data parsing to extract the timestamp and latitude and longitude coordinate data in the GNSS data stream for repackaging with the paving width data.
Specifically, in step 906, during paving by the paving machine, the vision camera samples the rear paved road surface. And acquiring a depth image of the visual camera, carrying out external reference calibration on the depth image, and paving the ground level in the data in a loose mode. During calibration, the vehicle body is driven to a flat road surface, and the vision camera is over against the road. Collecting data of road boundary and paving material area in depth image and calculating point cloud normal vector
If the vehicle body is right opposite to the wall surface and the ground is horizontal, the lower boundary of the screed coordinate system can be establishedNormal vector of ground
At this time, the singular value decomposition can be adopted to obtain the external parameter RT from the visual camera to the screed coordinate system:
V,U＝svd(H)；
R＝V×UT。
where H denotes a matrix composed of the four normal vectors, V, U denotes singular values calculated by svd (singular Value decomposition) algorithm, and R denotes a rotation matrix of extrinsic parameters. And (5) calculating a rotation matrix R and then configuring a translation amount T according to the installation position.
Further, a loose material area is extracted on the image returned by the vision sensor. The boundary of the paving area is calculated by adopting the area growth, the seed point is selected as the area (the loose paving area close to the screed plate) at the lower part of the middle part of the image, as shown in fig. 8b, the area indicated by 804 is the seed point selection area, the seed area is increased towards the left, the right and the upper part, and the increasing conditions are the color and the image brightness. And terminating if the color and brightness change gradient is too large. As shown in fig. 8b, the area denoted by 802 is the road boundary area.
In step 908, the edge positions are located according to the image, dense edge points are extracted from the depth image, and the loose width is calculated according to the point clouds on the two sides:
specifically, a random sampling consensus algorithm RANSAC (random SAmple consensus) algorithm is adopted to fit a left straight line, the point cloud is P, the fitted inner point is PL, and the straight line equation is y ═ kx + b, so that PL, k, b ═ RANSAC (P); and obtaining the mean value of the point clouds on the right side, and finally calculating the projection distance from the points to a linear equation on the left side, namely the current paving width W. And outputting the W to a display module, and drawing a paving width map by combining the GPS position.
Example fifteen
The present embodiment provides a readable storage medium having stored thereon a program or instructions which, when executed by a processor, implement the steps of the paving width detection method of any of the embodiments described above.
The present embodiment provides a readable storage medium, which stores a program or instructions that, when executed, can implement the steps of the paving width detection method according to any one of the above embodiments, and therefore, has all the beneficial effects of the paving width detection method, which will not be discussed herein.
Example sixteen
The embodiment provides a paver, including: the paving width detection device based on the vehicle-mounted image acquisition device in any one of the embodiments; and/or a readable storage medium as in the above embodiments.
The paver provided by this embodiment includes the paving width detection device based on the vehicle-mounted image capture device according to any of the above embodiments, and/or the readable storage medium according to the above embodiments, and therefore, the paver has all the beneficial effects of the paving width detection device and/or the readable storage medium, and details are not repeated.
In the description herein, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance unless explicitly stated or limited otherwise; the terms "connected," "mounted," "secured," and the like are to be construed broadly and include, for example, fixed connections, removable connections, or integral connections; may be directly connected or indirectly connected through an intermediate. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the description herein, the description of the terms "one embodiment," "some embodiments," "specific embodiments," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A paving width detection method based on a vehicle-mounted image acquisition device is characterized by comprising the following steps:
acquiring image information of a paved road area through a calibrated vehicle-mounted image acquisition device;
extracting an RGB image and a depth image of the paved road area from image information of the paved road area;
determining edge position information of a loose material area in a paved road area according to the RGB image;
extracting point cloud data corresponding to edge position information of the loose material area from the depth image;
and determining the paving width according to the point cloud data.
2. The paving width detection method according to claim 1, wherein the step of determining the edge position information of the loose material region in the paved road surface region according to the RGB image specifically includes:
determining a loose material area in the RGB image according to the image characteristic value;
determining corresponding edge position information according to the loose material area;
wherein the image feature values comprise any one or a combination of: color values, luminance values, gray values, texture values.
3. The paving width detection method according to claim 2, wherein the step of determining the loose material region in the RGB image according to the image feature value specifically includes:
and determining the loose material area by using a region growing algorithm by taking any point in the loose material area as a seed point and the change gradient of the image characteristic value as a growing condition, and stopping growing when the change gradient of the image characteristic value is larger than a preset value.
4. The paving width detection method according to claim 1, wherein the step of determining the paving width from the point cloud data specifically comprises:
the point cloud data comprises first point cloud data and second point cloud data, wherein the first point cloud data and the second point cloud data respectively correspond to first edge position information and second edge position information of the loose material area in the direction perpendicular to the extending direction of the road;
fitting a boundary straight line according to the first point cloud data;
and calculating the average distance from the second point cloud data to the boundary straight line, and recording the average distance as the paving width.
5. The paving width detection method as claimed in any one of claims 1 to 4, further comprising, before the step of acquiring image information of the paved road surface area by a calibrated on-board image acquisition device:
acquiring image information of a road surface area through the vehicle-mounted image acquisition device;
extracting a depth image of the road surface region from the image information of the road surface region;
extracting third point cloud data corresponding to a road surface area except a road boundary area and fourth point cloud data corresponding to the road boundary area from the depth image;
calculating a point cloud normal vector according to the third point cloud data and the fourth point cloud data;
calculating external parameters of the vehicle-mounted image acquisition device according to the point cloud normal vector and the reference normal vector so as to calibrate the vehicle-mounted image acquisition device;
the pavement area is a paved pavement area or a pavement area to be paved.
6. The paving width detection method according to any one of claims 1 to 4, further comprising:
acquiring position information of the paved road;
drawing a paving width map according to the position information of the paved road and the paving width;
and sending the paving width map to a display device for displaying.
7. The utility model provides a width detection device paves based on-vehicle image acquisition device which characterized in that includes:
a memory storing programs or instructions;
a processor implementing the steps of the on-board image capture device-based paving width detection method of any of claims 1-6 when executing the program or instructions.
8. The utility model provides a width detection device paves based on-vehicle image acquisition device which characterized in that includes:
the acquisition unit is used for acquiring image information of a paved road area through a calibrated vehicle-mounted image acquisition device;
the extraction unit is used for extracting the RGB image and the depth image of the paved road surface area from the image information of the paved road surface area;
the first processing unit is used for determining the edge position information of a loose material area in a paved road surface area according to the RGB image;
a second processing unit for extracting point cloud data corresponding to the edge position information from the depth image;
and the calculation unit is used for determining the paving width according to the point cloud data.
9. A readable storage medium on which a program or instructions are stored, which when executed by a processor implement the steps of the vehicle-mounted image capture device-based paving width detection method of any of claims 1-6.
10. A paving machine, comprising:
the vehicle-mounted image acquisition device-based paving width detection device of claim 7 or 8; and/or
The readable storage medium of claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111463709.1A CN114283121A (en) | 2021-12-02 | 2021-12-02 | Paving width detection method and device, readable storage medium and paver |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111463709.1A CN114283121A (en) | 2021-12-02 | 2021-12-02 | Paving width detection method and device, readable storage medium and paver |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114283121A true CN114283121A (en) | 2022-04-05 |
Family
ID=80870646
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111463709.1A Pending CN114283121A (en) | 2021-12-02 | 2021-12-02 | Paving width detection method and device, readable storage medium and paver |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114283121A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117091516A (en) * | 2022-05-12 | 2023-11-21 | 广州镭晨智能装备科技有限公司 | Method, system and storage medium for detecting thickness of circuit board protective layer |
-
2021
- 2021-12-02 CN CN202111463709.1A patent/CN114283121A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117091516A (en) * | 2022-05-12 | 2023-11-21 | 广州镭晨智能装备科技有限公司 | Method, system and storage medium for detecting thickness of circuit board protective layer |
| CN117091516B (en) * | 2022-05-12 | 2024-05-28 | 广州镭晨智能装备科技有限公司 | Method, system and storage medium for detecting thickness of circuit board protective layer |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7587081B2 (en) | Method for processing stereo vision data using image density | |
| CN107957408B (en) | Method for measuring soil suction by light reflection theory | |
| CN107316325B (en) | Airborne laser point cloud and image registration fusion method based on image registration | |
| EP2313737B1 (en) | System for adaptive three-dimensional scanning of surface characteristics | |
| EP3534326A1 (en) | Method and apparatus for merging infrared image and visible light image | |
| CN107843204A (en) | Side slope three-dimensional deformation monitoring method and system based on monitoring level video camera | |
| EP2659458B1 (en) | System and method for generating textured map object images | |
| US20100207936A1 (en) | Fusion of a 2d electro-optical image and 3d point cloud data for scene interpretation and registration performance assessment | |
| CN101044390B (en) | Sight distance measuring device and uses | |
| CN109472865B (en) | Free measurable panoramic reproduction method based on image model drawing | |
| CN111179335A (en) | Standing tree measuring method based on binocular vision | |
| CN114283121A (en) | Paving width detection method and device, readable storage medium and paver | |
| CN116342674A (en) | Method for calculating asphalt pavement construction depth by three-dimensional model | |
| Yin et al. | Individual tree parameters estimation for chinese fir (cunninghamia lanceolate (lamb.) hook) plantations of south china using UAV Oblique Photography: Possibilities and Challenges | |
| CN113340405B (en) | Bridge vibration mode measuring method, device and system | |
| CN205711654U (en) | A kind of detection device of the road surface breakage information of three-dimensional visualization | |
| CN111382591A (en) | Binocular camera ranging correction method and vehicle-mounted equipment | |
| JP2009052907A (en) | Foreign matter detecting system | |
| CN108776135B (en) | Multi-factor combined road fog-weather detection device | |
| CN114724104B (en) | Method, device, electronic equipment, system and medium for detecting visual recognition distance | |
| CN116563370A (en) | Distance measurement method and speed measurement method based on monocular computer vision | |
| CN115131360A (en) | Road pit detection method and device based on 3D point cloud semantic segmentation | |
| CN115267756A (en) | Monocular real-time distance measurement method based on deep learning target detection | |
| CN117218089B (en) | Asphalt pavement structure depth detection method | |
| CN109859315B (en) | Surface vegetation separation method of three-dimensional image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |