CN114202548A - Forklift pallet positioning method and device, storage medium and electronic equipment - Google Patents
Forklift pallet positioning method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN114202548A CN114202548A CN202010896387.9A CN202010896387A CN114202548A CN 114202548 A CN114202548 A CN 114202548A CN 202010896387 A CN202010896387 A CN 202010896387A CN 114202548 A CN114202548 A CN 114202548A
- Authority
- CN
- China
- Prior art keywords
- image
- forklift
- tray
- template
- pallet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 29
- 230000011218 segmentation Effects 0.000 claims abstract description 19
- 230000000007 visual effect Effects 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims description 2
- 238000013507 mapping Methods 0.000 abstract description 3
- 238000004422 calculation algorithm Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000003709 image segmentation Methods 0.000 description 6
- 238000001514 detection method Methods 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 239000003086 colorant Substances 0.000 description 3
- 238000012937 correction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012946 outsourcing Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/12—Computing arrangements based on biological models using genetic models
- G06N3/126—Evolutionary algorithms, e.g. genetic algorithms or genetic programming
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
- G06T2207/20032—Median filtering
Abstract
The invention discloses a method and a device for positioning a pallet of a forklift, a storage medium and electronic equipment, wherein the method comprises the following steps: acquiring a depth image acquired based on an RGB-D sensor; acquiring a planar image acquired by a camera; identifying a forklift tray by using the plane image, and determining the area of the forklift tray in the depth image; carrying out plane segmentation processing on the region of the forklift pallet in the depth image, and determining a plane set containing the forklift pallet; matching the planar image with a pre-prepared tray template to obtain a matched target tray template; matching the determined plane set containing the forklift pallet with a target pallet template, and marking the position of the target pallet template in a second target area; and converting the position of the target pallet template in the second target area into a coordinate in a three-dimensional space to obtain the position of the forklift pallet. The tray area is determined through image mapping, and the problem of large image identification data volume in the prior art is solved.
Description
Technical Field
The invention relates to the field of image processing and pattern recognition, in particular to a positioning method and device of a forklift pallet, a storage medium and electronic equipment.
Background
In recent two years, the proportion of the total logistics cost of China in GDP is kept at about 18%, the proportion is 2 times that of the total logistics cost of China in developed countries and is about 6.5 percent higher than the global average level, and the logistics cost of China is higher.
In automated and semi-automated warehousing systems, the identification and positioning of pallets plays an important role. The tray is a horizontal platform device for placing goods and products in the process of packaging, stacking, carrying and transporting, and is widely applied to the fields of production, circulation, storage and the like. The tray recognition means that a sensor is mounted on the forklift to detect and recognize the tray in the warehouse, and the positioning means that three-dimensional coordinate information of the tray relative to the forklift is calculated according to the sensor information on the basis of the recognition.
The current detection technology is divided into two types according to whether the tray needs to be modified or not:
1) the method for modifying the tray comprises the following steps: the tray end face is labeled, for example, by sticking artificial marks on different parts of the end face, for example, by sticking concentric circles with black and white spaces on both sides and in the middle of the end face of the tray, or by sticking reflective tapes with high reflectivity on the whole end face. Identifying these artificial markings in the sensor data using pattern recognition correlation techniques accomplishes the identification and positioning of the pallet.
2) The method without modifying the tray comprises the following steps: such methods use the existing features of the pallet itself to accomplish the identification, for example, detecting two notches in the end face of the pallet is a common method.
The prior art has the following disadvantages:
the method of labeling the end faces of the trays limits the circulation of the trays and is prone to wear during use of the trays. In addition, from the economic cost consideration, the application and popularization involve the transformation of a large number of existing trays, and the labor cost and the time cost are high.
The method for modifying the tray is not based on a plane laser radar which is installed horizontally, the visual field of the plane laser radar in the vertical direction is limited, the movement of the forklift is needed for identifying the tray, and the efficiency is low. In addition, in consideration of economic cost, the current laser radar has high manufacturing cost and is not beneficial to application and popularization.
In order to solve the above problems, the chinese application CN105976375A discloses a tray identification and positioning method based on RGB-D sensor, which includes: acquiring a depth image through a sensor; performing plane segmentation on the point cloud of the depth image to obtain one or more planes to form a plane set; determining from the set of planes an associated plane that is likely to contain a tray; and matching the related planes according to a preset tray template, identifying the position of the tray in the related planes and positioning. Although the requirements on the shooting illumination conditions are low, in the identification process, noise reduction, invalid point elimination, window judgment and plane set processing need to be carried out on the whole depth image, and the plane set needs to be matched with all templates when template matching is carried out, so that the data processing capacity in the whole process is large, the processing process is complex, and the identification efficiency is low.
Disclosure of Invention
Therefore, the technical problem to be solved by the present invention is to overcome the defects of large data processing amount and low recognition efficiency in the tray recognition process in the prior art, so as to provide a forklift tray positioning method, device, storage medium and electronic device.
In order to achieve the purpose, the invention provides the following scheme:
in a first aspect, an embodiment of the present invention provides a method for positioning a pallet of a forklift, including the following steps:
acquiring a depth image acquired based on an RGB-D sensor;
acquiring a planar image acquired by a camera, wherein the camera and the RGB-D sensor have the same shooting visual angle, and the size of the acquired planar image is the same as that of the depth image;
identifying a forklift tray by using the plane image, and determining the area of the forklift tray in the depth image;
performing plane segmentation processing on the region of the forklift pallet in the depth image to determine a plane set containing the forklift pallet;
matching the planar image with a pre-prepared tray template to obtain a matched target tray template;
matching the determined plane set containing the forklift pallet with the target pallet template, and marking the position of the target pallet template in the second target area;
and converting the position of the target pallet template in the second target area into a coordinate in a three-dimensional space to obtain the position of the forklift pallet.
In an embodiment, the identifying the forklift tray by using the plane image and determining the area of the forklift tray in the depth image includes:
performing image recognition on the plane image, and determining a first target area where the forklift tray is located in the plane image;
acquiring a coordinate interval of the first target area in the plane image;
and utilizing the coordinate interval to circle out a second target area where the forklift tray is located in the depth image, and using the second target area as the area where the forklift tray is located in the depth image.
In an embodiment, matching the planar image with a pre-prepared tray template to obtain a matched target tray template includes:
extracting a first image feature in the plane image and a second image feature of the tray template;
comparing the first image features and the second image features one by one, and calculating the similarity between each corresponding feature;
weighting and summing all the calculated similarities to obtain the similarity between the forklift pallet in the plane image and the pallet template;
and when the similarity reaches a preset similarity threshold, determining the tray template as the target tray template.
In one embodiment, matching the determined set of planes containing the forklift pallets with the target pallet template, marking the position of the target pallet template in the second target area comprises:
calculating the matching degree of the target tray template and the plane set;
when the matching between the target pallet template and the plane set reaches a preset matching threshold value, determining that the target pallet template is matched with the forklift pallet, and marking the position of the target pallet template in the second target area.
In a second aspect, an embodiment of the present invention provides a positioning device for a forklift pallet, including:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a depth image acquired based on an RGB-D sensor;
the second acquisition module is used for acquiring a plane image acquired by a camera, wherein the camera and the RGB-D sensor have the same shooting visual angle, and the size of the acquired plane image is the same as that of the depth image;
the determining module is used for identifying a forklift tray by using the plane image and determining the area of the forklift tray in the depth image;
the segmentation module is used for performing plane segmentation processing on the region of the forklift pallet in the depth image to determine a plane set containing the forklift pallet;
the matching module is used for matching the planar image with a pre-prepared tray template to obtain a matched target tray template;
the marking module is used for matching the determined plane set containing the forklift pallet with the target pallet template and marking the position of the target pallet template in the second target area;
and the positioning module is used for converting the position of the target pallet template in the second target area into a coordinate in a three-dimensional space to obtain the position of the forklift pallet.
In one embodiment, the determining module comprises:
the identification unit is used for carrying out image identification on the plane image and determining a first target area where the forklift tray is located in the plane image;
the acquisition unit is used for acquiring a coordinate interval of the first target area in the plane image;
and the delineating unit is used for delineating a second target area where the forklift tray is located in the depth image by using the coordinate interval as an area where the forklift tray is located in the depth image.
In one embodiment, the matching module comprises:
an extraction unit that extracts a first image feature in the planar image and a second image feature of the tray template;
the comparison unit is used for comparing the first image features with the second image features one by one and calculating the similarity between each corresponding feature;
the calculating unit is used for weighting and summing all the calculated similarities to obtain the similarity between the forklift pallet in the plane image and the pallet template;
and the confirming unit is used for confirming that the tray template is the target tray template when the similarity reaches a preset similarity threshold value.
In one embodiment, the marking module comprises:
the calculating unit is used for calculating the matching degree of the target tray template and the plane set;
and the marking unit is used for determining that the target tray template is matched with the forklift tray and marking the position of the target tray template in the second target area when the matching between the target tray template and the plane set reaches a preset matching threshold value.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions that, when executed by a processor, implement the tray identifying and locating method according to any one of the embodiments of the first aspect.
In a fourth aspect, an embodiment of the present invention provides an electronic device, including:
a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the pallet identifying and locating method according to any one of the embodiments of the first aspect.
The technical scheme of the invention has the following advantages: the method comprises the steps of acquiring a plane image by using a camera, identifying a forklift tray in the image, matching the forklift tray with a depth image acquired by an RGB-D sensor, determining the area of the forklift tray in the depth image, performing plane segmentation on the depth image area of the forklift tray obtained by matching, determining a plane set containing the forklift tray, matching the plane image with a tray template prepared in advance, acquiring a target tray template, matching the determined plane set containing the forklift tray with the target tray template, marking the position of the target tray template in a second target area, and converting the position into a three-dimensional space coordinate to obtain the position of the forklift tray. In the embodiment of the invention, the area where the tray is located is matched by combining the plane image and the depth image, and the image processing is carried out on the area, so that the image area needing to be processed is reduced, the data volume of the image processing is greatly reduced, meanwhile, the tray template matching is carried out through the plane image, the processing speed is high compared with the processing speed of a method of matching the binary image after the depth image processing with all templates, the number of depth image matching templates is reduced, and the image processing efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a positioning method for a pallet of a forklift truck according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for identifying a forklift tray by using a plane image and determining a region of the forklift tray in a depth image according to an embodiment of the present invention;
fig. 3 is a flowchart of a method for depth image segmentation based on random Hough transform according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an off-line preparation tray end face template according to an embodiment of the present invention;
fig. 5 is a flowchart of a method for matching a planar image with a pre-prepared tray template to obtain a matched target tray template according to an embodiment of the present invention;
fig. 6 is a flowchart of a method for marking a position of a target pallet template in a second target area by matching a determined set of planes containing pallet of a forklift with the target pallet template according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a positioning device for a pallet of a forklift truck according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a determination module of a positioning device for a forklift pallet according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a matching module of the positioning device of the forklift pallet according to the embodiment of the invention;
fig. 10 is a schematic structural diagram of a marking module of a positioning device of a forklift pallet according to an embodiment of the present invention;
fig. 11 is a block diagram of a specific example of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; the two elements may be directly connected or indirectly connected through an intermediate medium, or may be communicated with each other inside the two elements, or may be wirelessly connected or wired connected. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Example 1
The embodiment of the invention provides a positioning method of a forklift pallet, which comprises the following steps as shown in figure 1:
step S101, acquiring a depth image acquired based on an RGB-D sensor;
the Depth image is a common RGB three-channel color image + Depth Map, the RGB color mode is a color standard in the industry, and various colors are obtained by changing three color channels of red (R), green (G), and blue (B) and superimposing the three color channels on each other, RGB represents colors of the three channels of red, green, and blue, and the standard almost includes all colors that can be perceived by human vision, and is one of the most widely used color systems at present; in 3D computer graphics, a Depth Map (Depth Map) is an image or image channel containing information about the distance of the surface of a scene object from a viewpoint. Where the Depth Map is similar to a grayscale image except that each pixel value thereof is the actual distance of the sensor from the object. Usually, the RGB image and the Depth image are registered, so that there is a one-to-one correspondence between the pixel points.
In the embodiment of the invention, the depth image needs to be converted into point cloud, namely, an image coordinate system is converted into a world coordinate system by using a formula, wherein the formula is as follows:
in the embodiment of the invention, the depth image is acquired by using the RGB-D sensor, so that the distance relationship between each pixel point and the sensor can be visually distinguished, and whether the pixel point and other pixel points are in the same plane or not can be judged.
Step S102, acquiring a plane image collected by a camera, wherein the camera and the RGB-D sensor have the same shooting visual angle, and the size of the collected plane image is the same as that of the depth image;
after a camera acquires and acquires a planar image, binary processing is carried out on the planar image, each pixel on the image has only two possible values or gray scale states, and the binary image is generally represented by black and white, B & W and monochrome images.
The RGB-D sensor and the camera are arranged in parallel and have infinite distance, at the moment, the plane image acquired by the camera and the image acquired by the RGB-D sensor can be considered as two images shot at the same visual angle, the two images have the same size, and at the moment, the images can not be corrected; of course, it can also be considered that the shooting angles of the two images are approximately consistent, then the images shot by the camera can be subjected to micro-correction according to the images obtained by the RGB-D sensor, so as to obtain two images with completely consistent shooting angles and sizes.
In the embodiment of the invention, on the basis of acquiring the RGB-D depth image, the camera is used for acquiring the plane image, and two pictures with the same shooting visual angle and the same size are acquired, so that the area of the required image can be determined by utilizing the mapping relation in the subsequent operation process.
Step S103, identifying a forklift tray by using the plane image, and determining the area of the forklift tray in the depth image;
as the feature recognition technology of the plane image is mature, and the inventor finds that light rays are generally required in the use environment of the forklift, the features of the forklift pallet on the plane image shot by a CCD/CMOS camera and the like can still be clearly recognized. Therefore, in the embodiment of the invention, the area recognition can be performed by using the plane image to circle the approximate area of the forklift pallet to obtain a detection frame, and since the shooting angles of the plane image and the depth image are the same and the image size is the same, the detection frame can be directly mapped into the depth image, so that the area of the forklift pallet in the depth image can be determined.
After the depth image area of the forklift pallet is determined, the depth image of the area is generated into point clouds and the point clouds are fused into final point cloud output. Noise inevitably exists in an actual depth image acquired by any depth image acquisition equipment, so that the image needs to be smoothed, firstly, a 3x3 window is used for carrying out median filtering on the image to filter out random impulse noise, and then a Gaussian smoothing filter is used for filtering out other random noise and quantization noise to obtain a smoothed image.
As an alternative embodiment, as shown in fig. 2, the identifying the forklift tray by using the plane image and determining the area of the forklift tray in the depth image includes the following steps:
step S1031, carrying out image recognition on the plane image, and determining a first target area where the forklift pallet is located in the plane image;
step S1032, acquiring a coordinate interval of the first target area in the plane image;
because the shooting angles and the sizes of the plane image and the depth image are approximately the same, it can be considered that pixel points in the plane image can correspond to each other in the depth image one by one, so that the position of a first target area of the forklift pallet is identified and confirmed in the plane image, and a coordinate interval where the first target area is located is determined through coordinates.
And step S1033, a second target area where the forklift tray is located in the depth image is determined by utilizing the coordinate interval, and the second target area is used as the area where the forklift tray is located in the depth image.
And determining a second target area where the forklift pallet is located in the depth image according to the coordinate interval of the forklift pallet in the plane image through the mapping relation.
According to the embodiment of the invention, the position area of the forklift pallet in the depth image is confirmed by utilizing the position of the forklift pallet in the plane image, so that the identification process of the depth image on the position of the forklift pallet is greatly simplified, and the identified position area of the forklift pallet is accurate and has higher precision.
Step S104, performing plane segmentation processing on the region of the forklift pallet in the depth image, and determining a plane set containing the forklift pallet;
plane segmentation methods for depth images are generally classified into three categories:
the first type: based on the edge method, the basic idea is to use a proper convolution operator to perform convolution on an image so as to obtain a gradient image corresponding to the image. The advantages are that: in the traditional operator gradient detection, the corresponding edge image can be quickly obtained only by using a proper convolution kernel for convolution. The disadvantages are as follows: the image edges are not necessarily exact and the gradients of complex images do not only occur at the image edges but may also occur in color and texture within the image.
The second type: based on the region method, more common algorithms such as traditional algorithm combined genetic algorithm, region growing algorithm, region splitting and merging, watershed algorithm and the like are adopted. The deep learning segmentation algorithm based on the region and the semanteme is the main direction of more image segmentation results and research at present.
In the third category: the graph-based segmentation algorithm is simple to implement, high in speed and high in precision. Of course other segmentation algorithms may be used, such as: depth image segmentation based on random Hough transformation, depth image segmentation based on normal component edge fusion, and the like.
The embodiment of the invention takes the depth image segmentation based on random Hough transformation as an example, and as shown in figure 3, the method comprises the following steps:
step S201, quantizing the whole parameter space into a plurality of sub-areas;
step S202, randomly selecting three non-collinear points in an image;
step S203, if the distance between every two of the three points is between the preset maximum distance threshold and the preset minimum distance threshold, calculating the plane parameters determined by the three points, and obtaining the sub-region where the determined plane parameters are located, wherein the cumulative number of planes of the sub-region is added with 1;
step S204, if one of the distances between every two of the three points is not between the preset maximum distance threshold and the preset minimum distance threshold, re-selecting the three points;
step S205, if the accumulated plane number of a certain sub-region exceeds a pre-specified threshold value T, or the iteration number exceeds P, the sub-region parameter with the largest accumulated plane number is the found plane F; otherwise, go back to step S202.
The prior art already has a mature technology for depth image segmentation, and the embodiment of the present invention is only used as an example and is not limited thereto.
According to the embodiment of the invention, the depth image after the smoothing processing is segmented, the plane of the forklift tray in the depth image area is confirmed, and the subsequent positioning processing of the tray is facilitated.
Step S105, matching the planar image with a pre-prepared tray template to obtain a matched target tray template;
the pre-prepared tray template and the plane image can be binary images, and when a new template exists, only the binary image of the new tray template needs to be stored in the system template base. Or processing the two images into binary images during template matching, and then matching. In the embodiment of the invention, in order to better link the matching of the tray template matched by using the plane image and the plane set of the depth image with the tray template, a series of tray templates can be prepared in advance by adopting the plane image and used for matching the plane image to determine the target tray template; and then preparing a series of tray templates corresponding to the tray templates by using the depth images, and after determining the target tray template, performing positioning matching by using a template prepared by using the depth image data corresponding to the target tray template. Because the plane image matching algorithm is mature and has high speed, after the template tray template is determined by using the plane image matching algorithm, the plane set corresponding to the depth image does not need to be matched with all tray templates, and the efficiency of positioning and identifying is improved.
As shown in fig. 4, the lengths of the respective portions of the end faces of the template are measured and discretized into a grid, the side length of the grid is determined according to the resolution of the sensor and the required accuracy of the system, and the side length used in the illustration is 1 cm. The size of the binary image is the number of grids contained in the minimum outsourcing rectangle of the tray, and the assignment rule of pixel points of the binary image is as follows: if the grid corresponds to the end face (shaded part in the figure) of the tray, the pixel point corresponding to the binary image is assigned to be 1, otherwise, if the grid corresponds to the notch of the tray, the pixel point corresponding to the binary image is assigned to be 0.
As an alternative embodiment, as shown in fig. 5, the matching the planar image with a pre-prepared tray template to obtain a matched target tray template includes the steps of:
step S1051, extracting a first image characteristic in the plane image and a second image characteristic of the tray template;
because the images of the plane image and the tray template are binary images after being processed, the image characteristics of the binary images comprise texture change characteristics, boundary characteristics, gray change characteristics and the like.
Step S1052, comparing the first image feature and the second image feature one by one, and calculating a similarity between each corresponding feature;
using formulas
Wherein, theta is the ratio of the number of the tray template with the same pixel value in the current position and the corresponding plane image to the total number of the tray template pixels, and mu is the ratio of the number of the pixels with the pixel value of 1 in the template to the total number of the pixels.
Step S1053, weighting and summing all the calculated similarities to obtain the similarity between the forklift pallet in the plane image and the pallet template;
and step S1054, when the similarity reaches a preset similarity threshold, determining the pallet template as the target pallet template.
The similarity threshold may be a set value or a value obtained by a learning algorithm of the system, and the present invention is not limited herein. Marking the type and the position of the tray template with the similarity reaching a preset similarity threshold value in the plane image.
According to the embodiment of the invention, the plane image of the forklift pallet shot by the camera is matched with the pallet template prepared in advance, so that the type and the model of the forklift pallet can be determined, and the subsequent positioning in the depth image can be facilitated.
Step S106, matching the determined plane set containing the forklift pallet with the target pallet template, and marking the position of the target pallet template in the second target area;
as an alternative embodiment, as shown in fig. 6, the matching the determined plane set including the forklift pallet with the target pallet template and marking the position of the target pallet template in the second target area includes:
step S1061, calculating the matching degree of the target pallet template and the plane set;
because the binary image of the target pallet template and the point cloud image of the plane set both contain data such as gray scale and the like, the matching degree of the target pallet template and the plane set containing the forklift pallet can be comprehensively calculated according to numerical values such as edge contour, area size, gray state and the like.
Step S1062, when the matching between the target pallet template and the plane set reaches a preset matching threshold, determining that the target pallet template is matched with the forklift pallet, and marking the position of the target pallet template in the second target area.
In the embodiment of the invention, the binary image of the target pallet template is matched with the plane set containing the forklift pallets, so that the area position of the target pallet template in the depth image can be directly marked, and the target pallet template can be used for carrying out point cloud correction on the forklift pallets in the plane set, thereby being beneficial to obtaining more accurate point cloud data of the forklift pallets.
And S107, converting the position of the target pallet template in the second target area into a coordinate in a three-dimensional space to obtain the position of the forklift pallet.
The RGB picture in the RGB-D image provides the x, y coordinates in the pixel coordinate system, while the depth map directly provides the Z coordinate in the camera coordinate system, i.e. the camera-to-point distance.
According to the information of the RGB-D image and the internal reference of the camera, the coordinates of any pixel point in the camera coordinate system can be calculated.
According to the information of the RGB-D image and the internal and external parameters of the camera, the coordinates of any pixel point in the world coordinate system can be calculated.
In the field of view of the camera, the coordinates of the obstacle points in the camera coordinate system are the point cloud sensor data, i.e. the point cloud data in the camera coordinate system. The point cloud sensor data may be calculated from coordinates provided by the RGB-D image and camera parameters.
Coordinates of the obstacle points in all the world coordinate systems are point cloud map data, namely point cloud data in the world coordinate systems. The point cloud map data can be calculated according to coordinates provided by the RGB-D image and internal and external parameters of the camera.
The point cloud data under the camera coordinate system can obtain the X and Y coordinate values under the camera coordinate system according to the X and Y coordinates (i.e. u and v in a formula) under the pixel coordinate system provided by the RGB image and camera internal parameters. Meanwhile, the depth map directly provides the Z-coordinate value in the camera coordinate system. Further obtaining the coordinates under the camera coordinates
Coordinates of the obstacle point in the camera coordinates are point cloud sensor data, that is, point cloud data in the camera coordinate system.
According to the camera coordinate system P and the pixel coordinate system PuvThe following formula for the coordinates of points:
a coordinate conversion formula from a world coordinate system to a point under a pixel coordinate system is as follows:
obtaining a world coordinate system PwTo the pixel coordinate system PuvCoordinate relation of lower pointIs described. Thus, here, using the above equation, the pixel coordinates [ u, v ] are input]After the depth Z is reached, the coordinate P of the point in the world coordinate system can be obtainedwAnd coordinates of the obstacle points in all world coordinate systems are point cloud map data. According to the internal reference formula, the three-dimensional coordinate P ═ X, Y, Z of the point in the camera coordinate system can be calculated]Then, according to the homogeneous transformation matrix T or rotation matrix with camera and translation vector R, T, the three-dimensional coordinate P of said point in world coordinate system can be obtainedw＝[Xw,Yw,Zw]，Pw＝[Xw,Yw,Zw]Namely the point cloud calibrated under the world coordinate system.
According to the embodiment of the invention, the depth image coordinate data is converted by utilizing the information of the RGB-D image and the internal and external parameters of the camera, the coordinate position of the target pallet template in the depth image area under the camera coordinate system is converted into the coordinate under the world coordinate system, the position of the forklift pallet is obtained, and the positioning of the forklift pallet is favorably completed.
Example 2
An embodiment of the present invention provides a positioning device for a forklift pallet, as shown in fig. 7, including:
a first obtaining module 301, configured to obtain a depth image obtained based on an RGB-D sensor;
a second acquiring module 302, configured to acquire a planar image acquired by a camera, where the camera and the RGB-D sensor have the same shooting angle, and the size of the acquired planar image is the same as that of the depth image;
a determining module 303, configured to identify a forklift tray by using the plane image, and determine an area of the forklift tray in the depth image;
a segmentation module 304, configured to perform plane segmentation on an area where the forklift tray is located in the depth image, and determine a plane set including the forklift tray;
a matching module 305, configured to match the planar image with a pre-prepared tray template to obtain a matched target tray template;
a marking module 306, configured to match the determined plane set including the forklift pallet with the target pallet template, and mark a position of the target pallet template in the second target area;
and a positioning module 307, configured to convert the position of the target pallet template in the second target area into a coordinate in a three-dimensional space, so as to obtain the position of the forklift pallet.
According to the embodiment of the invention, a camera is used for acquiring a plane image, a forklift tray in the image is identified, the plane image is matched with a depth image acquired by an RGB-D sensor, the area of the forklift tray in the depth image is determined, the plane image area of the forklift tray obtained through matching is subjected to plane segmentation, a plane set containing the forklift tray is determined, the plane image is matched with a pre-prepared tray template, a target tray template is obtained, finally, the determined plane set containing the forklift tray is matched with the target tray template, the position of the target tray template in a second target area is marked, and the position is converted into a three-dimensional space coordinate, so that the position of the forklift tray is obtained. In the embodiment of the invention, the area where the tray is located is matched by combining the plane image and the depth image, and the image processing is carried out on the area, so that the image area needing to be processed is reduced, the data volume of the image processing is greatly reduced, meanwhile, the tray template matching is carried out through the plane image, the processing speed is high compared with the processing speed of a method of matching the binary image after the depth image processing with all templates, the number of depth image matching templates is reduced, and the image processing efficiency is improved.
Example 3
An embodiment of the present invention provides a determining module of a positioning device for a forklift pallet, as shown in fig. 8, including:
the recognition unit 3031 is configured to perform image recognition on the planar image, and determine a first target area where the forklift tray is located in the planar image;
an obtaining unit 3032, configured to obtain a coordinate interval of the first target region in the planar image;
a delineating unit 3033, configured to delineate, by using the coordinate interval, a second target region where the forklift tray is located in the depth image, as a region where the forklift tray is located in the depth image.
According to the embodiment of the invention, the position area of the forklift pallet in the depth image is confirmed by utilizing the position of the forklift pallet in the plane image, so that the identification process of the depth image on the position of the forklift pallet is greatly simplified, and the identified position area of the forklift pallet is accurate and has higher precision.
For a detailed description of the apparatus, reference may be made to the above method embodiments, which are not described herein again.
Example 4
The embodiment of the invention provides a matching module of a positioning device of a forklift pallet, as shown in fig. 9, comprising:
an extracting unit 3051, extracting a first image feature in the planar image and a second image feature of the tray template;
a comparing unit 3052, comparing the first image feature with the second image feature one by one, and calculating a similarity between each corresponding feature;
the first calculation unit 3053 performs weighted summation on all the calculated similarities to obtain the similarity between the forklift pallet in the plane image and the pallet template;
the confirming unit 3054, when the similarity reaches a preset similarity threshold, determines that the tray template is the target tray template.
According to the embodiment of the invention, the plane image of the forklift pallet shot by the camera is matched with the pallet template prepared in advance, so that the type and the model of the forklift pallet can be determined, and the subsequent positioning in the depth image can be facilitated.
For a detailed description of the apparatus, reference may be made to the above method embodiments, which are not described herein again.
Example 5
An embodiment of the present invention provides a marking module of a positioning device for a forklift pallet, as shown in fig. 10, including:
a second calculation unit 3061, which calculates the matching degree of the target pallet template and the plane set;
a marking unit 3062, configured to determine that the target pallet template matches the forklift pallet when the matching between the target pallet template and the plane set reaches a preset matching threshold, and mark the position of the target pallet template in the second target area.
In the embodiment of the invention, the binary image of the target pallet template is matched with the plane set containing the forklift pallets, so that the area position of the target pallet template in the depth image can be directly marked, and the target pallet template can be used for carrying out point cloud correction on the forklift pallets in the plane set, thereby being beneficial to obtaining more accurate point cloud data of the forklift pallets.
For a detailed description of the apparatus, reference may be made to the above method embodiments, which are not described herein again.
Example 6
In an embodiment of the present invention, an electronic device is further provided, where the electronic device may be a background server in the foregoing embodiment, and an internal structure diagram of the electronic device may be as shown in fig. 11. The electronic equipment comprises a processor, a memory and a network interface which are connected through a system bus, and further comprises a display screen and an input device. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic equipment comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used for connecting and communicating with an external electronic device through a network. The computer program is executed by a processor to realize the forklift pallet positioning method, the electronic equipment can also comprise a display screen and an input device, the display screen can be a liquid crystal display screen or an electronic ink display screen, the input device of the electronic equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the electronic equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
On the other hand, the electronic device may not include a display screen and an input device, and those skilled in the art will understand that the structure shown in fig. 11 is only a block diagram of a part of the structure related to the present application, and does not constitute a limitation to the electronic device to which the present application is applied, and a specific electronic device may include more or less components than those shown in the figure, or combine some components, or have a different arrangement of components.
In one embodiment, an electronic device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: acquiring a depth image acquired based on an RGB-D sensor; acquiring a planar image acquired by a camera, wherein the camera and the RGB-D sensor have the same shooting visual angle, and the size of the acquired planar image is the same as that of the depth image; identifying a forklift tray by using the plane image, and determining the area of the forklift tray in the depth image; performing plane segmentation processing on the region of the forklift pallet in the depth image to determine a plane set containing the forklift pallet; matching the planar image with a pre-prepared tray template to obtain a matched target tray template; matching the determined plane set containing the forklift pallet with the target pallet template, and marking the position of the target pallet template in the second target area; and converting the position of the target pallet template in the second target area into a coordinate in a three-dimensional space to obtain the position of the forklift pallet.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring a depth image acquired based on an RGB-D sensor; acquiring a planar image acquired by a camera, wherein the camera and the RGB-D sensor have the same shooting visual angle, and the size of the acquired planar image is the same as that of the depth image; identifying a forklift tray by using the plane image, and determining the area of the forklift tray in the depth image; performing plane segmentation processing on the region of the forklift pallet in the depth image to determine a plane set containing the forklift pallet; matching the planar image with a pre-prepared tray template to obtain a matched target tray template; matching the determined plane set containing the forklift pallet with the target pallet template, and marking the position of the target pallet template in the second target area; and converting the position of the target pallet template in the second target area into a coordinate in a three-dimensional space to obtain the position of the forklift pallet. It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
It should be understood that the above examples are only for clarity of illustration and are not intended to limit the embodiments. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. And obvious variations or modifications therefrom are within the scope of the invention.
Claims (10)
1. A positioning method of a forklift pallet is characterized by comprising the following steps:
acquiring a depth image acquired based on an RGB-D sensor;
acquiring a planar image acquired by a camera, wherein the camera and the RGB-D sensor have the same shooting visual angle, and the size of the acquired planar image is the same as that of the depth image;
identifying a forklift tray by using the plane image, and determining the area of the forklift tray in the depth image;
performing plane segmentation processing on the region of the forklift pallet in the depth image to determine a plane set containing the forklift pallet;
matching the planar image with a pre-prepared tray template to obtain a matched target tray template;
matching the determined plane set containing the forklift pallet with the target pallet template, and marking the position of the target pallet template in a second target area;
and converting the position of the target pallet template in the second target area into a coordinate in a three-dimensional space to obtain the position of the forklift pallet.
2. The method of claim 1, wherein identifying a forklift tray using the planar image and determining an area in the depth image in which the forklift tray is located comprises:
performing image recognition on the plane image, and determining a first target area where the forklift tray is located in the plane image;
acquiring a coordinate interval of the first target area in the plane image;
and utilizing the coordinate interval to circle out a second target area where the forklift tray is located in the depth image, and using the second target area as the area where the forklift tray is located in the depth image.
3. The method of claim 1, wherein matching the planar image with a pre-prepared pallet template to obtain a matched target pallet template comprises:
extracting a first image feature in the plane image and a second image feature of the tray template;
comparing the first image features and the second image features one by one, and calculating the similarity between each corresponding feature;
weighting and summing all the calculated similarities to obtain the similarity between the forklift pallet in the plane image and the pallet template;
and when the similarity reaches a preset similarity threshold, determining the tray template as the target tray template.
4. The method of claim 1, wherein using the determined set of planes containing the forklift pallets to match the target pallet template to mark the location of the target pallet template in the second target area comprises:
calculating the matching degree of the target tray template and the plane set;
when the matching between the target pallet template and the plane set reaches a preset matching threshold value, determining that the target pallet template is matched with the forklift pallet, and marking the position of the target pallet template in the second target area.
5. A positioning device for forklift pallets, comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a depth image acquired based on an RGB-D sensor;
the second acquisition module is used for acquiring a plane image acquired by a camera, wherein the camera and the RGB-D sensor have the same shooting visual angle, and the size of the acquired plane image is the same as that of the depth image;
the determining module is used for identifying a forklift tray by using the plane image and determining the area of the forklift tray in the depth image;
the segmentation module is used for performing plane segmentation processing on the region of the forklift pallet in the depth image to determine a plane set containing the forklift pallet;
the matching module is used for matching the planar image with a pre-prepared tray template to obtain a matched target tray template;
the marking module is used for matching the determined plane set containing the forklift pallet with the target pallet template and marking the position of the target pallet template in a second target area;
and the positioning module is used for converting the position of the target pallet template in the second target area into a coordinate in a three-dimensional space to obtain the position of the forklift pallet.
6. The apparatus of claim 5, wherein the determining module comprises:
the identification unit is used for carrying out image identification on the plane image and determining a first target area where the forklift tray is located in the plane image;
the acquisition unit is used for acquiring a coordinate interval of the first target area in the plane image;
and the delineating unit is used for delineating a second target area where the forklift tray is located in the depth image by using the coordinate interval as an area where the forklift tray is located in the depth image.
7. The apparatus of claim 5, wherein the matching module comprises:
an extraction unit that extracts a first image feature in the planar image and a second image feature of the tray template;
the comparison unit is used for comparing the first image features with the second image features one by one and calculating the similarity between each corresponding feature;
the first calculation unit is used for carrying out weighted summation on all the calculated similarities to obtain the similarity between the forklift pallet in the plane image and the pallet template;
and the confirming unit is used for confirming that the tray template is the target tray template when the similarity reaches a preset similarity threshold value.
8. The apparatus of claim 5, wherein the tagging module comprises:
the second calculation unit is used for calculating the matching degree of the target tray template and the plane set;
and the marking unit is used for determining that the target tray template is matched with the forklift tray and marking the position of the target tray template in the second target area when the matching between the target tray template and the plane set reaches a preset matching threshold value.
9. A computer-readable storage medium storing computer instructions which, when executed by a processor, implement the method of locating forklift pallets as claimed in any one of claims 1 to 4.
10. An electronic device, comprising:
a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the method of positioning a forklift pallet according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010896387.9A CN114202548A (en) | 2020-08-31 | 2020-08-31 | Forklift pallet positioning method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010896387.9A CN114202548A (en) | 2020-08-31 | 2020-08-31 | Forklift pallet positioning method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114202548A true CN114202548A (en) | 2022-03-18 |
Family
ID=80644285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010896387.9A Pending CN114202548A (en) | 2020-08-31 | 2020-08-31 | Forklift pallet positioning method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114202548A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315264A (en) * | 2023-11-30 | 2023-12-29 | 深圳市普拉托科技有限公司 | Tray detection method based on image recognition and related device |
-
2020
- 2020-08-31 CN CN202010896387.9A patent/CN114202548A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315264A (en) * | 2023-11-30 | 2023-12-29 | 深圳市普拉托科技有限公司 | Tray detection method based on image recognition and related device |
| CN117315264B (en) * | 2023-11-30 | 2024-03-08 | 深圳市普拉托科技有限公司 | Tray detection method based on image recognition and related device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109658398B (en) | Part surface defect identification and evaluation method based on three-dimensional measurement point cloud | |
| Lari et al. | An adaptive approach for the segmentation and extraction of planar and linear/cylindrical features from laser scanning data | |
| CN111340797A (en) | Laser radar and binocular camera data fusion detection method and system | |
| CN109801333B (en) | Volume measurement method, device and system and computing equipment | |
| CN110084243B (en) | File identification and positioning method based on two-dimensional code and monocular camera | |
| CN110473221B (en) | Automatic target object scanning system and method | |
| CN115546202B (en) | Tray detection and positioning method for unmanned forklift | |
| CN111986185A (en) | Tray detection and positioning method based on depth camera | |
| CN114331986A (en) | Dam crack identification and measurement method based on unmanned aerial vehicle vision | |
| CN111178193A (en) | Lane line detection method, lane line detection device and computer-readable storage medium | |
| CN113362385A (en) | Cargo volume measuring method and device based on depth image | |
| CN113096183A (en) | Obstacle detection and measurement method based on laser radar and monocular camera | |
| CN116279592A (en) | Method for dividing travelable area of unmanned logistics vehicle | |
| CN112734844A (en) | Monocular 6D pose estimation method based on octahedron | |
| CN116452852A (en) | Automatic generation method of high-precision vector map | |
| CN113128346B (en) | Target identification method, system and device for crane construction site and storage medium | |
| CN114396875A (en) | Rectangular parcel volume measurement method based on vertical shooting of depth camera | |
| CN114202548A (en) | Forklift pallet positioning method and device, storage medium and electronic equipment | |
| CN116309882A (en) | Tray detection and positioning method and system for unmanned forklift application | |
| CN116125489A (en) | Indoor object three-dimensional detection method, computer equipment and storage medium | |
| CN116309817A (en) | Tray detection and positioning method based on RGB-D camera | |
| CN115116026A (en) | Automatic tracking method and system for logistics carrying robot | |
| CN115100271A (en) | Method and device for detecting goods taking height, computer equipment and storage medium | |
| CN113808262A (en) | Building model generation system based on depth map analysis | |
| CN113095324A (en) | Classification and distance measurement method and system for cone barrel |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |