CN117496189B - Rectangular tray hole identification method and system based on depth camera - Google Patents

Rectangular tray hole identification method and system based on depth camera Download PDF

Info

Publication number
CN117496189B
CN117496189B CN202410001208.9A CN202410001208A CN117496189B CN 117496189 B CN117496189 B CN 117496189B CN 202410001208 A CN202410001208 A CN 202410001208A CN 117496189 B CN117496189 B CN 117496189B
Authority
CN
China
Prior art keywords
tray
max
contour
depth
contours
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202410001208.9A
Other languages
Chinese (zh)
Other versions
CN117496189A (en
Inventor
王枫淇
薄迎春
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China University of Petroleum East China
Original Assignee
China University of Petroleum East China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China University of Petroleum East China filed Critical China University of Petroleum East China
Priority to CN202410001208.9A priority Critical patent/CN117496189B/en
Publication of CN117496189A publication Critical patent/CN117496189A/en
Application granted granted Critical
Publication of CN117496189B publication Critical patent/CN117496189B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/469—Contour-based spatial representations, e.g. vector-coding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/36—Applying a local operator, i.e. means to operate on image points situated in the vicinity of a given point; Non-linear local filtering operations, e.g. median filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/752—Contour matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Nonlinear Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention belongs to the technical field of material tray recognition, and discloses a rectangular tray hole recognition method and system based on a depth camera, wherein the physical width of a tray hole is obtained through measurementPhysical height of tray holeCenter distance of two tray holesSetting initial parameters, and acquiring depth map data and color map data of a depth camera; fusing the obtained depth map data and color map data into comprehensive grayA degree map; extracting the outline of the image containing elements according to the obtained comprehensive gray level diagram; removing the outline which does not accord with the characteristics of the tray holes by adopting a gradual removal method; and taking the processed residual profile as the final identified tray hole profile, and recording the physical coordinates of the center of the profile. The invention provides a method for fusing depth information and color information into comprehensive gray information, which can improve the recognition accuracy when light is darker. The invention provides a novel self-adaptive threshold setting method, which can improve the recognition accuracy and the accuracy of tray holes in long distance.

Description

Rectangular tray hole identification method and system based on depth camera
Technical Field
The invention belongs to the technical field of material tray recognition, and particularly relates to a rectangular tray hole recognition method and system based on a depth camera.
Background
The material tray with rectangular holes has wide application in the industrial field. Along with the increasing demand of automatic handling, the demand of identifying and positioning the material tray based on visual perception technology is urgent. The depth camera can obtain object color and distance information, and is widely applied to industrial material tray identification. At present, the identification method of the material tray mainly comprises two main types: the method is a machine learning method, the method adopts tools such as an artificial neural network and the like to obtain a model of the tray to be identified through training, then the model is matched to identify whether a target object exists in an image or not and obtain the position of the target object, the method does not need to care about the specific characteristics of the target object, but a large number of pictures of the tray to be identified are required to be provided for training, the time and labor cost are high, and the calculation amount is large. The other is a contour matching method. The method firstly identifies the outline of an object in an image based on the information acquired by the depth camera, then determines whether a tray to be identified exists or not through outline characteristics, and calculates the pose of the tray (such as patent number 2022100594184, 2023101105090). The outline of the tray is easily influenced by the goods placed above the tray, so that the identification mode has strict requirements on the goods placing position above the tray. Secondly, the image is mostly scanned along the horizontal and vertical directions during contour extraction, which requires that the X plane of the camera and the plane of the object placed on the tray are always kept in parallel, and in order to achieve the problem, besides strict requirements on the installation position of the camera, the flatness of the field around the tray is required to be higher, otherwise, the inclination can be generated during the running process of the vehicle, so that the camera and the tray are not strictly positioned on the same plane, and at the moment, the contour scanning can generate larger errors. Again, contour recognition is greatly affected by light, and too strong or too dark light can lead to insufficient contour information, so that deviation occurs in contour recognition.
Through the above analysis, the problems and defects existing in the prior art are as follows:
(1) The existing material tray identification method based on machine learning needs to provide a large number of pictures of trays to be identified for training, the time and labor cost are high, and the implementation difficulty is high for the general industrial field.
(2) The existing material tray identification method based on contour matching has strict requirements on material placement, camera installation, ground flatness, light rays and the like, and is difficult to adapt to complex and changeable scenes and non-standardized industrial environments.
Disclosure of Invention
In order to overcome the problems in the related art, the disclosed embodiments of the present invention provide a rectangular tray hole recognition method and system based on a depth camera, and the technical scheme is as follows:
the invention is realized in such a way that a rectangular tray hole identification method based on a depth camera comprises the following steps:
s1, obtaining the physical width of the tray hole through measurementTray hole physical height +>Center distance of two tray holes>Setting initial parameters;
s2, acquiring depth map data and color map data of a depth camera;
s3, fusing the obtained depth map data and the color map data into a comprehensive gray map;
s4, extracting the outline of the image containing elements according to the obtained comprehensive gray level diagram;
S5, removing the outline which does not accord with the characteristics of the tray hole by adopting a gradual removal method;
s6, taking the processed residual profile as the final identified tray hole profile, and recording the physical coordinates of the center of the profile.
In step S1, the initial parameters include: actual physical width of tray holeTray hole actual physical height +>The actual distance between the centers of the two tray holes +.>Red component weighting factor->Green component weighting factor->Blue component weighting factor->，Moment similarity threshold +.>Pixel upper threshold +.>Pixel lower threshold +.>Tray hole width error thresholdTray hole height error threshold +.>Tray hole center distance error threshold +.>Tray Kong Qingjiao difference threshold +.>Gray level suppression threshold->Upper depth limit->Depth lower limit->Gray value weighting factor->。
In step S3, the obtained depth map data and color map data are fused into a comprehensive gray scale map, including:
s2.1, converting the color map to obtain gray map color components;
s2.2, converting the depth map to obtain a gray map depth component;
s2.3, superposing the obtained gray value color and depth component to obtain a comprehensive gray image.
Further, the formula for converting the color map to obtain the gray map color component is as follows:
；
In the method, in the process of the invention,for pixels +.>Corresponding gray-value color component,/>For rounding function, ++>Weighting coefficients for the red component +.>Weighting coefficients for the green component,/->Weighting coefficients for the blue component, ">Is pixel dot +.>Corresponding red component->Is pixel dot +.>Corresponding green component, < >>Is pixel dot +.>A corresponding blue component;
the formula for converting the depth map to obtain the depth component of the gray map is as follows:
；
in the method, in the process of the invention,for pixels +.>Corresponding gray value depth component,/>For pixels +.>The corresponding depth value is used to determine the depth value,for the upper depth value limit, ++>Is the lower limit of the depth value;
the formula for obtaining the comprehensive gray map by superposing the obtained gray value color and depth component is as follows:
；
in the method, in the process of the invention,for pixels +.>Corresponding integrated gray value +.>Weighting coefficients for the gray values.
In step S4, extracting the outline of the image-containing element from the obtained integrated gray-scale image, including:
s4.1, restraining high gray value points, and setting gray suppression threshold valuesWhen the pixel point is larger than the gray suppression threshold +.>At this time, let the pixel equal to the gray-scale suppression threshold +.>；
S4.2, performing Gaussian filtering on the processed image;
s4.3, calculating entropy of the processed image;
s4.4, determining a high thresholdLow threshold- >；
S4.5, will be high thresholdLow threshold->As a parameter, contour extraction is performed on the image processed in step S4.2.
In step S4.3, the formula for calculating entropy for the processed image is:
；
in the method, in the process of the invention,information entropy for a comprehensive gray-scale image, +.>For gray values in the integrated gray image +.>Probability of occurrence;
the high threshold is calculated according to the following formulaLow threshold->：
；
；
In the method, in the process of the invention,high threshold value actually used for contour extraction algorithm, < +.>Is natural constant (18)>Information entropy for a comprehensive gray-scale image, +.>For use +.>High threshold value calculated by algorithm, < >>A low threshold value actually used for the contour extraction algorithm;
in step S4.5, the high threshold value is setLow threshold->As a parameter, performing contour extraction on the image processed in step S4.2, including:
image pixel point after calculation processingThe gradient of (2) is calculated as:
；
in the method, in the process of the invention,is pixel dot +.>Corresponding gradient values,/->Is->Is pixel dot +.>Corresponding->Direction and->A gradient component of direction;
calculating gradient direction and pixel pointThe corresponding gradient direction calculation formula is:
；
in the method, in the process of the invention,is pixel dot +.>Gradient direction of->Is pixel dot +.>Corresponding->The direction of the gradient of the direction,is pixel dot +.>Corresponding->A gradient direction of the direction;
By means ofThe neighborhood is scanned over the entire image,pixel point along neighborhood +.>Gradient direction determination ++>Whether it is maximum in the gradient direction, if not, let the gradient value +.>And makes the following judgment: if gradient valueLet gradient value->The method comprises the steps of carrying out a first treatment on the surface of the If gradient value +>Let gradient value->The method comprises the steps of carrying out a first treatment on the surface of the If it isJudging->Is->If there is a point with a value of 255 in the neighborhood, if there is, let the gradient value +.>Otherwise, let gradient value ++>；/>Is the gradient lower threshold +_>Is the upper threshold of the gradientValues.
In step S5, a step-by-step elimination method is used to eliminate contours that do not conform to the tray hole features, including:
s5.1, obtaining minimum outer-wrapping rectangles of all the contours, and recording image coordinates of 4 vertexes of each minimum outer-wrapping rectangle;
s5.2, calculate the firstThe number of pixels in the minimum outline outsourcing rectangle>Wherein->Is the number of remaining contours; if->Or->Deleting the corresponding outline; />Is the upper limit threshold value of the pixel point, +.>Is a pixel lower threshold;
s5.3, calculating the residual outline and the minimum outsourcing rectangleMoment similarity->If->Deleting the corresponding outline; />Is->Moment similarity threshold;
s5.4, for each remaining contour, calculating the corresponding physical width of the contour according to the physical coordinates corresponding to the minimum outsourcing rectangle vertex And height->Remove not simultaneously satisfying->And->Is a contour of (2); />For tray hole width error threshold, +.>For tray hole height error threshold, +.>For measuring the physical width of the tray wells obtained, +.>Measuring the physical height of the obtained tray hole;
s5.5, performing contour matching on the rest contours, and removing contours with failed matching; calculating the center distance of two contours during matchingTwo contour centerlines and a camera->Included angle of plane->And->The method comprises the steps of carrying out a first treatment on the surface of the At the same time satisfy->Andthe two contours of the matching are regarded as successful, the contours which are successfully matched are reserved, and the contours which are failed to be matched are removed; />For the center distance error threshold of two tray holes, < +.>For tray Kong Qingjiao difference threshold, +.>Measuring the center distance of the two tray holes;
s5.6, selecting the best matching contour for the rest contours, and calculating each pair of matching contoursMoment similarity, select->The pair of contours with the smallest moment similarity is taken as the target contours.
In step S5.4, calculating the corresponding physical width of the outline according to the physical coordinates corresponding to the vertex of the minimum bounding rectangleAnd height->The calculation formula is as follows:
；
；
in the method, in the process of the invention,the physical widths of the upper side and the lower side of the minimum outsourcing rectangle of the tray hole are respectively; />The physical heights of the left side and the right side of the minimum outsourcing rectangle of the tray hole are respectively;
；
；
；
；
In the method, in the process of the invention,physical coordinates corresponding to four vertexes of the minimum enveloping rectangle of the tray hole outline respectively are +.>；
For any pixel pointCorresponding physical coordinates->The calculation formula of (2) is as follows:
；
in the method, in the process of the invention,for pixel position in the image coordinate system, +.>For pixels +.>Corresponding depth value, < > and->For the origin position of the image coordinate system,/->Is the focal length of the camera.
Further, the center distance of the two contoursThe calculation formula of (2) is as follows:
；
in the method, in the process of the invention,for the center distance of the two contours, +.>And->Respectively->Outline sum->The center physical coordinate of the outline minimum outsourcing rectangle;
two contour centerlines and cameraIncluded angle of plane->And->The calculation formula is as follows:
；
；
in the method, in the process of the invention,respectively is a profile->Image coordinates of left and lower end points of minimum bounding rectangle, +.>Respectively is a profile->The image coordinates of the left end point and the lower end point of the minimum outsourcing rectangle.
Another object of the present invention is to provide a depth camera-based rectangular tray hole recognition system for regulating and controlling the depth camera-based rectangular tray hole recognition method, the system comprising:
initializing module, obtaining physical width of tray hole by measurementTray hole physical height +>Center distance of two tray holes >Setting initial parameters;
the data acquisition module is used for acquiring depth map data and color map data of the depth camera;
the image fusion module is used for fusing the obtained depth map data and the color map data into a comprehensive gray map;
the contour extraction module is used for extracting the contour of the image containing elements according to the obtained comprehensive gray level map;
the profile screening module is used for removing the profiles which do not accord with the characteristics of the tray holes by adopting a gradual elimination method;
and the contour matching module is used for taking the residual contour after the processing as the contour of the tray hole finally obtained by recognition, and recording the physical coordinates of the center of the contour.
By combining all the technical schemes, the invention has the advantages and positive effects that: the invention provides a rectangular tray hole identification method based on a depth camera, which solves the problems of identification and positioning of rectangular tray holes. The rectangular tray hole identification method does not need to identify the whole tray, but directly identifies two rectangular tray holes on the tray; the tray hole is embedded in the side surface of the tray, and an obvious boundary exists between the tray hole and surrounding objects, so that the outline of the tray hole acquired by the depth camera is not interfered by objects above the tray; secondly, the inside of the tray hole is shielded by the tray shell, so that the influence of light is not easy to occur, and the interference of light fluctuation on the identification of the tray hole can be reduced; again, the tray aperture center distance is less than the tray width, which can reduce the need for the camera field angle. At present, the existing patent mainly identifies the industrial tray, and the technology of directly identifying the tray holes is available, so the invention provides the characteristics determination, the characteristic calculation, the characteristic identification flow and the like of the rectangular tray holes for the first time.
The invention only identifies the tray holes, the identification process is not easy to be interfered by light and goods on the tray, and larger errors in the identification process can be avoided. The invention provides a method for fusing depth information and color information into comprehensive gray information, which can improve the recognition accuracy when light is darker. Meanwhile, the invention provides a novel self-adaptive threshold setting method, which can improve the recognition accuracy of the tray holes in a long distance. The invention provides a tray Kong Shibie flow based on a gradual deleting method, and the tray placement position is not required to be considered in the identification process.
As inventive supplementary evidence of the claims of the present invention, the following important aspects are also presented:
(1) The automatic carrying is widely applied in the common industrial field, and the reliable and stable tray identification and positioning technology is a basic link for realizing the automatic carrying. At present, the cost of an intelligent carrier applied to the common industrial field is about 30 ten thousand yuan, the cost of a target identification module accounts for about 10% of the total cost, and as far as the chemical industry is concerned, chemical enterprises with the national scale of more than 28760 have the automatic material carrying requirement, 2 automatic carriers are configured for each enterprise to calculate, the market scale of the automatic carrier is about 172 hundred million yuan, the cost of the target identification module is about 17.2 hundred million yuan, and the market after the technical scheme of the invention is converted is calculated according to 15% of the market occupation rate of the chemical enterprises, and the market scale is about 2.6 hundred million yuan. According to the calculation of the yield rate of 10%, the expected yield of the invention in the chemical industry can reach 0.26 hundred million yuan, and the expected yield is higher in consideration of the fact that the invention can be popularized to other industries.
(2) At present, the tray identification technology is widely applied in the loading and unloading field (such as logistics industry) with higher standardization degree. But the application in the field of general process industry is less, mainly because the loading and unloading operation of different industrial enterprises is more different, and the standardization is difficult to achieve. The invention mainly aims at the characteristics of the tray in the process industry field, realizes the accurate identification and positioning of the tray in a non-standardized occasion, and has no technology in the domestic process industry field at present.
(3) The invention solves the problem of high-precision identification and positioning of a common tray in the field of process industry, and provides a favorable support for popularization and application of automatic material handling technology in the field. In addition, the recognition method has small calculated amount, can realize the recognition of 5-8 frames of dynamic pictures in 1 second, is superior to a machine learning algorithm in recognition efficiency, and can avoid the tedious work of sample collection, labeling and the like in the machine learning method.
(4) At present, tray identification research is mainly aimed at standardized scenes. The main method of identification is to identify the tray contour first and then determine the tray hole position. This method can quickly reduce the area to be identified, thereby reducing the amount of calculation. However, the method has strict requirements on the placement position of the materials on the tray. Compared with the main stream method, the method can directly identify the tray holes, but the method can solve the problem of difficulty in identifying the tray holes, and the identification result is not influenced by the outline of the tray, has no strict requirement on the placement position of the materials on the tray, and can be suitable for various non-standardized scenes. The automatic handling requirement in the industrial field is huge, and the industrial scene is different, and the standardization is difficult to realize. Therefore, the proposal provided by the invention has great application value for non-standardized automatic carrying process in the process industry field.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure;
FIG. 1 is a flow chart of a rectangular tray hole identification method based on a depth camera provided by an embodiment of the invention;
fig. 2 is a schematic diagram of a projection relationship between a tray hole and a camera plane in a camera coordinate system according to an embodiment of the present invention;
FIG. 3 is a schematic view of the inclination angle and center distance of a tray hole according to an embodiment of the present invention;
FIG. 4 is a flow chart of removal of non-tray hole contours by feature contrast provided by an embodiment of the present invention;
FIG. 5 is a flow chart of contour matching provided by an embodiment of the present invention;
fig. 6 is a gray scale image obtained by directly converting colors in a scene 1 according to an embodiment of the present invention;
FIG. 7 is a comprehensive gray scale map obtained by overlapping colors and depths in scene 1 according to an embodiment of the present invention;
fig. 8 is a contour diagram extracted from scene 1 according to an embodiment of the present invention;
FIG. 9 is a final identified tray hole profile in scenario 1 provided by an embodiment of the present invention;
fig. 10 is a gray scale chart obtained by directly converting colors in a scene 2 according to an embodiment of the present invention;
FIG. 11 is a comprehensive gray scale map obtained by overlapping colors and depths in a scene 2 according to an embodiment of the present invention;
FIG. 12 is a contour map extracted from scene 2 according to an embodiment of the present invention;
FIG. 13 is a final identified tray hole profile in scenario 2 provided by an embodiment of the present invention;
fig. 14 is a gray scale image obtained by directly converting colors in a scene 3 according to an embodiment of the present invention;
FIG. 15 is a comprehensive gray scale map obtained by overlapping colors and depths in scene 3 according to an embodiment of the present invention;
FIG. 16 is a contour map extracted from scene 3 according to an embodiment of the present invention;
FIG. 17 is a final identified tray hole profile in scenario 3 provided by an embodiment of the present invention;
fig. 18 is a gray scale image obtained by directly converting colors in a scene 4 according to an embodiment of the present invention;
FIG. 19 is a comprehensive gray scale map obtained by overlapping colors and depths in a scene 4 according to an embodiment of the present invention;
FIG. 20 is a contour map extracted from scene 4 according to an embodiment of the present invention;
FIG. 21 is a final identified tray hole profile in scenario 4 provided by an embodiment of the present invention;
fig. 22 is a gray scale image obtained by directly converting colors in a scene 5 according to an embodiment of the present invention;
FIG. 23 is a comprehensive gray scale map obtained by overlapping colors and depths in a scene 5 according to an embodiment of the present invention;
FIG. 24 is a contour map extracted from scene 5 provided by an embodiment of the present invention;
FIG. 25 is a final identified tray hole profile in scenario 5 provided by an embodiment of the present invention;
fig. 26 is a gray scale image obtained by directly converting colors in a scene 6 according to an embodiment of the present invention;
FIG. 27 is a comprehensive gray scale map obtained by overlapping colors and depths in a scene 6 according to an embodiment of the present invention;
FIG. 28 is a contour map extracted from scene 6 provided by an embodiment of the present invention;
FIG. 29 is a final identified tray hole profile in scenario 6 provided by an embodiment of the present invention;
fig. 30 is a gray scale image obtained by directly converting colors in a scene 7 according to an embodiment of the present invention;
FIG. 31 is a comprehensive gray scale map obtained by overlapping colors and depths in a scene 7 according to an embodiment of the present invention;
FIG. 32 is a contour map extracted from scene 7 provided by an embodiment of the present invention;
FIG. 33 is a final identified tray hole profile in scenario 7 provided by an embodiment of the present invention;
fig. 34 is a gray scale image obtained by directly converting colors in a scene 8 according to an embodiment of the present invention;
FIG. 35 is a comprehensive gray scale map obtained by overlapping colors and depths in a scene 8 according to an embodiment of the present invention;
FIG. 36 is a contour map extracted from scene 8 provided by an embodiment of the present invention;
FIG. 37 is a final identified tray hole profile in scenario 8 provided by an embodiment of the present invention;
Fig. 38 is a gray scale image obtained by directly converting colors in a scene 9 according to an embodiment of the present invention;
FIG. 39 is a comprehensive gray scale map obtained by overlapping colors and depths in a scene 9 according to an embodiment of the present invention;
FIG. 40 is a contour map extracted from scene 9 provided by an embodiment of the present invention;
FIG. 41 is a final identified tray hole profile in scenario 9 provided by an embodiment of the present invention;
FIG. 42 is a gray scale image of a scene 10 with direct color conversion according to an embodiment of the present invention;
FIG. 43 is a comprehensive gray scale map obtained by overlapping colors and depths in a scene 10 according to an embodiment of the present invention;
FIG. 44 is an extracted contour map of scene 10 provided by an embodiment of the present invention;
FIG. 45 is a final identified tray hole profile in scene 10 provided by an embodiment of the invention;
FIG. 46 shows the difference between the method of the present invention and the FAST-RCNN methodThe following comparison chart of the recognition success rate;
FIG. 47 shows the difference between the method of the present invention and the FAST-RCNN methodThe following is a graph of the recognition time.
Detailed Description
In order that the above objects, features and advantages of the invention will be readily understood, a more particular description of the invention will be rendered by reference to the appended drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The invention may be embodied in many other forms than described herein and similarly modified by those skilled in the art without departing from the spirit or scope of the invention, which is therefore not limited to the specific embodiments disclosed below.
The invention is mainly different from the prior art in that:
(1) The invention provides a tray hole direct identification method;
(2) The invention provides a color map and depth map fusion method which is suitable for the characteristics of tray holes;
(3) The invention provides a new self-adaptive threshold setting method;
(4) The invention determines the key physical characteristics of rectangular tray hole identification;
(5) The invention deduces a contour feature calculation method independent of the position and the posture of the tray based on a camera imaging principle;
(6) The invention provides a method and a process for gradually eliminating non-tray hole outlines.
Embodiment 1 as shown in fig. 1, the rectangular tray hole recognition method based on the depth camera provided by the embodiment of the invention comprises the following steps:
s1, obtaining the physical width of the tray hole through measurementTray hole physical height +>Center distance of two tray holes>Setting initial parameters;
s2, acquiring depth map data and color map data of a depth camera;
s3, fusing the obtained depth map data and the color map data into a comprehensive gray map;
s4, extracting the outline of the image containing elements according to the obtained comprehensive gray level diagram;
s5, removing the outline which does not accord with the characteristics of the tray hole by adopting a gradual removal method;
S6, taking the processed residual profile as the final identified tray hole profile, and recording the physical coordinates of the center of the profile.
Preferably, the step S3 of the present invention specifically comprises the following steps:
s301, converting the color map to obtain a gray map color component according to the following formula:
；
in the method, in the process of the invention,for pixels +.>Corresponding gray-value color component,/>For rounding function, ++>Weighting coefficients for the red component are preferablyA value of 0.299; />For the green component weighting factor, the preferred value is 0.587; />For the blue component weighting factor, the preferred value is 0.114; />Is pixel dot +.>Corresponding red component->Is pixel dot +.>A corresponding green component is used to represent the green component,is pixel dot +.>A corresponding blue component;
equation (1) is a conventional method of converting a color map into a gray map,typical values of (2) are:。
s302, converting the depth map to obtain a gray map depth component, wherein the formula is as follows:
；
in the method, in the process of the invention,for pixels +.>Corresponding gray value depth component,/>For pixels +.>The corresponding depth value is used to determine the depth value,for the upper depth value limit, ++>Is the lower limit of the depth value;
the formula (2) provides a gray level conversion method aiming at the characteristics of the tray hole, and the inner depth value of the tray hole is larger than the outer depth value because the inner part of the tray hole is empty; equation (2) makes the depth value inversely proportional to its converted gray value component, i.e., the larger the depth value, the smaller the converted gray value component (the darker the color). The conversion method ensures that the gray value at the outer side of the tray hole is higher than that at the inner side (namely, the outer side brightness is higher than that at the inner side), and can effectively improve the brightness around the tray hole, thereby solving the problem of coupling between the tray hole and the surrounding dark background when light is darker.
S303, superposing the obtained gray value color and depth component to obtain a comprehensive gray map, wherein the formula is as follows:
；
in the method, in the process of the invention,for pixels +.>Corresponding integrated gray value +.>Weighting coefficients for the gray values.
Equation (3) is provided for the characteristics of the tray holesAnd (3) synthesizing a gray level calculation formula, wherein the gray level value calculated by the formula (3) is used as basic data for subsequent contour recognition. This has two advantages: firstly, after the information of color, depth and the like is converted into comprehensive gray information, the data volume is greatly reduced, the data processing efficiency can be improved, and the requirement of data processing on hardware is reduced; secondly, depth and color information are integrated into comprehensive gray information, so that the edge of a tray hole can be highlighted more effectively, and the problem that the edge of the tray is difficult to identify when light is darker is solved. In the formula (3),it can be determined according to the light condition that +.>Conversely, the +.>。
Preferably, the step S4 of the present invention specifically comprises the following steps:
s401, restraining high gray value points, and setting gray suppression threshold valuesWhen the pixel point is larger than the gray suppression threshold +.>When the pixel point is equal to +.>The method comprises the steps of carrying out a first treatment on the surface of the Because the gray value of the tray hole is lower, the interference of highlight elements in the background can be removed by inhibiting the high gray value point;
S402, performing Gaussian filtering on the processed image;
s403, calculating entropy of the processed image, wherein the formula is as follows:
；
in the method, in the process of the invention,information entropy for a comprehensive gray-scale image, +.>For gray values in the integrated gray image +.>Probability of occurrence;
s404, determining a high thresholdLow threshold->；
；
；
In the method, in the process of the invention,high threshold value actually used for contour extraction algorithm, < +.>Is natural constant (18)>For use +.>High threshold value calculated by algorithm, < >>A low threshold value actually used for the contour extraction algorithm;
s405, high threshold valueLow thresholdValue->As a parameter, contour extraction is performed on the image processed in step S4.2.
First, the processed image pixel point is calculatedThe gradient of (2) is calculated as:
；
in the method, in the process of the invention,is pixel dot +.>Corresponding gradient values,/->Is->Is pixel dot +.>Corresponding->Direction and->A gradient component of direction;
secondly, calculating the gradient direction and the pixel pointThe corresponding gradient direction calculation formula is:
；
in the method, in the process of the invention,is pixel dot +.>Gradient direction of->Is pixel dot +.>Corresponding->The direction of the gradient of the direction,is pixel dot +.>Corresponding->A gradient direction of the direction;
finally, utilizeScanning the whole image in the neighborhood, and adding pixels in the neighborhood>Gradient direction determination ++>Whether it is maximum in the gradient direction, if not, let +. >And makes the following judgment: if->Let->The method comprises the steps of carrying out a first treatment on the surface of the If->Let->The method comprises the steps of carrying out a first treatment on the surface of the If->JudgingIs->If there is a point with a value of 255 in the neighborhood, if so, let +.>Otherwise, let->。
For the extraction algorithm, the effect of contour extraction and a high threshold valueIs closely related to (I)>Smaller contour segmentation is finer>The larger the contour segmentation the coarser. In general, a larger +.>Value, whereas the picture is more complex, a smaller +.>Values. During the running of the carrying vehicle, the images collected by the camera are changed in real time, so that the +.>The effect of contour recognition can be ensured. At presentConventional adaptive threshold setting methods generally employ the OTSU algorithm whose main idea is to determine +_by maximizing the inter-class variance>. This->The adaptive approach can meet the needs of most scenes, but is less effective for very small contour extraction. For tray hole identification, the OSTU algorithm is used to determine +.>The value can meet the contour segmentation requirement, but when the tray is far away from the camera, the tray hole has very small proportion in the image, and the OTSU algorithm is adopted to determine the +. >The profile extraction is too large to be fine enough, resulting in an inability to accurately extract the tray hole profile. To solve this problem, the present invention proposes an improved OSTU-based approach>Value adaptive determination method (equation (5)).
In general, when the camera is far from the tray, the camera vision is relatively wide because no tray is used for shielding, the acquired image contains more elements and is relatively complex, and when the camera is relatively close to the tray, the tray occupies relatively large images, the other elements occupy less space, and the elements contained in the image are relatively simple, so that the distance between the camera and the tray can be judged according to the complexity degree of the elements contained in the image. Information entropyIs one of indexes for measuring the complexity of the image, when +.>If the image is larger, the description image is more complex, otherwise, the description image is simpler. Therefore, in the formula (5) Middle use->For->The value is corrected. When the camera is far from the tray, the +.>Big (I)>Smaller, at this time->Significantly less than->When the camera is closer to the tray, < > the camera is>Larger at this timeApproach->. Due to->In the range of 0 to 1, therefore->The theoretical range of (2) is: />. Determining the threshold by equation (5) helps to improve the refinement of contour segmentation in complex scenes, which is advantageous for remote tray hole recognition.
Preferably, the step S5 of the present invention specifically comprises the following steps:
s501, obtaining minimum outer-wrapping rectangles of all the contours, and recording image coordinates of 4 vertexes of each minimum outer-wrapping rectangle;
s502, calculate the firstThe number of pixels in the minimum outline outsourcing rectangle>Wherein->Is the number of remaining contours; if->Or->Then delete the corresponding outline, ++>Is the upper limit threshold value of the pixel point, +.>Is a pixel lower threshold;
s503, calculating the residual outline and the minimum outsourcing rectangleMoment similarity->If->The corresponding contour is deleted, wherein +.>Is->Moment similarity threshold; />Smaller means that the outline has higher similarity with the minimum outsourcing rectangle, and the minimum outsourcing rectangle is because the outline of the tray hole is approximate rectangleThe similarity between the package rectangle and the outline shape is extremely high; the step can remove the outline which is obviously non-rectangular, so that the workload of subsequent calculation is greatly reduced;
s504, for each remaining outline, calculating the corresponding physical width of the outline according to the physical coordinates corresponding to the corresponding minimum outsourcing rectangle vertexAnd height->Remove not simultaneously satisfying->And->Is defined by the contour of (a),for tray hole width error threshold, +.>A tray hole height error threshold; according to the transformation relation between the physical coordinates of the camera and the coordinates of the image, < > >Can be calculated by the formula (9) and the formula (10):
；
；
wherein,
；
；
；‘
；
in the method, in the process of the invention,physical coordinates corresponding to four vertexes of the minimum enveloping rectangle of the tray hole outline respectively are +.>；
In fig. 2, the four vertex coordinates of the tray a are:，/>，/>，/>，the physical width of the upper and lower sides of the minimum bounding rectangle of the tray hole is respectively, namely ++in FIG. 2>Point to->Point and +.>Point to->Physical distance of the points; />The physical heights of the left and right sides of the minimum outsourcing rectangle of the tray hole are respectively, namely ++in FIG. 2>Point to->Point and +.>Point to->Physical distance of the points; in fig. 2, 1 is an industrial pallet; 2 is a tray hole A;3 is a tray hole B;4 is the projection of the tray hole A on the camera plane; 5 is tray hole A vertex +>The method comprises the steps of carrying out a first treatment on the surface of the 6 is tray hole A vertex->The method comprises the steps of carrying out a first treatment on the surface of the 7 is tray hole A vertex->The method comprises the steps of carrying out a first treatment on the surface of the 8 is tray hole A vertex->。
For any pixel pointWhich corresponds to physical coordinates +.>The calculation method is as follows:
；
in the formula (15) of the present invention,for pixel position in the image coordinate system, +.>For pixels +.>Corresponding depth value, < > and->For the origin position of the image coordinate system,/->Is the focal length of the camera.
Through S504, only rectangular contours conforming to the height and width of the tray holes are retained, other contours are culled;
s505, performing contour matching on the rest contours in S504, and removing contours with failed matching. Any side of the tray contains two rectangular tray holes, and only if two tray holes are identified at the same time, the tray holes can be identified successfully. The following features exist for two tray holes on either side of the tray of fig. 3: firstly, the physical distance between the centers of two rectangular holes is fixed; secondly, no matter how the trays are placed, two tray holes on the side surface of the same tray are opposite to the camera The inclination of the planes is uniform (in fig. 3, 1 is an industrial pallet, 9 is a pallet hole center line; 10 is a camera X plane). Contours conforming to these two features can be considered as a pair of matching contours. The contour matching uses a pairwise comparison (two contours are used for +.>Represented) is provided. The center distance of the two contours is calculated at the time of comparison (by +.>Representation), two contour centerlines and camera->Included angle of plane (use->Is->Representation). At the same time satisfyAnd->Is considered to be successful, the successful profile is retained, the failed profile is eliminated, < >>For the center distance error threshold of two tray holes, < +.>Is the tray Kong Qingjiao difference threshold.
Physical distance between two contour centersThe calculation method comprises the following steps:
；
in the method, in the process of the invention,and->Respectively->Outline sum->The center physical coordinate of the outline minimum outsourcing rectangle;
and->The calculation formula is as follows: />
；
；
In the method, in the process of the invention,respectively is a profile->Left side of minimum outsourcing rectangle->Down->Image coordinates of two endpoints, +.>Respectively is a profile->Left side of minimum outsourcing rectangle->Down->Image coordinates of both endpoints.
S506, selecting the best matching contour for the rest contours. Computing each pair of matching contoursMoment similarity, select->Moment similarity minimum (+) >The minimum moment similarity means that the two contours are most similar) as a target contour.
The steps S501-S506 of the invention adopt a gradual deletion method to remove the outline of the non-tray hole, and the method has two advantages:
(1) The first two steps can delete a large number of contours which do not meet the target requirement only by a small amount of calculation, and the subsequent steps only need to carry out fine feature calculation on fewer contours, so that the workload of image processing is saved to a great extent.
(2) A pair of best contours can be selected from a plurality of candidate contours through contour matching and best contour selection, so that the accuracy of the recognition of the tray holes is ensured.
(3) The feature calculation method of the formulas (9) to (18) is derived based on the camera imaging principle and the characteristics of the tray Kong Guyou, and the camera coordinate system and the world coordinate system are regarded as overlapping in calculation, so that the trouble of coordinate system conversion is avoided, the pose of the tray is not needed to be considered, and the calculation method has wide scene adaptability. In the formulas (9) to (16), feature calculation is performed based on physical coordinates, and the calculated width, height, distance, etc. correspond to actual physical dimensions of the object, and these physical dimensions are independent of the placement position and angle of the object. In equations (17), (18), the contour tilt angle is calculated using image coordinates, which may not coincide with the actual placement tilt angle of the object. However, in the feature discrimination, the inclination angle is not taken as a basis, but the inclination angle difference of two contours As a basis, for the actual tray hole outline, no matter the physical coordinates or the image coordinates are adopted for calculation, the inclination angle difference of the two tray hole outlines on the same side face of the tray is very close to 0 DEG, in this case, the inclination angle difference is calculated by adopting the physical coordinates and the image coordinates>The difference is small, and the image coordinates are used for calculating relative simplicitySingly, therefore, equation (17) and equation (18) employ image coordinate calculation.
The flow of steps S501 to S504 of the present invention is shown in FIG. 4, and the flow of step S505 is shown in FIG. 5.
In embodiment 2, in the embodiment of the invention, the resolution of the color map and the depth map of the camera are 640×360, the depth measurement distance is 10m at maximum, and the object is to identify the physical coordinates of the centers of two tray holes on the tray. The following is a detailed description of the steps.
S1: measuring the width, the height and the center distance of two tray holes, wherein the measuring result is as follows:，，/>. The initial parameters were set as follows: />，/>，/>，/>，，/>，/>，/>，/>，/>，，/>，/>。
S2: and acquiring a color image and a depth image acquired by the depth camera.
S3: and (3) converting the color map and the depth map into a comprehensive gray scale map according to formulas (1), (2) and (3).
S4: the threshold values are obtained by adopting the formulas (5) and (6), and the contour of the comprehensive gray level map obtained by the step S3 is obtained by adopting the contour extraction algorithm described by the step S405, and other contour extraction algorithms can be used in the contour extraction algorithm of the step, but the threshold values are set by adopting the methods of the formulas (5) and (6) only and are within the protection scope of the invention.
S5: a gradual removal process is used to remove contours that do not conform to the tray hole features. In this step, key features may also be reduced or added according to practical situations, such as adding features of contour area, depth difference between contour edge and contour center point, etc., or different feature combinations may be used to filter the non-tray hole contour. Non-adopted in selecting best matching contoursMoment similarity criteria, such as other criteria including a nearest criterion to the camera, a minimum criterion of the inclination angle of the contour, etc., or a combination of two or more criteria; in this step, some of the reject links or modification of the judgment criteria of some links may be reduced, for example, the operation of S501 or S502 is not performed, and S503 is directly entered, or the judgment criteria of S501 and S502 are modified; in this step, the partial feature calculation may also be performed by other methods, such as calculating +.>And->The method comprises the steps of carrying out a first treatment on the surface of the The foregoing modifications are considered as consistent with the spirit and principles of the invention.
S6: and (5) taking the residual profile in the step S5 as the final identified tray hole profile, and recording the physical coordinates of the tray hole profile.
The above experiments were performed in 10 different scenarios, respectively:
scene 1: the height difference between the camera and the tray hole is 0.5 m, the distance between the camera and the center of the tray is 0.75 m (the result is shown in figures 6-9), and the scene experiment can test the tray hole identification effect when the mounting position of the camera is lower and the distance is short; as can be seen by comparing fig. 6 and 7, the overall gray scale image obtained by color and depth conversion is brighter near the tray aperture than gray scale images using only color conversion (with similar effect for subsequent scene images);
Scene 2: the height difference between the camera and the tray hole is 0.5 m, the distance between the camera and the center of the tray is 2.55 m (the result is shown in figures 10-13), and the scene experiment can test the tray hole identification effect when the mounting position of the camera is lower and the distance is middle;
scene 3: the height difference between the camera and the tray hole is 0.5 m, the distance between the camera and the center of the tray is 6.55 m (the result is shown in figures 14-17), and the scene experiment can test the tray hole identification effect when the mounting position of the camera is lower and the distance is long;
scene 4: the height difference between the camera and the tray hole is 1.0 meter, the distance between the camera and the center of the tray is 0.95 meter (the result is shown in fig. 18-21), and the scene experiment can test the tray hole identification effect when the mounting position of the camera is higher and the camera is close;
scene 5: the height difference between the camera and the tray hole is 1.0 meter, the distance between the camera and the center of the tray is 3.35 meters (the result is shown in fig. 22-25), and the scene experiment can test the tray hole identification effect when the camera is arranged at a higher position and at a middle distance;
scene 6: the height difference between the camera and the tray hole is 1.0 meter, the distance between the camera and the center of the tray is 7.25 meters (the result is shown in fig. 26-29), and the scene experiment can test the tray hole identification effect when the mounting position of the camera is higher and the distance is long;
scene 7: the height difference between the camera and the tray hole is 1.0 meter, and the tray hole recognition effect is achieved when the camera and the tray are at a certain angle (the result is shown in fig. 30-33).
Scene 8: the height difference between the camera and the tray hole is 1.0 meter, and the tray hole recognition effect is achieved when the camera and the tray are at a certain angle (the result is shown in fig. 34-37).
The experiment of the scene 7 and the scene 8 can test the recognition effect of the camera at different angles with the tray when the installation position of the camera is higher;
scene 9: the height difference between the camera and the tray hole is 1.0 m, the distance between the camera and the center of the tray is 2.75 m, the light around the tray hole is darker (the result is shown in figures 38-41), and the scene experiment can test the tray hole identification effect when the light is darker; FIG. 38 is a gray scale image directly converted from a color image (where the tray hole is coupled to the surrounding background, and the camera has difficulty extracting the tray hole outline from the gray scale image); FIG. 39 is a composite gray scale map resulting from color and depth superposition (the composite gray scale map increases the brightness around the tray hole, making the tray hole outline significantly clearer than FIG. 38); fig. 40 is an extracted contour map, and fig. 41 is a final recognition result map. When the light around the tray hole is darker, the comprehensive gray level graph can obviously improve the brightness around the tray hole and highlight the outline of the tray hole, so that the difficulty of extracting the outline of the tray hole is reduced, and the accuracy of identifying the tray hole and the coordinate positioning precision are improved.
Scene 10: the height difference between the camera and the tray hole is 1.0 meter, the distance between the camera and the tray is 7.25 meters, the image contains more elements (the result is shown in fig. 42-45), and the scene experiment can test the tray hole identification effect when the camera is far away from the tray hole and the image elements in the visual field are more;
the above scenario verifies the recognition effect of the present invention under different conditions, including:
(1) The recognition effect of the camera when the camera is near, middle and far from the tray hole (scenes 1, 2 and 3 and scenes 4, 5 and 6);
(2) The recognition effect when the camera forms a certain angle with the tray hole (scenes 7 and 8);
(3) The light around the tray hole is darker, and the recognition effect (scene 9) is achieved when the tray hole is coupled with the surrounding background;
(4) The recognition effects when the mounting heights of the cameras are different, wherein scenes 1, 2 and 3 are near, middle and long-distance recognition effects when the mounting heights of the cameras are lower, and scenes 4, 5 and 6 are near, middle and long-distance recognition effects when the mounting heights of the cameras are higher;
(5) Contour extraction effect when there are more elements in the scene (scene 10). Experimental results show that the tray hole identification method can realize accurate identification when the camera and the tray hole are in different angles in long, medium and short distances, and the method has no specific requirements on the installation position and the height of the camera, and the tray hole position can be captured as long as the tray hole appears in the effective visual field and the visual range of the camera in the running process of the vehicle.
Objects in a wide range in front of the tray holes are arranged in scenes 1-9, so that other objects are basically absent, and the objects are calculated at the momentThe range is 0.70-0.82 (different scenes are different), namely the high threshold value obtained by the method is +.>Is that. In the scene 10, objects are randomly added, so that the image elements are added, the information entropy of the image is increased, and in this scene, the content is +.>About->. As can be seen from FIG. 44, due to +.>The reduction, the outline extraction is obviously finer than the previous scene, which is beneficial to the accurate extraction farDistance profile. Theoretically, in all scenarios, +.>Are all lower than->The more complex the scene, the more intuitive>Ratio->The more low. The front scene can also extract the remote tray hole contour, but the remote contour has a certain deformation, and the contour deformation of fig. 44 is obviously lighter than that of the front scene. Therefore, the self-adaptive threshold method provided by the invention has advantages in the aspect of remote contour extraction, in particular to the remote contour extraction of complex scenes.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
It will be clear to those skilled in the art that, for convenience and brevity of description, the method of the present invention will be described by taking the above scenario as an example, and the actual application may be a combination of different scenarios or use different parameters from the above scenario.
According to the embodiments of the present application, the present invention further provides a computer program for implementing the method, where the computer program can run with various operating systems (such as Windows, linux, android, etc.) and various computer devices (such as computers, mobile phones, etc.), and running the program can implement the steps in any of the embodiments described above. The program can also be used for carrying out online modification and setting on the corresponding parameters in the invention.
To further demonstrate the positive effects of the above embodiments, the present invention also providesThe success rate and accuracy of the identification are tested when the parameters take different values. />When it is a pure colour image +.>The time is a pure depth image, and fig. 46 and 47 show the difference +.>The value and the method of the invention are compared with the success rate and the recognition time of FAST-RCNN recognition. FAST-RCNN is a commonly used image recognition method based on machine learning technology in the field of artificial intelligence. The experiment adopts an Arm embedded processor, the CPU frequency is 1.50GHz, the memory is 4G, and the image resolution is 640 multiplied by 360. Only adjust +.>Value for each ofAnd dynamically acquiring 1000 pictures, and calculating the overall success rate of the pictures and the average time of the identification of each picture according to the identification result of the 1000 pictures. The accuracy here refers to the spatial distance of the actual position of the center of the tray hole from the identified position. The accuracy less than 2cm is regarded as successful recognition, otherwise, recognition failure is regarded as failure. The success rate is as follows: number of successful recognition/total number of recognition. As can be seen from FIG. 46, FAST-RCNN is for color pictures (++ >Larger value), the recognition effect is better, the recognition success rate can reach 98.5% at the highest, and the depth picture is (/ -or more)>Larger value) is poor in recognition effect, whenβThe recognition success rate is only 78.4% when the value is 0.
In contrast, the success rate of the method for identifying the color picture is slightly lower than that of FAST-RCNN, but whenProper value selectionAbout 0.5-0.7), the success rate of the invention can reach 98.2% at most, which is equivalent to FAST-RCNN. The success rate of the identification of the full depth image can reach 89.2 percent, which is far more than that of the FAST-RCNN identification method. The success rate of the FAST-RCNN identification method depends on the accuracy of learning sample labeling, the edges are relatively clear for color samples, manual labeling is easy, the labeling accuracy is high, the edges are fuzzy for samples blended with depth information, and the accuracy of manual labeling can be obviously reduced. This can significantly affect the learning effect of FAST-RCNN and the accuracy of the test, and how to implement accurate labeling of fuzzy edges remains an unresolved problem in the field of artificial intelligence.
The method provided by the invention avoids the difficulty of manual marking. As can be seen from fig. 47, the method of the present invention is significantly superior to the FAST-RCNN method in terms of recognition time, and almost all the existing artificial intelligence methods have the problem of large calculation amount, and in the case of low accuracy requirement, the problem can be solved by reducing the resolution of the image, while in the case of high accuracy requirement, other auxiliary measures such as defining a specific scene, assisting other sensors, etc. must be used. The calculated amount of the method provided by the invention is obviously smaller than that of the FAST-RCNN method, and the recognition time is reduced along with the increase of the depth information proportion. This is mainly due to the fact that color pictures are complex, a large amount of edge processing is needed during recognition, and many non-tray hole edges are obviously deformed after depth information is added, so that the color pictures are easy to filter. When the depth ratio is 0.5-0.7, the image recognition time of each frame is about 0.17 seconds, and the FAST-RCNN is about 0.42 seconds, so that the method has remarkable advantages in the aspect of recognition speed.
While the invention has been described with respect to what is presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Claims (9)

1. A rectangular tray hole identification method based on a depth camera is characterized by comprising the following steps:
s1, obtaining the physical width w of the tray hole through measurement 0 Physical height h of tray hole 0 Distance d between centers of two tray holes 0 Setting initial parameters;
s2, acquiring depth map data and color map data of a depth camera;
s3, fusing the obtained depth map data and the color map data into a comprehensive gray map;
s4, extracting the outline of the image containing elements according to the obtained comprehensive gray level diagram;
s5, removing the outline which does not accord with the characteristics of the tray hole by adopting a gradual removal method;
s6, taking the processed residual profile as the final identified tray hole profile, and recording the physical coordinates of the center of the profile;
in step S5, a step-by-step elimination method is used to eliminate contours that do not conform to the tray hole features, including:
S5.1, obtaining minimum outer-wrapping rectangles of all the contours, and recording image coordinates of 4 vertexes of each minimum outer-wrapping rectangle;
s5.2, calculating the number N of pixel points in the minimum outer wrapping rectangle of the kth outline k ,k＝1…C NUM Wherein C NUM Is the number of remaining contours; if N k >N MAX Or N k <N MIN Deleting the corresponding outline; n (N) MAX Is the upper limit threshold value of the pixel point, N MIN Is a pixel lower threshold;
s5.3, calculating the Hu moment similarity S between each residual contour and the minimum outsourcing rectangle k ,k＝1…C NUM If S k >S MAX Deleting the corresponding outline; s is S MAX Is a Hu moment similarity threshold;
s5.4, for each remaining contour, calculating the corresponding physical width w and height h of the contour according to the physical coordinates corresponding to the minimum outsourcing rectangle vertex, and removing the difference of meeting the condition of |w-w 0 |≤ε w,MAX And |h-h 0 |≤ε h,MAX Is a contour of (2); epsilon w,MAX Epsilon is the tray hole width error threshold h,MAX As a tray hole height error threshold,w 0 for the physical width of the tray hole, h 0 Physical height for the tray aperture;
s5.5, performing contour matching on the rest contours, and removing contours with failed matching; calculating the center distance d of two contours during matching AB Included angle gamma between two contour center lines and camera X plane A And gamma B The method comprises the steps of carrying out a first treatment on the surface of the At the same time satisfy |d AB -d 0 |≤ε d,MAX And |gamma A -γ B |≤ε γ,MAX The two contours of the matching are regarded as successful, the contours which are successfully matched are reserved, and the contours which are failed to be matched are removed; epsilon d,MAX Is the error threshold value of the center distance of the two tray holes epsilon γ,MAX Is tray Kong Qingjiao difference threshold, d 0 The center distance of the two tray holes is the same;
s5.6, selecting the best matching contour for the rest contours, calculating the Hu moment similarity of each pair of matching contours, and selecting a pair of contours with the minimum Hu moment similarity as target contours.
2. The depth camera-based rectangular tray hole recognition method according to claim 1, wherein in step S1, the initial parameters include: actual physical width w of tray hole 0 Actual physical height h of tray hole 0 Actual distance d between centers of two tray holes 0 Weighting coefficient a of red component 1 Green component weighting coefficient a 2 Weighting coefficient a of blue component 3 Hu moment similarity threshold S MAX Pixel upper limit threshold N MAX Pixel lower threshold value N MIN Tray hole width error threshold epsilon w,MAX Tray hole height error threshold epsilon h,MAX Tray hole center distance error threshold epsilon d,MAX Tray Kong Qingjiao difference threshold ε γ,MAX Gray level suppression threshold V MAX Upper depth limit D MAX Lower depth limit D MIN The gray value weighting coefficient beta.
3. The depth camera-based rectangular tray hole recognition method according to claim 1, wherein in step S3, fusing the obtained depth map data and color map data into a comprehensive gray map comprises:
S2.1, converting the color map to obtain gray map color components;
s2.2, converting the depth map to obtain a gray map depth component;
s2.3, superposing the obtained gray value color and depth component to obtain a comprehensive gray image.
4. A depth camera-based rectangular tray hole recognition method according to claim 3, wherein the formula for converting the color map to obtain gray-scale map color components is as follows:
V c (i,j)＝Round[a 1 R(i,j)+a 2 G(i,j)+a 3 B(i,j)]
wherein V is c (i, j) is the gray value color component corresponding to pixel (i, j), round is a rounding function, a 1 Weighting coefficients for the red component, a 2 Weighting coefficients for the green component, a 3 For the blue component weighting coefficient, R (i, j) is the red component corresponding to the pixel point (i, j), G (i, j) is the green component corresponding to the pixel point (i, j), and B (i, j) is the blue component corresponding to the pixel point (i, j);
the formula for converting the depth map to obtain the depth component of the gray map is as follows:
V D (i,j)＝Round[255(1-(D(i,j)-D MIN )/(D MAX -D MIN ))]
wherein V is D (i, j) is the gray value depth component corresponding to pixel (i, j), D (i, j) is the depth value corresponding to pixel (i, j), D MAX For the upper limit of depth value, D MIN Is the lower limit of the depth value;
the formula for obtaining the comprehensive gray map by superposing the obtained gray value color and depth component is as follows:
V(i,j)＝βV C (i,j)+(1-β)V D (i,j)
where V (i, j) is the integrated gray value corresponding to pixel (i, j), and β is the gray value weighting coefficient.
5. The depth camera-based rectangular tray hole recognition method according to claim 1, wherein in step S4, extracting the outline of the image-containing element from the obtained integrated gray scale map comprises:
s4.1, restraining high gray value points, and setting a gray suppression threshold V MAX When the pixel point is larger than the gray suppression threshold V MAX When the pixel point is equal to the gray level suppression threshold V MAX ；
S4.2, performing Gaussian filtering on the processed image;
s4.3, calculating entropy of the processed image;
s4.4, determining a high threshold V H Low threshold V L ；
S4.5, high threshold V H Low threshold V L As a parameter, contour extraction is performed on the image processed in step S4.2.
6. The depth camera-based rectangular tray hole recognition method according to claim 5, wherein in step S4.3, the formula for calculating entropy for the processed image is:
wherein E is the information entropy of the comprehensive gray level image, p k The probability of the occurrence of the gray value k in the comprehensive gray image;
the high threshold V is calculated according to the following formula H Low threshold V L ：
V H ＝Round(e -E V H,OSTU )
V L ＝0.5V H
Wherein V is H The high threshold value actually used for the contour extraction algorithm is E, which is a natural constant, E is the information entropy of the comprehensive gray level image, V H,OSTU For a high threshold calculated using the OSTU algorithm, V L A low threshold value actually used for the contour extraction algorithm;
in step S4.5, the high threshold V H Low threshold V L As a parameter, for the image processed in step S4.2Performing contour extraction, including:
the gradient of the pixel point (i, j) of the image after the calculation processing is calculated, and the calculation formula is as follows:
wherein GD (i, j) is a gradient value corresponding to the pixel point (i, j), GD X (i, j) and GD Y (i, j) is the gradient component in the X direction and the Y direction corresponding to the pixel point (i, j);
the gradient direction is calculated, and the gradient direction calculation formula corresponding to the pixel point (i, j) is as follows:
wherein θ (i, j) is the gradient direction of the pixel point (i, j), G x (i, j) is the gradient direction of the X direction corresponding to the pixel point (i, j), G y (i, j) is the gradient direction of the Y direction corresponding to the pixel point (i, j);
using 3×3 neighborhood to scan the whole image, judging whether GD (i, j) is maximum in gradient direction along gradient direction of pixel point (i, j) in neighborhood, if not, making gradient value GD (i, j) =0, and judging as follows: if the gradient value GD (i, j) is not less than V H Let the gradient value GD (i, j) =255; if the gradient value GD (i, j) is less than or equal to V L Let the gradient value GD (i, j) =0; if GD MIN <GD(i,j)<GD MAX Judging whether a point with a value of 255 exists in a 3×3 neighborhood of the GD (i, j), if so, making the gradient value GD (i, j) =255, otherwise, making the gradient value GD (i, j) =0; GD (graphics device) MIN Is a gradient lower threshold value GD MAX Is the gradient upper threshold.
7. The depth camera-based rectangular tray hole recognition method according to claim 1, wherein in step S5.4, the corresponding physical width w and height h of the outline are calculated according to the physical coordinates corresponding to the minimum outsourcing rectangular vertex, and the calculation formula is:
wherein w is 1 ,w 2 The physical widths of the upper side and the lower side of the minimum outsourcing rectangle of the tray hole are respectively; h is a 1 ,h 2 The physical heights of the left side and the right side of the minimum outsourcing rectangle of the tray hole are respectively;
wherein x is k ,y k ,z k The physical coordinates corresponding to the four vertexes of the minimum outsourcing rectangle of the tray hole outline are respectively k=0, 1,2 and 3;
for any pixel (i, j), the calculation formula of the corresponding physical coordinates (x, y, z) is:
where (i, j) is the pixel position in the image coordinate systemD (i, j) is the depth value corresponding to pixel (i, j), (i) 0 ,j 0 ) And f is the focal length of the camera.
8. The depth camera-based rectangular tray hole recognition method according to claim 1, wherein the center distance d of the two contours AB The calculation formula of (2) is as follows:
wherein d AB Is the center distance of the two contours, (x) A ,y A ,z A ) And (x) B ,y B ,z B ) The central physical coordinates of the minimum outer package rectangle of the A contour and the B contour are respectively;
Included angle gamma between two contour center lines and camera X plane A And gamma B The calculation formula is as follows:
wherein (i) A,0 ,j A,0 ),(i A,1 ,j A,1 ) The coordinates of the image of the left and the lower end points of the minimum outsourcing rectangle of the outline A are respectively, (i) B,0 ,j B,0 ),(i B,1 ,j B,1 ) The image coordinates of the left end point and the lower end point of the minimum outsourcing rectangle of the outline B are respectively.
9. A depth camera-based rectangular tray hole recognition system for regulating and controlling the depth camera-based rectangular tray hole recognition method according to any one of claims 1 to 8, the system comprising:
the module is initialized and the process is performed,for obtaining by measurement the physical width w of the tray aperture 0 Physical height h of tray hole 0 Distance d between centers of two tray holes 0 Setting initial parameters;
the data acquisition module is used for acquiring depth map data and color map data of the depth camera;
the image fusion module is used for fusing the obtained depth map data and the color map data into a comprehensive gray map;
the contour extraction module is used for extracting the contour of the image containing elements according to the obtained comprehensive gray level map;
the profile screening module is used for removing the profiles which do not accord with the characteristics of the tray holes by adopting a gradual elimination method;
the contour matching module is used for taking the residual contour after the processing as the contour of the tray hole obtained by final recognition and recording the physical coordinates of the center of the contour;
Wherein, the step-by-step elimination method is adopted to remove the outline which does not accord with the characteristics of the tray holes, and the step-by-step elimination method comprises the following steps:
obtaining the minimum outer-wrapping rectangle of each contour, and recording the image coordinates of 4 vertexes of each minimum outer-wrapping rectangle;
calculating the number N of pixel points in the minimum outer wrapping rectangle of the kth outline k ,k＝1…C NUM Wherein C NUM Is the number of remaining contours; if N k >N MAX Or N k <N MIN Deleting the corresponding outline; n (N) MAX Is the upper limit threshold value of the pixel point, N MIN Is a pixel lower threshold;
calculating the Hu moment similarity S of each residual contour and the minimum outsourcing rectangle k ,k＝1…C NUM If S k >S MAX Deleting the corresponding outline; s is S MAX Is a Hu moment similarity threshold;
for each remaining contour, calculating the corresponding physical width w and height h of the contour according to the physical coordinates corresponding to the minimum outsourcing rectangle vertex, and removing the difference to satisfy the condition of |w-w 0 |≤ε w,MAX And |h-h 0 |≤ε h,MAX Is a contour of (2); epsilon w,MAX Epsilon is the tray hole width error threshold h,MAX Is the tray hole height error threshold value，w 0 For the physical width of the tray hole, h 0 Physical height for the tray aperture;
performing contour matching on the rest contours, and removing contours with failed matching; calculating the center distance d of two contours during matching AB Included angle gamma between two contour center lines and camera X plane A And gamma B The method comprises the steps of carrying out a first treatment on the surface of the At the same time satisfy |d AB -d 0 |≤ε d,MAX And |gamma A -γ B |≤ε γ,MAX The two contours of the matching are regarded as successful, the contours which are successfully matched are reserved, and the contours which are failed to be matched are removed; epsilon d,MAX Is the error threshold value of the center distance of the two tray holes epsilon γ,MAX Is tray Kong Qingjiao difference threshold, d 0 The center distance of the two tray holes is the same;
and selecting the best matching contour from the rest contours, calculating the Hu moment similarity of each pair of matching contours, and selecting a pair of contours with the minimum Hu moment similarity as target contours.
CN202410001208.9A 2024-01-02 2024-01-02 Rectangular tray hole identification method and system based on depth camera Active CN117496189B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410001208.9A CN117496189B (en) 2024-01-02 2024-01-02 Rectangular tray hole identification method and system based on depth camera

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410001208.9A CN117496189B (en) 2024-01-02 2024-01-02 Rectangular tray hole identification method and system based on depth camera

Publications (2)

Publication Number Publication Date
CN117496189A CN117496189A (en) 2024-02-02
CN117496189B true CN117496189B (en) 2024-03-22

Family

ID=89673058

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410001208.9A Active CN117496189B (en) 2024-01-02 2024-01-02 Rectangular tray hole identification method and system based on depth camera

Country Status (1)

Country Link
CN (1) CN117496189B (en)

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105500370A (en) * 2015-12-21 2016-04-20 华中科技大学 Robot offline teaching programming system and method based on somatosensory technology
CN107578418A (en) * 2017-09-08 2018-01-12 华中科技大学 A kind of indoor scene profile testing method of confluent colours and depth information
CN109272577A (en) * 2018-08-30 2019-01-25 北京计算机技术及应用研究所 A kind of vision SLAM method based on Kinect
CN110070557A (en) * 2019-04-07 2019-07-30 西北工业大学 A kind of target identification and localization method based on edge feature detection
CN110648367A (en) * 2019-08-15 2020-01-03 大连理工江苏研究院有限公司 Geometric object positioning method based on multilayer depth and color visual information
CN111485475A (en) * 2020-04-23 2020-08-04 陕西国防工业职业技术学院 Pavement pit recognition method and device
CN114078220A (en) * 2022-01-19 2022-02-22 浙江光珀智能科技有限公司 Tray identification method based on depth camera
CN114863250A (en) * 2022-04-06 2022-08-05 淮阴工学院 Container lockhole identification and positioning method, system and storage medium
CN114972489A (en) * 2022-05-31 2022-08-30 珠海格力智能装备有限公司 Tray identification method and device, storage medium and intelligent forklift equipment
CN115424240A (en) * 2022-08-19 2022-12-02 南威软件股份有限公司 Ground obstacle detection method, system, medium, equipment and terminal
CN115909401A (en) * 2022-11-17 2023-04-04 内蒙古工业大学 Cattle face identification method and device integrating deep learning, electronic equipment and medium
FR3135322A1 (en) * 2022-05-06 2023-11-10 Institute of Oceanographic Instrumentation, Shandong Academy of Sciences Method for online measuring the height of an object based on profile matching
CN117115249A (en) * 2023-08-03 2023-11-24 上海工程技术大学 Container lock hole automatic identification and positioning system and method

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105500370A (en) * 2015-12-21 2016-04-20 华中科技大学 Robot offline teaching programming system and method based on somatosensory technology
CN107578418A (en) * 2017-09-08 2018-01-12 华中科技大学 A kind of indoor scene profile testing method of confluent colours and depth information
CN109272577A (en) * 2018-08-30 2019-01-25 北京计算机技术及应用研究所 A kind of vision SLAM method based on Kinect
CN110070557A (en) * 2019-04-07 2019-07-30 西北工业大学 A kind of target identification and localization method based on edge feature detection
CN110648367A (en) * 2019-08-15 2020-01-03 大连理工江苏研究院有限公司 Geometric object positioning method based on multilayer depth and color visual information
CN111485475A (en) * 2020-04-23 2020-08-04 陕西国防工业职业技术学院 Pavement pit recognition method and device
CN114078220A (en) * 2022-01-19 2022-02-22 浙江光珀智能科技有限公司 Tray identification method based on depth camera
CN114863250A (en) * 2022-04-06 2022-08-05 淮阴工学院 Container lockhole identification and positioning method, system and storage medium
FR3135322A1 (en) * 2022-05-06 2023-11-10 Institute of Oceanographic Instrumentation, Shandong Academy of Sciences Method for online measuring the height of an object based on profile matching
CN114972489A (en) * 2022-05-31 2022-08-30 珠海格力智能装备有限公司 Tray identification method and device, storage medium and intelligent forklift equipment
CN115424240A (en) * 2022-08-19 2022-12-02 南威软件股份有限公司 Ground obstacle detection method, system, medium, equipment and terminal
CN115909401A (en) * 2022-11-17 2023-04-04 内蒙古工业大学 Cattle face identification method and device integrating deep learning, electronic equipment and medium
CN117115249A (en) * 2023-08-03 2023-11-24 上海工程技术大学 Container lock hole automatic identification and positioning system and method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Monocular Depth Estimation with Multiscale Feature Fusion Networks;Chuqing Cao等;《2022 2nd International Conference on Algorithms, High Performance Computing and Artificial Intelligence (AHPCAI)》;20230407;742-746 *
基于图像特征分析的物体轮廓提取;王田等;《北京航空航天大学学报》;20151119;第42卷(第08期);1762-1768 *
面向智能装车卸车机器人的货物识别技术研究;余志文;《中国优秀硕士学位论文全文数据库 信息科技辑》;20190615;第2019年卷(第06期);I138-388 *

Also Published As

Publication number Publication date
CN117496189A (en) 2024-02-02

Similar Documents

Publication Publication Date Title
CN111415363B (en) Image edge identification method
CN106960208B (en) Method and system for automatically segmenting and identifying instrument liquid crystal number
EP0853293B1 (en) Subject image extraction method and apparatus
US20060262960A1 (en) Method and device for tracking objects in a sequence of images
CN110473221B (en) Automatic target object scanning system and method
US8744177B2 (en) Image processing method and medium to extract a building region from an image
CN110674812B (en) Civil license plate positioning and character segmentation method facing complex background
CN106934806A (en) It is a kind of based on text structure without with reference to figure fuzzy region dividing method out of focus
CN114693946A (en) Image anomaly detection method and device, computer equipment and storage medium
CN114938410B (en) Mobile phone screen picture-inspection partition method and system
CN112580383A (en) Two-dimensional code identification method and device, electronic equipment and storage medium
CN114241438B (en) Traffic signal lamp rapid and accurate identification method based on priori information
CN116052152A (en) License plate recognition system based on contour detection and deep neural network
CN114004858A (en) Method and device for identifying aviation cable surface code based on machine vision
CN117496189B (en) Rectangular tray hole identification method and system based on depth camera
CN111882576A (en) Method and device for classifying depth information of foreground pixels of video image and segmenting foreground
CN111667509A (en) Method and system for automatically tracking moving target under condition that target is similar to background color
CN115619796A (en) Method and device for obtaining photovoltaic module template and nonvolatile storage medium
CN114120320A (en) Image multi-target information identification method, system and medium
CN112488938A (en) Remote sensing image processing system
CN112489052A (en) Line structure light central line extraction method under complex environment
CN110826564A (en) Small target semantic segmentation method and system in complex scene image
CN113628142B (en) Picture sharpening processing system based on similarity simulation
CN117456371B (en) Group string hot spot detection method, device, equipment and medium
CN114494778B (en) Image acquisition processing system for remote monitoring of power equipment and control method thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant