WO2024077979A1 - 基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法 - Google Patents

基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法 Download PDF

Info

Publication number
WO2024077979A1
WO2024077979A1 PCT/CN2023/098787 CN2023098787W WO2024077979A1 WO 2024077979 A1 WO2024077979 A1 WO 2024077979A1 CN 2023098787 W CN2023098787 W CN 2023098787W WO 2024077979 A1 WO2024077979 A1 WO 2024077979A1
Authority
WO
WIPO (PCT)
Prior art keywords
contour
segmentation
solov2
mushroom
segment
Prior art date
Application number
PCT/CN2023/098787
Other languages
English (en)
French (fr)
Inventor
杨淑珍
朱浩宇
俞涛
Original Assignee
上海第二工业大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海第二工业大学 filed Critical 上海第二工业大学
Publication of WO2024077979A1 publication Critical patent/WO2024077979A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • A—HUMAN NECESSITIES
    • A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
    • A01G—HORTICULTURE; CULTIVATION OF VEGETABLES, FLOWERS, RICE, FRUIT, VINES, HOPS OR SEAWEED; FORESTRY; WATERING
    • A01G18/00—Cultivation of mushrooms
    • A01G18/70—Harvesting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/188—Vegetation

Definitions

  • the present invention relates to the field of smart agriculture and visual recognition technology, in particular to the field of automatic fruit and vegetable picking technology, and specifically to a method for high-precision contour segmentation and reconstruction of mushroom cluster fruiting bodies based on improved SOLOv2.
  • Mushrooms are a naturally high-clumping crop, and their fruiting bodies tend to grow densely and overlap each other.
  • the shape of mushrooms is easily changed from round to elliptical or irregular due to the limited space for dense growth.
  • the fruiting bodies also tend to change from vertical growth to an inclined posture. Therefore, it is difficult to achieve high-precision segmentation and reconstruction of the contour, resulting in low positioning accuracy of the center point of the fruiting body and low accuracy of the fruiting body size, which will reduce the success rate of automatic picking of mushroom clusters.
  • the main segmentation methods for fruiting bodies are segmentation algorithms based on traditional vision, which use circles or ellipses for reconstruction, and Boudingbox center point positioning methods based on target detection.
  • the segmentation and positioning accuracy of relatively sparse circular fruiting bodies is good, but the positioning effect of fruiting bodies in densely overlapping fruit clusters is not good.
  • Traditional vision has poor adaptability to the environment, poor robustness of recognition algorithms, high missed recognition rate for small and adhered fruiting bodies, and will mistakenly identify high-density circular mycelium clusters as fruiting bodies.
  • the edge segmentation accuracy of tilted, irregular and occluded fruiting bodies is not high.
  • the contour of the fruiting body reconstructed by a circle or ellipse alone, especially the contour of irregular and occluded fruiting bodies, is not highly consistent with the actual contour.
  • the center point positioning method based on target detection has greatly improved the robustness of fruiting body recognition, but because its target center is determined by Boundingbox, it is not very consistent with the edge of the entity, especially for tilted, complex, adhered and overlapping fruiting bodies, which will cause the size and center point position of the fruiting body to be offset from the actual one.
  • the purpose of the present invention is to provide a method for high-precision contour segmentation and reconstruction of mushroom cluster fruiting bodies based on an improved SOLOv2 (PR-SOLOv2 segmentation algorithm).
  • This method overcomes the complex situations of mushroom overlap, tilt, occlusion, uneven illumination, soil on the surface of mushroom caps, and the problem of fuzzy contour segmentation of densely adhered mushrooms during image acquisition.
  • the PR-SOLOv2 segmentation algorithm proposed in the present invention can not only accurately segment the edges of dense, overlapping, squeezed and tilted fruiting bodies and completely obtain their instance masks, but also can perform better contour fitting and reconstruction of fruiting bodies based on the contour reconstruction algorithm designed by the present invention when there are serious occlusions, squeeze deformations, etc. between mushroom fruiting bodies.
  • the purpose is to more accurately obtain the center point coordinates and shape size of the fruiting body, and provide accurate fruiting body data for subsequent picking or growth status monitoring.
  • the present invention firstly segments the mushroom cluster image by integrating the PR-SOLOv2 segmentation network of the SOLOv2 network and the PointRend module to obtain masks of each fruiting body that fit well with the actual fruiting body. Then, the mask contour data of each fruiting body is extracted, and the fruiting bodies are classified according to the curvature and radius of the mask contour of the fruiting body.
  • the least squares ellipse method is used to fit and reconstruct the fruiting bodies with regular shapes and relatively smooth contours, and the irregular contour fruiting bodies that are severely blocked or squeezed are reconstructed.
  • Step A Collect mushroom images
  • Step B Use the improved SOLOv2 algorithm to perform high-precision segmentation on the fruiting bodies in the mushroom cluster in the image, and obtain masks of each fruiting body that fits well with the actual fruiting body; wherein: the improved SOLOv2 algorithm is implemented based on the PR-SOLOv2 segmentation network that integrates the SOLOv2 network and the PointRend module; in the PR-SOLOv2 segmentation network, the PointRend module is set after the fully convolutional network FCN, the feature extraction backbone network of the SOLOv2 network, and at the forefront of the instance segmentation prediction branch.
  • the second and fourth layer features of the FCN network are respectively used as fine features and coarse features as inputs of the PointRend module.
  • the PointRend module uses a multi-layer perceptron to iteratively optimize each point based on the feature pixel points extracted from the feature map with low spatial resolution and the corresponding relatively fine high spatial resolution features, and then predicts the instance mask through the instance segmentation prediction branch;
  • Step C extracting edge contour data of each fruiting body mask
  • Step D Calculate the average curvature and length of the mask contour of the fruiting body, and divide the fruiting bodies into regular contour fruiting bodies and irregular contour fruiting bodies according to the combined size relationship between the contour length and the average curvature;
  • Step E The contour of the regular contour fruiting body is reconstructed by the least squares ellipse fitting method, and the contour of the irregular contour fruiting body is reconstructed by the longest contour extraction and classification reconstruction method based on corner point segmentation, so as to finally realize the segmentation and reconstruction of the mushroom cluster contour.
  • the PR-SOLOv2 segmentation network connects low-level features and high-level features layer by layer, and iteratively renders the output image in a coarse-to-fine manner, which can improve the blur and misjudgment of boundary pixels that may be caused by the original network structure and improve the accuracy of image segmentation edges.
  • step B when training the PR-SOLOv2 segmentation network, the training batch size is set to 4, the initial learning rate is set to 0.01, the weight decay factor is 0.0001, the momentum size is 0.9, and 5000 iterations of training are performed.
  • step D the curvature k of each point in the fruiting body mask contour is calculated according to formula 1), and the average curvature of all contour points is taken as the average curvature of the contour:
  • x′, y′ and x represent the first and second order derivatives of the x and y coordinates of the contour point, respectively.
  • step D according to the combined size relationship between the contour length and the average curvature, when the contour length of the fruiting body is greater than h (pixel points) and the average curvature is less than f, the fruiting body is a regular contour fruiting body, otherwise it is an irregular contour fruiting body; where: h and f are thresholds; in this article, for the segmentation of Agaricus bisporus, h is taken as 100 and f is taken as 0.103.
  • the longest contour extraction and classification reconstruction method based on corner point segmentation includes the following steps:
  • Step E1 Detect corner points where the contour shape suddenly changes
  • Step E2 by using the corner point coordinates, the coordinate line between the adjacent corner point coordinates is regarded as a sub-contour segment, thereby segmenting the entire mushroom contour into N sub-contour segments;
  • Step E3 Calculate the length of each contour segment and select the longest contour segment
  • R is the radius of curvature of the arc
  • h Arc is the arc height of the longest contour segment
  • z, v, and m are the three sides of the triangle formed by the two endpoints of the longest contour segment and any point on the segment
  • z is the length of the side connecting the two endpoints of the longest contour segment
  • p is the semi-perimeter of the triangle
  • S is the area of the triangle
  • Step E5 According to the different combinations of the arcuate curvature C of the longest contour segment and its length, different methods are used to reconstruct the contour of the irregular fruiting body; the details are as follows:
  • the present invention has the following beneficial effects:
  • the present invention integrates the SOLOv2 network structure with the PointRend neural network module to construct an improved SOLOv2 algorithm PR-SOLOv2, which iteratively renders the point in a coarse-to-fine manner by connecting low-level features and high-level features layer by layer.
  • the method of dyeing the output image improves the misjudgment of boundary pixels that may be caused by the original network structure, overcomes the fuzzy problem of the segmentation edge of dense fruit bodies, and improves the accuracy of image segmentation edges.
  • the present invention proposes a method for reconstructing the classified contour of mushroom fruiting bodies based on the high-precision instance segmentation mask contour.
  • this method it is proposed for the first time to use an instance segmentation algorithm to obtain a high-precision mask of the fruiting body and extract the mask contour data of each fruiting body as the basic data for fruiting body contour reconstruction.
  • the method of obtaining contour data from the mask is simpler and more effective than other contour search methods, and because the obtained mask contour data fits the contour edge more closely, the accuracy of the contour edge data is also very high.
  • the curvature and radius of the fruiting body mask contour are used to classify and reconstruct regular-shaped, smooth-contour fruiting bodies and irregular-contour fruiting bodies that are severely occluded or squeezed.
  • the present invention proposes a method for extracting and classifying the longest contour based on corner point segmentation.
  • the method extracts the longest contour segment in the contour based on the contour corner points, and then reconstructs the complete contour using circle and ellipse fitting according to the arc curvature and length of the longest contour segment.
  • This method can well restore irregular-contour fruiting bodies that are severely occluded or squeezed.
  • the present invention has high precision and high speed in segmenting and reconstructing the contours of densely overlapping mushroom cluster fruiting bodies.
  • the reconstructed contour edges have a high degree of fit with the actual edges of the mushrooms, and the center point positioning is more accurate. It is suitable for high-precision recognition and positioning of mushroom fruiting bodies such as Agaricus bisporus, Agaricus blazei, Flammulina velutipes, Oyster mushroom, Pleurotus eryngii, etc.
  • the present invention is also suitable for high-precision recognition and positioning of other spherical densely overlapping fruits.
  • Figure 1 is a flow chart of the high-precision contour segmentation and reconstruction method of mushroom cluster fruiting bodies based on SOLOv2.
  • Figure 2 is the original image of Agaricus bisporus.
  • Figure 3 is a diagram of the PR-SOLOv2 network structure.
  • Figure 4 is a comparison of the segmentation effects of the PR-SOLOv2 segmentation model
  • Figure 5 shows the mask contour obtained by PR-SOLOv2 segmentation.
  • FIG6 is an extracted mask contour map.
  • Figure 7 shows the more regular fruiting body outline fitted by the least squares ellipse.
  • FIG8 shows the effect of directly fitting an ellipse using an unprocessed mask contour on a fruiting body with an irregular contour shape that is severely blocked or severely squeezed and deformed.
  • Figure 9 shows the longest contour of a severely occluded irregular contour based on corner point extraction.
  • FIG. 10 is a diagram showing the reconstructed contour effect of an irregular contour that is severely occluded or squeezed.
  • FIG. 11 is a diagram showing the complete effect of reconstructing the outline of a fruit cluster based on the method of the present invention.
  • FIG1 The flow chart of the present invention is shown in FIG1 , and the specific implementation steps are as follows.
  • a depth camera Realsense D435 was used to collect images, which was installed at a height of 300 mm from the culture medium. The collected images are shown in Figure 2.
  • Agaricus bisporus images were collected at different growth stages, different light sources, different shooting angles, different fungus species, dense growth states, and sparse growth states.
  • the polygonal outer contours of the mushroom fruiting bodies were annotated on the training set images using the "Labelme" image data annotation software to achieve manual annotation of the target fruiting bodies and generate the corresponding JSON format files.
  • the image data enhancement of the collected fruit image original data training set was further performed by image brightness change, horizontal flip, vertical mirroring, random rotation, etc. to obtain more image data and image features. Finally, the image-enhanced images and the original data images were combined to form the final training data set.
  • the PointRend network module is added after the fully convolutional network FCN, the backbone network of the SOLOv2 network, and at the front end of the instance segmentation prediction branch.
  • the second and fourth layer features of the FCN network are selected as fine features and coarse features as the input of this module, respectively.
  • a multi-layer perceptron is used to continuously iteratively optimize each point, and the instance segmentation prediction branch is connected to predict the instance mask.
  • the operating environment for building the model using win10 operating system, 24GB running memory, 1 ⁇ GeForceRTX3090 graphics card, python3.7, PyTorch1.6, GPU operating architecture CUDA10.1, acceleration library CUDNN7.6.5, Facebook's open source framework Detectron2 as the model's operating framework.
  • the PR-SOLOv2 model was trained using the constructed dataset with the following training parameters: the training batch size was set to 4, the initial learning rate was set to 0.01, the weight decay factor was 0.0001, the momentum size was 0.9, and 5000 iterations of training were performed.
  • the collected image is segmented through the trained PR-SOLOv2 algorithm model to complete the segmentation of the Agaricus bisporus fruiting bodies in the image.
  • the segmentation effect of the PR-SOLOv2 segmentation model and the masks of each fruiting body are obtained, as shown in Figure 5.
  • FIG4 is the segmentation effect obtained by using the SOLOv2 algorithm before improvement.
  • the segmentation effect for dense mushroom clusters is not good, and the edge segmentation of densely adhered fruiting bodies is fuzzy.
  • the fruiting body 1 The segmentation of the boundary edges of fruiting bodies such as 2, 2 and 3, 3 and 4, 4 and 5, 6 and 7 is fuzzy.
  • FIG4(b) is the segmentation result of the PR-SOLOv2 algorithm model proposed in the present invention.
  • PR-SOLOv2 has high segmentation accuracy, with an average precision AP of 93.037%, an AP 50 of 99.056%, and an AP 75 of 95.249%, all of which are higher than other typical instance segmentation algorithms, as shown in Table 1.
  • the curvature k of each point in the sub-entity mask contour is calculated according to the following formula, and the average curvature of all contour points is taken as the average curvature of the contour.
  • x′, y′ and x", y" represent the first-order and second-order derivatives of the x and y coordinates of the contour point respectively.
  • the ellipse position parameters ( ⁇ , x0 , y0 ) and shape parameters (a, b) are calculated to achieve contour reconstruction.
  • the contour reconstruction effect of the fruiting body with regular shape and smooth contour edge is shown in Figure 7.
  • the coordinate line between adjacent corner point coordinates is regarded as a sub-contour segment, thereby dividing the entire mushroom contour into N sub-contour segments;
  • R is the radius of curvature of the arc
  • h Arc is the height of the arc of the longest contour segment
  • z, v, and m are the three sides of the triangle formed by the two endpoints of the longest contour segment and any point on the segment
  • z is the length of the side connecting the two endpoints of the longest contour segment
  • p is the semi-perimeter of the triangle
  • S is the area of the triangle.
  • n points (x j , y j ) are selected on the maximum contour, and the contour of the fruiting body is reconstructed using the least squares ellipse fitting method.
  • n points (x j , y j ) are selected on the maximum contour, and the contour of the fruiting body is reconstructed using the minimum distance circle fitting method.
  • the specific calculation method is to determine the parameters of the circle based on the sum of the absolute values of the distances from the n data points (x j , y j ) to the circle:
  • x c and y c are the center points of the fitted circle
  • r is the radius of the fitted circle
  • x c , y c and r that make f take the minimum value are the best fitting parameters
  • Figure 11 is a fruit cluster reconstructed contour effect diagram based on the method of the present invention.
  • the contour construction effect is very good, the constructed contour is very consistent with the actual contour, and the positioning accuracy and shape and size accuracy of the fruiting body are both high.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Mycology (AREA)
  • Environmental Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

本发明公开了一种基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法。首先通过PR-SOLOv2分割网络将蘑菇簇图像进行分割，获得各子实体掩膜。然后，提取各子实体掩膜轮廓数据，根据子实体掩膜轮廓的曲率和半径大小对子实体进行分类，进而根据分类结果，对形状规则、较平滑轮廓子实体采用最小二乘椭圆法进行拟合重构，对被严重遮挡或挤压的不规则轮廓子实体通过基于角点分割的最长轮廓提取及分类重构方法重构。本发明对密集重叠蘑菇簇子实体的轮廓分割和重构精度高、速度快，重构的轮廓边缘与蘑菇实际边缘贴合度较高，中心点定位更加精准，适用于双孢蘑菇、姬松茸、金针菇、海鲜菇、杏鲍菇等蘑菇子实体的高精度识别与定位。

Description

基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法 技术领域
本发明涉及智慧农业及视觉识别技术领域，特别是果蔬自动采摘技术领域，具体涉及一种基于改进的SOLOv2的蘑菇簇子实体高精度轮廓分割和重构方法。
背景技术
蘑菇是一种天然高丛生性作物，子实体之间极易密集并重叠生长在一起，蘑菇的形状因密集生长空间受限易从圆形长成椭圆或不规则形，子实体也容易由垂直生长变成倾斜姿态，因此，实现轮廓高精度分割和重构难度大，导致子实体的中心点定位精度以及子实体尺寸精度偏低，会降低蘑菇簇的自动采摘成功率。
目前针对子实体的分割方法主要有基于传统视觉的分割算法采用圆或椭圆进行重构和基于目标检测的Boudingbox中心点定位方法，对于相对稀疏圆形子实体的分割定位精度效果较好，但是对于密集重叠的果实簇中的子实体的定位效果并不好。传统视觉对环境适应性差识别算法鲁棒性差、对较小和粘连子实体漏识别率高、会将密度高的类圆形菌丝团误识别为子实体，对倾斜、不规则形和被遮挡子实体的边缘分割精度不高，而且，单用圆形或椭圆重构的子实体轮廓特别是对不规则形状及被遮挡子实体的轮廓与实际轮廓贴合程度不高。而基于目标检测的中心点定位方法，虽然对子实体识别的鲁棒性提高不少，但是由于其目标中心是依靠Boundingbox来确定的，与实体的边缘也不太贴合，尤其是对于倾斜、复杂粘连重叠子实体，会导致子实体的尺寸和中心点位置与实际产生一定偏移。
发明内容
针对现有技术存在的问题，本发明的目的在于提供一种基于改进的SOLOv2(PR-SOLOv2分割算法)的蘑菇簇子实体高精度轮廓分割和重构方法。本方法克服了图像采集时蘑菇重叠、倾斜、遮挡、光照不均匀、菇帽表面有土壤等复杂情况以及密集粘连蘑菇的轮廓分割模糊问题，通过本发明提出的PR-SOLOv2分割算法不仅能针对密集、重叠、挤压和倾斜子实体的边缘进行精准分割、完整获得其实例掩码，并且对于蘑菇子实体之间存在较严重遮挡、挤压变形等情况也能基于本发明所设计的轮廓重构算法对子实体进行较好地轮廓拟合重构，其目的是为了更加精准获得子实体的中心点坐标和形状尺寸，为后续的采摘或生长状态监测提供精准的子实体数据。
本发明首先通过融合SOLOv2网络及PointRend模块的PR-SOLOv2分割网络对蘑菇簇图像进行分割，获得与实际子实体贴合较好的各子实体掩膜。然后，提取各子实体掩膜轮廓数据，根据子实体掩膜轮廓的曲率和半径大小对子实体进行分类，对形状规则、较平滑轮廓子实体采用最小二乘椭圆法进行拟合重构，对被严重遮挡或挤压的不规则轮廓子实 体基于角点分割的最长轮廓提取及分类重构方法进行重构。本发明的技术方案具体介绍如下。
一种基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法，包括以下步骤：
步骤A：采集蘑菇图像；
步骤B：采用改进的SOLOv2算法对图像中蘑菇簇中的子实体进行高精度分割，获得与实际子实体贴合好的各子实体掩膜；其中：改进的SOLOv2算法基于融合SOLOv2网络及PointRend模块的PR-SOLOv2分割网络实现；PR-SOLOv2分割网络中，PointRend模块设置在SOLOv2网络的特征提取骨干网络全卷积网络FCN后、实例分割预测分支的最前端，FCN网络的第二层和第四层特征分别作为精细特征和粗糙特征作为PointRend模块的输入，PointRend模块根据在低空间分辨率的特征图和相应较为精细的高空间分辨率特征上提取到的特征像素点，采用多层感知机对每个点不断迭代优化，再通过实例分割预测分支来进行实例掩膜的预测；
步骤C：提取各子实体掩膜的边缘轮廓数据；
步骤D：计算子实体掩膜轮廓的平均曲率和长度，根据轮廓长度和平均曲率的组合大小关系，将子实体分为规则轮廓子实体和非规则轮廓子实体；
步骤E：采用最小二乘椭圆拟合法重构规则轮廓子实体的轮廓，采用基于角点分割的最长轮廓提取及分类重构方法重构非规则轮廓子实体的轮廓，最终实现蘑菇簇轮廓分割和重构。
上述步骤B中，PR-SOLOv2分割网络通过逐层连接低层特征和高层特征，以由粗至细的方式迭代渲染输出图像的方法，能改善原来网络结构可能对边界像素产生的模糊及误判，提高图像分割边缘的精度。
上述步骤B中，对PR-SOLOv2分割网络进行训练时，训练批次大小设置为4，初始学习率设置为0.01，权重衰减因子为0.0001，动量大小为0.9，进行5000次迭代训练。
上述步骤D中，按照式1)计算子实体掩膜轮廓中每个点的曲率k，并取所有轮廓点曲率的平均值作为该轮廓的平均曲率：
式中：x′、y′以及x"、y"分别表示轮廓点x，y坐标的一阶和二阶导数。
上述步骤D中，根据轮廓长度和平均曲率的组合大小关系，当子实体轮廓长度大于h(像素点)且平均曲率小于f时，子实体为规则轮廓子实体，否则为非规则轮廓子实体；其中：h、f是阈值；本文中，针对双孢蘑菇分割时，h取值为100，f取值为0.103。
上述步骤E中，基于角点分割的最长轮廓提取及分类重构方法包括以下步骤：
步骤E1：检测轮廓形状突变处的角点；
步骤E2：通过角点坐标，将相邻角点坐标之间的坐标连线视为一条子轮廓片段，从而将整个蘑菇轮廓分割成N段子轮廓片段；
步骤E3：计算各轮廓片段的长度，从中选出最长轮廓片段；
步骤E4：计算该最长轮廓片段的弓形曲率C，计算公式如下：




p＝1/(z+v+m)
其中，R为弓形曲率半径，hArc为最长轮廓片段的弓形高，z、v、m分别为最长轮廓片段的两端点与片段上任一点组成三角形的三边长，z为最长轮廓片段的两端点连接的边长，p为三角形的半周长，S为三角形的面积；
步骤E5：根据最长轮廓片段的弓形曲率C以及其长度大小组合不同，采用不同方式对非规则子实体进行轮廓重建；具体如下：
当最长轮廓片段的弓形曲率C大于等于r并且其长度大于等于q(像素点)时，在该片段上选取若干个点，采用最小二乘椭圆拟合法重建子实体轮廓，r、q为阈值；本文中，针对双孢蘑菇重构时，r取值为0.9，q取值为100。
当最长轮廓片段的弓形曲率C小于r或其长度小于q时，在该片段上选取若干个点，采用最小距离圆拟合的方法重建子实体轮廓。
与现有技术相比，本发明的有益效果在于：
(1)本发明对SOLOv2网络结构与PointRend神经网络模块进行融合，构建了改进的SOLOv2算法PR-SOLOv2,通过逐层连接低层特征和高层特征，以由粗至细的方式迭代渲 染输出图像的方法，改善原来网络结构可能对边界像素产生的误判，克服了密集果实子实体分割边缘的模糊问题，提高了图像分割边缘的精度。
(2)本发明提出了一种基于高精度实例分割掩膜轮廓的蘑菇子实体的分类轮廓重构方法。本方法中，首次提出采用实例分割算法获得子实体的高精度掩膜并提取各子实体掩膜轮廓数据作为子实体轮廓重构的基础数据。由掩膜获得轮廓数据方法比其它的轮廓搜索方法简单有效，而且由于所获得的掩膜轮廓的数据较贴合轮廓边缘，轮廓边缘数据的精度也很高。此外，通过子实体掩膜轮廓的曲率和半径大小对形状规则、较平滑轮廓子实体和被严重遮挡或挤压的不规则轮廓子实体进行分类重构。特别是针对不规则轮廓子实体本发明提出了基于角点分割的最长轮廓提取及分类重构方法，基于轮廓角点提取轮廓中的最长轮廓片段、然后根据最长轮廓片段的弓形曲率及长度分别采用圆及椭圆拟合重构完整轮廓方法，能很好的恢复被严重遮挡或严重挤压的不规则轮廓子实体。
本发明对密集重叠蘑菇簇子实体的轮廓分割和重构精度高、速度快，重构的轮廓边缘与蘑菇实际边缘贴合度较高，中心点定位更加精准，适用于双孢蘑菇、姬松茸、金针菇、海鲜菇、杏鲍菇等蘑菇子实体的高精度识别与定位。本发明还适用于其它类球型密集重叠果实的高精度识别与定位。
附图说明
图1为基于SOLOv2的蘑菇簇子实体高精度轮廓分割和重构方法流程图。
图2为双孢蘑菇原图像。
图3为PR-SOLOv2网络结构图。
图4为PR-SOLOv2分割模型的分割效果比较图
图5为经PR-SOLOv2分割获得的掩膜轮廓图。
图6为提取的掩膜轮廓图。
图7为最小二乘椭圆拟合的较规则子实体轮廓。
图8为受遮挡严重或被严重挤压变形的轮廓形状不太规则的子实体直接用未处理的掩膜轮廓进行椭圆拟合的效果。
图9为被严重遮挡的不规则轮廓基于角点提取的最长轮廓图。
图10为受严重遮挡或严重挤压的不规则轮廓的重构轮廓效果图。
图11为基于本发明方法的果实簇重构轮廓完整效果图。
具体实施方式
下面结合附图，以双孢蘑菇为例对本发明的技术方案作进一步说明。
本发明的流程图如图1所示，具体实施步骤如下。
1.构建双孢蘑菇子实体分割模型的数据集
采用深度相机RealsenseD435，安装在离培养基高度300mm处采集图像，采集的图像如图2所示。
采集不同生长时期，不同光源环境下、不同拍摄角度下、不同菌种下、密集生长状态和稀疏生长状态的双孢蘑菇图像，使用“Labelme”图像数据标注软件在训练集图像上标注出蘑菇子实体的多边形外轮廓，实现对目标子实体的人工标注，生成相应的JSON格式文件。为了提高双孢蘑菇训练模型的泛化能力和鲁棒性，进一步对采集的果实图像原始数据训练集进行图像亮度变化、水平翻转、镜像垂直、随机旋转等图像数据增强以获得更多的图像数据和图像特征。最后将图像增强的图像与原始数据图像集中在一起构成最终的训练数据集。
2、设计构建改进的SOLOv2的PR-SOLOv2网络
在SOLOv2网络的特征提取骨干网络全卷积网络FCN后、实例分割预测分支的最前端加入PointRend网络模块，并选取FCN网络的第二层和第四层特征分别作为精细特征和粗糙特征作为该模块的输入，根据在低空间分辨率的特征图和相应较为精细的高空间分辨率特征上提取到的特征像素点，采用多层感知机对每个点不断迭代优化，接入实例分割预测分支来进行实例掩膜的预测。通过这种逐层连接低层特征和高层特征，以由粗至细的方式迭代渲染输出图像的方法，改善原来网络结构可能对边界像素产生的误判而造成的轮廓分割模糊问题，提高图像分割边缘的精度。所设计的PR-SOLOv2算法的网络结构如图3所示。
3、训练PR-SOLOv2模型
构建模型的运行环境：采用win10操作系统，运行内存24GB，显卡为1×GeForceRTX3090，配置python3.7、PyTorch1.6、GPU运行架构CUDA10.1、加速库为CUDNN7.6.5下的Facebook的开源框架Detectron2作为模型的运行框架。
PR-SOLOv2模型采用构建的数据集进行训练，训练参数：训练批次大小设置为4，初始学习率设置为0.01，权重衰减因子为0.0001，动量大小为0.9，进行5000次迭代训练。
4、采集双孢蘑菇图像
5、将采集的图像通过训练出来的PR-SOLOv2算法模型完成对图像中双孢蘑菇子实体的分割，PR-SOLOv2分割模型的分割效果、获得各子实体掩膜，如图5所示。
进一步的，发明人基于SOLOv2算法和PR-SOLOv2算法对不同双孢蘑菇图像样本的分割效果进行比较，结果如图4所示，其中图4(a)图是未改进前即采用SOLOv2算法得分割效果，对于密集蘑菇簇分割效果不好，密集粘连子实体边缘分割模糊，例如子实体1 和2、2和3、3和4、4和5、6和7等子实体交界边缘分割处分割模糊。而图4(b)则是采用本发明提出的PR-SOLOv2算法模型分割的结果，由图中可见，即使是密集粘连子实体间的边缘分割精准、清晰，与实际子实体边缘非常贴合。PR-SOLOv2的分割精度高，平均精度AP可达93.037％，AP50高达99.056％，AP75可达95.249％，均高于其它典型实例分割算法，如表1所示。
表1
6.提取密集重叠子实体掩膜的边缘轮廓数据，如图6所示。
7.计算子实体掩膜轮廓的平均曲率和长度
按照以下公式计算子实体掩膜轮廓中每个点的曲率k，并取所有轮廓点曲率的平均值作为该轮廓的平均曲率。
其中，x′、y′以及x"、y"分别表示轮廓点x，y坐标的一阶和二阶导数。
8.根据轮廓长度和平均曲率的组合大小关系，对规则轮廓子实体和非规则轮廓子实体分别采用不同方法进行轮廓重构。
(1)若轮廓平均曲率和长度的组合关系满足条件实体轮廓长度大于100且平均曲率小于0.103，则说明该子实体是基本未受遮挡或被严重挤压变形的子实体，其轮廓边缘比较平滑，形状比较规则，取其上N个轮廓点为Pi(xi,yi)(i＝1,2,…,N)，根据最小二乘椭圆拟合目标函数(如下式)进行拟合运算，解得A,B,C,D,E的值，根据椭圆特性，计算出椭圆位置参数(θ,x0,y0)以及形状参数(a,b)，从而实现轮廓的重构。形状规则、轮廓边缘平滑子实体的轮廓重构效果如图7所示。
其中，欲使得F最小，需满足：
(2)若不满足实体轮廓长度大于100且平均曲率小于0.103的条件，则说明该子实体遮挡严重或被严重挤压变形，其轮廓形状不太规则、轮廓不太平滑，如果直接对其进行最小二乘椭圆拟合，如图8所示,箭头所指A1轮廓为拟合轮廓，拟合结果与实际轮廓相差较大；需要采用基于角点分割的最长轮廓提取及分类重构方法进行轮廓重构。具体实施步骤如下：
①检测轮廓形状突变处的角点；
②通过角点坐标，将相邻角点坐标之间的坐标连线视为一条子轮廓片段，从而将整个蘑菇轮廓分割成N段子轮廓片段；
③计算各轮廓片段的长度，从中选出最长轮廓片段；选出的最长轮廓片段。如图9所示，箭头指向的L1红色轮廓为最长轮廓。
④计算该最长轮廓片段的弓形曲率C，具体计算公式如下：




p＝1/(z+v+m)
其中，其中，R为弓形曲率半径，hArc为最长轮廓片段的弓形高，z、v、m分别为最长轮廓片段的两端点与片段上任一点组成三角形的三边长，z为最长轮廓片段的两端点连接的边长，p为三角形的半周长，S为三角形的面积。
然后，根据最大轮廓的弓形曲率以及长度大小组合不同，采用不同方式对较不规则子实体进行轮廓重建。当最大轮廓的弓形曲率较大大于0.9并且长度大于100，在最大轮廓上选取n个点(xj,yj)，采用最小二乘椭圆拟合法重建子实体轮廓。当最大轮廓的弓形曲率小于0.9或长度较短小于100时，在最大轮廓上选取n个点(xj,yj)，采用最小距离圆拟合的方法重建子实体轮廓，具体计算方法为根据n个数据点(xj,yj)到圆的距离绝对值的和来确定圆的参数：
其中，xc、yc为拟合后圆的中心点，r为拟合的圆半径，使得f取得最小值的xc、yc和r就是最佳拟合参数；
基于角点分割的最长轮廓提取及分类重构方法对受严重遮挡或严重挤压的不规则轮廓重构结果如图10所示，红色轮廓为重建的轮廓，被遮挡部分恢复较好，重构轮廓的边缘与实际子实体轮廓边缘拟合较好。
图11为基于本发明方法的果实簇重构轮廓效果图。由图中可见，密集粘连的蘑菇簇中，无论是形状较规则的子实体还是因被严重遮挡或被严重挤压变形或倾斜的不规则子实体的轮廓构建效果都很好，所构建的轮廓与实际轮廓很贴合，子实体的定位精度和形状尺寸精度都较高。
上述实施例只为说明本发明的技术构思及特点，其目的在于让熟悉此项技术的人士能够了解本发明的内容并据以实施，并不能以此限制本发明的保护范围。凡根据本发明精神实质所作的等效变化或修饰，都应涵盖在本发明的保护范围之内。

Claims (5)

  1. 一种基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法，其特征在于，包括以下步骤：
    步骤A：采集蘑菇图像；
    步骤B：采用改进的SOLOv2算法对图像中蘑菇簇中的子实体进行高精度分割，获得与实际子实体贴合好的各子实体掩膜；其中：改进的SOLOv2算法基于融合SOLOv2网络及PointRend模块的PR-SOLOv2分割网络实现；PR-SOLOv2分割网络中，PointRend模块设置在SOLOv2网络的特征提取骨干网络全卷积网络FCN后、实例分割预测分支的最前端，FCN网络的第二层和第四层特征分别作为精细特征和粗糙特征作为PointRend模块的输入，PointRend模块根据在低空间分辨率的特征图和相应较为精细的高空间分辨率特征上提取到的特征像素点，采用多层感知机对每个点不断迭代优化，再通过实例分割预测分支来进行实例掩膜的预测；
    步骤C：提取各子实体掩膜的边缘轮廓数据；
    步骤D：计算子实体掩膜轮廓的平均曲率和长度，根据轮廓长度和平均曲率的组合大小关系，将子实体分为规则轮廓子实体和非规则轮廓子实体；
    步骤E：采用最小二乘椭圆拟合法重构规则轮廓子实体的轮廓，采用基于角点分割的最长轮廓提取及分类重构方法重构非规则轮廓子实体的轮廓，最终实现蘑菇簇轮廓分割和重构。
  2. 根据权利要求1所述的基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法，其特征在于，步骤B中，对PR-SOLOv2分割网络进行训练时，训练批次大小设置为4，初始学习率设置为0.01，权重衰减因子为0.0001，动量大小为0.9，进行5000次迭代训练。
  3. 根据权利要求1所述的基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法，其特征在于，步骤D中，按照式1)计算子实体掩膜轮廓中每个点的曲率k，并取所有轮廓点曲率的平均值作为该轮廓的平均曲率：
    式中：x′、y′以及x"、y"分别表示轮廓点x，y坐标的一阶和二阶导数。
  4. 根据权利要求1所述的基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法，其特征在于，步骤D中，根据轮廓长度和平均曲率的组合大小关系，当子实体轮廓长度大于h且平均曲率小于f时，子实体为规则轮廓子实体，否则为非规则轮廓子实体；其中：h、f是阈值。
  5. 根据权利要求1所述的基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法，其特征在于，步骤E中，基于角点分割的最长轮廓提取及分类重构方法包括以下步骤：
    步骤E1：检测轮廓形状突变处的角点；
    步骤E2：通过角点坐标，将相邻角点坐标之间的坐标连线视为一条子轮廓片段，从而将整个蘑菇轮廓分割成N段子轮廓片段；
    步骤E3：计算各轮廓片段的长度，从中选出最长轮廓片段；
    步骤E4：计算该最长轮廓片段的弓形曲率C，计算公式如下：




    p＝1/(z+v+m)
    其中，R为弓形曲率半径，hArc为最长轮廓片段的弓形高，z、v、m分别为最长轮廓片段的两端点与片段上任一点组成三角形的三边长，z为最长轮廓片段的两端点连接的边长，p为三角形的半周长，S为三角形的面积；
    步骤E5：根据最长轮廓片段的弓形曲率C以及其长度大小组合不同，采用不同方式对非规则子实体进行轮廓重建；具体如下：
    当最长轮廓片段的弓形曲率C大于等于r并且其长度大于等于q时，在该片段上选取若干个点，采用最小二乘椭圆拟合法重建子实体轮廓，r、q为阈值；
    当最长轮廓片段的弓形曲率C小于r或其长度小于q时，在该片段上选取若干个点，采用最小距离圆拟合的方法重建子实体轮廓。
PCT/CN2023/098787 2022-10-13 2023-06-07 基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法 WO2024077979A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211252987.7A CN115601543A (zh) 2022-10-13 2022-10-13 基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法
CN202211252987.7 2022-10-13

Publications (1)

Publication Number Publication Date
WO2024077979A1 true WO2024077979A1 (zh) 2024-04-18

Family

ID=84847499

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/098787 WO2024077979A1 (zh) 2022-10-13 2023-06-07 基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法

Country Status (2)

Country Link
CN (1) CN115601543A (zh)
WO (1) WO2024077979A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115601543A (zh) * 2022-10-13 2023-01-13 上海第二工业大学（Cn） 基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013154062A1 (ja) * 2012-04-11 2013-10-17 日本電気株式会社 画像識別システム、画像識別方法、およびプログラム
CN110059663A (zh) * 2019-04-24 2019-07-26 扬州龙科信息科技有限公司 一种基于特征点检测的粘连蘑菇视觉识别与测量方法
CN111445482A (zh) * 2020-03-24 2020-07-24 上海第二工业大学 重叠双孢蘑菇的分割识别方法
CN113077453A (zh) * 2021-04-15 2021-07-06 华南理工大学 一种基于深度学习的电路板元器件缺陷检测方法
CN115601543A (zh) * 2022-10-13 2023-01-13 上海第二工业大学（Cn） 基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013154062A1 (ja) * 2012-04-11 2013-10-17 日本電気株式会社 画像識別システム、画像識別方法、およびプログラム
CN110059663A (zh) * 2019-04-24 2019-07-26 扬州龙科信息科技有限公司 一种基于特征点检测的粘连蘑菇视觉识别与测量方法
CN111445482A (zh) * 2020-03-24 2020-07-24 上海第二工业大学 重叠双孢蘑菇的分割识别方法
CN113077453A (zh) * 2021-04-15 2021-07-06 华南理工大学 一种基于深度学习的电路板元器件缺陷检测方法
CN115601543A (zh) * 2022-10-13 2023-01-13 上海第二工业大学（Cn） 基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法

Also Published As

Publication number Publication date
CN115601543A (zh) 2023-01-13

Similar Documents

Publication Publication Date Title
CN110097536B (zh) 基于深度学习和霍夫变换的六边形螺栓松动检测方法
CN111681197B (zh) 一种基于Siamese网络结构的遥感图像无监督变化检测方法
CN112598713A (zh) 一种基于深度学习的近岸海底鱼类检测、跟踪统计方法
CN108491786B (zh) 一种基于分级网络和聚类合并的人脸检测方法
CN105701483B (zh) 一种融合多光谱遥感数据和夜间灯光遥感数据的城市边界提取方法
CN108171752A (zh) 一种基于深度学习的海面船只视频检测与跟踪方法
CN103793708B (zh) 一种基于仿射校正的多尺度车牌精准定位方法
CN111753682B (zh) 一种基于目标检测算法的吊装区域动态监控方法
CN103810474A (zh) 一种基于多特征低秩矩阵表示的车牌检测方法
CN102842037A (zh) 一种基于多特征融合的车辆阴影消除方法
CN109559324A (zh) 一种线阵图像中的目标轮廓检测方法
CN111860587B (zh) 一种用于图片小目标的检测方法
CN108664838A (zh) 基于改进rpn深度网络的端到端的监控场景行人检测方法
WO2024077979A1 (zh) 基于改进的SOLOv2的蘑菇簇轮廓分割和重构方法
CN103886619A (zh) 一种融合多尺度超像素的目标跟踪方法
CN110991444A (zh) 面向复杂场景的车牌识别方法及装置
CN111754538B (zh) Usb表面缺陷检测的阈值分割方法
CN103971377A (zh) 基于先验形状水平集分割的建筑物提取方法
CN110399840A (zh) 一种快速的草坪语义分割及边界检测方法
CN111199245A (zh) 油菜害虫识别方法
CN114463525B (zh) 一种面向三维重建的大规模图像色彩一致性校正方法
Chen et al. Research on fast recognition method of complex sorting images based on deep learning
CN105354547A (zh) 一种结合纹理和彩色特征的行人检测方法
CN111178405A (zh) 一种融合多种神经网络的相似物体识别方法
CN113077021A (zh) 基于机器学习的电子病历多维挖掘方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23876190

Country of ref document: EP

Kind code of ref document: A1