WO2020172992A1 - 基于深度联想感知算法的受遮挡蔬果采收方法 - Google Patents

基于深度联想感知算法的受遮挡蔬果采收方法 Download PDF

Info

Publication number
WO2020172992A1
WO2020172992A1 PCT/CN2019/085439 CN2019085439W WO2020172992A1 WO 2020172992 A1 WO2020172992 A1 WO 2020172992A1 CN 2019085439 W CN2019085439 W CN 2019085439W WO 2020172992 A1 WO2020172992 A1 WO 2020172992A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target
occluded
harvesting
mature
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/085439
Other languages
English (en)
French (fr)
Inventor
刘成良
贡亮
汪韬
李旭东
林可
李彦明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jiao Tong University
Original Assignee
Shanghai Jiao Tong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jiao Tong University filed Critical Shanghai Jiao Tong University
Publication of WO2020172992A1 publication Critical patent/WO2020172992A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/0014—Image feed-back for automatic industrial control, e.g. robot with camera
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30128—Food products

Definitions

  • the invention relates to the field of image processing, in particular to a method for harvesting occluded fruits and vegetables based on a depth association perception algorithm.
  • the large-scale production of vegetable and fruit picking operations has a large labor gap.
  • the use of robots for automatic harvesting can improve production efficiency but has high technical requirements.
  • robots for accurately picking fruits and vegetables are rarely used on a large scale.
  • An important limiting factor is that robot picking requires that the target is unobstructed in the field of view.
  • the environment is complex and there are a large number of branches and leaves that block the target, reducing the success rate of robot picking.
  • the morphology of vegetables and fruits is changeable, and the harvesting control amount is difficult to extract, which further reduces the success rate of robot picking.
  • Japan's Matsushita Company has developed a tomato picking robot that can automatically pick tomatoes that are not covered by branches and leaves, are evenly hung, and are hung on one side. It is currently in the trial phase and plans to start sales in 2019.
  • the Chinese invention patent with publication number CN108319894A discloses a fruit recognition method and device based on deep learning.
  • the trained deep learning model is used to recognize the image input by the user, determine whether it contains the target fruit and identify all the target fruits. The specific location in the image.
  • the above two schemes require that the target is unobstructed in the field of view, and the redundant branches and leaves need to be removed. It takes a long time to judge whether the fruits and vegetables are mature. At the same time, the actual operation is seriously affected by the light, so the picking success rate is low and difficult widely used.
  • the purpose of the present invention is to provide a method for harvesting obscured fruits and vegetables based on a deep association perception algorithm.
  • a method for harvesting obscured fruits and vegetables based on a deep association perception algorithm includes:
  • Image acquisition steps acquire RGB channel color images, D channel depth images and I channel infrared images
  • Preprocessing steps unify the resolution of the collected images
  • Image cropping step remove the area outside the working area of the harvesting robot in the image with uniform resolution
  • Target recognition step Identify mature targets, and mark each pixel of mature targets in the cropped image
  • Occlusion judgment step determine whether the mature target is occluded, restore the four-channel RGBD image of the complete mature target when it is occluded, and directly obtain the four-channel RGBD image of the complete mature target without being occluded;
  • Pose extraction step According to the four-channel RGBD image of the complete mature target, extract the position and posture of the mature target to the harvesting robot.
  • the preprocessing step further includes removing noise in the image.
  • the image cropping step includes calculating the real coordinates of each point in the image, so as to eliminate areas other than the working area of the harvesting robot in the image with the uniform resolution.
  • the method of judging whether the mature target is blocked includes:
  • the method of restoring the four-channel RGBD image of the complete mature target includes: constructing a visual association model to restore the occluded mature target.
  • an obscured fruit and vegetable harvesting system based on a deep association perception algorithm includes:
  • Image acquisition module acquire RGB channel color image, D channel depth image and I channel infrared image
  • Preprocessing module unify the resolution of the collected images
  • Image cropping module exclude areas other than the working area of the harvesting robot in the image with uniform resolution
  • Target recognition module Recognize mature targets and mark each pixel of mature targets in the cropped image
  • Occlusion judging module Determine whether the mature target is occluded, restore the RGBD four-channel image of the complete mature target when it is occluded, and directly obtain the RGBD four-channel image of the complete mature target without being occluded;
  • Pose extraction module According to the four-channel RGBD image of the complete mature target, extract the position and posture of the mature target to the harvesting robot.
  • the preprocessing module further includes removing noise in the image.
  • the image cropping module includes calculating the real coordinates of each point in the image, so as to eliminate areas other than the working area of the harvesting robot in the image with the uniform resolution.
  • the method of judging whether the mature target is blocked includes:
  • the method of restoring the four-channel RGBD image of the complete mature target includes: constructing a visual association model to restore the occluded mature target.
  • the present invention has the following beneficial effects:
  • the present invention is based on RGBDI five-channel image, which is less disturbed by the environment and works more stable;
  • the training of the present invention is simple, and the training process is simplified by adopting transfer learning and similar network structure.
  • FIG. 1 is a flowchart of the present invention
  • Figure 2 is the algorithm framework diagram of target recognition
  • Figure 3 is the algorithm framework diagram of occluded target restoration
  • Figure 4 is a framework diagram of the algorithm for target pose extraction.
  • the present invention provides a method for harvesting occluded fruits and vegetables based on a deep association perception algorithm, including:
  • Image acquisition steps acquire RGB channel color images, D channel depth images and I channel infrared images
  • Preprocessing steps unify the resolution of the collected images and remove image noise
  • Image cropping steps Calculate the real coordinates of each point in the image, and eliminate the areas other than the working area of the harvesting robot in the image with uniform resolution;
  • Target recognition step Identify mature targets, and mark each pixel of mature targets in the cropped image
  • Occlusion judgment step determine whether the mature target is occluded, restore the four-channel RGBD image of the complete mature target when it is occluded, and directly obtain the four-channel RGBD image of the complete mature target without being occluded;
  • Pose extraction step According to the four-channel RGBD image of the complete mature target, extract the position (x, y, z) and posture ( ⁇ , ⁇ , ⁇ ) of the mature target to the harvesting robot as the harvesting control parameters.
  • a pixel-level target recognition algorithm is used.
  • This algorithm is a deep learning method, which uses multi-layer convolutional layers to extract image features.
  • Convolutional neural network is a kind of deep neural network. Its structure is often connected by neural networks with multiple hidden layers. Compared with traditional artificial neural networks, the number of layers is deeper and the deep features can be more effectively described. Its weight sharing and local connection network structure conform to the characteristics of biological neurons, which reduces the scale of the weight parameters of the network model. This feature makes the network show excellent performance when the input is a multi-dimensional image. The image basically does not require additional operations and can be directly used as the input of the network, thus avoiding the feature extraction operation in the traditional pattern recognition algorithm, and there is no need to manually design features.
  • the present invention introduces depth image D and infrared image I that are not interfered by illumination to study multiple tasks separately
  • the contribution of image features in each dimension is characterized by: 1. Little influence from light, high accuracy; 2. Complex algorithm object, large amount of information, and high difficulty. And designed a pixel-level target recognition algorithm to strengthen the robustness and accuracy of the visual model.
  • the input of the target recognition algorithm in this embodiment is the RGBDI image taken by the Kinect camera, and the output is the binary mask image.
  • the pixel value corresponding to the target in the image is 1, and the pixel value of other areas is 0.
  • the algorithm framework is shown in Figure 2. .
  • Conv means using a 3x3 convolution kernel to perform sliding convolution on the image to extract features; Pool means maximum pooling, and the maximum value of the data contained in the core is used as output, which will reduce the image size by 1/2; Relu means using relu The function activates the convolution result.
  • the formula is as in formula (1). If there is no non-linear activation operation, the linear network has insufficient ability to mine and represent the high-level features of the data.
  • Up-conv means deconvolution layer, which will double the image. The entire network has completed the image encoding-decoding process, through a series of feature extraction and feature activation operations, and finally output the target marked image.
  • Conv layer and Up-conv layer each convolution kernel consists of 9 numbers. Convolution kernels of different values will cause the network to get different results. In order to obtain the desired convolution kernel, a training set needs to be constructed to train the network. . Except for the RGB feature extraction network that uses the public vgg19 network weights, other networks are trained on the data set collected by the applicant and determine the weights. Using vgg19 pre-trained weights to extract image features is a common practice for small data sets, because self-built data sets often cannot meet the data requirements of large-scale network training.
  • Methods to determine whether a mature target is occluded include:
  • the three-dimensional coordinates of the target image determine whether there is occlusion. If there is no occlusion, the three-dimensional coordinates corresponding to the target pixel are continuous everywhere, and the depth value of the outer pixel of the target contour is greater than the depth value of the inner pixel; if there is occlusion, the occluder and the target form two adjacent image fields in the image, and The target contour for pixel-level target recognition must include the adjacent edges of the two image domains. Therefore, the three-dimensional coordinates of the pixels on both sides of the target contour pixel are not continuous, and the difference result will be much greater than the continuous case, and the depth value of the occluder must be less than the target depth value.
  • the contour pixels of the mature target obtained by target recognition are extracted, and the depth values and 3D coordinates of the pixels inside and outside the contour are extracted and then the difference is made. If the outer pixel depth value is less than the inner pixel depth value, and the three-dimensional coordinate difference is greater than the threshold, the target is occluded; otherwise, the target is not occluded.
  • the method of recovering the four-channel RGBD image of a complete mature target includes: constructing a visual association model to restore the occluded mature target.
  • the algorithm framework of the occlusion target shape restoration algorithm is similar to the target recognition algorithm framework. It receives the RGBD image of the occluded image. The difference is that the mask image obtained by the recognition network is added as the input auxiliary restoration process, and the output is the complete target RGBD image information.
  • the network architecture is shown in Figure 3.
  • the algorithm includes a complete feature extraction (coding) and decoding process to complete the occlusion target shape restoration task. Its input includes mask image, infrared image, depth image and RGB image, and separate feature extraction networks are used to encode different input images. After integration, a single decoding network is used to decode the complete target image.
  • the mask image, infrared image, and depth image use the same feature extraction network, that is, a multi-layer convolutional layer concatenates the image multiple times, and the convolution kernel is initialized randomly.
  • RGB images use transfer learning, directly use the feature extraction network of VGG19 for encoding, and the convolution kernel coefficients do not participate in training.
  • the decoding network is also composed of multiple convolutional layers, which are trained simultaneously with the coding network.
  • the position (x, y, z) is extracted to determine the end stop position and posture ( ⁇ , ⁇ , ⁇ ) to determine the end picking posture and improve the picking success rate, the fruit radius (r ) Determine the degree of opening and closing of the end and increase the success rate of picking.
  • the robot can adjust the best picking posture and grasping path.
  • the algorithm framework is shown in Figure 4.
  • the pose extraction algorithm includes a complete feature extraction (encoding) and decoding process to complete the occlusion target shape restoration task.
  • Its input includes the complete target depth image and RGB image, and separate feature extraction networks are used to encode and integrate different input images. Then use a single decoding network to decode the complete target image.
  • the convolutional layer of multi-layer concatenation of depth image performs repeated convolution on the image to extract features, and the convolution kernel is initialized randomly.
  • the gradient descent algorithm is used to continuously reduce the network output
  • the error between the actual pose and the convolution kernel coefficient is finally determined and the network has the ability to extract the complete target pose.
  • RGB images use transfer learning, directly use the feature extraction network of VGG19 for encoding, and the convolution kernel coefficients do not participate in training.
  • the decoding network is also composed of multiple convolutional layers, which are trained simultaneously with the coding network.
  • the present invention also provides a harvesting system for occluded fruits and vegetables based on the depth association perception algorithm, including:
  • Image acquisition module acquire RGB channel color image, D channel depth image and I channel infrared image
  • Preprocessing module unify the resolution of the collected images
  • Image cropping module exclude areas other than the working area of the harvesting robot in the image with uniform resolution
  • Target recognition module Recognize mature targets and mark each pixel of mature targets in the cropped image
  • Occlusion judging module Determine whether the mature target is occluded, restore the RGBD four-channel image of the complete mature target when it is occluded, and directly obtain the RGBD four-channel image of the complete mature target without being occluded;
  • Pose extraction module According to the four-channel RGBD image of the complete mature target, extract the position and posture of the mature target to the harvesting robot.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Robotics (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

一种基于深度联想感知算法的受遮挡蔬果采收方法，包括：采集RGB通道彩色图像、D通道深度图像和I通道红外图像；对采集到的图像统一分辨率；剔除统一分辨率后的图像中的采收机器人工作区域以外的区域；识别成熟目标，并在裁剪后的图像中标记出成熟目标的每个像素；判断成熟目标是否被遮挡，在被遮挡的情况下复原出完整成熟目标的RGBD四个通道的图像，在不被遮挡的情况下直接获取完整成熟目标的RGBD四个通道的图像；根据完整成熟目标的RGBD四个通道的图像，提取出成熟目标的位置和姿态至采收机器人。本方法受环境干扰小，工作更稳定；能够估计目标的完整RGBD信息；能够提高机器人采收成功率。还公开了相应的采收系统。

Description

基于深度联想感知算法的受遮挡蔬果采收方法 技术领域
本发明涉及图像处理领域，具体地，涉及基于深度联想感知算法的受遮挡蔬果采收方法。
背景技术
蔬果采摘作业规模化生产劳动力缺口大，使用机器人自动采收能提高生产效率但技术要求高。目前精确采摘蔬果的机器人鲜有大规模应用，一个重要的限制性因素是机器人采摘要求目标在视野中无遮挡，而实际农业环境中环境复杂，存在大量枝叶遮挡目标，降低了机器人采摘的成功率。另一方面，蔬果形态多变，采收控制量难以提取，进一步降低了机器人采摘成功率。
如日本松下公司开发了一种番茄采摘机器人，能够自动采摘无枝叶遮挡、挂果均匀、单侧挂果的番茄。目前处于试用阶段，计划于2019年开始销售。以及，如公开号为CN108319894A的中国发明专利公开了基于深度学习的水果识别方法及装置，利用训练后的深度学习模型对用户输入的图像进行识别，判断其中是否包含目标水果并且识别出所有目标水果在图像中的具体位置。上述两种方案均需要要求目标在视野中无遮挡，及需要将冗余枝叶全部去除，需要较长的判断蔬果是否成熟的时间，同时实际作业中受光照影响严重，因而采摘成功率低且难以广泛应用。
发明内容
针对现有技术中的缺陷，本发明的目的是提供一种基于深度联想感知算法的受遮挡蔬果采收方法。
根据本发明提供的一种基于深度联想感知算法的受遮挡蔬果采收方法，包括：
图像采集步骤：采集RGB通道彩色图像、D通道深度图像和I通道红外图像；
预处理步骤：对采集到的图像统一分辨率；
图像裁剪步骤：剔除统一分辨率后的图像中的采收机器人工作区域以外的区域；
目标识别步骤：识别成熟目标，并在裁剪后的图像中标记出成熟目标的每个像素；
遮挡判断步骤：判断成熟目标是否被遮挡，在被遮挡的情况下复原出完整成熟目标的RGBD四个通道的图像，在不被遮挡的情况下直接获取完整成熟目标的RGBD四个通道的图像；
位姿提取步骤：根据完整成熟目标的RGBD四个通道的图像，提取出成熟目标的位置和姿态至采收机器人。
较佳的，所述预处理步骤还包括去除图像中的噪声。
较佳的，所述图像裁剪步骤包括计算图像中每个点的真实坐标，从而剔除统一分辨率后的图像中的采收机器人工作区域以外的区域。
较佳的，判断成熟目标是否被遮挡的方法包括：
提取目标识别得到的成熟目标轮廓像素，并提取轮廓内、外侧像素的深度值及三维坐标然后作差，若存在外侧像素深度值小于内侧像素深度值，且三维坐标差值大于阈值，说明目标被遮挡；否则目标未被遮挡。
较佳的，复原出完整成熟目标的RGBD四个通道的图像的方法包括：构建视觉联想模型对被遮挡的成熟目标进行还原。
根据本发明提供的一种基于深度联想感知算法的受遮挡蔬果采收系统，包括：
图像采集模块：采集RGB通道彩色图像、D通道深度图像和I通道红外图像；
预处理模块：对采集到的图像统一分辨率；
图像裁剪模块：剔除统一分辨率后的图像中的采收机器人工作区域以外的区域；
目标识别模块：识别成熟目标，并在裁剪后的图像中标记出成熟目标的每个像素；
遮挡判断模块：判断成熟目标是否被遮挡，在被遮挡的情况下复原出完整成熟目标的RGBD四个通道的图像，在不被遮挡的情况下直接获取完整成熟目标的RGBD四个通道的图像；
位姿提取模块：根据完整成熟目标的RGBD四个通道的图像，提取出成熟目标的位置和姿态至采收机器人。
较佳的，所述预处理模块还包括去除图像中的噪声。
较佳的，所述图像裁剪模块包括计算图像中每个点的真实坐标，从而剔除统一分辨率后的图像中的采收机器人工作区域以外的区域。
较佳的，判断成熟目标是否被遮挡的方法包括：
提取目标识别得到的成熟目标轮廓像素，并提取轮廓内、外侧像素的深度值及三维坐标然后作差，若存在外侧像素深度值小于内侧像素深度值，且三维坐标差值大于阈值， 说明目标被遮挡；否则目标未被遮挡。
较佳的，复原出完整成熟目标的RGBD四个通道的图像的方法包括：构建视觉联想模型对被遮挡的成熟目标进行还原。
与现有技术相比，本发明具有如下的有益效果：
1、本发明基于RGBDI五通道图像，受环境干扰小，工作更稳定；
2、对被遮挡的目标进行复原，能够估计目标的完整RGBD信息；
3、提取得到目标的位置和姿态进行高效采摘，能够提高机器人采收成功率，降低果实损伤率；
4、剔除机器人工作区域外的图像，使图像处理时间降至1/9；同时克服了现有方法(算法)成熟度检测慢的缺点，将目标识别与成熟度判定统一到目标识别算法；
5、本发明训练简单，通过采用迁移学习与相似网络结构，简化训练过程。
附图说明
通过阅读参照以下附图对非限制性实施例所作的详细描述，本发明的其它特征、目的和优点将会变得更明显：
图1为本发明的流程图；
图2为目标识别的算法框架图；
图3为遮挡目标复原的算法框架图；
图4为目标位姿提取的算法框架图。
具体实施方式
下面结合具体实施例对本发明进行详细说明。以下实施例将有助于本领域的技术人员进一步理解本发明，但不以任何形式限制本发明。应当指出的是，对本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变化和改进。这些都属于本发明的保护范围。
如图1所示，本发明提供的一种基于深度联想感知算法的受遮挡蔬果采收方法，包括：
图像采集步骤：采集RGB通道彩色图像、D通道深度图像和I通道红外图像；
预处理步骤：对采集到的图像统一分辨率，去除图像噪声；
图像裁剪步骤：计算图像中每个点的真实坐标，剔除统一分辨率后的图像中的采收 机器人工作区域以外的区域；
目标识别步骤：识别成熟目标，并在裁剪后的图像中标记出成熟目标的每个像素；
遮挡判断步骤：判断成熟目标是否被遮挡，在被遮挡的情况下复原出完整成熟目标的RGBD四个通道的图像，在不被遮挡的情况下直接获取完整成熟目标的RGBD四个通道的图像；
位姿提取步骤：根据完整成熟目标的RGBD四个通道的图像，提取出成熟目标的位置(x，y，z)和姿态(α，β，γ)至采收机器人作为采收的控制参数。
其中，在目标识别步骤中，采用像素级目标识别算法。
该算法属于深度学习类方法，利用多层卷积层提取图像特征。卷积神经网络是深度神经网络的一种，其结构往往由多层隐藏层的神经网络连接而成，相比传统人工神经网络，层数更深，能更有效刻画深层特征。它的权值共享和局部连接网络结构符合生物神经元的特性，降低了网络模型的权值参数的规模。这个特点使得网络在输入是多维图像时表现出优异的性能，图像基本不需要额外操作，可直接作为网络的输入，从而避免了传统模式识别算法中的特征提取操作，无需手工设计特征。
然而现有目标检测网络多数基于RGB图像，图像信息简单，在农业作业场景下易受光照等因素干扰，因此本发明引入不受光照干扰的深度图像D及红外图像I，对多个任务分别研究各维度图像特征的贡献，特点为：1.受光照影响小，准确度高；2.算法对象复杂，信息量大，难度高。并设计了像素级目标识别算法，强化视觉模型鲁棒性及精度。
本实施例目标识别算法的输入为Kinect摄像机拍摄的RGBDI图像，输出为二值化掩码图像，图像中目标对应的像素值为1，其他区域像素值为0，其算法框架如图2所示。
其中，Conv表示使用3x3卷积核对图像进行滑动卷积，从而提取特征；Pool表示最大池化，将核内所包含数据的最大值作为输出，会使图像尺寸缩小1/2；Relu表示使用relu函数对卷积结果进行激活，公式如式(1)，若没有非线性激活操作，线性网络对数据高层特征的挖掘和表征能力不足。Up-conv表示解卷积层，会使图像放大一倍。整个网络完成了图像的编码-解码过程，通过一系列特征提取、特征激活运算，最后输出目标的标记图像。
ReLU(x)＝max(x，0)  (1)
其中Conv层，Up-conv层的每个卷积核由9个数字组成，不同值的卷积核会导致网络得到不同的结果，为了得到期望的卷积核，需要构建训练集对网络进行训练。除RGB 特征提取网络使用了公开的vgg19网络权重，其他网络通过申请人自行收集的数据集训练并确定权重。使用vgg19预训练的权重提取图像特征是小数据集的常用做法，因为自行搭建的数据集往往无法满足大型网络训练的数据量需求。
判断成熟目标是否被遮挡的方法包括：
利用目标图像三维坐标判断是否存在遮挡。若不存在遮挡，则目标像素对应的三维坐标处处连续，目标轮廓外侧像素的深度值处处大于内侧像素深度值；若存在遮挡，则遮挡物与目标在图像中构成两个相邻图像域，且像素级目标识别的目标轮廓必然包含两图像域的邻接边缘，因此目标轮廓像素两侧像素的三维坐标不连续，作差结果将远大于连续情况，且遮挡物的深度值必小于目标深度值。
基于上述分析，提取目标识别得到的成熟目标轮廓像素，并提取轮廓内、外侧像素的深度值及三维坐标然后作差。若存在外侧像素深度值小于内侧像素深度值，且三维坐标差值大于阈值，说明目标被遮挡；否则目标未被遮挡。
复原出完整成熟目标的RGBD四个通道的图像的方法包括：构建视觉联想模型对被遮挡的成熟目标进行还原。
遮挡目标形态复原算法的算法框架与目标识别算法框架类似，接收被遮挡图像的RGBD图像，区别在于附加了识别网络得到的掩码图像作为输入辅助还原过程，输出为完整目标的RGBD图像信息，其网络架构如图3所示。算法包含完整的特征提取(编码)与解码过程以完成遮挡目标形态复原任务，其输入包括掩码图像、红外图像、深度图像与RGB图像，对不同的输入图像分别使用单独的特征提取网络进行编码，整合后使用单一的解码网络解码得到完整目标图像。掩码图像、红外图像、深度图像使用相同的特征提取网络，即多层串联的卷积层对图像进行多次重复卷积，卷积核随机初始化，通过对已有的遮挡样本-完整样本的训练，使用梯度下降算法不断缩小网络输出与真实完整样本间的误差，最终确定卷积核系数并使网络具有复原真实完整目标的能力。RGB图像使用迁移学习，直接使用VGG19的特征提取网络进行编码，卷积核系数不参与训练。解码网络同样由多层卷积层构成，与编码网络同时训练。虽然该网络结构与识别网络基本一致，但由于作用不同，网络输出不同，因此网络输出层通道数不同；且训练标签不同，因此网络权重也不同，需要重新搭建网络进行训练。
传统的算法仅能提取蔬果的位置(x，y，z)，对应地限制了采摘机器人的自由程度与采摘成功率，对于复杂环境下受枝叶影响导致姿态异常的目标果实难以处理，且进一步容易导致鲜嫩果实受损。在本实施例的位姿提取步骤中，提取出位置(x，y，z)以 确定末端停靠位置、姿态(α，β，γ)以确定末端采摘姿态并提高采摘成功率、果实半径(r)以确定末端开合程度并提高采摘成功率。利用7维控制参数的提取，机器人可以调整最佳的采摘姿态及抓取路径，其算法框架如图4所示。
位姿提取算法包含完整的特征提取(编码)与解码过程以完成遮挡目标形态复原任务，其输入包括完整目标深度图像与RGB图像，对不同的输入图像分别使用单独的特征提取网络进行编码，整合后使用单一的解码网络解码得到完整目标图像。深度图像多层串联的卷积层对图像进行多次重复卷积提取特征，卷积核随机初始化，通过对已有的遮挡样本-七维位姿数据的训练，使用梯度下降算法不断缩小网络输出与真实位姿间的误差，最终确定卷积核系数并使网络具有完整目标位姿提取的能力。RGB图像使用迁移学习，直接使用VGG19的特征提取网络进行编码，卷积核系数不参与训练。解码网络同样由多层卷积层构成，与编码网络同时训练。
在上述一种基于深度联想感知算法的受遮挡蔬果采收方法的基础上，本发明还提供一种基于深度联想感知算法的受遮挡蔬果采收系统，包括：
图像采集模块：采集RGB通道彩色图像、D通道深度图像和I通道红外图像；
预处理模块：对采集到的图像统一分辨率；
图像裁剪模块：剔除统一分辨率后的图像中的采收机器人工作区域以外的区域；
目标识别模块：识别成熟目标，并在裁剪后的图像中标记出成熟目标的每个像素；
遮挡判断模块：判断成熟目标是否被遮挡，在被遮挡的情况下复原出完整成熟目标的RGBD四个通道的图像，在不被遮挡的情况下直接获取完整成熟目标的RGBD四个通道的图像；
位姿提取模块：根据完整成熟目标的RGBD四个通道的图像，提取出成熟目标的位置和姿态至采收机器人。
本领域技术人员知道，除了以纯计算机可读程序代码方式实现本发明提供的系统及其各个装置、模块、单元以外，完全可以通过将方法步骤进行逻辑编程来使得本发明提供的系统及其各个装置、模块、单元以逻辑门、开关、专用集成电路、可编程逻辑控制器以及嵌入式微控制器等的形式来实现相同功能。所以，本发明提供的系统及其各项装置、模块、单元可以被认为是一种硬件部件，而对其内包括的用于实现各种功能的装置、模块、单元也可以视为硬件部件内的结构；也可以将用于实现各种功能的装置、模块、单元视为既可以是实现方法的软件模块又可以是硬件部件内的结构。
以上对本发明的具体实施例进行了描述。需要理解的是，本发明并不局限于上 述特定实施方式，本领域技术人员可以在权利要求的范围内做出各种变化或修改，这并不影响本发明的实质内容。在不冲突的情况下，本申请的实施例和实施例中的特征可以任意相互组合。

Claims (10)

  1. 一种基于深度联想感知算法的受遮挡蔬果采收方法，其特征在于，包括：
    图像采集步骤：采集RGB通道彩色图像、D通道深度图像和I通道红外图像；
    预处理步骤：对采集到的图像统一分辨率；
    图像裁剪步骤：剔除统一分辨率后的图像中的采收机器人工作区域以外的区域；
    目标识别步骤：识别成熟目标，并在裁剪后的图像中标记出成熟目标的每个像素；
    遮挡判断步骤：判断成熟目标是否被遮挡，在被遮挡的情况下复原出完整成熟目标的RGBD四个通道的图像，在不被遮挡的情况下直接获取完整成熟目标的RGBD四个通道的图像；
    位姿提取步骤：根据完整成熟目标的RGBD四个通道的图像，提取出成熟目标的位置和姿态至采收机器人。
  2. 根据权利要求1所述的基于深度联想感知算法的受遮挡蔬果采收方法，其特征在于，所述预处理步骤还包括去除图像中的噪声。
  3. 根据权利要求1所述的基于深度联想感知算法的受遮挡蔬果采收方法，其特征在于，所述图像裁剪步骤包括计算图像中每个点的真实坐标，从而剔除统一分辨率后的图像中的采收机器人工作区域以外的区域。
  4. 根据权利要求1所述的基于深度联想感知算法的受遮挡蔬果采收方法，其特征在于，判断成熟目标是否被遮挡的方法包括：
    提取目标识别得到的成熟目标轮廓像素，并提取轮廓内、外侧像素的深度值及三维坐标然后作差，若存在外侧像素深度值小于内侧像素深度值，且三维坐标差值大于阈值，说明目标被遮挡；否则目标未被遮挡。
  5. 根据权利要求1所述的基于深度联想感知算法的受遮挡蔬果采收方法，其特征在于，复原出完整成熟目标的RGBD四个通道的图像的方法包括：构建视觉联想模型对被遮挡的成熟目标进行还原。
  6. 一种基于深度联想感知算法的受遮挡蔬果采收系统，其特征在于，包括：
    图像采集模块：采集RGB通道彩色图像、D通道深度图像和I通道红外图像；
    预处理模块：对采集到的图像统一分辨率；
    图像裁剪模块：剔除统一分辨率后的图像中的采收机器人工作区域以外的区域；
    目标识别模块：识别成熟目标，并在裁剪后的图像中标记出成熟目标的每个像素；
    遮挡判断模块：判断成熟目标是否被遮挡，在被遮挡的情况下复原出完整成熟目标的RGBD四个通道的图像，在不被遮挡的情况下直接获取完整成熟目标的RGBD四个通道的图像；
    位姿提取模块：根据完整成熟目标的RGBD四个通道的图像，提取出成熟目标的位置和姿态至采收机器人。
  7. 根据权利要求6所述的基于深度联想感知算法的受遮挡蔬果采收系统，其特征在于，所述预处理模块还包括去除图像中的噪声。
  8. 根据权利要求6所述的基于深度联想感知算法的受遮挡蔬果采收系统，其特征在于，所述图像裁剪模块包括计算图像中每个点的真实坐标，从而剔除统一分辨率后的图像中的采收机器人工作区域以外的区域。
  9. 根据权利要求6所述的基于深度联想感知算法的受遮挡蔬果采收系统，其特征在于，判断成熟目标是否被遮挡的方法包括：
    提取目标识别得到的成熟目标轮廓像素，并提取轮廓内、外侧像素的深度值及三维坐标然后作差，若存在外侧像素深度值小于内侧像素深度值，且三维坐标差值大于阈值，说明目标被遮挡；否则目标未被遮挡。
  10. 根据权利要求6所述的基于深度联想感知算法的受遮挡蔬果采收系统，其特征在于，复原出完整成熟目标的RGBD四个通道的图像的方法包括：构建视觉联想模型对被遮挡的成熟目标进行还原。
PCT/CN2019/085439 2019-02-25 2019-05-05 基于深度联想感知算法的受遮挡蔬果采收方法 Ceased WO2020172992A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201910138052.8 2019-02-25
CN201910138052 2019-02-25
CN201910168396.3 2019-03-06
CN201910168396.3A CN110033487A (zh) 2019-02-25 2019-03-06 基于深度联想感知算法的受遮挡蔬果采收方法

Publications (1)

Publication Number Publication Date
WO2020172992A1 true WO2020172992A1 (zh) 2020-09-03

Family

ID=67235073

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/085439 Ceased WO2020172992A1 (zh) 2019-02-25 2019-05-05 基于深度联想感知算法的受遮挡蔬果采收方法

Country Status (2)

Country Link
CN (1) CN110033487A (zh)
WO (1) WO2020172992A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114677675A (zh) * 2022-04-15 2022-06-28 河北农业大学 一种基于双目视觉的苹果分级方法
CN116935378A (zh) * 2023-08-02 2023-10-24 石河子大学 一种基于高效聚合网络的杨梅果实识别方法
CN119111266A (zh) * 2024-09-11 2024-12-13 湖大粤港澳大湾区创新研究院(广州增城) 一种疏果机器人及其疏果方法
CN119526383A (zh) * 2024-10-18 2025-02-28 湖南信息职业技术学院 一种茶叶嫩芽采摘机器人智能控制系统及方法

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110941991B (zh) * 2019-10-28 2023-05-23 成都华迈通信技术有限公司 一种多通道结构化数据采集系统及数据采集方法
CN113221603A (zh) * 2020-02-05 2021-08-06 中国电信股份有限公司 一种检测监控设备被异物遮挡的方法及装置
CN111340030B (zh) * 2020-02-14 2021-12-21 北京市商汤科技开发有限公司 图像处理方法及装置、电子设备、计算机可读存储介质
CN112464747B (zh) * 2020-11-10 2024-07-23 广州富港生活智能科技有限公司 基于图像采集设备的身高检测方法及装置
CN112528826B (zh) * 2020-12-04 2024-02-02 江苏省农业科学院 一种基于3d视觉感知的采摘装置的控制方法
CN113099210B (zh) * 2021-03-31 2023-01-10 东莞中国科学院云计算产业技术创新与育成中心 三维图像还原方法、装置、计算机设备和存储介质
CN116758535B (zh) * 2023-05-06 2026-02-13 浙江光仑科技有限公司 基于rgb-d传感器的水果采摘方法、系统及计算机
CN117501990B (zh) * 2023-11-30 2024-06-07 北京市农林科学院智能装备技术研究中心 用于遮挡环境下水果自动采摘的定位和识别方法及系统
CN119360366B (zh) * 2024-12-25 2025-05-30 山东大学 基于多传感器融合的室外苹果识别定位方法及系统
CN120359903B (zh) * 2025-06-19 2025-08-22 农业农村部南京农业机械化研究所 基于图像处理的多品种蔬菜收获机动态识别与喂入控制系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101683037A (zh) * 2008-09-26 2010-03-31 中国农业大学 一种果实采摘机器人的控制器
US20110071675A1 (en) * 2009-09-22 2011-03-24 Gm Global Technology Operations, Inc. Visual perception system and method for a humanoid robot
CN105259832A (zh) * 2015-10-26 2016-01-20 重庆大学 一种基于多目立体视觉的智能采棉机控制系统
KR20170011503A (ko) * 2015-07-23 2017-02-02 성균관대학교산학협력단 파프리카 수확 시스템
CN108781762A (zh) * 2018-04-10 2018-11-13 河海大学文天学院 一种高效草莓采摘机器人及采摘方法

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103310218B (zh) * 2013-05-21 2016-08-10 常州大学 一种重叠遮挡果实精确识别方法
CN103295018B (zh) * 2013-05-21 2016-04-13 常州大学 一种枝叶遮挡果实精确识别方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101683037A (zh) * 2008-09-26 2010-03-31 中国农业大学 一种果实采摘机器人的控制器
US20110071675A1 (en) * 2009-09-22 2011-03-24 Gm Global Technology Operations, Inc. Visual perception system and method for a humanoid robot
KR20170011503A (ko) * 2015-07-23 2017-02-02 성균관대학교산학협력단 파프리카 수확 시스템
CN105259832A (zh) * 2015-10-26 2016-01-20 重庆大学 一种基于多目立体视觉的智能采棉机控制系统
CN108781762A (zh) * 2018-04-10 2018-11-13 河海大学文天学院 一种高效草莓采摘机器人及采摘方法

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
DING, YI: "Basic research on vision system technology of apple picking robot", MASTER THESIS, 1 December 2014 (2014-12-01), pages 1 - 86, XP009522873 *
GU, BAOXING: "Research on Intelligent Mobile Fruit Picking Robot", DOCTORAL DISSERTATION, 1 September 2012 (2012-09-01), pages 1 - 174, XP009522874 *
WANG, XINZHONG: "Studies on Information Acquisition and Path Planning of Greenhouse Tomato Harvesting Robot with Selective Harvesting Operation", DOCTORAL DISSERTATION, 1 November 2012 (2012-11-01), pages 1 - 156, XP009522861 *
ZHAO, WENMIN: "Image Recognition and Posture Detection of Ripe Apple Based on Machine Vision", MASTER THESIS, 1 June 2012 (2012-06-01), pages 1 - 75, XP009522863, DOI: 20190605155139X *
ZHOU, XIAOJUN: "Study on Mature Fruit Location and Obstacle Detection for Citrus Picking Robots", MASTER'S THESIS, 1 June 2009 (2009-06-01), pages 1 - 77, XP009522860 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114677675A (zh) * 2022-04-15 2022-06-28 河北农业大学 一种基于双目视觉的苹果分级方法
CN116935378A (zh) * 2023-08-02 2023-10-24 石河子大学 一种基于高效聚合网络的杨梅果实识别方法
CN119111266A (zh) * 2024-09-11 2024-12-13 湖大粤港澳大湾区创新研究院(广州增城) 一种疏果机器人及其疏果方法
CN119526383A (zh) * 2024-10-18 2025-02-28 湖南信息职业技术学院 一种茶叶嫩芽采摘机器人智能控制系统及方法

Also Published As

Publication number Publication date
CN110033487A (zh) 2019-07-19

Similar Documents

Publication Publication Date Title
WO2020172992A1 (zh) 基于深度联想感知算法的受遮挡蔬果采收方法
Wu et al. Detection and counting of banana bunches by integrating deep learning and classic image-processing algorithms
Zhuang et al. Computer vision-based localisation of picking points for automatic litchi harvesting applications towards natural scenarios
Zhang et al. Computer vision‐based tree trunk and branch identification and shaking points detection in Dense‐Foliage canopy for automated harvesting of apples
Wei et al. Automatic method of fruit object extraction under complex agricultural background for vision system of fruit picking robot
Liu et al. A method of segmenting apples at night based on color and position information
Häni et al. Apple counting using convolutional neural networks
Malik et al. Detection and counting of on-tree citrus fruit for crop yield estimation
Qingchun et al. Design of structured-light vision system for tomato harvesting robot
Lv et al. A method to obtain the near-large fruit from apple image in orchard for single-arm apple harvesting robot
Yu et al. An object-based visual attention model for robotic applications
CN107016357A (zh) 一种基于时间域卷积神经网络的视频行人检测方法
CN112990103B (zh) 一种基于机器视觉的串采二次定位方法
Wang et al. Strawberry ripeness classification method in facility environment based on red color ratio of fruit rind
Arefi et al. Development of an expert system based on wavelet transform and artificial neural networks for the ripe tomato harvesting robot
CN105574514A (zh) 温室生西红柿自动识别方法
CN111783693A (zh) 果蔬采摘机器人的智能识别方法
CN105844213B (zh) 一种绿色果实识别方法
Ogorodnikova et al. Method of ripe tomato detecting for a harvesting robot
James et al. Citdet: A benchmark dataset for citrus fruit detection
Beegam et al. Hybrid consensus and recovery block-based detection of ripe coffee cherry bunches using RGB-D sensor
Iparraguirre-Villanueva et al. Disease identification in crop plants based on convolutional neural networks
CN120495661A (zh) 一种基于改进yolov8s-seg模型的番茄植株分割方法
Chiatti et al. Surgical fine-tuning for grape bunch segmentation under visual domain shifts
CN105160354A (zh) 基于稀疏表示的苹果病害识别方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19916748

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19916748

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC DATED 04.02.2022 (EPO FORM 1205A)

122 Ep: pct application non-entry in european phase

Ref document number: 19916748

Country of ref document: EP

Kind code of ref document: A1