WO2021227289A1 - 一种基于深度学习的复杂背景低质量二维条码检测方法 - Google Patents

一种基于深度学习的复杂背景低质量二维条码检测方法 Download PDF

Info

Publication number
WO2021227289A1
WO2021227289A1 PCT/CN2020/109776 CN2020109776W WO2021227289A1 WO 2021227289 A1 WO2021227289 A1 WO 2021227289A1 CN 2020109776 W CN2020109776 W CN 2020109776W WO 2021227289 A1 WO2021227289 A1 WO 2021227289A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
dimensional barcode
dimensional
deep learning
images
Prior art date
Application number
PCT/CN2020/109776
Other languages
English (en)
French (fr)
Inventor
何卫平
魏晓红
李亮
Original Assignee
南京翱翔信息物理融合创新研究院有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 南京翱翔信息物理融合创新研究院有限公司 filed Critical 南京翱翔信息物理融合创新研究院有限公司
Publication of WO2021227289A1 publication Critical patent/WO2021227289A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
    • G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
    • G06K7/14—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation using light without selection of wavelength, e.g. sensing reflected white light
    • G06K7/1404—Methods for optical code recognition
    • G06K7/1408—Methods for optical code recognition the method being specifically adapted for the type of code
    • G06K7/1417—2D bar codes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
    • G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
    • G06K7/14—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation using light without selection of wavelength, e.g. sensing reflected white light
    • G06K7/1404—Methods for optical code recognition
    • G06K7/1439—Methods for optical code recognition including a method step for retrieval of the optical code
    • G06K7/1443—Methods for optical code recognition including a method step for retrieval of the optical code locating of the code in an image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
    • G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
    • G06K7/14—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation using light without selection of wavelength, e.g. sensing reflected white light
    • G06K7/1404—Methods for optical code recognition
    • G06K7/1439—Methods for optical code recognition including a method step for retrieval of the optical code
    • G06K7/1452—Methods for optical code recognition including a method step for retrieval of the optical code detecting bar code edges
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality

Definitions

  • the invention belongs to the technical field of image processing, and specifically relates to a method for detecting a low-quality two-dimensional bar code with a complex background based on deep learning.
  • Chinese Patent Publication No. CN 107066914 A discloses a method and system for positioning a two-dimensional bar code image in a complex background.
  • the method includes: preprocessing the two-dimensional bar code image to obtain the pre-processed two-dimensional bar code image;
  • the preprocessed two-dimensional barcode image is subjected to contour extraction processing to obtain the barcode contour of the two-dimensional barcode image and use it as a candidate barcode area; determine whether each candidate barcode area is rectangular, and remove non-rectangular candidate barcode areas;
  • the candidate area of the rectangular center is overlapped and removed to obtain a candidate barcode area with parallel opposite sides; and the positioning of the two-dimensional barcode image is obtained according to the candidate barcode area with parallel opposite sides.
  • the purpose of the present invention is to address the above-mentioned shortcomings of the prior art, and provide a deep learning based deep learning that expands the scope of application of barcode detection, and can accurately detect and locate the conditions of blur, partial occlusion, abrasion, and uneven illumination of two-dimensional barcodes.
  • the complex background low-quality two-dimensional bar code detection method is to address the above-mentioned shortcomings of the prior art, and provide a deep learning based deep learning that expands the scope of application of barcode detection, and can accurately detect and locate the conditions of blur, partial occlusion, abrasion, and uneven illumination of two-dimensional barcodes.
  • a low-quality 2D barcode detection method based on deep learning with complex background the steps are as follows.
  • Step 1 Collect images and make them into a two-dimensional barcode data set in VOC2007 format.
  • Step 2 Use the AR glasses on the mobile end or the industrial camera on the fixed end to obtain the image of the two-dimensional barcode, and send the obtained image to the server.
  • Step 3 Perform deblurring preprocessing on the received image through an existing algorithm based on sparse convolution kernel estimation.
  • Step 4 Based on the estimated value of the convolution kernel in step 3, use the existing non-convex optimization algorithm to perform the final restoration process on the preprocessed image.
  • Step 5 Use the convolutional neural network to perform feature extraction on the restored image containing the two-dimensional barcode to obtain a feature map of the two-dimensional barcode image.
  • Step 6 Use the improved Faster-RCNN algorithm to detect the acquired feature maps, and finally obtain the coordinates and angles of the two-dimensional barcode image and the two-dimensional barcode frame.
  • step 1 The specific steps of step 1 are: (a) Collect 5000 original images taken, and randomly select 500 images as the test set in VOC2007; (b) Data enhancement and expansion of the remaining 4500 images to 18000 images are used as VOC2007 (C) Mark the images containing 2D barcodes in the training set with a marking tool, obtain the position and category information of the 2D barcodes, and convert them into TFrecord format.
  • step 2 In the mobile terminal, the operator wears AR glasses, faces the work scene, selects the scene to be acquired, and uses gestures or the camera that comes with the AR glasses to shoot; (b) In the fixed end, multiple industrial cameras are evenly distributed in the scene to ensure that the industrial cameras can capture images of the entire scene.
  • step 3 The specific steps of step 3 are: (a) Use Gaussian filtering to pre-smooth the image to construct the edge of the image; (b) Select the image with the edge for convolution kernel estimation; (c) Use the predicted sharp edge gradient as the space A priori, to guide the initial restoration of the image.
  • step 5 The specific steps of step 5 are: (a) use the pre-training data of the convolutional neural network on the VOC2007 format data set to enhance the generalization ability of the two-dimensional barcode data set; (b) continuously adjust and modify the learning rate during the training process , Training batches and related parameters of the number of iterations; (c) use the adjusted image as the input image to obtain the feature map of the two-dimensional barcode image through the feature extractor.
  • the convolutional neural network is a VGG16 model
  • the VGG16 model includes 13 convolutional layers and 4 pooling layers.
  • the present invention can accurately detect and locate the two-dimensional barcode image with blur, partial occlusion, abrasion, and uneven illumination.
  • the operator of the mobile terminal of the present invention can perform 2D barcode detection by wearing AR glasses.
  • the camera on the AR glasses acquires images of the work scene, and then transmits the acquired images to the server for calculation.
  • the server will detect The results are sent back to the AR glasses, and the results are rendered to the AR glasses, combining virtual and real, giving the operator a more intuitive experience, which can greatly improve work efficiency.
  • the fixed-end image acquisition method can be used for two-dimensional bar code detection under complex and large background conditions.
  • the Faster-RCNN frame regression model is improved, and the angle regression is added, making it more suitable for two-dimensional Bar code detection can accurately detect two-dimensional bar codes.
  • the present invention can accurately detect and locate the two-dimensional barcode image with blur, partial occlusion, abrasion, and uneven illumination.
  • the operator of the mobile terminal of the present invention can perform 2D barcode detection by wearing AR glasses.
  • the camera on the AR glasses acquires images of the work scene, and then transmits the acquired images to the server for calculation.
  • the server will detect The results are sent back to the AR glasses, and the results are rendered to the AR glasses, combining virtual and real, giving the operator a more intuitive experience, which can greatly improve work efficiency.
  • the fixed-end image acquisition method can be used for two-dimensional bar code detection under complex and large background conditions.
  • the Faster-RCNN frame regression model is improved, and the angle regression is added, making it more suitable for two-dimensional Bar code detection can accurately detect two-dimensional bar codes.
  • Figure 1 is a flow chart of the present invention.
  • FIG. 2 is a flowchart of the mobile terminal image acquisition of the present invention.
  • Fig. 3 is a flowchart of the fixed-end image acquisition of the present invention.
  • the present invention includes it.
  • Step 1 Collect images and make them into a two-dimensional barcode data set in VOC2007 format.
  • Step 2 Use the AR glasses on the mobile terminal or the industrial camera on the fixed terminal to acquire the image of the two-dimensional barcode, and send the acquired image to the server, where the detection information acquired at the mobile terminal is displayed on the AR eyes, and the final acquired at the fixed terminal The detection information is displayed on the display screen of the server.
  • Step 3 Perform deblurring preprocessing on the received image through the existing algorithm based on sparse convolution kernel estimation.
  • the flow chart of the convolution kernel initialization algorithm is shown in Table 1.
  • Step 4 Based on the estimated value of the convolution kernel in step 3, the existing non-convex optimization algorithm is used to perform the final restoration processing on the preprocessed image.
  • the optimization process of the convolution kernel is shown in Table 2.
  • ISD is an iterative method. At the beginning of each iteration, the previously estimated kernel is used Form partial support, that is, put large value elements into the collection , And all other elements belong to the set , Is structured as.
  • That Index in with All are positive numbers, which are constantly changing in iterations, thus forming partial support.
  • the kernel value of is assigned to , Each collection
  • the elements in the optimization will reduce the loss, which leads to the adaptive kernel optimization process, and the minimization formula is.
  • the minimization formula (2) is used to refine the spread function (PSF), and its threshold is gently applied to the function through adaptive regularization. This threshold allows the energy to be concentrated on important values, thereby automatically maintaining the sparseness of the PSF, which is better Service deblurring process.
  • Step 5 Use the convolutional neural network to perform feature extraction on the restored image containing the two-dimensional barcode to obtain a feature map of the two-dimensional barcode image.
  • Step 6 Use the improved Faster-RCNN algorithm to detect the acquired feature maps, and finally obtain the coordinates and angles of the two-dimensional barcode image and the two-dimensional barcode frame.
  • step 1 The specific steps of step 1 are as follows.
  • (C) Mark the images containing 2D barcodes in the training set with a marking tool, obtain the position and category information of the 2D barcodes, and convert them into TFrecord format.
  • step 2 The specific steps of step 2 are as follows.
  • the operator wears AR glasses, faces the work scene, selects the scene to be acquired, and uses gestures or the camera that comes with the AR glasses to shoot.
  • (B) In the fixed end, distribute multiple industrial cameras evenly in the scene, for example: select three industrial cameras to ensure that the three industrial cameras can capture images of the entire scene.
  • step 3 The specific steps of step 3 are as follows.
  • (A) Use Gaussian filtering to pre-smooth the image to construct the edge of the image.
  • the Gaussian filtering has the following manifestations: (3) Among them, with Are the first derivative and the second derivative, Represents Gaussian smoothed input image for iterative update The initial input.
  • step 5 The specific steps of step 5 are as follows.
  • the convolutional neural network is the VGG16 model.
  • the VGG16 model includes 13 convolutional layers and 4 pooling layers.
  • (B) Constantly adjust and modify the relevant parameters of the learning rate, training batches, and number of iterations during the training process.
  • (C) Use the adjusted image as the input image to obtain the feature image of the two-dimensional barcode image through the feature extractor.
  • the feature image passes through 4 pooling layers and the size is 2 convolution kernels.
  • Step 6 Use the improved Faster-RCNN algorithm to detect the acquired feature maps, and finally obtain the coordinates and angles of the two-dimensional barcode image and the two-dimensional barcode frame.
  • the anchor point in the improved Faster-RCNN algorithm has three barcode sizes of 8, 16, and 32 in scale.
  • the shape of the conventional two-dimensional barcode is square, so the anchor point ratio is taken as 1, and the two-dimensional barcode rotates.
  • Angle is , , , 0,, with Among the 6 positive and negative samples, the optional angle of the two-dimensional bar code is the angle between the predicted bar code and the original bar code. The smaller the angle, the more accurate the prediction.
  • Combining the size of the anchor point and the rotation angle of the two-dimensional bar code select the most suitable anchor point for the frame selection and display of the two-dimensional bar code, so it is generated at each point on the extracted feature map An anchor, where the anchor point is used to determine the position information of the two-dimensional bar code, and the angle of rotation of the two-dimensional bar code is used to determine its direction information.
  • the positive and negative samples include a positive sample and a negative sample.
  • the positive sample is defined as the intersection ratio between the candidate frame of the rotated rectangle and the original marker frame is greater than 0.7, and the included angle with the meta marker frame is less than .
  • the negative sample is defined as the intersection ratio of the candidate frame of the rotated rectangle and the original marked frame is less than 0.3 or the intersection ratio of the candidate frame of the rotated rectangle and the original marked frame is greater than 0.7, and the included angle with the original marked frame is greater than , Samples in other situations do not participate in training.
  • the loss function of the candidate frame adopts the form of multi-task loss, and the specific definition is as follows.
  • Is a classification label for the case of a two-dimensional barcode , Only the background , Is the probability value calculated by the Softmax function, Is the tagged array value, Is the final prediction parameter of the network.
  • the trade-off between these two parameters is through the parameter To control.
  • the classification loss is defined as.
  • the loss function is used to regress the region of interest of the two-dimensional barcode, which is defined as follows.
  • the scale-invariant hyperparameters are calculated as follows.
  • Step 1 Collect images and make them into a two-dimensional barcode data set in VOC2007 format.
  • (C) Mark the images containing 2D barcodes in the training set with a marking tool, obtain the position and category information of the 2D barcodes, and convert them into TFrecord format.
  • Step 2 Use the AR glasses on the mobile end or the industrial camera on the fixed end to obtain the image of the two-dimensional barcode, and send the obtained image to the server.
  • the operator wears AR glasses, faces the work scene, selects the scene to be acquired, and uses gestures or the camera that comes with the AR glasses to shoot.
  • Step 3 Perform deblurring preprocessing on the received image through an existing algorithm based on sparse convolution kernel estimation.
  • Step 4 Based on the estimated value of the convolution kernel in step 3, use the existing non-convex optimization algorithm to perform the final restoration process on the preprocessed image.
  • Step 5 Use the convolutional neural network to perform feature extraction on the restored image containing the two-dimensional barcode to obtain a feature map of the two-dimensional barcode image.
  • (B) Constantly adjust and modify the relevant parameters of the learning rate, training batches, and number of iterations during the training process.
  • (C) Use the adjusted image as the input image to obtain the feature map of the two-dimensional barcode image through the feature extractor.
  • Step 6 Use the improved Faster-RCNN algorithm to detect the acquired feature maps, and finally obtain the coordinates and angles of the two-dimensional barcode image and the two-dimensional barcode frame.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Toxicology (AREA)
  • Electromagnetism (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

一种基于深度学习的复杂背景低质量二维条码检测方法，步骤如下：步骤1，收集图像并制作成VOC2007格式的二维条码数据集；步骤2，采用移动端的AR眼镜或固定端的工业相机进行二维条码的图像获取；步骤3，对接收到的图像通过现有的基于稀疏卷积核估计的算法进行预处理；步骤4，基于步骤3中的卷积核估计值，运用非凸优化算法对图像进行最后的复原处理；步骤5，利用卷积神经网络对复原后含有二维条码的图像进行特征提取；步骤6，对获取到的特征图利用改进的Faster-RCNN算法进行检测，最后得到二维条码图像和二维条码边框的坐标以及角度。上述方法对二维条码存在的模糊、磨损以及光照不均等问题均可准确检测定位。

Description

一种基于深度学习的复杂背景低质量二维条码检测方法 技术领域
本发明属于图像处理技术领域，具体涉及一种基于深度学习的复杂背景低质量二维条码检测方法。
 背景技术
随着“工业4.0”和“中国制造2025”概念的不断深入，智能化和自动化逐渐成为企业生产所关注的重点。在信息物理融合系统中，信息世界和物理世界的联系越来越密切，整个系统都需要对周围环境有一个“感知”，这个过程就是由自动识别技术实现的。其中零件直接标刻Data Matrix技术在工业应用中更加广泛，因此快速准确检测识别产品表面的二维条码，是实现产品信息追踪管理的前提，同时也是提升物流管理效率、实现产品生产过程信息获取以及产品实时追踪的重点。
在条码检测方面，企业还是使用传统的图像处理方式来进行二维条码的检测。有利用L边对二维条码进行检测定位，利用Hough变换或者Radon变换检测直线边缘得到“L”形寻边区的位置和旋转角度，但是要想取得较好的变换精度，这种方法计算量大，如果变换精读太小降低了计算量，但是又无法取得较好的检测效果。传统的图像处理方法需要针对不同的条码检测情况设计不同的图像特征，这是一项耗费人力物力的工作，而且设计的图像特征也不具有通用性；据统计，85%的识读时间都消耗在寻找可能识读的位置上，针对复杂背景中的条码或者微小二维条码的鲁棒性较差，条码的检测时间长，识读准确率低。
中国专利公开号CN 107066914 A公开了一种复杂背景下的二维条码图像定位方法及系统，所述方法包括：对二维条码图像进行预处理，获取预处理后的二维条码图像；对所述预处理后的二维条码图像进行轮廓提取处理，获取二维条码图像的条码轮廓，并作为候选条码区；判断每个候选条码区是否为矩形，去除非矩形的候选条码区；对保留下来的候选区进行矩形中心重叠去除处理，获取对边平行的候选条码区；根据所述对边平行的候选条码区获取二维条码图像的定位。上述现有技术解决了复杂背景下完整二维条码图像定位的问题，但是对于标刻在零件表面的二维条码，在零件运输过程中，难免会因为磨损或者油污导致二维条码的辨识度降低，导致条码检测的难度会加大，甚至无法达到检测定位要求，难以检测定位到二维条码。
 技术问题
本发明的目的就是针对上述现有技术的不足，提供一种扩大了条码检测的适用范围，对二维条码存在模糊、部分遮挡、磨损以及光照不均的情况均可准确检测定位的基于深度学习的复杂背景低质量二维条码检测方法。
 技术解决方案
本发明采用的技术方案如下。
一种基于深度学习的复杂背景低质量二维条码检测方法，步骤如下。
步骤1：收集图像并制作成VOC2007格式的二维条码数据集。
步骤2：采用移动端的AR眼镜或固定端的工业相机进行二维条码的图像获取，并将获取的图像发送给服务器。
步骤3：对接收到的图像通过现有的基于稀疏卷积核估计的算法进行去模糊的预处理。
步骤4：基于步骤3中的卷积核估计值，运用现有的非凸优化算法对预处理后的的图像进行最后的复原处理。
步骤5：利用卷积神经网络对复原之后含有二维条码的图像进行特征提取，得到二维条码图像的特征图。
步骤6：对获取到的特征图利用改进的Faster-RCNN算法进行检测，最后得到二维条码图像和二维条码边框的坐标以及角度。
步骤1的具体步骤为：（a）收集拍摄到的5000张原始图像，随机选取500张图像作为VOC2007内的测试集；（b）剩余的4500张图像进行数据增强扩充到18000张图像作为VOC2007内的训练集；（c）对训练集中的包含有二维条码的图像用标记工具进行标注，获取二维条码的位置和类别信息，并将其转化为TFrecord格式。
步骤2的具体步骤为：（a）在移动端中，操作者配戴AR眼镜，面对工作场景，选定要获取的场景后，利用手势或者AR眼镜自带的相机进行拍摄；（b）在固定端中，将多个工业相机均匀分布在场景之中，确保工业相机可以对整个场景进行图像拍摄。
步骤3的具体步骤为：（a）采用高斯滤波对图像进行预平滑处理来构造图像的边缘；（b）选择具有边缘的图像进行卷积核估计；（c）使用预测的锐利边缘梯度作为空间先验，指导图像的初步恢复。
步骤5的具体步骤为：（a）利用卷积神经网络在VOC2007格式的数据集上的预训练数据来增强二维条码数据集的泛化能力；（b）在训练过程中不断调整修改学习率、训练批次和迭代次数的相关参数；（c）将调整后的图像作为输入图经过特征提取器获取二维条码图像的特征图。
步骤5中，所述卷积神经网络为 VGG16模型，所述VGG16模型包含13个卷积层和4个池化层。
本发明的有益效果有。
（1）本发明对二维条码图像存在模糊、部分遮挡、磨损以及光照不均的情况均可准确检测定位。
（2）本发明移动端的操作人员可以通过佩戴AR眼镜即可进行二维条码检测，AR眼镜上的相机对工作场景进行中图像获取，然后将获取的图片传输到服务器进行运算，服务器将检测的结果回传给AR眼镜，并将结果渲染到AR眼镜上，将虚实进行结合，给予操作者更加直观的感受，可以极大提高工作效率。
（3）在大视野范围下采用固定端的图像获取方式可以进行复杂大背景情况下的二维条码检测，改进了Faster-RCNN的边框回归模型，增加了角度的回归，使其更加适用于二维条码检测，可以准确检测二维条码。
 有益效果
（1）本发明对二维条码图像存在模糊、部分遮挡、磨损以及光照不均的情况均可准确检测定位。
（2）本发明移动端的操作人员可以通过佩戴AR眼镜即可进行二维条码检测，AR眼镜上的相机对工作场景进行中图像获取，然后将获取的图片传输到服务器进行运算，服务器将检测的结果回传给AR眼镜，并将结果渲染到AR眼镜上，将虚实进行结合，给予操作者更加直观的感受，可以极大提高工作效率。
（3）在大视野范围下采用固定端的图像获取方式可以进行复杂大背景情况下的二维条码检测，改进了Faster-RCNN的边框回归模型，增加了角度的回归，使其更加适用于二维条码检测，可以准确检测二维条码。
 附图说明
图1为本发明的流程图。
图2为本发明的移动端图像获取流程图 。
图3为本发明的固定端图像获取流程图。
 本发明的实施方式
通过以下说明和实施例对本发明的基于深度学习的复杂背景低质量二维条码检测方法作进一步的说明。
如图1-3所示，本发明它包括。
步骤1：收集图像并制作成VOC2007格式的二维条码数据集。
步骤2：采用移动端的AR眼镜或固定端的工业相机进行二维条码的图像获取，并将获取的图像发送给服务器，其中移动端最后获取的检测信息在AR眼睛上进行显示，固定端最后获取的检测信息在服务器的显示屏上进行显示。
步骤3：对接收到的图像通过现有的基于稀疏卷积核估计的算法进行去模糊的预处理，其卷积核初始化算法流程表如表1所示。
表1 。
Figure 34277dest_path_image001
。
 步骤4：基于步骤3中的卷积核估计值，运用现有的非凸优化算法对预处理后的的图像进行最后的复原处理，其卷积核优化流程如表2所示。
表2。
Figure 115366dest_path_image002
。
ISD是一个迭代的方法，在每次迭代的开始，使用先前估计的核
Figure 714843dest_path_image003
形成部分支持，也就是说，将大值元素放入集合
Figure 414946dest_path_image004
中，而所有其他的元素都属于集合
Figure 517900dest_path_image005
，
Figure 645256dest_path_image005
被构造为。
                                              
Figure 469380dest_path_image006
                                                      （1）。
其
Figure 973174dest_path_image007
中的索引
Figure 71580dest_path_image008
和
Figure 619105dest_path_image009
均为正数，在迭代中不断变化，从而形成部分支持。以
Figure 678327dest_path_image003
的升序对所有元素进行排序，并计算每两个相邻元素之间的差
Figure 235080dest_path_image010
，
Figure 797779dest_path_image011
，然后从
Figure 516205dest_path_image010
开始依次检查这些差，搜寻满足
Figure 921779dest_path_image012
的第一个元素，
Figure 767375dest_path_image013
是内核宽度，
Figure 967937dest_path_image014
返回
Figure 607997dest_path_image003
的最大值，随后将位置
Figure 891079dest_path_image008
的内核值分配给
Figure 274787dest_path_image015
，每个集合
Figure 936713dest_path_image016
中的元素在优化中将减少损失，从而导致自适应内核优化过程，最小化式为。
                                    
Figure 996941dest_path_image017
                               （2）。
最小化式（2）用于扩散函数（PSF）细化，其阈值通过自适应正则化轻柔地应用到函数中，该阈值使能量可以集中在重要值上，从而自动保持PSF稀疏性，更好服务去模糊过程。
步骤5：利用卷积神经网络对复原之后含有二维条码的图像进行特征提取，得到二维条码图像的特征图。
步骤6：对获取到的特征图利用改进的Faster-RCNN算法进行检测，最后得到二维条码图像和二维条码边框的坐标以及角度。
步骤1的具体步骤为。
（a）收集拍摄到的5000张原始图像，随机选取500张图像作为VOC2007内的测试集。
（b）剩余的4500张图像进行数据增强扩充到18000张图像作为VOC2007内的训练集，图像原始的分辨率均为2592*1944，包含了各种遮挡、光照不均、磨损、油污等情况。
（c）对训练集中的包含有二维条码的图像用标记工具进行标注，获取二维条码的位置和类别信息，并将其转化为TFrecord格式。
步骤2的具体步骤为。
（a）在移动端中，操作者配戴AR眼镜，面对工作场景，选定要获取的场景后，利用手势或者AR眼镜自带的相机进行拍摄。
（b）在固定端中，将多个工业相机均匀分布在场景之中，例如：选取三个工业相机，确保三个工业相机可以对整个场景进行图像拍摄。
步骤3的具体步骤为。
（a）采用高斯滤波对图像进行预平滑处理来构造图像的边缘，该高斯滤波具有以下表现形式：（3）其中，
Figure 518053dest_path_image018
和
Figure 689140dest_path_image019
分别为一阶导数和二阶导数，
Figure 612096dest_path_image020
表示高斯平滑的输入图像，用于迭代更新
Figure 857875dest_path_image021
的初始输入。
（b）选择具有边缘的图像进行卷积核估计，用一种新的准则来选择用于估计的边缘，定义为。
                                               
Figure 725337dest_path_image022
                                     （4）。
其中
Figure 716427dest_path_image023
表示模糊图像，
Figure 743157dest_path_image024
是一个以像素
Figure 551713dest_path_image025
为中心的
Figure 640892dest_path_image026
的窗口，常数0.5是为了防止在平坦区域产生较大的
Figure 170094dest_path_image027
。狭窄物体（尖峰）带符号的
Figure 785752dest_path_image028
大部分会在
Figure 374996dest_path_image029
中抵消，
Figure 344613dest_path_image030
是
Figure 536560dest_path_image031
中绝对梯度大小的总和，它估计窗口中图像结构的强度，
Figure 757457dest_path_image032
的值较小表示涉及到尖峰或平坦区域，这会导致中和许多梯度分量，然后使用掩码排除较小
Figure 32450dest_path_image032
值窗口的像素，由此可得掩模
Figure 971587dest_path_image033
的表达式为。
                                
Figure 357438dest_path_image034
                         （5）。
其中
Figure 963999dest_path_image035
是Heaviside阶跃函数，对于负值输出零，其他情况输出1，
Figure 675472dest_path_image036
是 一个阈值，用于核估计的最终选择边缘确定为。
                           
Figure 367485dest_path_image037
                   （6）。
其中
Figure 28798dest_path_image038
表示滤波图像，
Figure 224287dest_path_image039
是梯度幅值的阈值，等式（6）排除幅度的一部分，这取决于幅度
Figure 841082dest_path_image040
和先前的掩模
Figure 754811dest_path_image033
，该选择过程减小了随后的卷积核估计中的歧义。经过不同迭代计算的
Figure 826673dest_path_image041
映射表明，包括更多的边并不一定有益于卷积核估计，优化可能会被误导，尤其是在前几次迭代中，因此，图像边缘选择过程对于减少混乱至关重要。
（c）使用预测的锐利边缘梯度作为空间先验，指导图像的初步恢复。
步骤5的具体步骤为。
（a）利用卷积神经网络在VOC2007格式的数据集上的预训练数据来增强二维条码数据集的泛化能力，其中卷积神经网络为VGG16模型，该VGG16模型包括13个卷积层和4个池化层。
（b）在训练过程中不断调整修改学习率、训练批次和迭代次数的相关参数。
（c）将调整后的图像作为输入图经过特征提取器获取二维条码图像的特征图，该特征图经过4个池化层，大小都为2卷积核，将得到大小为输入图1/16的特征图，从而提高检测的效率。
步骤6：对获取到的特征图利用改进的Faster-RCNN算法进行检测，最后得到二维条码图像和二维条码边框的坐标以及角度。
所述改进的Faster-RCNN算法中的锚点在尺度上具有8、16和32三种条码尺寸，常规的二维条码的形状为正方形，故锚点比例取为1，而二维条码旋转的角度为
Figure 657094dest_path_image042
，
Figure 664364dest_path_image043
，
Figure 314658dest_path_image044
，0，，
Figure 65576dest_path_image045
和
Figure 360291dest_path_image046
这6个正负样本，其中二维条码选装的角度为预测条码和原始条码之间的夹角，夹角越小则预测越精准。
结合锚点的尺寸和二维条码旋转的角度，选取最合适的锚点进行二维条码的框选和显示，所以在提取的特征图上的每个点生成
Figure 790660dest_path_image047
个锚，其中锚点用于确定二维条码的位置信息，而二维条码旋转的角度用于确定其方向信息。
所述正负样本包括正样本和负样本，所述正样本的定义为旋转矩形的候选框与原标记框的交并比大于0.7，且与元标记框的夹角小于
Figure 678981dest_path_image048
。
所述负样本的定义为旋转矩形的候选框与原标记框的交并比小于0.3或者旋转矩形的候选框与原标记框的交并比大于0.7，且与原标记框的夹角大于
Figure 482858dest_path_image048
，其他情况的样本不参与训练。
其中对候选框的损失函数采用多任务损失的形式，具体定义如下。
                                     
Figure 773025dest_path_image049
                                  （7）。
Figure 512311dest_path_image050
是分类标签，对于有二维条码的情况下
Figure 871617dest_path_image051
，只有背景的情况下
Figure 229917dest_path_image052
，
Figure 623858dest_path_image053
是经过Softmax函数计算的概率值，
Figure 143833dest_path_image054
是标记的数组值，
Figure 865801dest_path_image055
是网络最后的预测参数，这两个参数的权衡通过参数
Figure 268270dest_path_image056
来控制。而分类损失定义为。
                                                  
Figure 267451dest_path_image057
                                               （8）。
对于候选框，采用
Figure 473173dest_path_image058
损失函数来对二维条码的感兴趣区域进行回归，定义如下。
                                 
Figure 557803dest_path_image059
                                  （9）。
其中，
Figure 116961dest_path_image060
。
尺度不变的超参数计算如下。
                                            
Figure 219915dest_path_image061
                         （10）。
                                          
Figure 612850dest_path_image062
                         （11）。
其中
Figure 434044dest_path_image063
和
Figure 672259dest_path_image064
分别表示预测框，锚和标注真值的参数，不同形式的
Figure 505086dest_path_image065
和
Figure 55540dest_path_image066
表示规则同
Figure 114763dest_path_image067
，
Figure 140357dest_path_image068
，其中
Figure 234215dest_path_image069
，可以确保
Figure 828007dest_path_image070
。
实施例。
步骤1：收集图像并制作成VOC2007格式的二维条码数据集。
具体实现步骤如下。
（a）收集拍摄到的5000张原始图像，随机选取500张图像作为VOC2007内的测试集。
（b）剩余的4500张图像进行数据增强扩充到18000张图像作为VOC2007内的训练集。
（c）对训练集中的包含有二维条码的图像用标记工具进行标注，获取二维条码的位置和类别信息，并将其转化为TFrecord格式。
步骤2：采用移动端的AR眼镜或固定端的工业相机进行二维条码的图像获取，并将获取的图像发送给服务器。
具体实现步骤如下。
（a）在移动端中，操作者配戴AR眼镜，面对工作场景，选定要获取的场景后，利用手势或者AR眼镜自带的相机进行拍摄。
（b）在固定端中，将三个工业相机均匀分布在场景之中，确保三个工业相机可以对整个场景进行图像拍摄。
步骤3：对接收到的图像通过现有的基于稀疏卷积核估计的算法进行去模糊的预处理。
具体实现步骤如下。
（a）采用高斯滤波对图像进行预平滑处理来构造图像的边缘。
（b）选择具有边缘的图像进行卷积核估计。
（c）使用预测的锐利边缘梯度作为空间先验，指导图像的初步恢复。
步骤4：基于步骤3中的卷积核估计值，运用现有的非凸优化算法对预处理后的的图像进行最后的复原处理。
步骤5：利用卷积神经网络对复原之后含有二维条码的图像进行特征提取，得到二维条码图像的特征图。
具体实现步骤如下。
（a）利用卷积神经网络在VOC2007格式的数据集上的预训练数据来增强二维条码数据集的泛化能力。
（b）在训练过程中不断调整修改学习率、训练批次和迭代次数的相关参数。
（c）将调整后的图像作为输入图经过特征提取器获取二维条码图像的特征图。
步骤6：对获取到的特征图利用改进的Faster-RCNN算法进行检测，最后得到二维条码图像和二维条码边框的坐标以及角度。
本发明涉及的其它未说明部分与现有技术相同。

Claims (6)

  1. 一种基于深度学习的复杂背景低质量二维条码检测方法，其特征是，步骤如下：
    步骤1：收集图像并制作成VOC2007格式的二维条码数据集；
    步骤2：采用移动端的AR眼镜或固定端的工业相机进行二维条码的图像获取，并将获取的图像发送给服务器；
    步骤3：对接收到的图像通过现有的基于稀疏卷积核估计的算法进行去模糊的预处理；
    步骤4：基于步骤3中的卷积核估计值，运用现有的非凸优化算法对预处理后的的图像进行最后的复原处理；
    步骤5：利用卷积神经网络对复原之后含有二维条码的图像进行特征提取，得到二维条码图像的特征图；
    步骤6：对获取到的特征图利用改进的Faster-RCNN算法进行检测，最后得到二维条码图像和二维条码边框的坐标以及角度。
  2. 根据权利要求1所述的基于深度学习的复杂背景低质量二维条码检测方法，其特征是，步骤1的具体步骤为：（a）收集拍摄到的5000张原始图像，随机选取500张图像作为VOC2007内的测试集；（b）剩余的4500张图像进行数据增强扩充到18000张图像作为VOC2007内的训练集；（c）对训练集中的包含有二维条码的图像用标记工具进行标注，获取二维条码的位置和类别信息。
  3. 根据权利要求1所述的基于深度学习的复杂背景低质量二维条码检测方法，其特征是，步骤2的具体步骤为：（a）在移动端中，操作者配戴AR眼镜，面对工作场景，选定要获取的场景后，利用手势或者AR眼镜自带的相机进行拍摄；（b）在固定端中，将多个工业相机均匀分布在场景之中，确保工业相机可以对整个场景进行图像拍摄。
  4. 根据权利要求1所述的基于深度学习的复杂背景低质量二维条码检测方法，其特征是，步骤3的具体步骤为：（a）采用高斯滤波对图像进行预平滑处理来构造图像的边缘；（b）选择具有边缘的图像进行卷积核估计；（c）使用预测的锐利边缘梯度作为空间先验，指导图像的初步恢复。
  5. 根据权利要求1所述的基于深度学习的复杂背景低质量二维条码检测方法，其特征是，步骤5的具体步骤为：（a）利用卷积神经网络在VOC2007格式的数据集上的预训练数据来增强二维条码数据集的泛化能力；（b）在训练过程中不断调整修改学习率、训练批次和迭代次数的相关参数；（c）将调整后的图像作为输入图经过特征提取器获取二维条码图像的特征图。
  6. 根据权利要求1所述的基于深度学习的复杂背景低质量二维条码检测方法，其特征是，步骤5中，所述卷积神经网络为 VGG16模型，所述VGG16模型包含13个卷积层和4个池化层。
PCT/CN2020/109776 2020-05-14 2020-08-18 一种基于深度学习的复杂背景低质量二维条码检测方法 WO2021227289A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010405518.9A CN111753572A (zh) 2020-05-14 2020-05-14 一种基于深度学习的复杂背景低质量二维条码检测方法
CN202010405518.9 2020-05-14

Publications (1)

Publication Number Publication Date
WO2021227289A1 true WO2021227289A1 (zh) 2021-11-18

Family

ID=72674383

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/109776 WO2021227289A1 (zh) 2020-05-14 2020-08-18 一种基于深度学习的复杂背景低质量二维条码检测方法

Country Status (2)

Country Link
CN (1) CN111753572A (zh)
WO (1) WO2021227289A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114139564A (zh) * 2021-12-07 2022-03-04 Oppo广东移动通信有限公司 二维码检测方法、装置、终端设备及检测网络的训练方法
CN114943238A (zh) * 2022-07-26 2022-08-26 北京紫光青藤微系统有限公司 用于条码识读设备的测试设备及其测试方法
CN117036767A (zh) * 2023-04-10 2023-11-10 广东奥普特科技股份有限公司 二维码识别方法、装置、设备和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346850A (zh) * 2011-10-13 2012-02-08 西北工业大学 复杂金属背景下DataMatrix条码区域定位方法
WO2016160912A1 (en) * 2015-03-30 2016-10-06 Temptime Corporation A two dimensional barcode with dynamic environmental data system, method, and apparatus
CN107066914A (zh) * 2017-03-13 2017-08-18 广州卓德信息科技有限公司 一种复杂背景下的二维条码图像定位方法及系统
CN108416412A (zh) * 2018-01-23 2018-08-17 浙江瀚镪自动化设备股份有限公司 一种基于多任务深度学习的物流复合码识别方法
CN109165538A (zh) * 2018-07-18 2019-01-08 北京飞搜科技有限公司 基于深度神经网络的条形码检测方法及装置
CN110610174A (zh) * 2019-07-16 2019-12-24 北京工业大学 复杂条件下银行卡号识别方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104091315B (zh) * 2014-07-22 2017-02-15 中国科学技术大学 一种车牌图像去模糊的方法及系统
CN107967482A (zh) * 2017-10-24 2018-04-27 广东中科南海岸车联网技术有限公司 图标识别方法及装置
CN109961049B (zh) * 2019-03-27 2022-04-26 东南大学 一种复杂场景下香烟品牌识别方法
CN110969589B (zh) * 2019-12-03 2023-06-02 重庆大学 基于多流注意对抗网络的动态场景模糊图像盲复原方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346850A (zh) * 2011-10-13 2012-02-08 西北工业大学 复杂金属背景下DataMatrix条码区域定位方法
WO2016160912A1 (en) * 2015-03-30 2016-10-06 Temptime Corporation A two dimensional barcode with dynamic environmental data system, method, and apparatus
CN107066914A (zh) * 2017-03-13 2017-08-18 广州卓德信息科技有限公司 一种复杂背景下的二维条码图像定位方法及系统
CN108416412A (zh) * 2018-01-23 2018-08-17 浙江瀚镪自动化设备股份有限公司 一种基于多任务深度学习的物流复合码识别方法
CN109165538A (zh) * 2018-07-18 2019-01-08 北京飞搜科技有限公司 基于深度神经网络的条形码检测方法及装置
CN110610174A (zh) * 2019-07-16 2019-12-24 北京工业大学 复杂条件下银行卡号识别方法

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114139564A (zh) * 2021-12-07 2022-03-04 Oppo广东移动通信有限公司 二维码检测方法、装置、终端设备及检测网络的训练方法
CN114139564B (zh) * 2021-12-07 2024-05-07 Oppo广东移动通信有限公司 二维码检测方法、装置、终端设备及检测网络的训练方法
CN114943238A (zh) * 2022-07-26 2022-08-26 北京紫光青藤微系统有限公司 用于条码识读设备的测试设备及其测试方法
CN117036767A (zh) * 2023-04-10 2023-11-10 广东奥普特科技股份有限公司 二维码识别方法、装置、设备和存储介质

Also Published As

Publication number Publication date
CN111753572A (zh) 2020-10-09

Similar Documents

Publication Publication Date Title
CN111179225B (zh) 一种基于灰度梯度聚类的试纸表面纹理缺陷检测方法
CN106875381B (zh) 一种基于深度学习的手机外壳缺陷检测方法
WO2021227289A1 (zh) 一种基于深度学习的复杂背景低质量二维条码检测方法
CN109977723B (zh) 大票据图片文字识别方法
CN113592845A (zh) 一种电池涂布的缺陷检测方法及装置、存储介质
WO2020206850A1 (zh) 基于高维图像的图像标注方法和装置
CN113658131B (zh) 一种基于机器视觉巡游式环锭纺断纱检测方法
CA3136674C (en) Methods and systems for crack detection using a fully convolutional network
CN108171102A (zh) 一种基于视觉的零件快速识别方法
Babahajiani et al. Object recognition in 3D point cloud of urban street scene
JP2023533644A (ja) マシン視覚に基づく透明度検出方法
CN112883986A (zh) 一种复杂背景下的静态红外靶灯识别方法
Niu et al. Data augmentation on defect detection of sanitary ceramics
CN112669300A (zh) 瑕疵检测方法及装置、计算机设备和存储介质
CN113989604A (zh) 基于端到端深度学习的轮胎dot信息识别方法
CN108961262A (zh) 一种复杂场景下的条码定位方法
CN113705564B (zh) 一种指针式仪表识别读数方法
CN112686872B (zh) 基于深度学习的木材计数方法
CN105374045B (zh) 一种基于形态学的图像特定形状尺寸目标快速分割方法
Dai et al. An Improved ORB Feature Extraction Algorithm Based on Enhanced Image and Truncated Adaptive Threshold
CN117649589A (zh) 一种基于改进YOLO-V5s模型的LNG卸料臂目标识别方法
CN113033386A (zh) 一种基于高分辨率遥感影像的输电线路通道隐患识别方法及系统
CN117132540A (zh) 一种基于分割模型的pcb电路板缺陷后处理方法
CN111243006A (zh) 一种基于图像处理的液滴接触角及尺寸的测量方法
CN109741302B (zh) 基于机器视觉的sd卡形态识别系统和方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20935337

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20935337

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20935337

Country of ref document: EP

Kind code of ref document: A1