WO2019136760A1 - 图像语义分割方法、可编程逻辑电路、系统及电子设备 - Google Patents

图像语义分割方法、可编程逻辑电路、系统及电子设备 Download PDF

Info

Publication number
WO2019136760A1
WO2019136760A1 PCT/CN2018/072674 CN2018072674W WO2019136760A1 WO 2019136760 A1 WO2019136760 A1 WO 2019136760A1 CN 2018072674 W CN2018072674 W CN 2018072674W WO 2019136760 A1 WO2019136760 A1 WO 2019136760A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processing module
convolution
deconvolution
feature data
Prior art date
Application number
PCT/CN2018/072674
Other languages
English (en)
French (fr)
Inventor
肖梦秋
Original Assignee
深圳鲲云信息科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳鲲云信息科技有限公司 filed Critical 深圳鲲云信息科技有限公司
Priority to CN201880002143.2A priority Critical patent/CN109564684B/zh
Priority to PCT/CN2018/072674 priority patent/WO2019136760A1/zh
Priority to US16/962,444 priority patent/US11636665B2/en
Publication of WO2019136760A1 publication Critical patent/WO2019136760A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/15—Correlation function computation including computation of convolution operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2431—Multiple classes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/19—Recognition using electronic means
    • G06V30/191—Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
    • G06V30/19173—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/26—Techniques for post-processing, e.g. correcting the recognition result
    • G06V30/262—Techniques for post-processing, e.g. correcting the recognition result using context analysis, e.g. lexical, syntactic or semantic context
    • G06V30/274—Syntactic or semantic context, e.g. balancing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30181—Earth observation
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present invention relates to the field of image processing, and in particular to an image semantic segmentation method, a programmable logic circuit, a system, and an electronic device.
  • the image semantic segmentation technology based on deep learning can quickly segment the image semantically, and cluster the pixels with the same semantics at different positions in the image into one classification and mark them with the same color.
  • the current image semantic segmentation is realized through software programming, and the processing speed needs to be improved.
  • the object of the present invention is to provide an image semantic segmentation method, a programmable logic circuit, a system and an electronic device, and implement image semantic segmentation through a programmable logic circuit, thereby improving the processing speed of image semantic segmentation.
  • the present invention provides an image semantic segmentation method, the method being applied to a programmable logic circuit; the programmable logic circuit comprising: a convolution processing module, and a deconvolution processing module;
  • the method includes: the convolution processing module receives an image that needs to be semantically segmented, and performs convolution operation processing on the image to generate feature data of each feature tile of the image; the deconvolution processing module Performing a deconvolution operation process on each of the feature data to respectively obtain a tile region having the same size as the feature tile of each feature data; determining a degree of approximation of each of the feature data and each preset semantic class, and Each of the feature data is classified into a preset semantic category with the highest degree of similarity; wherein each of the preset semantic category associations has a fill color corresponding to the one-to-one correspondence; and preset semantics where each of the feature data is located
  • the colors associated with the categories are color-filled for each of the corresponding tile regions to enable semantic segmentation of the images
  • the programmable logic circuit further includes: a shared cache module electrically connected to the convolution processing module and the deconvolution processing module; the method further comprising: the convolution The processing module and the deconvolution processing module transmit or receive data in a time-multiplexed manner through the shared cache module.
  • the method further includes: the convolution processing module performs convolution operation processing on the basis of the previous convolution operation processing on the image, thereby increasing the number of convolution layers The number of the feature data is reduced.
  • the method further includes: the deconvolution processing module performs deconvolution operation processing on the basis of the previous deconvolution operation processing on the image, until all the convolution operations are performed.
  • the layers of the arithmetic processing are dimensionally restored one by one.
  • the present invention provides a programmable logic circuit, including: a convolution processing module, and a deconvolution processing module; the convolution processing module is configured to receive an image that needs to be semantically segmented, Performing a convolution operation process on the image to generate feature data of each feature tile of the image; the deconvolution processing module is configured to perform deconvolution operation processing on each of the feature data to respectively Obtaining a tile area of the same size as the feature tile of each of the feature data; determining a degree of approximation of each of the feature data and each preset semantic category, and classifying each of the feature data into a preset with the highest degree of similarity In the semantic category, each of the preset semantic category associations has a fill color corresponding to the one-to-one correspondence; and the color associated with the preset semantic category in which the feature data is located is performed for each corresponding tile area Color padding to achieve semantic segmentation of the image.
  • the programmable logic circuit further includes: a shared cache module electrically connected to the convolution processing module and the deconvolution processing module for the convolution processing module and the The deconvolution processing module transmits or receives data in a time-multiplexed manner.
  • the convolution processing module is further configured to: perform convolution operation processing on the basis of the previous convolution operation processing on the image, thereby reducing the number of convolution layers by The number of feature data.
  • the deconvolution processing module is further configured to perform deconvolution operation processing on the basis of the previous deconvolution operation processing on the image until all convolution operation processing The layers are restored one by one.
  • the present invention provides an image semantic segmentation system, comprising: the programmable logic circuit, the central processing unit, and the memory electrically connected to the central processing unit;
  • the convolution processing module is electrically connected to the central processing unit and the memory;
  • the deconvolution processing module is electrically connected to the central processing unit and the memory;
  • the programmable logic circuit further includes the sharing In the case of a cache module, the shared cache module is also electrically connected to the memory.
  • the present invention provides an electronic device comprising: the image semantic segmentation system as described above.
  • the image semantic segmentation method, the programmable logic circuit, the system and the electronic device of the invention realize the image semantic segmentation processing through the hardware circuit, and the speed of the image semantic segmentation processing is greatly improved.
  • FIG. 1 is a schematic structural diagram of an image semantic segmentation system according to an embodiment of the present invention.
  • FIGS. 2A and 2B are respectively shown diagrams showing the simulation original image and its processing by the image semantic segmentation system of the present invention.
  • the invention provides a programmable logic circuit (FPGA circuit) capable of realizing an image semantic segmentation task, an image semantic segmentation system including the same, and an electronic device including the image semantic segmentation system, in addition to providing an application An image semantic segmentation method for the programmable logic circuit.
  • FPGA circuit programmable logic circuit
  • the invention greatly improves the processing speed of image semantic segmentation by adopting a hardware circuit.
  • FIG. 1 shows an architectural diagram of an image semantic segmentation system.
  • the image semantic segmentation system includes: a programmable logic circuit, and a central processing unit (ARM CPU) and a memory (DDR) connected to the programmable logic circuit.
  • the programmable logic circuit mainly includes a convolution processing module and a deconvolution processing module, and the convolution processing module and the deconvolution processing module are electrically connected between the central processing unit and the memory, of course, the central processing.
  • the memory and the memory are also electrically connected, thereby implementing a call to the memory.
  • the convolution processing module and the deconvolution processing module implement a final connection with the central processing unit through a bus bridge (AXI4-Lite Bridge) and an interconnection (Interconnect 0); a convolution processing module
  • the deconvolution processing module is first electrically connected to the respective direct memory access modules (DMA 0 and DMA 1) and then interconnected (Interconnect 1) to achieve the final connection with the memory.
  • a shared cache module is electrically connected between the convolution processing module and the deconvolution processing module, thereby facilitating the simultaneous transmission and reception of data by the two.
  • the electrical connection between the convolution processing module and the deconvolution processing module and the central processing unit and the memory is not limited to the embodiment, and those skilled in the art can fully The actual application scenario is selected or transformed.
  • the circuit structure of the convolution processing module includes, for example, a parameter buffer, an input buffer, a convolution operation circuit, and an output buffer, wherein the parameter buffer and the output buffer are respectively connected to the convolution operation circuit.
  • the convolution processing module is electrically connected to an external memory that stores data to be processed and weight parameters.
  • the parameter buffer is configured to receive and output a weight parameter;
  • the input buffer includes: a plurality of connected line buffers for receiving and outputting data to be processed; wherein each of the line buffers outputs one bit of data Collecting a column of data output;
  • the convolution operation circuit is configured to receive the to-be-processed data from the input buffer, receive a weight parameter from the parameter buffer, perform a convolution operation, and output a convolution operation result;
  • the output buffer is configured to receive the convolution operation result and output the convolution operation result to the external memory.
  • the convolution operation circuit includes a plurality of convolution kernels and an adder tree running in parallel.
  • Each of said convolution kernels includes a multiplier for performing a convolution operation; an adder tree is used to accumulate output results of a plurality of said multipliers; each of said convolver inputs is in the form of a K ⁇ K matrix
  • the pixel data is outputted pixel by bit by a convolution operation according to the input pixel data and the weight parameter.
  • the convolver further includes a pooling operation circuit connected between the output buffer and the external memory for pooling the convolution operation result and outputting to the external memory.
  • the convolution processing module of this embodiment After receiving the image that needs to be semantically segmented, the convolution processing module of this embodiment performs convolution operation processing on the image to obtain feature data of each feature tile of the image.
  • the convolution processing module traverses the image by a preset convolution kernel (such as a 3 ⁇ 3 filter matrix), so that for each pixel of the image, its neighborhood pixel and filter matrix can be calculated.
  • the product of the corresponding elements and then add these product values as the value of the pixel position.
  • the present invention refers to a collection of pixels and their neighboring pixels as feature tiles, and the calculated value of the pixel location is referred to as feature data.
  • the deconvolution processing module of the present embodiment performs deconvolution operation processing on each of the feature data to obtain a tile region of the same size as the feature tile of each of the feature data.
  • the region of the "all-in-one" tile in the convolution process is restored.
  • the circuit structure of the deconvolution processing module includes, for example, a parameter buffer, an input buffer, a deconvolution operation circuit, and an output buffer, wherein the parameter buffer and the output buffer are respectively connected to the deconvolution operation circuit.
  • the deconvolution processing module is electrically connected to an external memory that stores data to be processed and weight parameters.
  • the parameter buffer is configured to receive and output the weight parameter;
  • the input buffer includes: a plurality of connected line buffers for receiving and outputting the to-be-processed data; wherein each of the line buffers outputs One bit of data is aggregated to form a column of data output;
  • the deconvolution operation circuit is configured to receive the to-be-processed data from the input buffer, receive a weight parameter from the parameter buffer, and perform a deconvolution operation according to Outputting a result of the deconvolution operation;
  • the output buffer is configured to receive the result of the deconvolution operation and output the result of the deconvolution operation to the external memory.
  • the deconvolution operation circuit includes: a plurality of deconvolution cores running in parallel, an adder tree.
  • Each of said deconvolution cores includes a multiplier for performing a deconvolution operation; an adder tree is used to accumulate output results of a plurality of said multipliers; each of said deconvolution inputs is K ⁇ K
  • the pixel data in the form of a matrix outputs pixel data bit by bit according to the input pixel data and the weight parameter through a deconvolution operation.
  • the deconvolution device further includes a pooling operation circuit connected between the output buffer and the external memory for pooling the deconvolution operation result and outputting to the external memory.
  • the following describes in detail how the deconvolution processing module of the present embodiment determines which semantic category the restored tile region belongs to, and what color is filled for it.
  • each of the feature data and each of the preset semantic categories is respectively determined, so that each of the feature data is classified into a preset semantic category with the highest degree of similarity.
  • the predefined semantic categories are: housing, road, and vegetation.
  • Each category is set with a range of values.
  • Each feature data is compared with the range values, and the closer to which value range. , which category it is classified into.
  • each of the preset semantic categories is associated with a fill color corresponding to one-to-one correspondence, for example, the blue corresponding to the house type, the gray corresponding to the road type, and the green corresponding to the vegetation type.
  • the color of the tile region restored by the feature data should be determined, for example, the tile region restored by the feature data classified as the house class is filled with blue.
  • the tile area restored by the feature data classified as the road class is filled with gray, and the tile area restored by the feature data classified as the vegetation class is filled with green.
  • FIG. 2a shows a photograph of a bird's-eye view in a certain street scene, which includes various objects such as houses, roads, vegetation, etc.
  • FIG. 2b is a semantic segmentation effect diagram realized by the image semantic segmentation system proposed by the present invention. It distinguishes different objects in a street image by different colors.
  • the convolution processing module performs multiple convolution processing on the image, that is, performs convolution operation processing on the basis of the previous convolution operation processing on the image, and sequentially cycles through The increase in the number of convolution layers reduces the amount of feature data.
  • the deconvolution processing module when restoring the tile area, the deconvolution processing module also needs to perform deconvolution operation processing on the basis of the previous deconvolution operation processing on the image until all the convolution operation processing maps are performed. The layers are dimensionally restored one by one.
  • the present invention also provides an image semantic segmentation method, which is performed by the programmable logic circuit of any of the foregoing embodiments. Since the technical features in the foregoing embodiments can also be applied to the embodiment of the present method, the description thereof will not be repeated.
  • the image semantic segmentation method of this embodiment mainly includes the following steps:
  • the convolution processing module receives an image that needs to be semantically segmented, and performs convolution operation processing on the image to generate feature data of each feature tile of the image.
  • the deconvolution processing module performs deconvolution operation processing on each of the feature data to respectively obtain a tile region of the same size as the feature tile of each feature data; and determine each of the feature data and each Predetermining the degree of approximation of the semantic categories, and classifying each of the feature data into a preset semantic category with the highest degree of similarity; wherein each of the preset semantic category associations has a fill color corresponding to the one-to-one correspondence.
  • the color associated with each of the corresponding tile regions is color-filled with the color associated with the preset semantic category in which the feature data is located to implement semantic segmentation of the image.
  • the convolution processing module and the deconvolution processing module transmit or receive data in a time-multiplexed manner through the shared cache module.
  • the convolution processing module performs multiple convolution processing on the image, that is, performs convolution operation processing on the basis of the previous convolution operation processing on the image, and sequentially cycles through
  • the increase in the number of convolution layers reduces the amount of feature data.
  • the deconvolution processing module also needs to perform deconvolution operation processing on the basis of the previous deconvolution operation processing on the image until all the convolution operation processing maps are performed. The layers are dimensionally restored one by one.
  • the image semantic segmentation method, the programmable logic circuit, the system and the electronic device of the invention realize the image semantic segmentation processing through the hardware circuit, improve the image processing speed, and effectively overcome various shortcomings in the prior art. With a high degree of industrial use.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Software Systems (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biophysics (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Image Analysis (AREA)

Abstract

一种图像语义分割方法、可编程逻辑电路、系统及电子设备。可编程逻辑电路包括卷积处理模块和反卷积处理模块。卷积处理模块接收图像后对其进行卷积运算处理，以生成图像的各个特征图块的特征数据；反卷积处理模块对各特征数据进行反卷积运算处理，以分别得到与各特征数据的特征图块同尺寸的图块区域；判断各特征数据与各预设语义类别的近似程度，并将各特征数据归为与其近似程度最高的预设语义类别中；其中，各预设语义类别关联有与其一一对应的各填充色彩；用各特征数据所在的预设语义类别所关联的色彩为对应的各图块区域进行色彩填充以实现对图像的语义分割。该方法通过硬件电路实现图像语义分割，极大程度上提高了数据处理速度。

Description

图像语义分割方法、可编程逻辑电路、系统及电子设备 技术领域
本发明涉及图像处理领域，特别是涉及图像语义分割方法、可编程逻辑电路、系统及电子设备。
背景技术
随着深度学习技术的发展，计算机对图像的语义分割能力逐步增强。利用基于深度学习的图像语义分割技术可以快速地对图像进行语义化分割，把图像中不同位置具有相同语义的像素点聚类成一个分类，并以相同的颜色进行标记。然而，现阶段的图像语义分割都是通过软件编程实现，处理速度有待提升。
发明内容
鉴于以上所述现有技术的缺点，本发明的目的在于提供图像语义分割方法、可编程逻辑电路、系统及电子设备，通过可编程逻辑电路来实现图像语义分割，从而提高图像语义分割的处理速度。
为实现上述目的及其他相关目的，本发明提供一种图像语义分割方法，所述方法应用于可编程逻辑电路；所述可编程逻辑电路包括：卷积处理模块、及反卷积处理模块；所述方法包括：所述卷积处理模块接收需要进行语义分割的图像，并对所述图像进行卷积运算处理，以生成所述图像的各个特征图块的特征数据；所述反卷积处理模块对各所述特征数据进行反卷积运算处理，以分别得到与各所述特征数据的特征图块同尺寸的图块区域；判断各所述特征数据与各预设语义类别的近似程度，并将各所述特征数据归为与其近似程度最高的预设语义类别中；其中，各所述预设语义类别关联有与其一一对应的各填充色彩；用各所述特征数据所在的预设语义类别所关联的色彩为对应的各所述图块区域进行色彩填充以实现对所述图像的语义分割。
于本发明一实施例中，所述可编程逻辑电路还包括：共享缓存模块，电性连接于所述卷积处理模块及所述反卷积处理模块；所述方法还包括：所述卷积处理模块及所述反卷积处理模块通过所述共享缓存模块分时复用地发送或接收数据。
于本发明一实施例中，所述方法还包括：所述卷积处理模块在前一次对所述图像进行卷积运算处理的基础上再进行卷积运算处理，从而通过卷积层数的增加减少所述特征数据的数量。
于本发明一实施例中，所述方法还包括：所述反卷积处理模块在前一次对所述图像进行反卷积运算处理的基础上再进行反卷积运算处理，直至所有经卷积运算处理的图层逐一得到尺寸还原。
为实现上述目的及其他相关目的，本发明提供一种可编程逻辑电路，包括：卷积处理模块、及反卷积处理模块；所述卷积处理模块，用于接收需要进行语义分割的图像，并对所述图像进行卷积运算处理，以生成所述图像的各个特征图块的特征数据；所述反卷积处理模块，用于对各所述特征数据进行反卷积运算处理，以分别得到与各所述特征数据的特征图块同尺寸的图块区域；判断各所述特征数据与各预设语义类别的近似程度，并将各所述特征数据归为与其近似程度最高的预设语义类别中；其中，各所述预设语义类别关联有与其一一对应的各填充色彩；用各所述特征数据所在的预设语义类别所关联的色彩为对应的各所述图块区域进行色彩填充以实现对所述图像的语义分割。
于本发明一实施例中，所述可编程逻辑电路还包括：共享缓存模块，电性连接于所述卷积处理模块及所述反卷积处理模块，以供所述卷积处理模块及所述反卷积处理模块分时复用地发送或接收数据。
于本发明一实施例中，所述卷积处理模块还用于：在前一次对所述图像进行卷积运算处理的基础上再进行卷积运算处理，从而通过卷积层数的增加减少所述特征数据的数量。
于本发明一实施例中，所述反卷积处理模块还用于：在前一次对所述图像进行反卷积运算处理的基础上再进行反卷积运算处理，直至所有经卷积运算处理的图层逐一得到尺寸还原。
为实现上述目的及其他相关目的，本发明提供一种图像语义分割系统，包括：如上任一所述的可编程逻辑电路、中央处理器、及与所述中央处理器电性连接的内存；所述卷积处理模块电性连接所述中央处理器及所述内存；所述反卷积处理模块电性连接所述中央处理器及所述内存；在所述可编程逻辑电路还包括所述共享缓存模块的情况下，所述共享缓存模块还电性连接所述内存。
为实现上述目的及其他相关目的，本发明提供一种电子设备，包括：如上任一所述的图像语义分割系统。
如上所述，本发明的图像语义分割方法、可编程逻辑电路、系统及电子设备，通硬件电路实现了图像语义分割处理，极大程度上提高了图像语义分割处理的速度。
附图说明
图1显示为本发明一实施例中的图像语义分割系统的架构示意图。
图2A和图2B分别显示为仿真原始图及其经本发明的图像语义分割系统处理后的效果图。
具体实施方式
以下通过特定的具体实例说明本发明的实施方式，本领域技术人员可由本说明书所揭露的内容轻易地了解本发明的其他优点与功效。本发明还可以通过另外不同的具体实施方式加以实施或应用，本说明书中的各项细节也可以基于不同观点与应用，在没有背离本发明的精神下进行各种修饰或改变。需说明的是，在不冲突的情况下，以下实施例及实施例中的特征可以相互组合。
需要说明的是，以下实施例中所提供的图示仅以示意方式说明本发明的基本构想，遂图式中仅显示与本发明中有关的组件而非按照实际实施时的组件数目、形状及尺寸绘制，其实际实施时各组件的型态、数量及比例可为一种随意的改变，且其组件布局型态也可能更为复杂。
本发明提供能够实现图像语义分割任务的可编程逻辑电路(FPGA电路)、包含该可编程逻辑电路的图像语义分割系统，以及包含该图像语义分割系统的电子设备，除此之外，还提供应用于该可编程逻辑电路的图像语义分割方法。相比于现有通过软件实现图像语义分割的方式，本发明通过采用硬件电路大大提高了图像语义分割的处理速度。
图1显示了一种图像语义分割系统的架构示意图。在该图像语义分割系统中包括：可编程逻辑电路，以及连接于该可编程逻辑电路之外的中央处理器(ARM CPU)、内存(DDR)。详细的，该可编程逻辑电路主要包括卷积处理模块及反卷积处理模块，无论是卷积处理模块还是反卷积处理模块都会电性连接于中央处理器及内存之间，当然，中央处理器与内存也是电性连接的，从而实现对内存的调用。
在图1所示的实施例中，卷积处理模块和反卷积处理模块通过总线桥接(AXI4-Lite Bridge)和互连(Interconnect 0)来实现与中央处理器的最终连接；卷积处理模块和反卷积处理模块先电性连接各自的直接内存存取模块(DMA 0和DMA 1)再通过互连(Interconnect 1)来实现与内存的最终连接。在另一种实施例中，卷积处理模块和反卷积处理模块之间还电性连接有共享缓存模块，从而有利于二者分时复用地发送或接受数据。
需要说明的是，本发明的图像语义分割系统，其卷积处理模块和反卷积处理模块与中央处理器和内存的电性连接并不受限于本实施例，本领域技术人员完全可以根据实际应用场景进行选择或变换。
所述卷积处理模块的电路结构例如包括：参数缓存器、输入缓存器、卷积运算电路及输出缓存器，其中，参数缓存器、输出缓存器分别连接卷积运算电路。所述卷积处理模块电性连接至外部存储器，所述外部存储器存储有待处理数据及权重参数。
所述参数缓存器用于接收并输出权重参数；所述输入缓存器包括：多个相连的行缓存器，用于接收并输出待处理数据；其中，各所述行缓存器每输出一位数据则集合形成一列数据输出；所述卷积运算电路用于从所述输入缓存器接收所述待处理数据、从所述参数缓存器接收权重参数，据以进行卷积运算并输出卷积运算结果；所述输出缓存器用于接收所述卷积运算结果并将该卷积运算结果向所述外部存储器输出。
所述卷积运算电路包括：多个并行运行的卷积核、加法器树。每一个所述卷积核包含用于进行卷积运算的乘法器；加法器树用于对多个所述乘法器的输出结果进行累加；每一个所述卷积器输入K×K矩阵形式的像素数据，根据输入的像素数据和所述权重参数经过卷积运算逐位输出像素数据。所述卷积器还包括池化运算电路，连接于所述输出缓存器和所述外部存储器之间，用于对所述卷积运算结果进行池化后向外部存储器输出。
本实施例的卷积处理模块在接收到需要进行语义分割的图像后，对所述图像进行卷积运算处理，从而得到所述图像的各个特征图块的特征数据。详细而言，卷积处理模块通过预设卷积核(如3×3滤波器矩阵)遍历所述图像，从而对于图像的每一个像素点，都能计算出它的邻域像素和滤波器矩阵的对应元素的乘积，然后将这些乘积值相加，作为该像素位置的值。为了方便说明，本发明将像素点及其邻域像素的集合称为特征图块，且将计算得到的该像素位置的值称为特征数据。
本实施例的反卷积处理模块对各所述特征数据进行反卷积运算处理，以分别得到与各所述特征数据的特征图块同尺寸的图块区域。承接前述，也即是将卷积过程中经“多合一”的图块区域再还原回去。对于还原后的图块区域，我们需要依照其语义类别为其填充色彩，从而让人们能够直观地辨识其语义。
所述反卷积处理模块的电路结构例如包括：参数缓存器、输入缓存器、反卷积运算电路及输出缓存器，其中，参数缓存器、输出缓存器分别连接反卷积运算电路。所述反卷积处理模块电性连接至外部存储器，所述外部存储器存储有待处理数据及权重参数。
所述参数缓存器用于接收并输出所述权重参数；所述输入缓存器包括：多个相连的行缓存器，用于接收并输出所述待处理数据；其中，各所述行缓存器每输出一位数据则集合形成一列数据输出；所述反卷积运算电路用于从所述输入缓存器接收所述待处理数据、从所述参数缓存器接收权重参数，据以进行反卷积运算并输出反卷积运算结果；所述输出缓存器用于 接收所述反卷积运算结果并将该反卷积运算结果向所述外部存储器输出。
所述反卷积运算电路包括：多个并行运行的反卷积核、加法器树。每一个所述反卷积核包含用于进行反卷积运算的乘法器；加法器树用于对多个所述乘法器的输出结果进行累加；每一个所述反卷积器输入K×K矩阵形式的像素数据，根据输入的像素数据和所述权重参数经过反卷积运算逐位输出像素数据。所述反卷积器还包括池化运算电路，连接于所述输出缓存器和所述外部存储器之间，用于对所述反卷积运算结果进行池化后向外部存储器输出。
以下将详细说明本实施例的反卷积处理模块如何判断还原后的图块区域属于何种语义类别，以及为其填充何种色彩。
首先，分别判断各所述特征数据与各预设语义类别的近似程度，从而将各所述特征数据归为与其近似程度最高的预设语义类别中。举例来说，预先定义的语义类别有：房屋类、道路类、植被类，每个类别设置有一个数值范围，将每个特征数据分别与这些范围值进行比对，其与哪个数值范围越接近，就将其归入哪个类别其中。其次，每个预设语义类别都关联有与其一一对应的填充色彩，例如：房屋类对应蓝色、道路类对应灰色、植被类对应绿色等。当特征数据的语义类别确定后，以该特征数据所还原出的图块区域应当填充何种色彩也就被确定，例如：向归为房屋类的特征数据所还原出的图块区域填充蓝色，向归为道路类的特征数据所还原出的图块区域填充灰色，向归为植被类的特征数据所还原出的图块区域填充绿色。
图2a显示了某个街道场景下的俯视视角照片，该照片中包括了房屋、道路、植被等多种对象，图2b则是通过本发明提出的图像语义分割系统而实现的语义分割效果图，其通过不同的色彩将街道图片中的不同对象区分开来。
由上述实施例可见，一张图像在经过卷积核遍历后会产生大量的特征数据，从而提高了语义类别判断的工作量。在另一实施例中，所述卷积处理模块会对图像进行多次卷积处理，也即在前一次对图像进行卷积运算处理的基础上再进行卷积运算处理，依次循环，从而通过卷积层数的增加来减少特征数据的数量。对应的，在还原图块区域时，所述反卷积处理模块也需在前一次对图像进行反卷积运算处理的基础上再进行反卷积运算处理，直至所有经卷积运算处理的图层逐一得到尺寸还原。
需要说明的是，凡是包含前述任一实施例的图像语义分割系统的电子设备，也应在本发明所要保护的范围之内，由于篇幅所限，在此不一一列举。
与上述系统实施例相对应的，本发明还提供图像语义分割方法，该方法由前述任一实施例中的可编程逻辑电路执行。由于前述实施例中的技术特征也能应用于本方法实施例，因而不再重复赘述。本实施例的图像语义分割方法主要包括如下步骤：
首先，所述卷积处理模块接收需要进行语义分割的图像，并对所述图像进行卷积运算处理，以生成所述图像的各个特征图块的特征数据。
其次，所述反卷积处理模块对各所述特征数据进行反卷积运算处理，以分别得到与各所述特征数据的特征图块同尺寸的图块区域；判断各所述特征数据与各预设语义类别的近似程度，并将各所述特征数据归为与其近似程度最高的预设语义类别中；其中，各所述预设语义类别关联有与其一一对应的各填充色彩。
再次，用各所述特征数据所在的预设语义类别所关联的色彩为对应的各所述图块区域进行色彩填充以实现对所述图像的语义分割。
对于包括共享缓存模块的可编程逻辑电路，所述卷积处理模块及所述反卷积处理模块通过所述共享缓存模块分时复用地发送或接收数据。
在另一实施例中，所述卷积处理模块会对图像进行多次卷积处理，也即在前一次对图像进行卷积运算处理的基础上再进行卷积运算处理，依次循环，从而通过卷积层数的增加来减少特征数据的数量。对应的，在还原图块区域时，所述反卷积处理模块也需在前一次对图像进行反卷积运算处理的基础上再进行反卷积运算处理，直至所有经卷积运算处理的图层逐一得到尺寸还原。
需要说明的是，本发明的保护范围不限于本实施例列举的图像语义分割方法的步骤执行顺序，凡是根据实际需要进行灵活地顺序调整、步骤更替、删减、增加等做的变形，都包括在本发明的保护范围内。
综上所述，本发明的图像语义分割方法、可编程逻辑电路、系统及电子设备，通过硬件电路实现了图像语义分割处理，提高了图像处理速度，有效克服了现有技术中的种种缺点而具高度产业利用价值。
上述实施例仅例示性说明本发明的原理及其功效，而非用于限制本发明。任何熟悉此技术的人士皆可在不违背本发明的精神及范畴下，对上述实施例进行修饰或改变。因此，举凡所属技术领域中具有通常知识者在未脱离本发明所揭示的精神与技术思想下所完成的一切等效修饰或改变，仍应由本发明的权利要求所涵盖。

Claims (10)

  1. 一种图像语义分割方法，其特征在于，所述方法应用于可编程逻辑电路；所述可编程逻辑电路包括：卷积处理模块、及与其电性连接的反卷积处理模块；所述方法包括：
    所述卷积处理模块接收需要进行语义分割的图像，并对所述图像进行卷积运算处理，以生成所述图像的各个特征图块的特征数据；
    所述反卷积处理模块对各所述特征数据进行反卷积运算处理，以分别得到与各所述特征数据的特征图块同尺寸的图块区域；判断各所述特征数据与各预设语义类别的近似程度，并将各所述特征数据归为与其近似程度最高的预设语义类别中；其中，各所述预设语义类别关联有与其一一对应的各填充色彩；用各所述特征数据所在的预设语义类别所关联的色彩为对应的各所述图块区域进行色彩填充以实现对所述图像的语义分割。
  2. 根据权利要求1所述的方法，其特征在于，所述可编程逻辑电路还包括：共享缓存模块，电性连接于所述卷积处理模块及所述反卷积处理模块；所述方法还包括：所述卷积处理模块及所述反卷积处理模块通过所述共享缓存模块分时复用地发送或接收数据。
  3. 根据权利要求1所述的方法，其特征在于，还包括：所述卷积处理模块在前一次对所述图像进行卷积运算处理的基础上再进行卷积运算处理，从而通过卷积层数的增加减少所述特征数据的数量。
  4. 根据权利要求3所述的方法，其特征在于，还包括：所述反卷积处理模块在前一次对所述图像进行反卷积运算处理的基础上再进行反卷积运算处理，直至所有经卷积运算处理的图层逐一得到尺寸还原。
  5. 一种可编程逻辑电路，其特征在于，包括：卷积处理模块、及与其电性连接的反卷积处理模块；其中，
    所述卷积处理模块，用于接收需要进行语义分割的图像，并对所述图像进行卷积运算处理，以生成所述图像的各个特征图块的特征数据；
    所述反卷积处理模块，用于对各所述特征数据进行反卷积运算处理，以分别得到与各所述特征数据的特征图块同尺寸的图块区域；判断各所述特征数据与各预设语义类别的近似程度，并将各所述特征数据归为与其近似程度最高的预设语义类别中；其中，各所述预设语义类别关联有与其一一对应的各填充色彩；用各所述特征数据所在的预设语义类别所关联的色彩为对应的各所述图块区域进行色彩填充以实现对所述图像的语义分割。
  6. 根据权利要求5所述的可编程逻辑电路，其特征在于，还包括：共享缓存模块，电性连接于所述卷积处理模块及所述反卷积处理模块，以供所述卷积处理模块及所述反卷积处理模 块分时复用地发送或接收数据。
  7. 根据权利要求5所述的可编程逻辑电路，其特征在于，所述卷积处理模块还用于：在前一次对所述图像进行卷积运算处理的基础上再进行卷积运算处理，从而通过卷积层数的增加减少所述特征数据的数量。
  8. 根据权利要求7所述的可编程逻辑电路，其特征在于，所述反卷积处理模块还用于：在前一次对所述图像进行反卷积运算处理的基础上再进行反卷积运算处理，直至所有经卷积运算处理的图层逐一得到尺寸还原。
  9. 一种图像语义分割系统，其特征在于，包括：中央处理器、及与其电性连接的内存，以及如权利要求5至8中任一所述的可编程逻辑电路；其中，所述卷积处理模块电性连接所述中央处理器及所述内存；所述反卷积处理模块电性连接所述中央处理器及所述内存；在所述可编程逻辑电路还包括所述共享缓存模块的情况下，所述共享缓存模块还电性连接所述内存。
  10. 一种电子设备，其特征在于，包括：如权利要求9所述的图像语义分割系统。
PCT/CN2018/072674 2018-01-15 2018-01-15 图像语义分割方法、可编程逻辑电路、系统及电子设备 WO2019136760A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201880002143.2A CN109564684B (zh) 2018-01-15 2018-01-15 图像语义分割方法、可编程逻辑电路、系统及电子设备
PCT/CN2018/072674 WO2019136760A1 (zh) 2018-01-15 2018-01-15 图像语义分割方法、可编程逻辑电路、系统及电子设备
US16/962,444 US11636665B2 (en) 2018-01-15 2018-01-15 Streaming image semantic segmentation method, logical integrated circuit system and electronic device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/072674 WO2019136760A1 (zh) 2018-01-15 2018-01-15 图像语义分割方法、可编程逻辑电路、系统及电子设备

Publications (1)

Publication Number Publication Date
WO2019136760A1 true WO2019136760A1 (zh) 2019-07-18

Family

ID=65872622

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/072674 WO2019136760A1 (zh) 2018-01-15 2018-01-15 图像语义分割方法、可编程逻辑电路、系统及电子设备

Country Status (3)

Country Link
US (1) US11636665B2 (zh)
CN (1) CN109564684B (zh)
WO (1) WO2019136760A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110689475A (zh) * 2019-09-10 2020-01-14 浪潮电子信息产业股份有限公司 一种图像数据处理方法、系统、电子设备及存储介质
CN111782529A (zh) * 2020-06-30 2020-10-16 平安国际智慧城市科技股份有限公司 辅助诊断系统的测试方法、装置、计算机设备及存储介质
CN112288748A (zh) * 2019-07-25 2021-01-29 银河水滴科技（北京）有限公司 一种语义分割网络训练、图像语义分割方法及装置

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110910400A (zh) * 2019-10-29 2020-03-24 北京三快在线科技有限公司 图像处理方法、装置、存储介质及电子设备
CN113435412B (zh) * 2021-07-26 2022-09-20 张晓寒 一种基于语义分割的水泥布料区域检测方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7298877B1 (en) * 2001-11-20 2007-11-20 Icad, Inc. Information fusion with Bayes networks in computer-aided detection systems
CN104504734A (zh) * 2014-09-16 2015-04-08 浙江工业大学 一种基于语义的图像颜色传输方法
CN106709568A (zh) * 2016-12-16 2017-05-24 北京工业大学 基于深层卷积网络的rgb‑d图像的物体检测和语义分割方法
CN106709924A (zh) * 2016-11-18 2017-05-24 中国人民解放军信息工程大学 基于深度卷积神经网络和超像素的图像语义分割方法
CN107016415A (zh) * 2017-04-12 2017-08-04 合肥工业大学 一种基于全卷积网络的彩色图像色彩语义分类方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5905757A (en) * 1996-10-04 1999-05-18 Motorola, Inc. Filter co-processor
JP5184824B2 (ja) * 2007-06-15 2013-04-17 キヤノン株式会社 演算処理装置及び方法
WO2016197303A1 (en) * 2015-06-08 2016-12-15 Microsoft Technology Licensing, Llc. Image semantic segmentation
US11074492B2 (en) * 2015-10-07 2021-07-27 Altera Corporation Method and apparatus for performing different types of convolution operations with the same processing elements
US10402689B1 (en) * 2017-04-04 2019-09-03 Snap Inc. Generating an image mask using machine learning
US10410353B2 (en) * 2017-05-18 2019-09-10 Mitsubishi Electric Research Laboratories, Inc. Multi-label semantic boundary detection system
CN107369204B (zh) * 2017-07-27 2020-01-07 北京航空航天大学 一种从单幅照片恢复出场景基本三维结构的方法
US10410350B2 (en) * 2017-10-30 2019-09-10 Rakuten, Inc. Skip architecture neural network machine and method for improved semantic segmentation
WO2019127507A1 (zh) * 2017-12-29 2019-07-04 深圳市大疆创新科技有限公司 数据处理方法、设备、dma控制器及计算机可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7298877B1 (en) * 2001-11-20 2007-11-20 Icad, Inc. Information fusion with Bayes networks in computer-aided detection systems
CN104504734A (zh) * 2014-09-16 2015-04-08 浙江工业大学 一种基于语义的图像颜色传输方法
CN106709924A (zh) * 2016-11-18 2017-05-24 中国人民解放军信息工程大学 基于深度卷积神经网络和超像素的图像语义分割方法
CN106709568A (zh) * 2016-12-16 2017-05-24 北京工业大学 基于深层卷积网络的rgb‑d图像的物体检测和语义分割方法
CN107016415A (zh) * 2017-04-12 2017-08-04 合肥工业大学 一种基于全卷积网络的彩色图像色彩语义分类方法

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112288748A (zh) * 2019-07-25 2021-01-29 银河水滴科技（北京）有限公司 一种语义分割网络训练、图像语义分割方法及装置
CN112288748B (zh) * 2019-07-25 2024-03-01 银河水滴科技（北京）有限公司 一种语义分割网络训练、图像语义分割方法及装置
CN110689475A (zh) * 2019-09-10 2020-01-14 浪潮电子信息产业股份有限公司 一种图像数据处理方法、系统、电子设备及存储介质
CN111782529A (zh) * 2020-06-30 2020-10-16 平安国际智慧城市科技股份有限公司 辅助诊断系统的测试方法、装置、计算机设备及存储介质
CN111782529B (zh) * 2020-06-30 2023-08-01 深圳平安智慧医健科技有限公司 辅助诊断系统的测试方法、装置、计算机设备及存储介质

Also Published As

Publication number Publication date
CN109564684B (zh) 2023-05-02
CN109564684A (zh) 2019-04-02
US20210082119A1 (en) 2021-03-18
US11636665B2 (en) 2023-04-25

Similar Documents

Publication Publication Date Title
WO2019136760A1 (zh) 图像语义分割方法、可编程逻辑电路、系统及电子设备
US10445638B1 (en) Restructuring a multi-dimensional array
US11157592B2 (en) Hardware implementation of convolutional layer of deep neural network
US20230325348A1 (en) Performing concurrent operations in a processing element
WO2018036293A1 (zh) 图像分割方法、装置及全卷积网络系统
WO2021164234A1 (zh) 图像处理方法以及图像处理装置
JP4168125B2 (ja) データ処理システムおよび方法
US11157764B2 (en) Semantic image segmentation using gated dense pyramid blocks
WO2019201656A1 (en) Method for accelerating operations and accelerator apparatus
WO2020199476A1 (zh) 基于脉动阵列的神经网络加速方法、装置、计算机设备及存储介质
US10521700B2 (en) Methods and systems for converting a line drawing to a rendered image
JP6684951B2 (ja) 人工知能推論演算装置
KR20180125843A (ko) (국문)다양한 cnn 모델에 적용 가능한 하드웨어 분류기
WO2019216376A1 (ja) 演算処理装置
TWI778895B (zh) 顯著性檢測方法及其模型的訓練方法、設備及電腦可讀儲存介質
CN112799599B (zh) 一种数据存储方法、计算核、芯片和电子设备
Chien et al. Distributed computing in IoT: System-on-a-chip for smart cameras as an example
CN114494087A (zh) 无监督的多尺度视差/光流融合
CN110717864B (zh) 一种图像增强方法、装置、终端设备及计算机可读介质
US11704894B2 (en) Semantic image segmentation using gated dense pyramid blocks
WO2023115408A1 (zh) 图像处理装置和方法
CN112990305B (zh) 一种遮挡关系的确定方法、装置、设备及存储介质
US20200218960A1 (en) Convolution operator system to perform concurrent convolution operations
Mefenza et al. Interface based memory synthesis of image processing applications in fpga
Khalil et al. A hardware design and implementation for accelerating motion detection using (System On Chip) SOC

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18900449

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 16/11/2020)

122 Ep: pct application non-entry in european phase

Ref document number: 18900449

Country of ref document: EP

Kind code of ref document: A1