WO2019041265A1 - 特征提取电路和图像处理集成电路 - Google Patents

特征提取电路和图像处理集成电路 Download PDF

Info

Publication number
WO2019041265A1
WO2019041265A1 PCT/CN2017/100033 CN2017100033W WO2019041265A1 WO 2019041265 A1 WO2019041265 A1 WO 2019041265A1 CN 2017100033 W CN2017100033 W CN 2017100033W WO 2019041265 A1 WO2019041265 A1 WO 2019041265A1
Authority
WO
WIPO (PCT)
Prior art keywords
circuit
sub
feature
point
feature point
Prior art date
Application number
PCT/CN2017/100033
Other languages
English (en)
French (fr)
Inventor
高明明
颜钊
杨康
李涛
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to CN201780004680.6A priority Critical patent/CN108496186A/zh
Priority to PCT/CN2017/100033 priority patent/WO2019041265A1/zh
Publication of WO2019041265A1 publication Critical patent/WO2019041265A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components

Definitions

  • the present application relates to the field of image processing, and more particularly to feature extraction circuits and image processing integrated circuits.
  • the ORB (ORiented BRIEF) algorithm is an algorithm described based on Features from Accelerated Segment Test and BRIEF (Binary Robust Independent Elementary Features).
  • the ORB algorithm is widely used in the field of image processing, especially for positioning and navigation in the field of machine vision.
  • the steps of the FAST feature point detection of the ORB algorithm are all based on hardware implementation or all based on software implementation.
  • some steps lose flexibility, the portability is poor, and the calculation effect is not good.
  • feature points selected based on hardware-implemented feature point selection steps may be too concentrated and cannot be adjusted.
  • the calculation pressure of the system is large, and the real-time calculation is poor.
  • the present application provides a feature extraction circuit and an image processing integrated circuit, which can reduce the computational pressure of the system, improve the real-time performance of the calculation, and improve the flexibility, portability and calculation effect of feature point extraction.
  • a feature extraction circuit comprising a pyramid sub-circuit connected by a pipeline form, a feature point detection sub-unit and a feature point selection sub-unit, wherein the pyramid sub-circuit is used to establish at least an image to be processed a layer of pyramid; the feature point detection subunit is used for At least one candidate point is detected in the pyramid; the feature point selection subunit is implemented by a processor, and the feature point is selected from the candidate point according to the candidate point and a preset rule.
  • the feature extraction circuit of the first aspect is constructed by a hardware circuit, and the feature points are selected by the processor through software, which can reduce the computational pressure of the system, improve the real-time performance of the calculation, and improve the flexibility and portability of feature point extraction. Sex and calculation effects.
  • a descriptor calculation circuit comprising a second memory controller and a point pair extraction sub-circuit and a descriptor generation sub-circuit connected by a pipeline form, the second memory controller being used for
  • the coordinate information of the feature point is read in the memory;
  • the point pair extraction sub-circuit is configured to determine, according to the coordinate information of the feature point read from the memory, the corresponding point region corresponding to the feature point in the image to be processed a pair of points;
  • the descriptor generation sub-circuit is configured to determine a descriptor of the feature point based on the pixel values of the plurality of point pairs.
  • the second sub-calculation circuit calculates the descriptor of the feature point by the hardware circuit, which can reduce the calculation pressure of the system and improve the real-time and portability of the calculation.
  • an image processing integrated circuit comprising the feature extraction circuit of the first aspect.
  • an image processing integrated circuit comprising the sub-computing circuit of the second aspect.
  • an image processing integrated circuit comprising the feature extraction circuit of the first aspect and the sub-computation circuit of the second aspect.
  • a computing chip comprising the image processing integrated circuit of any of the third to fifth aspects.
  • a computing device comprising the image processing integrated circuit of any of the third to fifth aspects.
  • FIG. 1 is a schematic block diagram of a feature extraction circuit of an embodiment of the present application.
  • FIG. 2 is a schematic diagram of candidate point detection in one embodiment of the present application.
  • FIG. 3 is a schematic block diagram of a feature extraction circuit of an embodiment of the present application.
  • FIG. 4 is a schematic block diagram of a feature extraction circuit of another embodiment of the present application.
  • FIG. 5 is a schematic block diagram of a feature extraction circuit according to still another embodiment of the present application.
  • Figure 6 is a schematic block diagram depicting a sub-computing circuit of one embodiment of the present application.
  • FIG. 7 is a schematic diagram of point pair extraction in accordance with an embodiment of the present application.
  • FIG. 8 is a schematic block diagram depicting a sub-computing circuit of another embodiment of the present application.
  • the embodiment of the present application provides a solution in which hardware and software are combined to implement feature extraction.
  • the software can be implemented by using an embedded computer, a personal computer (PC), and a processor, for example, a central processing unit (CPU) or a graphics processing unit (GPU); This can be done with the High-Level Synthesis (HLS) or similar tools of the Vivado Design Suite.
  • PC personal computer
  • processor for example, a central processing unit (CPU) or a graphics processing unit (GPU); This can be done with the High-Level Synthesis (HLS) or similar tools of the Vivado Design Suite.
  • HLS High-Level Synthesis
  • the embodiment of the present application provides a feature extraction circuit.
  • 1 is a schematic block diagram of a feature extraction circuit 100 of one embodiment of the present application.
  • the feature extraction circuit 100 may include a pyramid sub-circuit 110, a feature point detection sub-unit 120, and a feature point selection sub-unit 130 connected by a pipeline.
  • the pyramid sub-circuit 110 is configured to establish at least one layer of pyramids for the image 10 to be processed;
  • the feature point detection sub-unit 120 is configured to detect at least one candidate point in each layer of pyramids;
  • the feature point selection sub-unit 130 is implemented by the processor for Candidate points and preset rules, feature points are selected from candidate points.
  • the feature extraction circuit of the embodiment of the present application is implemented by a hardware circuit, and the feature points are selected by the processor through software, which can reduce the calculation pressure of the system, improve the real-time performance of the calculation, and improve the flexibility of feature point extraction. Portability and computational effects.
  • the feature extraction circuit of the embodiment of the present application can be implemented based on a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC).
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • the pyramid sub-circuit 110 can be used to solve the problem that the feature points extracted by the algorithm do not satisfy the dimensional continuity. In other words, the pyramid sub-circuit 110 establishes a pyramid to extract the feature points in the images at different scales to achieve the effect of the scale change.
  • Pyramid sub-circuit 110 can create a simple downsampled pyramid, or resize to create images of various sizes.
  • the pyramid created by the pyramid sub-circuit 110 may be one layer or multiple layers. The number of layers in the pyramid can be pre-configured or changed by the processor.
  • the pyramid sub-circuit 110 may use a Gaussian filtering method or a Gaussian filtering method to establish a pyramid. The embodiment of the present application does not limit the use of the above case.
  • the related data of the pyramid established by the pyramid sub-circuit 110 may be written into the off-chip memory (or cache) through a memory controller (for example, the first memory controller which will be mentioned later). Subsequent calculations are used, for example, for grayscale centroids for subsequent computational images (which may be based on images of any layer of pyramids).
  • the image 10 to be processed may be stored in an off-chip memory that is read from memory into the pyramid sub-circuit 110 by, for example, a first memory controller.
  • the image 10 to be processed may be derived from a circuit or unit of the preceding stage.
  • the source of the image to be processed in the embodiment of the present application is not limited, and the image from which the pyramid sub-circuit 110 reads the image to be processed can be controlled by the configuration signal.
  • the feature point detection sub-unit 120 is configured to detect at least one candidate point in each layer of the pyramid.
  • 2 is a schematic diagram of candidate point detection in one embodiment of the present application.
  • the feature point detection sub-unit 120 performs candidate point detection based on the following principle. As shown in FIG. 2, one pixel P is selected from an image (which may be an image based on any layer of pyramids), and a gray value of I p is assumed, and a suitable threshold t is set. When the absolute value of the difference between the gray values of the two points is greater than t, the two points are considered to be different. Consider 16 pixel points around the pixel point P.
  • n can be a positive integer
  • the value of n can be configured according to the configuration signal.
  • the feature point detecting subunit 120 may detect candidate points from each layer of pyramids by using at least one of FAST5-8, FAST7-12, FAST9-16, and Harris corner detection.
  • embodiments of the present application are not limited to these detection methods.
  • the feature point detection sub-unit 120 may be implemented based on software, for example, by a processor; the feature point detection sub-unit 120 may also be implemented based on hardware, for example, the feature point detection sub-unit 120 is a feature point detection sub-circuit.
  • the related data of the candidate points detected by the feature point detection sub-unit 120 may pass through a memory controller (for example, the first memory that will be mentioned later) The controller) is written to the off-chip memory for subsequent calculations.
  • a memory controller for example, the first memory that will be mentioned later
  • Feature point selection sub-unit 130 may be based on software, such as implemented by a processor.
  • the feature point selection sub-unit 130 may select a feature point from the candidate points according to the candidate point and the preset rule.
  • Feature points The selection sub-unit 130 (processor) may also adjust the preset rule based on the position distribution of the candidate points and/or based on the system load, and select feature points from the candidate points.
  • the processor may select a point with a more distributed distribution from the candidate points as a feature point according to the position distribution of the candidate point; the processor may also perform a tradeoff according to the current system load, and select an appropriate number of appropriate positions as the feature point. .
  • the feature point selection sub-unit 130 may adopt different preset rules when processing different frames of the image. That is, the processor can dynamically adjust to select a suitable number of appropriate locations as feature points according to the criticality of the frame or the conditions of the pyramid layers.
  • the feature point selection sub-unit 130 can also perform the selection of the feature points by using other factors, which is not limited in this embodiment of the present application.
  • Feature point selection is based on software for a more flexible configuration.
  • the feature point selection sub-unit 130 may write the coordinate information of the selected feature points into the off-chip memory (or cache) for subsequent calculation, for example, for subsequent calculation of the feature points. Describe the use of the child.
  • the feature point selection sub-unit 130 is further configured to write coordinate information of the selected feature points into the memory in a preset order. This ensures that subsequent hardware reads in a preset order when reading data, which can reduce the bandwidth of random access memory and improve system performance.
  • the feature extraction circuit 100 may further include a response calculation sub-unit 140 for calculating a response result of each candidate point, and the feature point selection sub-unit 130 is configured to: according to the candidate point and the preset rule, The feature points are selected from the candidate points, and the feature point selection sub-unit 130 is configured to select the feature points from the candidate points according to the response result of the candidate points calculated by the response calculation sub-unit 140 and the preset rule.
  • the feature point selection sub-unit 130 may include a response calculation sub-unit, which is the same as or similar to the response calculation sub-unit 140, and is not limited in this embodiment.
  • the response calculation sub-unit 140 may calculate its response result, or response size, for each candidate point.
  • the response calculation sub-unit 140 may use a non-maximum suppression algorithm to remove portions of candidate points at denser locations of candidate points.
  • the response calculation sub-unit 140 may also be a response result obtained by other algorithms for the feature point selection sub-unit 130 to select a feature point, which is not limited in this embodiment of the present application.
  • the response calculation sub-unit 140 calculates the response result
  • the image to be processed or the original image
  • the response calculation sub-unit 140 performs a response maximum value search in the corresponding region of the image (ie, non-maximum suppression).
  • the embodiment of the present application does not limit this.
  • the response calculation sub-unit 140 can be implemented based on software, such as by a processor; the response calculation sub-unit 140 can also be implemented based on hardware, such as the response calculation sub-unit 140 calculating the sub-circuit in response.
  • the response result of the candidate point calculated by the response calculation sub-unit 140 may pass through a memory controller (for example, the first memory controller which will be mentioned later) ) is written to the off-chip memory for subsequent calculations.
  • a memory controller for example, the first memory controller which will be mentioned later
  • the feature extraction circuit of the embodiment of the present application is implemented by combining hardware and software, and data exchange between hardware and software can be implemented by using a memory.
  • the hardware can read data from the memory or directly receive data from the previous stage hardware; after the hardware processes the data, the related data is stored in the memory through the memory controller, and the software reads the relevant data from the memory for subsequent processing.
  • the feature extraction circuit 100 of the embodiment of the present application may further include a first memory controller 150.
  • the first memory controller 150 may be connected to other sub-circuits or sub-units in the feature extraction circuit 100 through a pipeline form, or may be connected by other forms and other sub-circuits or sub-units in the feature extraction circuit 100. This example does not limit this.
  • FIG. 3 is a schematic block diagram of a feature extraction circuit 100 of one embodiment of the present application.
  • the feature point detection sub-unit 120 and the response calculation sub-unit 140 are both implemented by a processor, and the first memory controller 150 is configured to write at least one layer of pyramid into the memory 200, and the feature point detection sub-unit 120 At least one candidate point is detected in each layer of the pyramid, including: the feature point detecting sub-unit 120 is configured to read at least one layer of the pyramid written in the memory 200 from the memory 200, and detect at least one in each layer of the pyramid read.
  • the memory 200 can be disposed in the feature extraction circuit 100, and can be disposed in the feature extraction circuit 100. This is not limited.
  • the pyramid sub-circuit 110 is implemented by hardware, and the feature point detection sub-unit 120, the response calculation sub-unit 140, and the feature point selection sub-unit 130 are all implemented by a processor.
  • the pyramid sub-circuit 110 can read the image to be processed from the off-chip memory 200 through the first memory controller 150 (not shown in FIG. 3); the pyramid sub-circuit 110 can also be processed by the circuit or unit of the previous stage. Image (not shown in Figure 3).
  • the pyramid sub-circuit 110 establishes at least one layer of pyramids for the image to be processed.
  • the pyramid sub-circuit 110 writes the associated data of the established pyramid into the memory 200 through the first memory controller 150.
  • the pyramid sub-circuit 110 can feed back the processed information to the processor, so that the processor reads the relevant data from the memory for subsequent processing.
  • the processor (feature point detecting sub-unit 120) reads at least one layer of the pyramid written in the memory 200 from the memory 200, and detects at least one candidate point in each of the read pyramids.
  • the processor (response calculation sub-unit 140) calculates the response result for each candidate point.
  • the processor feature point selection sub-unit 130) selects the feature points from the candidate points according to the response result of the candidate points calculated by the response calculation sub-unit 140 and the preset rule.
  • the processor may write the coordinate information of the selected feature points into the memory 200 in a preset order for subsequent calculation (not shown in FIG. 3), for example, for subsequent calculation of the feature points.
  • the descriptor is used.
  • the response calculation sub-unit 140 is an optional unit, and the processor (the feature point selection sub-unit 130) can directly select the feature points from the candidate points according to the candidate points and the preset rule, which is not limited in this embodiment of the present application. .
  • the feature point detection sub-unit 120 is based on hardware as the feature point detection sub-circuit 120, and the response calculation sub-unit 140 is implemented by a processor, and the first memory controller 150 is configured to write at least one candidate point into the memory 200.
  • the response calculation sub-unit 140 is configured to calculate a response result of each candidate point, including: the response calculation sub-unit 140 is configured to read at least one candidate point from the memory 200, and calculate a response result of each candidate point read.
  • the memory 200 can be disposed in the feature extraction circuit 100, and can be disposed in the feature extraction circuit 100, which is not limited in this embodiment of the present application.
  • the pyramid sub-circuit 110 is implemented by hardware, the feature point detection sub-unit 120 is based on hardware as the feature point detection sub-circuit 120, and the response calculation sub-unit 140 and the feature point selection sub-unit 130 are all implemented by a processor.
  • the pyramid sub-circuit 110 can read the image to be processed from the off-chip memory 200 through the first memory controller 150 (not shown in FIG. 4); the pyramid sub-circuit 110 can also be processed by the circuit or unit of the previous stage. Image (not shown in Figure 4).
  • Pyramid subcircuit 110 At least one layer of pyramid is created for the image to be processed.
  • the feature point detecting sub-circuit 120 detects at least one candidate point in each layer of the pyramid.
  • the feature point detecting sub-circuit 120 writes the related data of the detected candidate points into the memory 200 through the first memory controller 150. After completing the processing, the feature point detecting sub-circuit 120 can feed back the processed information to the processor, so that the processor reads the relevant data from the memory for subsequent processing.
  • the processor (response calculation sub-unit 140) reads the relevant data of the candidate points written in the memory 200 from the memory 200, and calculates the response result of each candidate point.
  • the processor feature point selection sub-unit 130) selects the feature points from the candidate points according to the response result of the candidate points calculated by the response calculation sub-unit 140 and the preset rule.
  • the processor may write the coordinate information of the selected feature points into the memory 200 in a preset order for subsequent calculation (not shown in FIG. 4), for example, for subsequent calculation of the feature points.
  • the descriptor is used.
  • the response calculation sub-unit 140 is an optional unit, and the processor (the feature point selection sub-unit 130) can directly select the feature points from the candidate points according to the candidate points and the preset rule, which is not limited in this embodiment of the present application. .
  • FIG. 5 is a schematic block diagram of a feature extraction circuit 100 in accordance with yet another embodiment of the present application.
  • the feature point detection sub-unit 120 is based on the hardware as the feature point detection sub-circuit 120, and the response calculation sub-unit 140 calculates the sub-circuit 140 based on the hardware, and the first memory controller 150 is configured to use the response result of the candidate point.
  • the feature point selection sub-unit 130 is configured to select a feature point from the candidate points according to the candidate point and the preset rule, and the feature point selection sub-unit 130 is configured to read the candidate point from the memory 200. In response to the result, the feature points are selected from the candidate points according to the response result of the read candidate points and the preset rule.
  • the memory 200 can be disposed in the feature extraction circuit 100, and can be disposed in the feature extraction circuit 100, which is not limited in this embodiment of the present application.
  • the pyramid sub-circuit 110 is implemented by hardware, the feature point detecting sub-unit 120 is based on hardware as the feature point detecting sub-circuit 120, and the response calculating sub-unit 140 is based on hardware as the response calculating sub-circuit 140, and the feature point selecting sub-unit 130 is processed by Implemented.
  • the pyramid sub-circuit 110 can read the image to be processed from the off-chip memory 200 through the first memory controller 150 (not shown in FIG. 5); the pyramid sub-circuit 110 can also be processed by the circuit or unit of the previous stage. Image (not shown in Figure 5).
  • the pyramid sub-circuit 110 establishes at least one layer of pyramids for the image to be processed.
  • the feature point detecting sub-circuit 120 detects at least one candidate point in each layer of the pyramid.
  • the response calculation sub-circuit 140 calculates the response result of each candidate point.
  • the response calculation sub-circuit 140 writes the response result of each candidate point into the memory 200 through the first memory controller 150.
  • the response calculation sub-circuit 140 is finished After processing, the processed information can be fed back to the processor, so that the processor can read the relevant data from the memory for subsequent processing.
  • the processor feature point selection sub-unit 130 reads the response result of each candidate point from the memory 200, and selects the feature point from the candidate points according to the response result of the candidate point and the preset rule.
  • the response calculation sub-circuit 140 is optional. In the absence of the response calculation sub-circuit 140, the feature point detection sub-circuit 120 writes the relevant data of the detected candidate points to the first memory controller 150. Memory 200. After completing the processing, the feature point detecting sub-circuit 120 can feed back the processed information to the processor, so that the processor reads the relevant data from the memory for subsequent processing.
  • the processor can read related data of the candidate points written in the memory 200 from the memory 200, and the processor (feature point selection sub-unit 130) selects from the candidate points according to the candidate points and the preset rule.
  • the feature points are selected, which are not limited in this embodiment of the present application.
  • the processor can write the coordinate information of the selected feature points into the memory 200 in a preset order for subsequent calculation (not shown in FIG. 5), for example, for subsequent calculation of the feature points.
  • the descriptor is used.
  • the algorithms of the various sub-circuits or sub-units of the feature extraction circuit 100 may be flexibly upgraded, in other words, the atomic circuit or atomic unit may be flexibly replaced with an upgraded sub-circuit or sub-unit for an upgrade of the algorithm. After the algorithm is upgraded, the partially hardened logic can still accelerate the algorithm.
  • each sub-circuit or sub-unit of the embodiment of the present application is connected by a pipeline, which can improve the processing performance of the processor, thereby improving system performance.
  • FIG. 6 is a schematic block diagram depicting a sub-computing circuit 600 in accordance with an embodiment of the present application.
  • the description sub-computing circuit 600 may include a second memory controller 610 and a point pair extraction sub-circuit 620 and a descriptor generation sub-circuit 630 connected by a pipeline, and the second memory controller 610 is used for the slave memory 200.
  • the coordinate information of the feature point is read; the point pair extraction sub-circuit 620 is configured to determine a plurality of point pairs in the point region corresponding to the feature point in the image to be processed according to the coordinate information of the feature point read from the memory 200
  • the descriptor generation sub-circuit 630 is configured to determine a descriptor of the feature point based on the pixel values of the plurality of pairs of points.
  • the sub-calculation circuit of the embodiment of the present application calculates the descriptor of the feature point by the hardware circuit, which can reduce the calculation pressure of the system and improve the real-time and portability of the calculation.
  • the sub-computing circuit of the embodiment of the present application can be implemented based on an FPGA or an ASIC or the like.
  • the coordinate information of the feature point may be calculated by the circuit or unit of the previous stage and stored in the memory 200.
  • the coordinate information of the feature point may be obtained by the processor through software calculation and stored in the memory 200.
  • the coordinate information of the feature point may be information of the position of the feature point in the corresponding image to be processed.
  • the image to be processed may be the original image to be processed (which may be read from the memory by the second memory controller 610); the image to be processed may also be any layer image of the pyramid (may be controlled by the second memory)
  • the 610 is read from the memory, and is also directly read from the pyramid sub-circuit of the previous stage, which is not limited in this embodiment. Where the image to be processed is read can be controlled by the configuration signal.
  • the point pair extraction sub-circuit 620 can read a plurality of rows (for example, 49 rows) of data from the image to be processed according to the position of the feature point at the position of the image, and then select a region of the point corresponding to the feature point.
  • the data of the pixel points of the dot area is also called block data.
  • the point pair extraction sub-circuit 620 can determine a plurality of point pairs directly from the puncturing area, and can also determine a plurality of point pairs from the puncturing area according to a certain rule or algorithm.
  • the second memory controller 610 is configured to read coordinate information of the feature point from the memory, and the second memory controller 610 is configured to read coordinate information of the plurality of feature points from the memory in a preset order.
  • the coordinate information of the feature points can be calculated by the circuits or units of the preceding stage and stored in the memory 200 in a preset order.
  • the coordinate information of the feature points may be calculated by the processor through software and stored in the memory 200 in a preset order. In this way, the descriptor calculation circuit 600 reads the coordinate information of the feature points in a preset order, which can reduce the bandwidth of the random access memory and improve the performance of the system.
  • FIG. 7 is a schematic diagram of point pair extraction in accordance with an embodiment of the present application.
  • the feature point is K
  • the point area corresponding to the feature point K may be a circular area having the feature point K as a center and d as a radius.
  • the point of the feature point K may not be the center or the center of the feature point K.
  • the shape of the point area may be other than a circle or an irregular shape, which is not limited in the embodiment of the present application. Assume that the four pairs of points selected as shown in Figure 5 are marked as:
  • the descriptor generation sub-circuit 630 determines the descriptor of the feature point based on the pixel values of the plurality of pairs of points. Specifically, it may be to define an operation T.
  • I A represents the pixel value of the pixel point A
  • I B represents the pixel value of the pixel point B
  • FIG. 8 is a schematic block diagram depicting a sub-computing circuit 600 in accordance with another embodiment of the present application.
  • the descriptor calculation circuit 600 shown in FIG. 8 may further include a centroid calculation sub-circuit 640 and a main direction calculation sub-circuit 650 connected by a pipeline form, a centroid calculation sub-
  • the circuit 640 is configured to determine a point-taking area from the image according to the coordinate information of the feature point, and calculate coordinates of the gray-scale centroid of the point-taking area;
  • the main-direction calculating sub-circuit 650 is configured to use the coordinate information of the feature point and the coordinate of the grayscale centroid Calculating a main direction corresponding to the feature point;
  • the point pair extraction sub-circuit 620 is configured to determine a plurality of point pairs in the puncturing area corresponding to the feature point in the image to be processed according to the coordinate information of the feature point read from the memory 200
  • the method includes: the point pair extraction sub-
  • the centroid calculation sub-circuit 640 is a hardware circuit that determines a point-taking area from the image based on the coordinate information of the feature point, and calculates the coordinates of the gray-scale centroid of the point-taking area.
  • the point area corresponding to the feature point K may be a circular area having the feature point K as a center and d as a radius.
  • the gray center of mass can be calculated.
  • C is the coordinate of the gray center of mass, then C can be expressed as:
  • m00 is the sum of the gray values of all the pixels in the point region
  • m10 is the sum of the gray values of all the pixels on the x coordinate (abscissa) in the point region
  • m01 is the y coordinate in the point region ( The sum of the gray values of all the pixels on the ordinate).
  • centroid calculation sub-circuit 640 is a hardware circuit
  • the parallelism of the hardware accelerator can be fully utilized, and the gray scale centroid can be quickly calculated.
  • the main direction can be calculated using the CORDIC algorithm.
  • the primary direction calculation sub-circuit 650 can be a hardware circuit to take a configurable number of iterations, for example 20 times, to calculate the final principal direction; a lookup table can also be used to find the result of the sine or cosine to obtain ⁇ .
  • the point pair extracting sub-circuit 620 corresponds to the main direction calculated by the main direction calculating sub-circuit 650 based on the coordinate information of the feature point read from the memory 200 and the feature point in the image. Multiple point pairs are determined within the point area.
  • the point pair extraction sub-circuit 620 can read the coordinate S of the point pair corresponding to the feature point K from the point-taking area, and perform the following operations:
  • the descriptor generation sub-circuit 630 reads the data of the corresponding pixel point according to the rotated coordinates to generate a descriptor.
  • the second memory controller 610 may be connected to the other sub-circuits in the sub-calculation circuit 600 through the pipeline form, and the other sub-circuit connections in the sub-calculation circuit 600 may be described in other forms. .
  • the embodiment of the present application further provides an image processing integrated circuit including the foregoing feature extraction circuit 100.
  • the embodiment of the present application further provides an image processing integrated circuit including the foregoing sub-computing circuit 600.
  • the embodiment of the present application further provides an image processing integrated circuit including the foregoing feature extraction circuit 100 and the foregoing description sub-calculation circuit 600.
  • the embodiment of the present application further provides a computing chip, which includes the foregoing image processing integrated circuit.
  • the embodiment of the present application further provides a computing device, which includes the foregoing image processing integrated circuit.
  • the embodiment of the present application can be applied to the field of an aircraft, especially a drone, and the computing device can For the flight control chip.
  • circuits, sub-circuits, and sub-units of various embodiments of the present application is merely illustrative. Those of ordinary skill in the art will appreciate that the circuits, sub-circuits, and sub-units of the various examples described in the embodiments disclosed herein can be further separated or combined.
  • the computer program product includes one or more computer instructions.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be from a website site, computer, server or data center Transmission to another website site, computer, server, or data center by wire (eg, coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).
  • the computer readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a high-density digital video disc (DVD)), or a semiconductor medium (for example, a solid state hard disk (Solid State Disk, SSD)) and so on.
  • a magnetic medium for example, a floppy disk, a hard disk, a magnetic tape
  • an optical medium for example, a high-density digital video disc (DVD)
  • DVD high-density digital video disc
  • semiconductor medium for example, a solid state hard disk (Solid State Disk, SSD)
  • the size of the sequence numbers of the foregoing processes does not mean the order of execution sequence, and the order of execution of each process should be determined by its function and internal logic, and should not be applied to the embodiment of the present application.
  • the implementation process constitutes any limitation.
  • B corresponding to A means that B is associated with A, and B can be determined according to A.
  • determining B from A does not mean that B is only determined based on A, and that B can also be determined based on A and/or other information.
  • the disclosed systems, devices, and methods may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Abstract

一种特征提取电路和图像处理集成电路，该特征提取电路包括通过管线形式连接的金字塔子电路、特征点检测子单元和特征点选取子单元，金字塔子电路用于对待处理的图像建立至少一层金字塔；特征点检测子单元用于在每层金字塔中检测至少一个候选点；特征点选取子单元由处理器实现，用于根据候选点和预设规则，从候选点中选取出特征点。该特征提取电路建立金字塔由硬件电路实现，选取特征点由处理器通过软件实现，既能够降低系统的计算压力、提高计算的实时性，又可以提高特征点提取的灵活性、可移植性和计算效果。

Description

特征提取电路和图像处理集成电路
版权申明
本专利文件披露的内容包含受版权保护的材料。该版权为版权所有人所有。版权所有人不反对任何人复制专利与商标局的官方记录和档案中所存在的该专利文件或者该专利披露。
技术领域
本申请涉及图像处理领域，尤其涉及特征提取电路和图像处理集成电路。
背景技术
ORB(ORiented BRIEF)算法是基于FAST特征点检测(Features from Accelerated Segment Test)与BRIEF(二进制鲁棒独立基础特征，Binary Robust Independent Elementary Features)描述的算法。ORB算法被广泛应用于图像处理领域，尤其可以应用于机器视觉领域的定位与导航等。
现有的方案中，ORB算法的FAST特征点检测各步骤全部基于硬件实现或者全部基于软件实现。在ORB算法的FAST特征点检测各步骤全部基于硬件实现的情况下，某些步骤会丧失灵活性，可移植性差，计算效果不好。例如，基于硬件实现的特征点选取步骤选取出的特征点可能会过于集中，并且无法调整。在ORB算法的FAST特征点检测各步骤全部基于软件实现的情况下，系统的计算压力较大，计算的实时性较差。
发明内容
本申请提供了一种特征提取电路和图像处理集成电路，既能够降低系统的计算压力、提高计算的实时性，又可以提高特征点提取的灵活性、可移植性和计算效果。
第一方面，提供了一种特征提取电路，该特征提取电路包括通过管线形式连接的金字塔子电路、特征点检测子单元和特征点选取子单元，该金字塔子电路用于对待处理的图像建立至少一层金字塔；该特征点检测子单元用于 在该金字塔中检测至少一个候选点；该特征点选取子单元由处理器实现，用于根据该候选点和预设规则，从该候选点中选取出特征点。
第一方面的特征提取电路建立金字塔由硬件电路实现，选取特征点由处理器通过软件实现，既能够降低系统的计算压力、提高计算的实时性，又可以提高特征点提取的灵活性、可移植性和计算效果。
第二方面，提供了一种描述子计算电路，该描述子计算电路包括第二内存控制器以及通过管线形式连接的点对提取子电路和描述子生成子电路，该第二内存控制器用于从内存中读取特征点的坐标信息；该点对提取子电路用于根据从该内存中读取的该特征点的坐标信息，在待处理的图像中该特征点对应的取点区域内确定多个点对；该描述子生成子电路用于根据该多个点对的像素值，确定该特征点的描述子。
第二方面的描述子计算电路由硬件电路计算特征点的描述子，能够降低系统的计算压力，提高计算的实时性和可移植性。
第三方面，提供了一种图像处理集成电路，该图像处理集成电路包括第一方面的特征提取电路。
第四方面，提供了一种图像处理集成电路，该图像处理集成电路包括第二方面的描述子计算电路。
第五方面，提供了一种图像处理集成电路，该图像处理集成电路包括第一方面的特征提取电路和第二方面的描述子计算电路。
第六方面，提供了一种计算芯片，包括第三至第五任一方面的图像处理集成电路。
第七方面，提供了一种计算设备，包括第三至第五任一方面的图像处理集成电路。
附图说明
图1是本申请一个实施例的特征提取电路的示意性框图。
图2是本申请一个实施例的候选点检测的示意图。
图3是本申请一个实施例的特征提取电路的示意性框图。
图4是本申请另一个实施例的特征提取电路的示意性框图。
图5是本申请又一个实施例的特征提取电路的示意性框图。
图6是本申请一个实施例的描述子计算电路的示意性框图。
图7是本申请一个实施例的点对提取的示意图。
图8是本申请另一个实施例的描述子计算电路的示意性框图。
具体实施方式
下面将结合附图，对本申请实施例中的技术方案进行描述。
除非另有定义，本文所使用的所有的技术和科学术语与属于本申请的技术领域的技术人员通常理解的含义相同。本文中在本申请的说明书中所使用的术语只是为了描述具体的实施例的目的，不是旨在于限制本申请。
本申请实施例提供了一种硬件与软件相结合以实现特征提取的方案。其中，软件可以使用嵌入式、个人计算机(Personal Computer，PC)和处理器，例如，中央处理器(Central Processing Unit，CPU)或图形处理器(Graphics Processing Unit，GPU)等各种平台实现；硬件可以通过Vivado设计套件的高层次综合(High-Level Synthesis，HLS)或类似的工具实现。
本申请实施例提供了一种特征提取电路。图1是本申请一个实施例的特征提取电路100的示意性框图。如图1所示，特征提取电路100可以包括通过管线(pipeline)形式连接的金字塔子电路110、特征点检测子单元120和特征点选取子单元130。金字塔子电路110用于对待处理的图像10建立至少一层金字塔；特征点检测子单元120用于在每层金字塔中检测至少一个候选点；特征点选取子单元130由处理器实现，用于根据候选点和预设规则，从候选点中选取出特征点。
本申请实施例的特征提取电路建立金字塔由硬件电路实现，选取特征点由处理器通过软件实现，既能够降低系统的计算压力、提高计算的实时性，又可以提高特征点提取的灵活性、可移植性和计算效果。
本申请实施例的特征提取电路可以基于现场可编程门阵列(Field Programmable Gate Array，FPGA)或专用集成电路(Application Specific Integrated Circuit，ASIC)等实现。
金字塔子电路110可以用来解决通过算法提取到的特征点不满足尺寸一直性的问题。换而言之，金字塔子电路110建立金字塔，通过在不同尺度下的图像中提取特征点以达到满足尺度变化的效果。金字塔子电路110可以建立简单的降采样的金字塔，也可以通过调整大小(resize)建立各种尺寸(size)的图片。金字塔子电路110建立的金字塔可以是一层的，也可以是多层的， 金字塔的层数可以通过处理器预先配置或更改可选。金字塔子电路110建立金字塔可以采用高斯滤波法，也可以不采用高斯滤波法。本申请实施例对采用上述哪种情况不作限定。
可选地，金字塔子电路110建立的金字塔的相关数据可以通过内存控制器(例如，下文中将会提到的第一内存控制器)写入到片外的内存(或缓存中)中，供后续计算使用，例如供后续计算图像(可以是基于任一层金字塔的图像)的灰度质心使用。
应理解，待处理的图像10，或者称为原始图像可以存储在片外的内存中，由例如第一内存控制器从内存中读出到金字塔子电路110。或者，待处理的图像10可以来源于前级的电路或单元。本申请实施例对待处理的图像的来源不作限定，金字塔子电路110从哪里读取待处理的图像可以通过配置信号来控制。
特征点检测子单元120用于在每层金字塔中检测至少一个候选点。图2是本申请一个实施例的候选点检测的示意图。特征点检测子单元120进行候选点检测可以基于以下原理。如图2所示，从图像(可以是基于任一层金字塔的图像)中选取一个像素点P，假设其灰度值为Ip，设定一个合适的阈值t。当两个点的灰度值之差的绝对值大于t时，认为这两个点不相同。考虑该像素点P周围的16个像素点。如果这16个像素点中有连续的n(例如，n＝12)个像素点都和像素点P不同，那么像素点P就是一个角点，即候选点。其中，其中，n可以为正整数，n的值可以根据配置信号来配置。n具体地，本申请实施例中，特征点检测子单元120可以采用FAST5-8、FAST7-12、FAST9-16和Harris角点检测中的至少一种，从每层金字塔中检测出候选点，但本申请实施例不仅限于这些检测方法。
可选地，特征点检测子单元120可以基于软件，例如由处理器实现；特征点检测子单元120也可以基于硬件实现，例如特征点检测子单元120为特征点检测子电路。
可选地，在特征点检测子单元120基于硬件实现的情况下，特征点检测子单元120检测出的候选点的相关数据可以通过内存控制器(例如，下文中将会提到的第一内存控制器)写入到片外的内存中，供后续计算使用。
特征点选取子单元130可以基于软件，例如由处理器实现。特征点选取子单元130可以根据候选点和预设规则，从候选点中选取出特征点。特征点 选取子单元130(处理器)也可以基于候选点的位置分布和/或基于系统负荷，调整预设规则，从候选点中选取特征点。具体而言，处理器可以根据候选点的位置分布，从候选点中选取分布较为分散的点作为特征点；处理器也可以根据当前的系统负荷进行权衡，选择适当数量适当位置的点作为特征点。可选地，特征点选取子单元130(处理器)在处理图像的不同帧时可以采取不同的预设规则。即，处理器可以根据帧的关键性或者金字塔各层的情况，可以动态的调整以选择适当数量适当位置的点作为特征点。
当然，特征点选取子单元130(处理器)还可以综合其他因素进行特征点的选取，本申请实施例对此不作限定。特征点选取基于软件可以实现更灵活的配置。
可选地，特征点选取子单元130(处理器)可以将选取的特征点的坐标信息写入到片外的内存(或缓存中)中，供后续计算使用，例如供后续计算这些特征点的描述子使用。
可选地，特征点选取子单元130(处理器)还可以用于将选取的特征点的坐标信息按预设顺序写入内存中。这样可以确保后续硬件在读取数据时，按预设顺序读取，可以降低随机访问内存的带宽，能够提高系统的性能。
可选地，特征提取电路100还可以包括响应计算子单元140，响应计算子单元140用于计算每个候选点的响应结果，特征点选取子单元130用于根据候选点和预设规则，从候选点中选取出特征点，可以包括：特征点选取子单元130用于根据响应计算子单元140计算的候选点的响应结果和预设规则，从候选点中选取出特征点。
可选地，特征点选取子单元130可以包括响应计算子单元，该响应计算子单元功能与上述响应计算子单元140相同或相似，本申请实施例对次不作限定。
响应计算子单元140可以为每个候选点计算其响应结果，或称为响应大小。具体而言，响应计算子单元140可以使用非极大值抑制算法，以去除候选点较密集位置的部分候选点。例如，响应结果可以是候选点Q与其周围m(例如，m＝16)个候选点偏差的绝对值的和。继而，特征点选取子单元130可以根据响应计算子单元140计算的候选点的响应结果和前文描述的预设规则，从候选点中选取出特征点。例如，在位置比较临近的候选点中，选取出响应结果较大的s(例如，s＝1)个候选点作为特征点。其中，m和s可以为 正整数，m和s的值可以根据配置信号来配置。响应计算子单元140也可以是通过其他算法得到的响应结果，以用于特征点选取子单元130选取特征点，本申请实施例对此不作限定。
应理解，响应计算子单元140计算响应结果时，还可以参考待处理的图像(或者原始图像)，以便于响应计算子单元140在图像的相应区域进行响应最大值搜索(即非极大值抑制)，本申请实施例对此不作限定。
可选地，响应计算子单元140可以基于软件，例如由处理器实现；响应计算子单元140也可以基于硬件实现，例如响应计算子单元140为响应计算子电路。
可选地，在响应计算子单元140基于硬件实现的情况下，响应计算子单元140计算出的候选点的响应结果可以通过内存控制器(例如，下文中将会提到的第一内存控制器)写入到片外的内存中，供后续计算使用。
应理解，本申请实施例的特征提取电路由硬件与软件相结合实现，硬件与软件之间交换数据可以通过内存来实现。硬件可以从内存中读取数据，或者直接从前级的硬件中接收数据；硬件处理完数据后，将相关数据通过内存控制器存储在内存中，软件从内存中读取相关数据进行后续处理。因此，本申请实施例的特征提取电路100还可以包括第一内存控制器150。第一内存控制器150可以通过pipeline形式和特征提取电路100中的其他各个子电路或子单元连接，也可以通过其他形式和特征提取电路100中的其他各个子电路或子单元连接，本申请实施例对此不作限定。
还应理解，本申请各实施例的硬件处理完成后如果需要由软件(处理器)进行后续处理，硬件在完成处理后，可以将处理完成的信息反馈给处理器，以便于处理器从内存中读取相关数据进行后续处理。
下面以几个具体的例子详细说明本申请实施例的特征提取电路。
图3是本申请一个实施例的特征提取电路100的示意性框图。在该实施例中，特征点检测子单元120和响应计算子单元140均由处理器实现，第一内存控制器150用于将至少一层金字塔写入内存200中，特征点检测子单元120用于在每层金字塔中检测至少一个候选点，包括：特征点检测子单元120用于从内存200中读取写入内存200的至少一层金字塔，并在读取的每层金字塔中检测至少一个候选点。内存200可以独立于特征提取电路100，设置在特征提取电路100外，也可以设置在特征提取电路100中，本申请实施例 对此不作限定。
具体地，金字塔子电路110由硬件实现，特征点检测子单元120、响应计算子单元140和特征点选取子单元130均由处理器实现。金字塔子电路110可以通过第一内存控制器150从片外的内存200中读取待处理的图像(图3中未示出)；金字塔子电路110也可以前级的电路或单元得到待处理的图像(图3中未示出)。金字塔子电路110对待处理的图像建立至少一层金字塔。金字塔子电路110将建立的金字塔的相关数据通过第一内存控制器150写入到内存200中。金字塔子电路110在完成处理后，可以将处理完成的信息反馈给处理器，以便于处理器从内存中读取相关数据进行后续处理。处理器(特征点检测子单元120)从内存200中读取写入内存200的至少一层金字塔，并在读取的每层金字塔中检测至少一个候选点。处理器(响应计算子单元140)计算每个候选点的响应结果。处理器(特征点选取子单元130)根据响应计算子单元140计算的候选点的响应结果和预设规则，从候选点中选取出特征点。处理器(特征点选取子单元130)可以将选取的特征点的坐标信息按预设顺序写入到内存200中，供后续计算使用(图3中未示出)，例如供后续计算这些特征点的描述子使用。这里，响应计算子单元140为可选的单元，处理器(特征点选取子单元130)可以直接根据候选点和预设规则，从候选点中选取出特征点，本申请实施例对此不作限定。
图4是本申请另一个实施例的特征提取电路100的示意性框图。在该实施例中，特征点检测子单元120基于硬件为特征点检测子电路120，响应计算子单元140由处理器实现，第一内存控制器150用于将至少一个候选点写入内存200中，响应计算子单元140用于计算每个候选点的响应结果，包括：响应计算子单元140用于从内存200中读取至少一个候选点，并计算读取的每个候选点的响应结果。内存200可以独立于特征提取电路100，设置在特征提取电路100外，也可以设置在特征提取电路100中，本申请实施例对此不作限定。
具体地，金字塔子电路110由硬件实现，特征点检测子单元120基于硬件为特征点检测子电路120，响应计算子单元140和特征点选取子单元130均由处理器实现。金字塔子电路110可以通过第一内存控制器150从片外的内存200中读取待处理的图像(图4中未示出)；金字塔子电路110也可以前级的电路或单元得到待处理的图像(图4中未示出)。金字塔子电路110 对待处理的图像建立至少一层金字塔。特征点检测子电路120在每层金字塔中检测至少一个候选点。特征点检测子电路120将检测出的候选点的相关数据通过第一内存控制器150写入到内存200中。特征点检测子电路120在完成处理后，可以将处理完成的信息反馈给处理器，以便于处理器从内存中读取相关数据进行后续处理。处理器(响应计算子单元140)从内存200中读取写入内存200的候选点的相关数据，并计算每个候选点的响应结果。处理器(特征点选取子单元130)根据响应计算子单元140计算的候选点的响应结果和预设规则，从候选点中选取出特征点。处理器(特征点选取子单元130)可以将选取的特征点的坐标信息按预设顺序写入到内存200中，供后续计算使用(图4中未示出)，例如供后续计算这些特征点的描述子使用。这里，响应计算子单元140为可选的单元，处理器(特征点选取子单元130)可以直接根据候选点和预设规则，从候选点中选取出特征点，本申请实施例对此不作限定。
图5是本申请又一个实施例的特征提取电路100的示意性框图。在该实施例中，特征点检测子单元120基于硬件为特征点检测子电路120，响应计算子单元140基于硬件为响应计算子电路140，第一内存控制器150用于将候选点的响应结果写入内存200中，特征点选取子单元130用于根据候选点和预设规则，从候选点中选取出特征点，包括：特征点选取子单元130用于从内存200中读取候选点的响应结果，并根据读取的候选点的响应结果和预设规则，从候选点中选取出特征点。内存200可以独立于特征提取电路100，设置在特征提取电路100外，也可以设置在特征提取电路100中，本申请实施例对此不作限定。
具体地，金字塔子电路110由硬件实现，特征点检测子单元120基于硬件为特征点检测子电路120，响应计算子单元140基于硬件为响应计算子电路140，特征点选取子单元130均由处理器实现。金字塔子电路110可以通过第一内存控制器150从片外的内存200中读取待处理的图像(图5中未示出)；金字塔子电路110也可以前级的电路或单元得到待处理的图像(图5中未示出)。金字塔子电路110对待处理的图像建立至少一层金字塔。特征点检测子电路120在每层金字塔中检测至少一个候选点。响应计算子电路140计算每个候选点的响应结果。响应计算子电路140将每个候选点的响应结果通过第一内存控制器150写入到内存200中。响应计算子电路140在完 成处理后，可以将处理完成的信息反馈给处理器，以便于处理器从内存中读取相关数据进行后续处理。处理器(特征点选取子单元130)从内存200中读取每个候选点的响应结果，根据候选点的响应结果和预设规则，从候选点中选取出特征点。
应理解，响应计算子电路140为可选的，在不存在响应计算子电路140的情况下，特征点检测子电路120将检测出的候选点的相关数据通过第一内存控制器150写入到内存200中。特征点检测子电路120在完成处理后，可以将处理完成的信息反馈给处理器，以便于处理器从内存中读取相关数据进行后续处理。处理器(特征点选取子单元130)可以从内存200中读取写入内存200的候选点的相关数据，处理器(特征点选取子单元130)根据候选点和预设规则，从候选点中选取出特征点，本申请实施例对此不作限定。
处理器(特征点选取子单元130)可以将选取的特征点的坐标信息按预设顺序写入到内存200中，供后续计算使用(图5中未示出)，例如供后续计算这些特征点的描述子使用。
应理解，特征提取电路100的各个子电路或子单元的算法可以是灵活升级的，换句话说，可以针对算法的升级，使用升级后的子电路或子单元灵活地替换原子电路或原子单元。在算法升级后，部分硬化的逻辑仍然可以起到加速算法的效果。
还应理解，本申请实施例的各个子电路或子单元通过管线(pipeline)形式连接，可以提高处理器的处理性能，进而提高系统性能。
现有的方案中，ORB算法的BRIEF描述的各步骤通常全部基于软件实现的情况下，系统的计算压力较大，计算的实时性较差。对此，本申请实施例还提供了一种描述子计算电路。图6是本申请一个实施例的描述子计算电路600的示意性框图。如图6所示，描述子计算电路600可以包括第二内存控制器610以及通过管线形式连接的点对提取子电路620和描述子生成子电路630，第二内存控制器610用于从内存200中读取特征点的坐标信息；点对提取子电路620用于根据从内存200中读取的特征点的坐标信息，在待处理的图像中特征点对应的取点区域内确定多个点对；描述子生成子电路630用于根据多个点对的像素值，确定特征点的描述子。
本申请实施例的描述子计算电路由硬件电路计算特征点的描述子，能够降低系统的计算压力，提高计算的实时性和可移植性。
本申请实施例的描述子计算电路可以基于FPGA或ASIC等实现。
其中，特征点的坐标信息可以是前级的电路或单元计算得到并存储在内存200中的。可选地，特征点的坐标信息可以是处理器通过软件计算得到并存储在内存200中的。特征点的坐标信息可以是该特征点在相应的待处理的图像中的位置的信息。待处理的图像可以是待处理的原始图像(可以是通过第二内存控制器610从内存中读取的)；待处理的图像也可以是金字塔的任一层图像(可以是通过第二内存控制器610从内存中读取的，也可以是直接从前级的金字塔子电路读取的，本申请实施例对此不作限定)。待处理的图像从哪里被读出可以通过配置信号来控制。
点对提取子电路620可以根据特征点在图像的位置，从待处理的图像中读取若干行(例如49行)数据，再从中选取特征点对应的取点区域。取点区域的像素点的数据也叫作块(block)数据。点对提取子电路620可以从取点区域内直接确定多个点对，也可以根据一定的规则或算法从取点区域内确定多个点对，本申请实施例对此不作限定。
可选地，第二内存控制器610用于从内存中读取特征点的坐标信息，包括：第二内存控制器610用于按预设顺序从内存中读取多个特征点的坐标信息。换句话说，特征点的坐标信息可以是前级的电路或单元计算得到并按预设顺序存储在内存200中的。可选地，特征点的坐标信息可以是处理器通过软件计算得到并按预设顺序存储在内存200中的。这样，描述子计算电路600按预设顺序读取特征点的坐标信息，可以降低随机访问内存的带宽，能够提高系统的性能。
图7是本申请一个实施例的点对提取的示意图。具体地，如图7所示，假设特征点为K，特征点K对应的取点区域可以为以特征点K为圆心，以d为半径的圆形区域。在该圆形区域内以某一预设规则或模式选取N个点对。这里为方便说明，以N＝4为例，实际应用中N可以取256、512或者其他数值。即实际应用中，可以在取点区域内取256个点对、512个点对或者其他数量个点对。应理解，图7所示的特征点K对应的取点区域仅为示例。特征点K对应的取点区域也可以不以特征点K为圆心或中心，取点区域的形状也可以为圆形以外的其他形状，或者为不规则形状，本申请实施例对此不作限定。假设如图5所示选取的4个点对分别标记为：
K1(A,B)、K2(A,B)、K3(A,B)和K4(A,B)。
描述子生成子电路630根据多个点对的像素值，确定特征点的描述子。具体可以是定义操作T。
Figure PCTCN2017100033-appb-000001
其中，IA表示像素点A的像素值，IB表示像素点B的像素值。
分别对已选取的点对进行T操作，例如：
T(K1(A,B))＝1
T(K2(A,B))＝0
T(K3(A,B))＝1
T(K4(A,B))＝1
将得到的结果进行组合，得到该特征点K的描述子为1011。
图8是本申请另一个实施例的描述子计算电路600的示意性框图。可选地，为了使生成的描述子具有旋转不变性，如图8所示的描述子计算电路600还可以包括通过管线形式连接的质心计算子电路640和主方向计算子电路650，质心计算子电路640用于根据特征点的坐标信息从图像中确定取点区域，并计算取点区域的灰度质心的坐标；主方向计算子电路650用于根据特征点的坐标信息和灰度质心的坐标，计算特征点对应的主方向；点对提取子电路620用于根据从内存200中读取的特征点的坐标信息，在待处理的图像中特征点对应的取点区域内确定多个点对，包括：点对提取子电路620用于根据从内存200中读取的特征点的坐标信息和主方向计算子电路650计算的主方向，在图像中特征点对应的取点区域内确定多个点对。
质心计算子电路640为硬件电路，根据特征点的坐标信息从图像中确定取点区域，并计算取点区域的灰度质心的坐标。例如，仍以特征点K为例，特征点K对应的取点区域可以为以特征点K为圆心，以d为半径的圆形区域。根据取点区域中所有像素点的灰度值，可以计算出灰度质心。C是灰度质心的坐标，则C可以表示为：
Figure PCTCN2017100033-appb-000002
其中，m00是取点区域中所有像素点的灰度值的和；m10是取点区域中x坐标(横坐标)上所有像素点的灰度值的和；m01是取点区域中y坐标(纵坐标)上所有像素点的灰度值的和。
其中，在质心计算子电路640为硬件电路时，可以充分利用硬件加速器的并行性，快速计算出灰度质心。
主方向计算子电路650根据特征点K的坐标信息和灰度质心的坐标C，计算特征点对应的主方向θ，其中，θ＝arctan(m01,m10)。
具体而言，可以使用CORDIC算法计算出主方向。主方向计算子电路650可以为硬件电路，从而采取可配置的迭代次数，例如20次，计算出最终的主方向；也可以使用查找表查找出正弦或余弦的结果从而得到θ。
主方向计算子电路650计算主方向的同时，点对提取子电路620根据从内存200中读取的特征点的坐标信息和主方向计算子电路650计算的主方向，在图像中特征点对应的取点区域内确定多个点对。具体地，点对提取子电路620可以从取点区域中，读取特征点K对应的点对的坐标S，进行如下运算：
Figure PCTCN2017100033-appb-000003
Sθ＝RθS
Sθ为旋转后的坐标，描述子生成子电路630根据旋转后的坐标读取出对应的像素点的数据，生成描述子。
第二内存控制器610可以通过pipeline形式和描述子计算电路600中的其他各个子电路连接，也可以通过其他形式描述子计算电路600中的其他各个子电路连接，本申请实施例对此不作限定。
本申请实施例还提供一种图像处理集成电路，该图像处理集成电路包括前述特征提取电路100。
本申请实施例还提供一种图像处理集成电路，该图像处理集成电路包括前述描述子计算电路600。
本申请实施例还提供一种图像处理集成电路，该图像处理集成电路包括前述特征提取电路100和前述描述子计算电路600。
本申请实施例还提供一种计算芯片，该计算芯片包括前述的图像处理集成电路。
本申请实施例还提供一种计算设备，该计算设备包括前述的图像处理集成电路。
本申请实施例可以应用在飞行器，尤其是无人机领域，该计算设备可以 为飞控芯片。
应理解，本申请各实施例的电路、子电路、子单元的划分只是示意性的。本领域普通技术人员可以意识到，本文中所公开的实施例描述的各示例的电路、子电路和子单元，能够再行拆分或组合。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(Digital Subscriber Line，DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如，软盘、硬盘、磁带)、光介质(例如，高密度数字视频光盘(Digital Video Disc，DVD))、或者半导体介质(例如，固态硬盘(Solid State Disk，SSD))等。
应理解，说明书通篇中提到的“一个实施例”或“一实施例”意味着与实施例有关的特定特征、结构或特性包括在本申请的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”或“在一实施例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
应理解，在本申请实施例中，“与A相应的B”表示B与A相关联，根据A可以确定B。但还应理解，根据A确定B并不意味着仅仅根据A确定B，还可以根据A和/或其它信息确定B。
应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表 示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (18)

  1. 一种特征提取电路，其特征在于，所述特征提取电路包括通过管线形式连接的金字塔子电路、特征点检测子单元和特征点选取子单元，
    所述金字塔子电路用于对待处理的图像建立至少一层金字塔；
    所述特征点检测子单元用于在所述金字塔中检测至少一个候选点；
    所述特征点选取子单元由处理器实现，用于根据所述候选点和预设规则，从所述候选点中选取出特征点。
  2. 根据权利要求1所述的特征提取电路，其特征在于，所述特征提取电路还包括响应计算子单元，所述响应计算子单元用于计算每个所述候选点的响应结果，
    所述特征点选取子单元用于根据所述候选点和预设规则，从所述候选点中选取出特征点，包括：
    所述特征点选取子单元用于根据所述响应计算子单元计算的所述候选点的所述响应结果和预设规则，从所述候选点中选取出特征点。
  3. 根据权利要求2所述的特征提取电路，其特征在于，所述特征提取电路还包括第一内存控制器。
  4. 根据权利要求3所述的特征提取电路，其特征在于，所述特征点检测子单元和所述响应计算子单元均由处理器实现，所述第一内存控制器用于将所述至少一层金字塔写入内存中，
    所述特征点检测子单元用于在每层所述金字塔中检测至少一个候选点，包括：
    所述特征点检测子单元用于从所述内存中读取所述写入所述内存的至少一层金字塔，并在所述读取的每层所述金字塔中检测所述至少一个候选点。
  5. 根据权利要求3所述的特征提取电路，其特征在于，所述特征点检测子单元基于硬件为特征点检测子电路，所述响应计算子单元由处理器实现，所述第一内存控制器用于将所述至少一个候选点写入内存中，
    所述响应计算子单元用于计算每个所述候选点的响应结果，包括：
    所述响应计算子单元用于从所述内存中读取所述至少一个候选点，并计算所述读取的每个所述候选点的所述响应结果。
  6. 根据权利要求3所述的特征提取电路，其特征在于，所述特征点检测子单元基于硬件为特征点检测子电路，所述响应计算子单元基于硬件为响应计算子电路，所述第一内存控制器用于将所述候选点的所述响应结果写入内存中，
    所述特征点选取子单元用于根据所述候选点和预设规则，从所述候选点中选取出特征点，包括：
    所述特征点选取子单元用于从所述内存中读取所述候选点的所述响应结果，并根据所述读取的所述候选点的所述响应结果和所述预设规则，从所述候选点中选取出特征点。
  7. 根据权利要求1至6中任一项所述的特征提取电路，其特征在于，所述特征点选取子单元用于基于所述候选点的位置分布和/或基于系统负荷，根据所述候选点的所述响应结果，从所述候选点中选取所述特征点。
  8. 根据权利要求7所述的特征提取电路，其特征在于，所述特征点选取子单元在处理图像的不同帧时采取不同的预设规则。
  9. 根据权利要求1所述的特征提取电路，其特征在于，所述特征点选取子单元还用于将选取的所述特征点的坐标信息按预设顺序写入内存中。
  10. 一种描述子计算电路，其特征在于，所述描述子计算电路包括第二内存控制器以及通过管线形式连接的点对提取子电路和描述子生成子电路，
    所述第二内存控制器用于从内存中读取特征点的坐标信息；
    所述点对提取子电路用于根据从所述内存中读取的所述特征点的坐标信息，在待处理的图像中所述特征点对应的取点区域内确定多个点对；
    所述描述子生成子电路用于根据所述多个点对的像素值，确定所述特征点的描述子。
  11. 根据权利要求10所述的描述子计算电路，其特征在于，所述描述子计算电路还包括通过管线形式连接的质心计算子电路和主方向计算子电路，
    所述质心计算子电路用于根据所述特征点的坐标信息从所述图像中确定所述取点区域，并计算所述取点区域的灰度质心的坐标；
    所述主方向计算子电路用于根据所述特征点的坐标信息和所述灰度质心的坐标，计算所述特征点对应的主方向；
    所述点对提取子电路用于根据从所述内存中读取的所述特征点的坐标信息，在待处理的图像中所述特征点对应的取点区域内确定多个点对，包括：
    所述点对提取子电路用于根据从所述内存中读取的所述特征点的坐标信息和所述主方向计算子电路计算的所述主方向，在所述图像中所述特征点对应的所述取点区域内确定所述多个点对。
  12. 根据权利要求10或11所述的描述子计算电路，其特征在于，所述第二内存控制器用于从内存中读取特征点的坐标信息，包括：
    所述第二内存控制器用于按预设顺序从所述内存中读取多个所述特征点的所述坐标信息。
  13. 一种图像处理集成电路，其特征在于，所述图像处理集成电路包括权利要求1至9中任一项所述的特征提取电路。
  14. 一种图像处理集成电路，其特征在于，所述图像处理集成电路包括权利要求10至12中任一项所述的描述子计算电路。
  15. 一种图像处理集成电路，其特征在于，所述图像处理集成电路包括权利要求1至9中任一项所述的特征提取电路和权利要求10至12中任一项所述的描述子计算电路。
  16. 一种计算芯片，其特征在于，包括权利要求13至15中任一项所述的图像处理集成电路。
  17. 一种计算设备，其特征在于，包括权利要求13至15中任一项所述的图像处理集成电路。
  18. 根据权利要求17所述的计算设备，其特征在于，所述计算设备为飞控芯片。
PCT/CN2017/100033 2017-08-31 2017-08-31 特征提取电路和图像处理集成电路 WO2019041265A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201780004680.6A CN108496186A (zh) 2017-08-31 2017-08-31 特征提取电路和图像处理集成电路
PCT/CN2017/100033 WO2019041265A1 (zh) 2017-08-31 2017-08-31 特征提取电路和图像处理集成电路

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/100033 WO2019041265A1 (zh) 2017-08-31 2017-08-31 特征提取电路和图像处理集成电路

Publications (1)

Publication Number Publication Date
WO2019041265A1 true WO2019041265A1 (zh) 2019-03-07

Family

ID=63344789

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/100033 WO2019041265A1 (zh) 2017-08-31 2017-08-31 特征提取电路和图像处理集成电路

Country Status (2)

Country Link
CN (1) CN108496186A (zh)
WO (1) WO2019041265A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113033590A (zh) * 2019-12-25 2021-06-25 杭州海康机器人技术有限公司 图像特征匹配方法、装置、图像处理设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104851094A (zh) * 2015-05-14 2015-08-19 西安电子科技大学 一种基于rgb-d的slam算法的改进方法
US20160012312A1 (en) * 2014-07-08 2016-01-14 Canon Kabushiki Kaisha Image processing method, image processing apparatus, program, and recording medium
CN106952215A (zh) * 2017-02-27 2017-07-14 珠海全志科技股份有限公司 一种图像金字塔特征提取电路、装置及方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8406507B2 (en) * 2009-01-14 2013-03-26 A9.Com, Inc. Method and system for representing image patches
CN104850851A (zh) * 2015-04-22 2015-08-19 福州大学 一种具有尺度不变性的orb特征点匹配方法
CN106504229B (zh) * 2016-09-30 2019-11-12 上海联影医疗科技有限公司 图像中特征点的检测方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160012312A1 (en) * 2014-07-08 2016-01-14 Canon Kabushiki Kaisha Image processing method, image processing apparatus, program, and recording medium
CN104851094A (zh) * 2015-05-14 2015-08-19 西安电子科技大学 一种基于rgb-d的slam算法的改进方法
CN106952215A (zh) * 2017-02-27 2017-07-14 珠海全志科技股份有限公司 一种图像金字塔特征提取电路、装置及方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113033590A (zh) * 2019-12-25 2021-06-25 杭州海康机器人技术有限公司 图像特征匹配方法、装置、图像处理设备及存储介质

Also Published As

Publication number Publication date
CN108496186A (zh) 2018-09-04

Similar Documents

Publication Publication Date Title
CN111815755B (zh) 虚拟物体被遮挡的区域确定方法、装置及终端设备
CN107533771B (zh) 通过3d模型重建进行网格简化
CN108596944B (zh) 一种提取运动目标的方法、装置及终端设备
CN108765340B (zh) 模糊图像处理方法、装置及终端设备
US9466118B2 (en) Image segmentation device, image segmentation method, and depth map generating method
CN109215037B (zh) 目标图像分割方法、装置及终端设备
CN108765315B (zh) 图像补全方法、装置、计算机设备及存储介质
US20200335197A1 (en) Method and apparatus for classification of lesion based on learning data applying one or more augmentation methods in lesion information augmented patch of medical image
EP2954457A1 (en) System and method for identifying similarities in different images
CN110288710B (zh) 一种三维地图的处理方法、处理装置及终端设备
WO2019128504A1 (zh) 一种台球比赛中的图像处理方法、装置及终端设备
WO2022093378A1 (en) Head position extrapolation based on a 3d model and image data
JP6937782B2 (ja) 画像処理方法及びデバイス
WO2019041265A1 (zh) 特征提取电路和图像处理集成电路
WO2023093260A1 (zh) 指令处理装置、方法、计算机设备及存储介质
KR101499124B1 (ko) 공유 메모리를 이용한 영상 처리 방법 및 장치
GB2555168A (en) Using labels to track high-frequency offsets for patch-matching algorithms
CN111754435A (zh) 图像处理方法、装置、终端设备及计算机可读存储介质
WO2023109086A1 (zh) 文字识别方法、装置、设备及存储介质
US10346948B2 (en) Graphics processing method and device
WO2023092997A1 (zh) 图像特征点选取方法及装置、设备、存储介质和程序产品
CN112487943B (zh) 关键帧去重的方法、装置和电子设备
CN112348069B (zh) 数据增强方法、装置、计算机可读存储介质及终端设备
WO2021139169A1 (zh) 卡片识别方法、装置、设备及存储介质
CN113436063A (zh) 图像处理方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17923774

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17923774

Country of ref document: EP

Kind code of ref document: A1