WO2020216249A1 - 一种基于多处理器的目标跟踪处理方法 - Google Patents

一种基于多处理器的目标跟踪处理方法 Download PDF

Info

Publication number
WO2020216249A1
WO2020216249A1 PCT/CN2020/086124 CN2020086124W WO2020216249A1 WO 2020216249 A1 WO2020216249 A1 WO 2020216249A1 CN 2020086124 W CN2020086124 W CN 2020086124W WO 2020216249 A1 WO2020216249 A1 WO 2020216249A1
Authority
WO
WIPO (PCT)
Prior art keywords
tracked
feature points
data
dsps
dsp
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/086124
Other languages
English (en)
French (fr)
Inventor
赵尔鑫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Autel Robotics Co Ltd
Original Assignee
Autel Robotics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Autel Robotics Co Ltd filed Critical Autel Robotics Co Ltd
Publication of WO2020216249A1 publication Critical patent/WO2020216249A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00—Digital computers in general; Data processing equipment in general
    • G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
    • G06F15/161—Computing infrastructure, e.g. computer clusters, blade chassis or hardware partitioning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform

Definitions

  • This application relates to the technical field of data processing, and in particular to a multi-processor-based target tracking processing method.
  • the detection and tracking of moving targets is an important part of digital image processing technology. People continue to develop new algorithms for the detection and tracking of moving targets. However, due to the computational complexity of the algorithm and the constraints of the processing speed of the hardware platform, it is difficult to balance the two key performance indicators of moving target tracking: real-time and accuracy. grasp. In order to achieve the real-time requirements in the moving target tracking system, some measures need to be taken to improve the processing speed of the algorithm, one is to optimize the algorithm, and the other is to improve the hardware system. At present, there are many moving target tracking algorithms, and each has its own advantages.
  • the pyramid optical flow algorithm improves the matching accuracy of large motion images by downsampling the original image. However, due to the large demand for pyramid optical flow resources, the algorithm is more complex and the processing efficiency It is low, and it is difficult to achieve real-time requirements for realizing multiple simultaneous tracking on the CPU of the existing embedded platform.
  • the present invention provides a multi-processor-based target tracking processing method, which overcomes the disadvantage of poor real-time performance when multiple channels are simultaneously tracked on the CPU of the existing embedded platform in the prior art.
  • the embodiment of the present invention provides a multi-processor-based target tracking processing method, including: acquiring original image data to be tracked; establishing an image pyramid based on the original image data, and extracting each resolution layer in the image pyramid
  • the feature points that need to be tracked determine the number of required processors according to the upper limit of the processor processing data and the number of feature points that need to be tracked; according to the number of required processors, the feature to be tracked Points are allocated to multiple processors for optical flow calculations.
  • the plurality of processors includes a CPU and a plurality of DSPs, and the step of determining the number of processors required according to the upper limit of the data processed by the processors and the number of feature points to be tracked, Including: obtaining the total number of feature points to be tracked in each resolution layer of the image pyramid; obtaining the upper limit of each DSP processing data; determining the total number of feature points to be tracked and the upper limit of each DSP processing data
  • the number of DSPs required and the amount of data of the feature points that need to be tracked are processed by the CPU.
  • the step of determining the number of DSPs required and the amount of data of the feature points to be tracked by the CPU according to the number of feature points to be tracked and the upper limit of data processed by each DSP includes: Divide the total number of feature points to be tracked by the upper limit of each DSP processing data, obtain the quotient and remainder, and judge whether the remainder is 0; when the remainder is 0, the number of DSPs required is the number of quotients, and the CPU processes the features to be tracked The data amount of the point is 0; when the remainder is not 0, the number of DSPs required is the number of quotient plus one, and the data amount of the feature points to be tracked by the CPU is 0.
  • the step of determining the number of DSPs required and the amount of data of the feature points to be tracked by the CPU according to the number of feature points to be tracked and the upper limit of data processed by each DSP includes: Divide the total number of feature points to be tracked by the upper limit of each DSP processing data to obtain the quotient and remainder to determine whether the quotient is greater than the actual number of DSPs; when the quotient is greater than the actual number of DSPs, the CPU processes the feature point data to be tracked The amount is the amount of data other than the amount of feature point data to be tracked by the DSP.
  • the number of DSPs required is the number of quotient; when the quotient is less than the actual number of DSPs and the remainder is not 0, the number of DSPs required is The number of quotients increases by one.
  • the required number of DSPs is the actual number.
  • the feature points of the pyramid layer with the smallest resolution are allocated to multiple processors for optical flow calculation.
  • the step of allocating the feature points to be tracked to multiple processors for optical flow calculation according to the number of required processors includes: each processor calculates and allocates to the current frame image Each feature point of the current pyramid layer and the interpolated sub-pixel of the preset window; calculate the first derivative in the first direction, the first derivative in the second direction, and the spatial gradient matrix of all pixels in the preset window, the first direction , The second direction is perpendicular to each other; iteratively calculate the interpolation sub-pixels of each feature point and the preset window for the next frame of image information; according to the interpolation between each feature point in the current frame of image and the previous frame of image window and the preset window Sub-pixel, calculate the pixel value error corresponding to each feature point; obtain the residual matrix according to the pixel value error corresponding to each feature point and the first derivative in the first direction and the first derivative in the second direction of the previous frame of image Obtain the coordinate value offset of each feature point according to the spatial gradient matrix and the residual matrix; determine whether the coordinate value
  • the method further includes: determining whether the current pyramid level is the original resolution level, and when it is the original resolution level When the optical flow calculation ends.
  • an embodiment of the present invention provides a computer-readable storage medium that stores computer instructions, and the computer instructions are used to make the computer execute the Multi-processor target tracking processing method.
  • the multi-processor-based target tracking processing method obtains the original image data required for target tracking through a CPU, establishes an image pyramid based on the original image data, and extracts the feature points to be tracked in each resolution layer of the image pyramid , According to the upper limit of processor processing data and the number of feature points to be tracked, determine the number of processors required, and assign the feature points to be tracked to multiple processors for optical flow calculation according to the number of processors required.
  • the embodiment of the present invention utilizes the advantages of parallel operation of the multi-core wide vector register DSP and combines the independence of feature point calculation to perform load balancing. Under the premise of meeting the accuracy requirements, the pyramid optical flow realization process is optimized, so that the overall operating efficiency of the system is high and real-time performance is improved.
  • FIG. 1 is a flowchart of a specific example of a multi-processor-based target tracking processing method provided by an embodiment of the present invention
  • FIG. 2 is a schematic diagram of the framework of the CPU and DSP hardware connection provided by an embodiment of the present invention
  • FIG. 3 is a flowchart of a specific example of determining the number of processors provided by an embodiment of the present invention.
  • FIG. 4 is a flowchart of a specific example of optical flow calculation performed by each processor according to an embodiment of the present invention.
  • the terms “installed”, “connected” and “connected” should be understood in a broad sense, unless otherwise clearly specified and limited. For example, they can be fixed or detachable. Connected or integrally connected; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of the two components, which can be a wireless connection or a wired connection connection.
  • Connected or integrally connected it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of the two components, which can be a wireless connection or a wired connection connection.
  • the specific meaning of the above-mentioned terms in the present invention can be understood in specific situations.
  • the embodiment of the present invention provides a multi-processor-based target tracking processing method. As shown in FIG. 1, the multi-processor based target tracking processing method includes the following steps:
  • Step S1 Obtain the original image data that needs to be tracked, and build an image pyramid based on the original image data, and extract the feature points to be tracked in each resolution layer of the image pyramid.
  • the original image data that needs to be tracked externally is obtained by the CPU, and an image pyramid is established for the original image data and the feature points that need to be tracked in each resolution layer are advanced.
  • the CPU after acquiring the original image, the CPU performs preprocessing such as filtering and denoising before establishing an image pyramid.
  • Step S2 Determine the number of processors required according to the upper limit of the processor processing data and the number of feature points to be tracked.
  • load balancing is performed on the CPU side, and according to the current number of feature points to be tracked and the DSP (digital signal processor, digital signal processor) wide vector register, the maximum amount of data can be processed at a time, and different feature point calculation tasks are allocated to each A DSP, when the amount of data processed by each DSP cannot meet the current number of feature points that need to be tracked, the CPU processes the amount of data that each DSP cannot process.
  • DSP digital signal processor, digital signal processor
  • the hardware connection relationship between the CPU and the DSP is used.
  • the DSP includes: DSP CORE, IRAM (Instruction RAM DSP, memory for storing instruction sets), DRAM (Dynamic Random Access Memory, and buffer memory ICache (when debugging DSP instructions, data will be stored in the buffer memory ICache).
  • the CPU loads the DSP program into IRAM, DRAM and DDR to run it Get up, and then the CPU informs the DSP that there is a task to be processed through an interrupt, and finally the DSP finishes processing and then uses an interrupt to notify the CPU of the status of the task completion.
  • the built-in IDMA in the chip is responsible for importing image data from DDR to DRAM.
  • DSP Core obtains data from DRAM for processing, writes the processing results back to DRAM, and then uses IDMA to export the data to DDR.
  • HiSilicon Hi3559CV100SOC can be used.
  • This chip provides efficient and abundant computing resources, including 4 Vision DSPs ( Figure 2 only uses the internal structure of one DSP and the connection relationship with the CPU as an example).
  • This series of DSPs is a dedicated processor for accelerating vision processing, with programmable capabilities. Based on Vision DSP, it can develop basic calculation functions and realize complex algorithms for intelligent analysis.
  • Step S3 According to the number of required processors, the feature points to be tracked are allocated to multiple processors for optical flow calculation.
  • each DSP CORE and CPU loads the corresponding number of feature points that need to be tracked, and starts from the feature point of the pyramid layer with the smallest resolution and is allocated to multiple processors for optical flow calculation.
  • multiple processors include CPUs and multiple DSPs.
  • the process of executing step S2 specifically includes:
  • Step S21 Obtain the total number of feature points to be tracked in each resolution layer of the image pyramid.
  • Step S22 Obtain the upper limit of data processed by each DSP.
  • the adopted Vision DSP has a vector register with a width of 512 Bit, and a single instruction cycle can execute 32 16 Bit data or 16 32 Bit data operations. This is just an example, and the invention is not limited to this.
  • Step S23 According to the number of feature points to be tracked and the upper limit of data processed by each DSP, the number of DSPs required and the data volume of feature points to be tracked by the CPU are determined.
  • only multiple DSPs are used to process the total number of feature points that need to be tracked, and the total number of feature points to be tracked is divided by the upper limit of each DSP processing data to obtain the quotient and remainder.
  • the actual number of DSPs is known (for example, the aforementioned HiSilicon processing chip includes 4 Vision DSPs), according to the total number of feature points to be tracked, the actual number of DSPs, and the amount of data processed by each DSP The upper limit is used to determine the number of feature point calculations to be tracked by each DSP and whether the CPU is required to perform feature point calculations to be tracked.
  • it includes: dividing the total number of feature points to be tracked by the upper limit of each DSP processing data, obtaining the quotient and the remainder, and judging whether the quotient is greater than the actual number of DSPs; when the quotient is greater than the actual number of DSPs, indicating the need
  • the number of processed feature points is greater than the actual number of DSPs that can be processed.
  • the CPU is required to assist in processing part of the feature point data.
  • the amount of feature point data that needs to be tracked by the CPU processing is the actual number of DSPs that can be processed.
  • the number of DSPs required is the number of quotients; when the quotient is less than the actual number of DSPs and the remainder is not 0, the number of DSPs required is the number of quotients plus One.
  • the quotient is equal to the actual number of DSPs, the required number of DSPs is the actual number. For example, if 280 feature points need to be tracked currently, DSP CORE0 processes 1 to 64 feature points, DSP CORE1 processes 65 to 129 feature points, DSP CORE2 processes 130 to 194 feature points, DSP CORE3 processes 195 to 259 feature points, and CPU processes 260 to 280 feature points.
  • performing the optical flow calculation in step S3, as shown in FIG. 4, specifically includes:
  • Step S31 Each processor calculates the interpolation sub-pixels of each feature point assigned to the current pyramid layer of the current frame image and the preset window.
  • the embodiment of the present invention takes a DSP as an example for description.
  • the feature points allocated by the DSP to the current pyramid layer of the current frame image are (p 1 , p 2 ,..., p 3 ), with the feature point coordinates as the center, the DSP uses DMA loads the first two feature points p 1 (x 1 , y 1 ) and p 2 (x 2 , y 2 ) in the reference image to specify the window size (xW x /2, yW y /2, x+W x /2, In y+W y /2), the commonly used window size in optical flow algorithm is 5 ⁇ 5, 7 ⁇ 7, 13 ⁇ 13, 21 ⁇ 21, and the register with DSP width of 512bit can load 32 16bit pixel data, so Each time one line of pixel data in the window is loaded into the DSP register, if there are less than 32 data, it will be filled; 2 lines of pixel data in the window are continuously loaded, using the DSP instruction set to directly operate on the wide register,
  • Step S32 Calculate the first derivative in the first direction, the first derivative in the second direction, and the spatial gradient matrix of all pixels in the preset window, where the first direction and the second direction are perpendicular to each other.
  • the first direction refers to the X direction and the second direction refers to the Y direction.
  • the following formula (1) is used to calculate the X-direction gradient and the Y-direction gradient of each pixel in the window, and the following formula (2) Calculate the spatial gradient matrix of each feature point:
  • Step S33 Iteratively calculate the interpolation sub-pixels of each feature point and the preset window on the image information of the next frame.
  • each processor After each processor completes the processing of the feature points of the current pyramid layer of the current frame image, it starts to process the feature points of the current pyramid layer of the current frame image.
  • Step S34 Calculate the pixel value error corresponding to each feature point according to the interpolation sub-pixels of each feature point in the current frame image and the previous frame image window and the preset window.
  • the pixel difference between the reference image and the corresponding point in the current image window is calculated according to the following formula (3):
  • Step S35 Obtain a residual matrix according to the pixel value error corresponding to each feature point and the first derivative in the first direction and the first derivative in the second direction of the previous frame of image.
  • the residual matrix in each feature point window is calculated according to the following formula (4):
  • Step S36 Obtain the coordinate value offset of each feature point according to the spatial gradient matrix and the residual matrix.
  • the coordinate value offset of each feature point is calculated according to the following formula (5):
  • Step S37 Determine whether the offset of each feature point coordinate value is smaller than a preset threshold.
  • the preset threshold value can be adaptively set according to actual engineering needs.
  • Step S38 When the offset of the coordinate value of each feature point is less than the preset threshold, the coordinate of each feature point is updated, and the calculation of the next pyramid level is performed.
  • each element of the gradient matrix and the residual matrix is 32 bits, so the 4 elements of the spatial gradient matrix of every 16 feature points are loaded into 4 DSP vector registers, corresponding to the 2 elements of the residual matrix of the feature points Loaded to 2 DSP vector registers respectively, according to matrix multiplication, the coordinates of 16 feature points can be updated at a time, and the updated feature point coordinates are:
  • the method further includes:
  • Step S39 Determine whether the current pyramid level is the original resolution level, and end the optical flow calculation when it is the original resolution level.
  • the original resolution layer is the last processed layer of the pyramid.
  • the current pyramid layer is the original resolution layer.
  • each feature point of each pyramid layer of the current frame image is different from the previous frame image.
  • the feature points are all calculated by optical flow, and the target tracking process is completed.
  • the original image data for target tracking is acquired through the CPU, and the image pyramid is established according to the original image data, and the target tracking of each resolution layer in the image pyramid is extracted Feature points, according to the upper limit of data processed by the processor and the number of feature points to be tracked, determine the number of processors required, and assign the feature points to be tracked to multiple processors for optical flow calculation.
  • the embodiment of the present invention uses The multi-core wide vector register DSP has the advantages of parallel operation, combined with the independence of feature point calculation for load balancing, and optimizes the pyramid optical flow realization process under the premise of meeting the accuracy requirements, so that the overall operating efficiency of the system is high and the real-time performance is improved.
  • An embodiment of the present invention also provides a computer-readable storage medium, and computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions can execute any of the foregoing methods in Embodiment 1 of the multi-processor target tracking processing method .
  • the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviation: HDD) or solid-state drive (Solid-State Drive, SSD), etc.; the storage medium may also include a combination of the foregoing types of memories.
  • the embodiments of the present invention can be provided as a method or a computer program product. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
  • a computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

本发明公开了一种基于多处理器的目标跟踪处理方法，通过CPU获取需进行目标跟踪的原始图像数据，并根据原始图像数据建立图像金字塔，并提取图像金字塔中各个分辨率层的需跟踪的特征点，根据处理器处理数据的上限量及需跟踪的特征点的数量，确定所需处理器的数量，将需跟踪的特征点分配给多个处理器进行光流计算，本发明实施例利用多核宽向量寄存器DSP的并行运算优势，结合特征点计算独立性进行负载均衡，在满足精度要求前提下，优化金字塔光流实现过程，使得系统整体运行效率高，提高实时性。

Description

一种基于多处理器的目标跟踪处理方法
本申请要求于2019年4月23日提交中国专利局、申请号为201910329474.3、申请名称为“一种基于多处理器的目标跟踪处理方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据处理技术领域，具体涉及一种基于多处理器的目标跟踪处理方法。
背景技术
运动目标的检测、跟踪是数字图像处理技术的一个重要部分。人们在运动目标的检测和跟踪上不断开发新的算法，但是由于算法的计算量上和硬件平台的处理速度的约束，使得运动目标跟踪的两个关键性能指标：实时性与准确度的平衡难以掌握。为了实现运动目标跟踪系统中的实时性要求，则需要采取一些措施来提高算法的运算处理速度，一是在算法上进行优化，二是在硬件系统上进行改善。目前运动目标跟踪的算法很多，各自有各自的优点，金字塔光流算法借助对原始图像进行下采样提高了图像大运动的匹配精度，但由于金字塔光流资源需求较大，算法较复杂，处理效率较低，在现有嵌入式平台的CPU上实现多路同时跟踪，很难达到实时性的要求。
发明内容
因此，本发明提供一种基于多处理器的目标跟踪处理方法，克服了现有技术中现有嵌入式平台的CPU上实现多路同时跟踪时实时性差的不足。
本发明实施例提供一种基于多处理器的目标跟踪处理方法，包括：获取需进行目标跟踪的原始图像数据；根据所述原始图像数据建立图像金字塔，并提取所述图像金字塔中各个分辨率层的需跟踪的特征点；根据处理器处理数据的上限量及所述需跟踪的特征点的数量，确定所需处理器的数量；根据所述所需处理器的数量将所述需跟踪的特征点分配给多个处理器进行光流计算。
在一实施例中，所述多个处理器包括CPU及多个DSP，所述根据处理器处理数据的上限量及所述需跟踪的特征点的数量，确定所需处理器的数量的步骤，包括：获取所述图像金字塔各个分辨率层的需跟踪的特征点的总数量；获取每个DSP处理数据的上限量；根据需跟踪的特征点数量及每个DSP处理数据的上限量，确定所需DSP的数量及CPU处理需跟踪的特征点的数据量。
在一实施例中，所述根据需跟踪的特征点数量及每个DSP处理数据的上限 量，确定所需DSP的数量及CPU处理需跟踪的特征点的数据量的步骤，包括：将所述需跟踪的特征点的总数量除以每个DSP处理数据的上限量，获取商及余数，判断余数是否为0；当余数为0，所需DSP数量为商的数量，CPU处理需跟踪的特征点的数据量为0；当余数不为0时，所需DSP数量为商加一的数量，CPU处理需跟踪的特征点的数据量为0。
在一实施例中，所述根据需跟踪的特征点数量及每个DSP处理数据的上限量，确定所需DSP的数量及CPU处理需跟踪的特征点的数据量的步骤，包括：将所述需跟踪的特征点的总数量除以每个DSP处理数据的上限量，获取商及余数，判断商是否大于DSP的实际数量；当商大于DSP的实际数量时，CPU处理需跟踪的特征点数据量为DSP处理需跟踪的特征点数据量之外的数据量。
在一实施例中，当商小于DSP的实际数量时且余数为0时，所需DSP的数量商的数量；当商小于DSP的实际数量时且余数不为0时，所需DSP的数量为商的数量加一。
在一实施例中，当商等于DSP的实际数量时，所需DSP的数量为实际数量。
在一实施例中，从分辨率最小的金字塔层的特征点开始分配给多个处理器进行光流计算。
在一实施例中，所述根据所述所需处理器的数量将所述需跟踪的特征点分配给多个处理器进行光流计算的步骤，包括：每个处理器计算分配到当前帧图像当前金字塔层的每个特征点与预设窗口的插值子像素；计算所述预设窗口所有像素第一方向的一阶导数、第二方向的一阶导数以及空间梯度矩阵，所述第一方向、第二方向相互垂直；对下一帧图像信息进行迭代计算每个特征点与预设窗口的插值子像素；根据当前帧图像与上一帧图像窗口内每个特征点与预设窗口的插值子像素，计算每个特征点对应的像素值误差；根据所述每个特征点对应的像素值误差及上一帧图像第一方向的一阶导数和第二方向的一阶导数获取残差矩阵；根据所述空间梯度矩阵和所述残差矩阵获取每个特征点坐标值偏移量；判断每个特征点坐标值偏移量是否小于预设阈值；当所述每个特征点坐标值偏移量小于预设阈值时，更新每个特征点坐标，进行下一金字塔层的计算。
在一实施例中，所述当所述每个特征点坐标值偏移量小于预设阈值时的步骤之后，还包括：判断当前金字塔层数是否为原始分辨率层，当为原始分辨率层时结束光流计算。
第二方面，本发明实施例提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使所述计算机执行本发明实施例第一方面提供的基于多处理器的目标跟踪处理方法。
本发明技术方案，具有如下优点：
本发明提供的基于多处理器的目标跟踪处理方法，通过CPU获取需进行目标跟踪的原始图像数据，并根据原始图像数据建立图像金字塔，并提取图像金字塔中各个分辨率层的需跟踪的特征点，根据处理器处理数据的上限量及需跟 踪的特征点的数量，确定所需处理器的数量，根据所需处理器的数量将需跟踪的特征点分配给多个处理器进行光流计算，本发明实施例利用多核宽向量寄存器DSP的并行运算优势，结合特征点计算独立性进行负载均衡，在满足精度要求前提下，优化金字塔光流实现过程，使得系统整体运行效率高，提高实时性。
附图说明
为了更清楚地说明本发明具体实施方式或现有技术中的技术方案，下面将对具体实施方式或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明实施例提供的基于多处理器的目标跟踪处理方法一个具体示例的流程图；
图2为本发明实施例提供的CPU与DSP硬件连接的框架示意图；
图3为本发明实施例提供的确定处理器的数量一个具体示例的流程图；
图4为本发明实施例提供的各处理器进行光流计算的一个具体示例的流程图。
具体实施方式
下面将结合附图对本发明的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
在本发明的描述中，需要说明的是，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或一体地连接；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，还可以是两个元件内部的连通，可以是无线连接，也可以是有线连接。对于本领域的普通技术人员而言，可以具体情况理解上述术语在本发明中的具体含义。
此外，下面所描述的本发明不同实施方式中所涉及的技术特征只要彼此之间未构成冲突就可以相互结合。
实施例1
本发明实施例提供一种基于多处理器的目标跟踪处理方法，如图1所示，该基于多处理器的目标跟踪处理方法，包括如下步骤：
步骤S1:获取需进行目标跟踪的原始图像数据，并根据原始图像数据建立图像金字塔，并提取图像金字塔中各个分辨率层的需跟踪的特征点。
本发明实施例中，通过CPU获取外部需要进行目标跟踪的原始图像数据， 并对原始图像数据建立图像金字塔及提前各个分辨率层需跟踪的特征点。在一可选的实施例中，CPU在获取原始图像后进行滤波除噪等预处理后，再建立图像金字塔。
步骤S2:根据处理器处理数据的上限量及需跟踪的特征点的数量，确定所需处理器的数量。
本发明实施例中，在CPU侧进行负载均衡，根据当前需要追踪的特征点数和DSP(digital signal processor，数字信号处理器)宽向量寄存器一次最多处理数据量，分配不同的特征点计算任务给每一个DSP，当各个DSP处理的数据量不能满足当前需要追踪的特征点数时，CPU处理各个DSP不能处理的数据量。
在一具体实施例中，如图2所示，为CPU与DSP的硬件连接关系，以一个DSP为例，DSP包括：DSP CORE、IRAM(Instruction RAM DSP，存放指令集的内存)、DRAM(Dynamic Random Access Memory，动态随机存取存储器)、及缓冲存储器ICache(在调试DSP指令时，会有数据存入到缓冲存储器ICache)，首先由CPU加载DSP的程序到IRAM、DRAM和DDR里使其运行起来，，然后CPU通过中断通知DSP有任务要处理，最后DSP处理完后再用中断通知CPU任务完成的状态。芯片中内置的IDMA负责将图像数据从DDR导入到DRAM，DSP CORE从DRAM获取数据进行处理，处理结果写回DRAM，再用IDMA将数据导出到DDR。
在本发明实施例中可以采用海思Hi3559CV100SOC，该芯片提供了高效且丰富的计算资源，包括4个Vision DSP(图2中仅以一个DSP内部结构及与CPU连接关系的作为举例)。该系列DSP是给视觉处理加速的专用处理器，具有可编程的能力，基于Vision DSP既可以开发基础运算功能，也可以实现智能分析的复杂算法。
步骤S3:根据所需处理器的数量将需跟踪的特征点分配给多个处理器进行光流计算。
本发明实施例中，每个DSP CORE和CPU加载相应的需要追踪的特征点数，从分辨率最小的金字塔层的特征点开始分配给多个处理器进行光流计算。
在本发明实施中，多个处理器包括CPU及多个DSP，如图3所示，执行步骤S2的过程具体包括：
步骤S21：获取图像金字塔各个分辨率层的需跟踪的特征点的总数量。
步骤S22：获取每个DSP处理数据的上限量。
本发明实施例中，采用的Vision DSP具备512Bit宽度的向量寄存器，单个指令周期可以执行32个16Bit的数据或者16个32Bit的数据运算，仅以此举例，本发明并不以此为限。
步骤S23：根据需跟踪的特征点数量及每个DSP处理数据的上限量，确定所需DSP的数量及CPU处理需跟踪的特征点的数据量。
在一具体实施例中，只利用多个DSP对需要需跟踪的特征点的总数量进行处理，将需跟踪的特征点的总数量除以每个DSP处理数据的上限量，获取商及余数，根据商和余数来确定DSP的数量。具体包括：将需跟踪的特征点的总数量除以每个DSP处理数据的上限量，获取商及余数，判断余数是否为0；当余数为0时，所需DSP数量为商的数量；当余数不为0时，所需DSP数量为商加一的数量。例如，当前需要追踪280个特征点，每个DSP处理的上限量为70个，那么需要4个DSP就可以完成处理；当每个DSP处理的上限量为65个时，需要5个DSP才可以完成处理。
在另一具体实施例中，DSP实际数量的已知(例如上述的海思处理芯片包括4个Vision DSP)，根据需跟踪的特征点的总数量、DSP的实际数量及每个DSP处理数据的上限量，来确定每个DSP进行需跟踪的特征点计算的数量以及是否需要CPU进行需跟踪的特征点运算。具体包括：将所述需跟踪的特征点的总数量除以每个DSP处理数据的上限量，获取商及余数，判断商是否大于DSP的实际数量；当商大于DSP的实际数量时，说明需处理的特征点的数量大于实际数量的DSP所能处理的数量，此时需要CPU协助处理部分特征点数据，CPU处理需跟踪的特征点数据量为实际数量的DSP所能处理的需跟踪的特征点数据量之外的数据量。当商小于DSP的实际数量时且余数为0时，所需DSP的数量即为商的数量；当商小于DSP的实际数量时且余数不为0时，所需DSP的数量为商的数量加一。当商等于DSP的实际数量时，所需DSP的数量为实际数量。例如当前需要追踪280个特征点，则DSP CORE0处理1～64特征点，DSP CORE1处理65～129特征点，DSP CORE2处理130～194特征点，DSP CORE3处理195～259特征点，CPU处理260～280特征点。
在一实施例中，执行步骤S3中光流计算，如图4所示，具体包括：
步骤S31：每个处理器计算分配到当前帧图像当前金字塔层的每个特征点与预设窗口的插值子像素。
本发明实施例以一个DSP为例进行说明，例如，该DSP分配到当前帧图像当前金字塔层的特征点为(p 1,p 2,…,p 3)，以特征点坐标为中心，DSP利用DMA加载参考图像中前两个特征点p 1(x 1,y 1)和p 2(x 2,y 2)指定窗口大小(x-W x/2,y-W y/2,x+W x/2,y+W y/2)内，光流算法中常用窗口大小为5×5,7×7,13×13,21×21，而DSP宽度为512bit的寄存器可以加载32个16bit的像素数据，所以每次加载窗口内的一行像素数据到DSP寄存器中，不足32个数据则补齐；连续加载窗口内2行像素数据，使用DSP指令集对宽寄存器直 接操作，按照线性插值的方法计算像素插值，直到窗口内所有像素插值计算完成。
步骤S32：计算预设窗口所有像素第一方向的一阶导数、第二方向的一阶导数以及空间梯度矩阵，所述第一方向、第二方向相互垂直。
本发明实施例中，第一方向是指X方向、第二方向是指Y方向，利用以下公式(1)计算窗口内每个像素的X方向梯度和Y方向梯度，及利用以下公式(2)计算每个特征点的空间梯度矩阵：
Figure PCTCN2020086124-appb-000001
Figure PCTCN2020086124-appb-000002
步骤S33：对下一帧图像信息进行迭代计算每个特征点与预设窗口的插值子像素。
本发明实施例中，各个处理器完成当前帧图像当前金字塔层的特征点的处理后，开始对当下一帧图像当前金字塔层的特征点进行处理。
步骤S34：根据当前帧图像与上一帧图像窗口内每个特征点与预设窗口的插值子像素，计算每个特征点对应的像素值误差。
本发明实施例中，根据以下公式(3)计算参考图像和当前图像窗口内对应点的像素差：
δI k(x,y)＝I(x,y)-J(x+Δx,y+Δy)    (3)，
步骤S35：根据每个特征点对应的像素值误差及上一帧图像第一方向的一阶导数和第二方向的一阶导数获取残差矩阵。
本发明实施中，根据以下公式(4)计算每个特征点窗口内的残差矩阵：
Figure PCTCN2020086124-appb-000003
步骤S36：根据空间梯度矩阵和残差矩阵获取每个特征点坐标值偏移量。
本发明实施例中，根据以下公式(5)计算每个特征点坐标值偏移量：
Figure PCTCN2020086124-appb-000004
步骤S37：判断每个特征点坐标值偏移量是否小于预设阈值。
本发明实施例中，预设阈值可根据实际工程需要做适应设置。
步骤S38：当每个特征点坐标值偏移量小于预设阈值时，更新每个特征点坐标，进行下一金字塔层的计算。
本发明实施例中，梯度矩阵和残差矩阵每个元素为32bit，所以每16个特征点的空间梯度矩阵4个元素分别加载到4个DSP向量寄存器，对应特征点的残差矩阵2个元素分别加载到2个DSP向量寄存器，按照矩阵乘法，可以一次更新16个特征点的坐标，更新特征点坐标为：
Figure PCTCN2020086124-appb-000005
在一较佳实施例中，当每个特征点坐标值偏移量小于预设阈值时的步骤之后，还包括：
步骤S39：判断当前金字塔层数是否为原始分辨率层，当为原始分辨率层时结束光流计算。
本发明实施例中，原始分辨率层为金字塔最后处理的一层，当判断当前金字塔层数是否为原始分辨率层时，说明当前帧图像个金字塔层的的各个特征点与上一帧图像的特征点都进行了光流计算，完成了目标跟踪的处理。
本发明实施例提供的基于多处理器的目标跟踪处理方法，通过CPU获取需进行目标跟踪的原始图像数据，并根据原始图像数据建立图像金字塔，并提取图像金字塔中各个分辨率层的需跟踪的特征点，根据处理器处理数据的上限量及需跟踪的特征点的数量，确定所需处理器的数量，将需跟踪的特征点分配给多个处理器进行光流计算，本发明实施例利用多核宽向量寄存器DSP的并行运 算优势，结合特征点计算独立性进行负载均衡，在满足精度要求前提下，优化金字塔光流实现过程，使得系统整体运行效率高，提高实时性。
实施例2
本发明实施例还提供一种计算机可读存储介质，计算机可读存储介质上存储有计算机可执行指令，该计算机可执行指令可执行上述任意方法实施例1中的多处理器的目标跟踪处理方法。其中，所述存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)、随机存储记忆体(Random Access Memory，RAM)、快闪存储器(Flash Memory)、硬盘(Hard Disk Drive，缩写：HDD)或固态硬盘(Solid-State Drive，SSD)等；所述存储介质还可以包括上述种类的存储器的组合。
本领域内的技术人员应明白，本发明的实施例可提供为方法或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
显然，上述实施例仅仅是为清楚地说明所作的举例，而并非对实施方式的限定。对于所属领域的普通技术人员来说，在上述说明的基础上还可以做出其它不同形式的变化或变动。这里无需也无法对所有的实施方式予以穷举。而由此所引申出的显而易见的变化或变动仍处于本发明创造的保护范围之中。

Claims (10)

  1. 一种基于多处理器的目标跟踪处理方法，其特征在于，包括：
    获取需进行目标跟踪的原始图像数据，并根据所述原始图像数据建立图像金字塔，并提取所述图像金字塔中各个分辨率层的需跟踪的特征点；
    根据处理器处理数据的上限量及所述需跟踪的特征点的数量，确定所需处理器的数量；
    根据所述所需处理器的数量将所述需跟踪的特征点分配给多个处理器进行光流计算。
  2. 根据权利要求1所述的基于多处理器的目标跟踪处理方法，其特征在于，所述多个处理器包括CPU及多个DSP，所述根据处理器处理数据的上限量及所述需跟踪的特征点的数量，确定所需处理器的数量的步骤，包括：
    获取所述图像金字塔各个分辨率层的需跟踪的特征点的总数量；
    获取每个DSP处理数据的上限量；
    根据需跟踪的特征点数量及每个DSP处理数据的上限量，确定所需DSP的数量及CPU处理需跟踪的特征点的数据量。
  3. 根据权利要求2所述的基于多处理器的目标跟踪处理方法，其特征在于，所述根据需跟踪的特征点数量及每个DSP处理数据的上限量，确定所需DSP的数量及CPU处理需跟踪的特征点的数据量的步骤，包括：
    将所述需跟踪的特征点的总数量除以每个DSP处理数据的上限量，获取商及余数，判断余数是否为0；
    当余数为0，所需DSP数量为商的数量，CPU处理需跟踪的特征点的数据量为0；
    当余数不为0时，所需DSP数量为商加一的数量，CPU处理需跟踪的特征点的数据量为0。
  4. 根据权利要求2所述的基于多处理器的目标跟踪处理方法，其特征在于，所述根据需跟踪的特征点数量及每个DSP处理数据的上限量，确定所需DSP的数量及CPU处理需跟踪的特征点的数据量的步骤，包括：
    将所述需跟踪的特征点的总数量除以每个DSP处理数据的上限量，获取商及余数，判断商是否大于DSP的实际数量；
    当商大于DSP的实际数量时，CPU处理需跟踪的特征点数据量为DSP处理需跟踪的特征点数据量之外的数据量。
  5. 根据权利要求4所述的基于多处理器的目标跟踪处理方法，其特征在于，
    当商小于DSP的实际数量时且余数为0时，所需DSP的数量商的数量；
    当商小于DSP的实际数量时且余数不为0时，所需DSP的数量为商的数量加一。
  6. 根据权利要求1所述的基于多处理器的目标跟踪处理方法，其特征在 于，
    当商等于DSP的实际数量时，所需DSP的数量为实际数量。
  7. 根据权利要求1所述的基于多处理器的目标跟踪处理方法，其特征在于，从分辨率最小的金字塔层的特征点开始分配给多个处理器进行光流计算。
  8. 根据权利要求1所述的基于多处理器的目标跟踪处理方法，其特征在于，所述根据所述所需处理器的数量将所述需跟踪的特征点分配给多个处理器进行光流计算的步骤，包括：
    每个处理器计算分配到当前帧图像当前金字塔层的每个特征点与预设窗口的插值子像素；
    计算所述预设窗口所有像素第一方向的一阶导数、第二方向的一阶导数以及空间梯度矩阵，所述第一方向、第二方向相互垂直；
    对下一帧图像信息进行迭代计算每个特征点与预设窗口的插值子像素；
    根据当前帧图像与上一帧图像窗口内每个特征点与预设窗口的插值子像素，计算每个特征点对应的像素值误差；
    根据所述每个特征点对应的像素值误差及上一帧图像第一方向的一阶导数和第二方向的一阶导数获取残差矩阵；
    根据所述空间梯度矩阵和所述残差矩阵获取每个特征点坐标值偏移量；
    判断每个特征点坐标值偏移量是否小于预设阈值；
    当所述每个特征点坐标值偏移量小于预设阈值时，更新每个特征点坐标，进行下一金字塔层的计算。
  9. 根据权利要求8所述的基于多处理器的目标跟踪处理方法，其特征在于，所述当所述每个特征点坐标值偏移量小于预设阈值时的步骤之后，还包括：
    判断当前金字塔层数是否为原始分辨率层，当为原始分辨率层时结束光流计算。
  10. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使所述计算机执行如权利要求1-9中任一项所述的基于多处理器的目标跟踪处理方法。
PCT/CN2020/086124 2019-04-23 2020-04-22 一种基于多处理器的目标跟踪处理方法 Ceased WO2020216249A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910329474.3A CN110120063A (zh) 2019-04-23 2019-04-23 一种基于多处理器的目标跟踪处理方法
CN201910329474.3 2019-04-23

Publications (1)

Publication Number Publication Date
WO2020216249A1 true WO2020216249A1 (zh) 2020-10-29

Family

ID=67521303

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/086124 Ceased WO2020216249A1 (zh) 2019-04-23 2020-04-22 一种基于多处理器的目标跟踪处理方法

Country Status (2)

Country Link
CN (1) CN110120063A (zh)
WO (1) WO2020216249A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110120063A (zh) * 2019-04-23 2019-08-13 深圳市道通智能航空技术有限公司 一种基于多处理器的目标跟踪处理方法
CN112842257B (zh) * 2019-11-12 2022-12-06 磅客策(上海)机器人有限公司 一种血管定位方法及装置
CN113079336A (zh) * 2020-01-03 2021-07-06 深圳市春盛海科技有限公司 高速影像的录像方法及装置
CN112991386A (zh) * 2021-02-20 2021-06-18 浙江欣奕华智能科技有限公司 光流跟踪装置
CN116848509A (zh) * 2021-12-31 2023-10-03 华为技术有限公司 一种计算任务处理装置、方法及电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101923718A (zh) * 2009-06-12 2010-12-22 新奥特（北京）视频技术有限公司 基于粒子滤波和光流矢量的视频目标跟踪方法的优化方法
CN103177455A (zh) * 2013-03-20 2013-06-26 南京理工大学 基于多核dsp的klt运动目标跟踪算法的实现方法
US20160063328A1 (en) * 2008-05-09 2016-03-03 Intuvision Inc. Video tracking systems and methods employing cognitive vision
CN107358938A (zh) * 2017-08-14 2017-11-17 绍兴市上虞东虞塑料电器有限公司 一种可调变调夹
CN110120063A (zh) * 2019-04-23 2019-08-13 深圳市道通智能航空技术有限公司 一种基于多处理器的目标跟踪处理方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6192156B1 (en) * 1998-04-03 2001-02-20 Synapix, Inc. Feature tracking using a dense feature array
CN102202216A (zh) * 2011-05-13 2011-09-28 南京信息工程大学 一种鼠虫监控装置及采用其进行图像跟踪识别的方法
CN102523374B (zh) * 2011-12-19 2014-02-19 北京理工大学 一种实时并行的电子稳像系统设计方法
CN104317768B (zh) * 2014-10-15 2017-02-15 中国人民解放军国防科学技术大学 面向cpu+dsp异构系统的矩阵乘加速方法
CN106599918B (zh) * 2016-12-13 2019-12-17 开易（深圳）科技有限公司 车辆追踪方法及系统
CN107358238A (zh) * 2017-03-01 2017-11-17 郑州云海信息技术有限公司 一种提取图像特征信息的方法及系统
WO2019000396A1 (zh) * 2017-06-30 2019-01-03 深圳市大疆创新科技有限公司 光流跟踪装置和方法
CN109509212B (zh) * 2018-09-30 2023-11-24 惠州市德赛西威汽车电子股份有限公司 目标跟踪处理方法、电子设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160063328A1 (en) * 2008-05-09 2016-03-03 Intuvision Inc. Video tracking systems and methods employing cognitive vision
CN101923718A (zh) * 2009-06-12 2010-12-22 新奥特（北京）视频技术有限公司 基于粒子滤波和光流矢量的视频目标跟踪方法的优化方法
CN103177455A (zh) * 2013-03-20 2013-06-26 南京理工大学 基于多核dsp的klt运动目标跟踪算法的实现方法
CN107358938A (zh) * 2017-08-14 2017-11-17 绍兴市上虞东虞塑料电器有限公司 一种可调变调夹
CN110120063A (zh) * 2019-04-23 2019-08-13 深圳市道通智能航空技术有限公司 一种基于多处理器的目标跟踪处理方法

Also Published As

Publication number Publication date
CN110120063A (zh) 2019-08-13

Similar Documents

Publication Publication Date Title
WO2020216249A1 (zh) 一种基于多处理器的目标跟踪处理方法
AU2017232186B2 (en) Fast and robust image alignment for burst mode
US11023206B2 (en) Dot product calculators and methods of operating the same
Shen et al. Fastvggt: Training-free acceleration of visual geometry transformer
CN103020620B (zh) 基于cpu和gpu协同处理的遥感影像正射校正方法
WO2021163928A1 (zh) 光流获取方法和装置
CN107358238A (zh) 一种提取图像特征信息的方法及系统
CN107657599A (zh) 基于混合粒度划分和动态负载分配的遥感图像融合系统并行实现方法
CN111797996A (zh) 样本补全方法、装置、设备及可读存储介质
WO2020125062A1 (zh) 一种图像融合方法及相关装置
WO2024027039A1 (zh) 数据处理方法、装置、设备和可读存储介质
US10152310B2 (en) Fusing a sequence of operations through subdividing
CN112634366B (zh) 位置信息的生成方法、相关装置及计算机程序产品
CN116593504B (zh) Ct成像方法、装置、设备及存储介质
WO2020062546A1 (zh) 目标跟踪处理方法、电子设备
CN112052861B (zh) 深度卷积神经网络有效感受野的计算方法及存储介质
WO2017210868A1 (zh) 数据处理方法、装置及系统
CN110648356A (zh) 一种基于视觉数字信号处理的多目标跟踪处理优化方法
WO2023109086A1 (zh) 文字识别方法、装置、设备及存储介质
CN114217955A (zh) 一种数据处理方法、设备及计算机可读存储介质
CN107292876B (zh) 基于岩土材料变形时空非均匀特征的dscm分析方法
CN105022889B (zh) 一种基于视频的快速模板匹配gpu实现方法
CN116052233A (zh) 一种神经网络优化方法、装置、计算设备及存储介质
GB2639729A (en) Interleaved processing of template data and search data according to a search window
CN115375740B (zh) 位姿确定方法和三维模型的生成方法、装置、设备、介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20794546

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20794546

Country of ref document: EP

Kind code of ref document: A1