CN205486304U - Portable realtime graphic object detection of low -power consumption and tracking means - Google Patents

Abstract

The utility model provides a portable realtime graphic object detection of low -power consumption and tracking means can be from master control cell culture process and online observation under space and equal adverse circumstances. Does wherein the image acquisition unit include camera interface module, image preprocessing module, AXI including PL subsystem and PS subsystem the DMA module, the camera interface module is used for the parameter control to camera data reading and image resolution, collection frame per second, data format, is the image preprocessing module used for flowing the preliminary treatment operation of colour correction, histogram equalization, space transform, AXI to of collection image is the DMA module used for through AXI the HP interface is realized the autonomic passback of image data to DDR3 RAM, does image accelerator module include restructural image accelerator, AXI the DMA module, restructural image accelerator is used for according to image object detection of the work phase dynamic loading of system or image target following accelerator, AXI is the DMA module used for through AXI the ACP interface is realized the autonomic passback of image data to DDR3 RAM.

Description

Low Power Consumption Portable real time imaging object detecting and tracking device

Technical field

This utility model belongs to image object detection and tracking technique field, relates to a kind of Low Power Consumption Portable real time imaging Object detecting and tracking device.

Background technology

Image object detection and tracking are the important component parts of computer vision technique, mainly by extracting target object Color, profile, texture, the feature such as kinematic parameter, it is achieved it is identified, mates or position.Secondary light source is for target The region of feature carries out visible or infrared light irradiation, to obtain more visible target image；Video camera realizes target image Collection；Image acquisition transmission circuit is used for reading, transmitting video camera acquired image data；Image processing equipment comprises meter The plurality of optional forms such as calculation machine, embedded system, are mainly used in disposing and operation image enhancing, target following scheduling algorithm.

Image processing equipment be image object detection with follow the tracks of system core component, its by Camera Link, The EBIs such as LVDS, USB, IEEE 1394, control the collection transmission of view data, then by built-in firmware, software or Application specific hardware modules, processes view data, completes the tasks such as image enhaucament, target detection, target following.Owing to target characteristic carries Take/model, sample matches/classification, the process such as tracking parameter calculating/renewal are generally of higher computation complexity, for protecting On the premise of barrier real-time performance of tracking, providing broader algorithm improvement space as far as possible, real-time and data processing rate are images The major design of processing equipment or type selecting index.

At present, the detection of major part image object mainly relies on graphics workstation or high-performance calculation mechanism with tracking system Build, use MicrosoftVisual Studio and OpenCV to carry out algorithm software design, and using 50ms/ frame as defining system Whether possesses the rate-valve value of real-time.As general processor, the multiplication ability of graphics workstation or high-performance computer CPU The most limited, research worker generally uses the general-purpose computing system framework of CPU PCIe GPU, with the isomery of GPU aiding CPU Operational pattern, it is achieved the Accelerating running to image object detect and track algorithm.Such as 2014, geology mineral portion of France J.Gance etc. devise mobile-object tracking system based on surface DSM, and landslide is detected early warning automatically, use Gigabit Ethernet gathers view data, and its central computer is the graphics workstation being equipped with Intel XeonE5-1607 processor. 2015, Middle East University of Science and Technology of Turkey Ilker etc. used the CPU PCIe GPU isomery computing framework of general purpose computer, it is achieved High operand, the TLD algorithm of long time-tracking, when 1920 × 1080 image in different resolution are processed, can realize relative to The acceleration that pure software scheme is 10.8 times.Use graphics workstation or high-performance computer as image processing equipment, can effectively protect Barrier image object detection and the real-time demand of the system of tracking, have the advantage that design is easy, software and hardware extensibility is strong.So And, such solution system construction cost and volume power consumption are higher, are only applicable to carry out the indoor application field of wired connection Scape.Additionally, the NVIDIA series GPU program compatibility of occuping market main body is poor, limit CPU PCIe GPU isomery computing The portability of program under framework.

Since the nineties in 20th century, the calculated performance of all kinds of embedded processing devices and collection rise the most rapidly, and It is widely used at image procossing and computer vision field.As shown in table 1, for the image mesh of small-sized movable platform Mark detection with follow the tracks of application demand, the embedded image processing equipment solution of main flow, mainly include use DSP, FPGA, The single embedded processing devices such as ARM (Advanced RISC Machines), and DSP FPGA, ARM DSP, SoC etc. are different Structure embedded processing systems.

DSP is a class towards controlling in real time or the application specific processor of signal processing, relative to general processor, its equipped with Special multiplier or multiplicaton addition unit, it is possible to fully meet the application demand of image procossing.Image object based on DSP detection with Track system generally uses C language to develop, and realizes high rate burst communication by built-in hardware multiplier, has real-time, exploitation Motility and the program maintainability advantage such as preferably.But, the power consumption of High Performance DSP is higher, and cost performance is limited, software portable Property poor, do not support the general-purpose operating system and multi-task parallel operation.At present, image object based on DSP detection and tracking in the industry System, is mainly directed towards the simple function platform such as guided missile, monitoring device.

FPGA, by lookup table technology based on RAM (RandomAccess Memory), can realize specific function numeral The deployment able to programme of circuit.Due to the circuit parallel of its uniqueness, when the view data that process pixel quantity is huge, have very Strong innate advantage.But, FPGA exploitation technical threshold higher, and its add parallel fast mode be only applicable to context data without Association, flow process better simply intensive operations link, therefore, it is difficult to use it to realize complicated exquisite high robust image object inspection Survey and track algorithm.At present, image object based on FPGA detection and the system of tracking, be based primarily upon flow process simple early stage in the industry Algorithm, the robustness of detect and track, accuracy are poor.

ARM, as a class low-power consumption, general processor, can carry the embedded OSs such as Linux, WinCE, in exploitation The aspects such as motility, program maintainability, software portability, multi-task parallel operation have significant advantage.But, its floating-point Computing and multiplying ability are poor, it is impossible to fully ensure the real-time of image object detection and the system of tracking.At present, base in the industry Image object in ARM detects and the system of tracking, is based primarily upon the simple algorithm that the amount of calculation such as color, texture is relatively low, detection and Robustness, the accuracy followed the tracks of are poor.

Owing to the embedded processing devices such as DSP, FPGA, ARM are respectively arranged with pluses and minuses, for fully meet image object detection with Track system in the demand of the aspects such as real-time, accuracy, power consumption, extensibility, and equilibrium development difficulty, cost, reliability, can The factors such as autgmentability.In recent years, occur in that the most successively based on the isomery such as DSP FPGA, ARM DSP, SoC embedded processing system Image object detection and the system of tracking of system.Comprise the dissimilar processing apparatus of two or more in heterogeneous processing system, pass through Task resolution, according to subtask feature, calls particular procedure device and processes, to merge the advantage of various processor part.But It is that DSP FPGA and ARM DSP heterogeneous processing system use sheet external bus to carry out the data interaction between two processing apparatus, its Traffic rate is relatively low, less stable, and overall engineering practical value is relatively low.SoC can be integrated on monolithic integrated circuit chip The dissimilar processing apparatus of two or more, and realize the mutual and allotment between processing apparatus by bus on chip, relative to DSP FPGA and ARM dsp system, it has outstanding advantage at aspects such as volume power consumption, stability, traffic rates.But, existing The most SoC processors having use does not has master-slave between double-core structure, i.e. processing apparatus, need to coordinate in a program Task between processing apparatus is distributed, is ensured its concurrent working, just can play the maximum treatment efficiency of SoC, system configuration and journey Sequence design difficulty is higher.

2011, Xilinx company was proposed full Programmable Technology Zynq-7000SoC of support.Including with ARM Cortex- PS (Processing System) subsystem centered by A9 general processor and the PL centered by FPGA (Programmable Logic) subsystem.Relative to existing SoC product, it is can directly run the PS subsystem of operating system System is for main side, it is achieved to the configuration of PL, programs, reconstruct；Peripheral hardware able to programme with PL subsystem as arm processor, can be used for structure Build computing accelerating module, image coprocessor or data/control interface.Particular, it is important that PS subsystem and PL subsystem it Between can pass through AXI (Advanced eXtensible Interface) bus, through interfaces such as AXI-GP, AXI-HP, AXI-ACP Realize the high-speed data coupling of 400-800MB/s.The primary and secondary structure of Zynq-7000SoC avoids appointing of double-core structure SoC Business scheduling and a coordination difficult problem, in its sheet, high speed AXI EBI ensure that the stability of system, high efficiency, and it is based on general embedding The PS subsystem entering formula processor and standard operation system ensure that versatility and the exploitability of system, its PL based on FPGA Subsystem can provide the parallel acceleration to intensive operations link.Meanwhile, by Xilinx Vivado HLS (High Level Synthesis) instrument, can use C/C++ language, in PL subsystem, it is achieved accelerate graphics coprocessor or image algorithm The FPGA of IP kernel quickly designs deployment, reduces the technological development threshold of system.

The versatility, stability, the real-time that are possessed in image procossing application in view of Zynq-7000SoC, can develop Property etc. advantage, from release since, achieve application in fields such as images steganalysis rapidly.2014, Institutes Of Technology Of Nanjing Wang Shuling etc. use Zynq-7000SoC design to achieve Real time face detection system, but it only used PS subsystem, system The algorithm used is relatively simple, and real-time is poor.2014, Zhuhai radio and TV university Yang Xiao peace etc. used Zynq-7000SoC to set Meter achieves high-speed image sampling and real time processing system, but it only uses PL subsystem to achieve data acquisition interface, and not Relate to image operation and process.2014, the employing Zynq-7000SoC designs such as Southern Yangtze University Wang Zhi is refined achieved dense optical flow Method, it uses the technology that software-hardware synergism processes, but is only limitted to the extraction of image Optical-flow Feature, is not directed to image object detection With the application of the high-level abstractions such as tracking.2014, the vertical treasured of the Maritime Affairs University Of Dalian etc. used Zynq-7000SoC design to achieve car Board identification system, but it only uses PL subsystem to achieve collection and the pretreatment of image, and be not directed to image object detection with The high-level abstractions application such as tracking.2015, Institutes Of Technology Of Taiyuan Jiao used the most by force etc. Zynq-7000SoC design to achieve embedding Formula digital image processing system, but it only verifies in PL subsystem that the simple image such as Sobel filter process function, and do not relate to And software-hardware synergism image procossing.

Utility model content

The purpose of this utility model is to provide a kind of Low Power Consumption Portable real time imaging object detecting and tracking device, energy Enough Autonomous Control cell cultivation process and online observations under space and equal adverse circumstances.

The technical solution realizing this utility model purpose is:

A kind of Low Power Consumption Portable real time imaging object detecting and tracking device, including PL subsystem and PS subsystem, PS AXI-ACP interface, AXI-HP interface and DMA channel is used to realize in the middle of view data and computing between subsystem and PL subsystem Data mutual；Wherein PL subsystem includes image acquisition units and graphics acceleration unit；Wherein:

Image acquisition units includes utilizing camera interface module, image pre-processing module, AXI-DMA module；Described photographic head Interface module is for photographic head digital independent and image resolution ratio, acquisition frame rate, the state modulator of data form, and image is pre- Processing module is for gathered image stream color correction, histogram equalization, the pretreatment operation of spatial alternation, AXI-DMA mould Block is for realizing view data to the autonomous passback of DDR3RAM by AXI-HP interface；

Graphics acceleration unit includes restructural image accelerator, AXI-DMA module；Restructural image accelerator is used for basis The detection of system working stage dynamic load image object or tracking image target accelerator, it is achieved the acceleration to algorithms of different； AXI-DMA module is for realizing view data to the autonomous passback of DDR3RAM by AXI-ACP interface；

AXI-DMA module in image acquisition units and graphics acceleration unit is all by AXI-Lite interconnection module and AXI- GP interface connects.

The beneficial effects of the utility model:

(1) this utility model uses the isomery computing framework that software and hardware combines, and builds figure in PL subsystem (FPGA) As parallel accelerating module, perform the algorithm link that computing is intensive；Operation image mesh in a software form in PS subsystem (ARM) Mark detection and the main body software of track algorithm；Utilize data interaction that DMA and AXI bus realizes between PS and PL subsystem and appoint Business allotment.So that this system is provided simultaneously with versatility and the exploitability of ARM-Linux embedded system, FPGA system is parallel The real-time of computing, the low-power consumption of SoC system, portability, it is possible to fully meet the low merit such as autonomous robot, smart mobile phone Consumption, the application demand of complex function mobile platform.

(2) this utility model uses C/C++ language design software, uses C++, VHDL/Verilog HDL language to carry out FPGA firmware design, it is the essence design standard of electronic information industry software and hardware respectively, therefore relative to other solution There is more excellent portability.

(3) this utility model is higher with track algorithm accelerator resource consumption for using the detection of FPGA design image object Problem, use dynamic partial reconfigurable technology, image accelerator Optimization of Word Length algorithm based on simulated annealing to carry out excellent Change, effectively reduce system resources consumption and power consumption.

Accompanying drawing explanation

Fig. 1 is image object of the present utility model detection and tracking apparatus structure block diagram；

Fig. 2 is the detection of this utility model image object and tracking device design flow diagram；

Fig. 3 is the detection of this utility model image object and tracking device workflow diagram.

Detailed description of the invention

Develop simultaneously embodiment below in conjunction with the accompanying drawings, is described in detail this utility model.

This utility model is a kind of low-power consumption, portable real time imaging object detecting and tracking system, and it is mainly directed towards The low-power consumption such as autonomous robot, smart mobile phone, the mobile platform of complex function.

Major design flow process of the present utility model is as shown in Figure 2.

The first step, for application demand, uses the advanced language programmings such as C/C++ to realize towards Zynq-7000PS subsystem Image object detection and the track algorithm software code of (ARM-Linux platform).At detection-phase, image object detection is used to calculate Method realizes the identification to interesting target；At tracking phase, tracking image target algorithm is used to realize detection gained target Consecutive tracking.

Second step, is some relatively independent links by algorithm partition, during by timer to the operation of each algorithm link Between be estimated, and analyze its characteristic such as arithmetic type, data association further.

3rd step, determines the software and hardware embodiment of each algorithm link according to second step assessment result, uses PS end software Realize algorithm main body control flow process, use the computing intensity link of PL end restructural image accelerator Parallel Implementation algorithm.

4th step, uses Vivado HLS instrument, with C Plus Plus designed image target detection/Track Accelerator IP kernel.

5th step, allows category as the resource consumption of image object detection/Track Accelerator IP kernel exceedes system, uses mould Intend annealing algorithm accelerator word length is optimized.Finally, software and firmware module being packaged and disposed, completion system sets Meter.

Image object detection/Track Accelerator IP kernel designed by this utility model, with the image of large data capacity Data, for processing object, therefore can cause higher FPGA resource consumption.For avoiding drawing because PL subsystem resource overuses Stability declines sent out, the problem such as power consumption rising, this utility model is in the 5th step employing simulated annealing pair of design cycle Image object detection/Track Accelerator is optimized.

Assuming that image object detection/Track Accelerator relates generally to the computing of n group variable, take when each variable is floating number Operation result is canonical solution.This algorithm is expected to quantization error and is less than under 1.5% premise, tries to achieve the variable floating number of shortest word length Representation w_best=[w_bestword,1,w_bestint,1,w_bestword,n,w_bestint,n], to save FPGA resource consumption.Wherein w_bestword,nIt is the total word length of the shortest fixed-point number of the n-th variable, w_bestint,nIt it is the shortest fixed-point number integer part of the n-th variable Word length.This Solve problems is also denoted as:

$\left.\begin{array}{c}\min Cost\left(w\right)\\ \begin{array}{cc}s.t.& 1-QuantizerError\left(w\right)>{\mathrm{rate}}_{thresh}\end{array}\end{array}\right\}---\left(1\right)$

Wherein Cost (w) represents that solving result is consumed resource during w, and QuantizerError (w) expression solves knot Fruit is quantization error during w, rate_thresholdFor computing accuracy rate threshold value 98.5% (i.e. 100%-1.5%).

The first step, it is temp that this algorithm will arrange initial temperature₀, and use random number array to initialize lookup center vector w_centerWith optimal solution vector w_best, and calculate computing accuracy rate rate of correspondence_centerAnd rate_best。

Second step, entrance is circulated by this algorithm, and in each cyclic process, temperature temp all can be with under ratio decayScale Drop once, until being reduced to threshold value temp_thresh, that terminates algorithm solves execution.

3rd step, this algorithm will search center vector w_centerNear randomly choose one group of solution w with Markov chain_try, and Calculate computing accuracy rate of its correspondence_try。

4th step, works as w_tryMeet computing accuracy requirement, and Cost (w_try)<Cost(w_best), then use w_tryUpdate w_best。

5th step, works as w_tryMeet computing accuracy requirement, and Cost (w_try)<Cost(w_center), then use w_tryUpdate w_center；Otherwise, with exp (-(Cost (w_try)-Cost(w_center))/temp) it is that probability uses w_tryUpdate w_center。

Such as table 1, using single Gaussian Background model is that image object detection algorithm is verified.IP kernel is carried out through algorithm 1 Optimization of Word Length, the resource consumption of image object detection accelerator IP kernel is remarkably decreased, and testing result is basically unchanged, and demonstrates this The effectiveness of algorithm.

Table 1 Low Power Consumption Portable real time imaging object detecting and tracking system Optimization of Word Length contrasts

The Low Power Consumption Portable real time imaging target detection that this utility model design realizes follows the tracks of principle of device block diagram such as Fig. 1 Shown in, it uses monolithic Zynq-7000SoC to realize the single-chip integration to system.Image detection algorithm and image tracking algorithm Main body, is deployed in PS subsystem with the form of ARM-Linux built-in system software；Image detection algorithm and image trace are calculated The computing intensity link of method, and each class interface, be deployed in PL subsystem with the form of FPGA firmware, and pass through AXI-GP Interface is called by the programme-control run in PS subsystem；AXI-ACP interface, AXI-is used between PS subsystem and PL subsystem HP interface and DMA channel realize the mutual of view data and computing intermediate data.Isomery computing by this kind of FPGA auxiliary ARM Framework, it is achieved system versatility is equilibrium with real-time.

FPGA firmware in PL subsystem mainly includes image acquisition units and graphics acceleration unit, and it is right that the former mainly realizes Interface Controller, image acquisition and the pretreatment of external OV7670CMOS minisize pick-up head, the latter mainly realize image object detection/ That follows the tracks of accelerates function parallel.

Image acquisition units comprises utilizing camera interface module, image pre-processing module, AXI-DMA module etc.: photographic head connects Mouth die block mainly realizes the state modulator such as photographic head digital independent and image resolution ratio, acquisition frame rate, data form, image Pretreatment module mainly realizes the pretreatment operation such as gathered image stream color correction, histogram equalization, spatial alternation, AXI- Dma module is mainly realized view data to the autonomous passback of DDR3RAM by AXI-GP interface.

Image accelerating module mainly comprises restructural image accelerator, AXI-DMA module.Restructural image accelerator is PL The a piece of restructural region divided in subsystem, it dynamically can add according to system working stage (detection-phase or tracking phase) Carry image object detection or tracking image target accelerator, it is achieved the acceleration to algorithms of different.

Between each IP kernel of PL subsystem internal administration and functional module, the 32bitAXI-Stream bus of standard is all used to enter Row interconnection, has higher efficiency and versatility.The AXI-DMA module of image acquisition units and graphics acceleration unit is passed through respectively AXI-HP interface and AXI-ACP interface realize the data interaction with PS subsystem.On the one hand, by the isolation between interface, it is to avoid The communication efficiency that bus conflict causes declines；On the other hand, the transmission of computing intermediate data directly can be synchronized to PS by AXI-ACP In the processor cache of subsystem, frequent, the high efficiency interactive being more beneficial between software program and firmware.

It is equipped with ARM-Linux embedded OS in PS subsystem, supports that many algorithms, multi-task parallel operate, can Fully meet the application demand of the complex function platform such as autonomous robot, smart mobile phone.In the solid-state memory that it controls, Each driver of modules module (.ko file) and firmware program module (.bit file or .bin file) is stored with document form, It is beneficial to system administration, upgrading and function based on dynamic reconfigurable switching.

The Low Power Consumption Portable real time imaging target detection that this utility model design realizes follows the tracks of device workflow sketch As shown in Figure 3.Its workflow can be divided into detection-phase and tracking phase:

The first step, after system start-up, resets and initializes each device Drivers, and being loaded extremely by target detection accelerator Restructural image accelerator.

Second step, image acquisition units collection, pretreatment image, and through AXI-DMA channel transfer to PS subsystem DDR3 internal memory.

3rd step, the image detection algorithm in PS subsystem is by reception and processes image, performs inverse in a software form Intensive algorithm link, and call graphics acceleration unit execution computing intensity algorithm link.

4th step, according to the result of software-hardware synergism detection, as found interesting target, then will perform dynamic part can weigh Structure operates, and loads target following accelerator to restructural image accelerator, and starts tracking image target algorithm.

5th step, the image tracking algorithm in PS subsystem is by reception and processes image, performs inverse in a software form Intensive algorithm link, and call graphics acceleration unit execution computing intensity algorithm link.

6th step, system, by continuous positioning target location, terminates until following the tracks of.

This utility model selects Gaussian Background model to be image object detection algorithm, and selecting compression track algorithm is image mesh Mark track algorithm.Use PS subsystem software to realize the target designation link of Gaussian Background model, use the inspection of PL subsystem objectives Survey accelerator and realize the foreground detection link of Gaussian Background model；PS subsystem software is used to realize the candidate of compression track algorithm The links such as sample collection, integrogram calculating, compressed sensing, grader renewal, use PL subsystem objectives detection accelerator to realize The Naive Bayes Classifier link of compression track algorithm.Relative to high-performance computer (Intel Core i7-4712MQ, 8GB DDR3RAM, Windows 764-bit), the built-in real time image object detecting and tracking system designed by this utility model System, can realize higher processing speed, and weight only 125g, volume only 118 × 98 × 45mm, peak power only 2.99W, it is possible to Fully meet low-power consumption, portable application demand.

In sum, these are only preferred embodiment of the present utility model, be not intended to limit of the present utility model Protection domain.All within spirit of the present utility model and principle, any modification, equivalent substitution and improvement etc. made, all should wrap Within being contained in protection domain of the present utility model.

Claims (1)

1. a Low Power Consumption Portable real time imaging object detecting and tracking device, including PL subsystem and PS subsystem, it is special Levy and be: between PS subsystem and PL subsystem, use AXI-ACP interface, AXI-HP interface and DMA channel to realize view data Mutual with computing intermediate data；Wherein PL subsystem includes image acquisition units and graphics acceleration unit；Wherein:

Image acquisition units includes utilizing camera interface module, image pre-processing module, AXI-DMA module；Described utilizing camera interface Module is for photographic head digital independent and image resolution ratio, acquisition frame rate, the state modulator of data form, Image semantic classification Module is for gathered image stream color correction, histogram equalization, the pretreatment operation of spatial alternation, and AXI-DMA module is used In being realized view data to the autonomous passback of DDR3RAM by AXI-HP interface；

Graphics acceleration unit includes restructural image accelerator, AXI-DMA module；Restructural image accelerator is for according to system The detection of working stage dynamic load image object or tracking image target accelerator, it is achieved the acceleration to algorithms of different；AXI-DMA Module is for realizing view data to the autonomous passback of DDR3RAM by AXI-ACP interface；

AXI-DMA module in image acquisition units and graphics acceleration unit is all connect with AXI-GP by AXI-Lite interconnection module Mouth connects.