CN108154523A - A kind of real-time modeling method system and method in airborne photoelectric platform - Google Patents
A kind of real-time modeling method system and method in airborne photoelectric platform Download PDFInfo
- Publication number
- CN108154523A CN108154523A CN201711422365.3A CN201711422365A CN108154523A CN 108154523 A CN108154523 A CN 108154523A CN 201711422365 A CN201711422365 A CN 201711422365A CN 108154523 A CN108154523 A CN 108154523A
- Authority
- CN
- China
- Prior art keywords
- target
- data
- image
- visible images
- infrared
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
Abstract
The present invention provides the real-time modeling method system and methods in a kind of airborne photoelectric platform, belong to intelligent video processing technology field, and system includes image capture module, image decoder module, data communication module and target tracking module.Image capture module is responsible for the acquisition of visible images and infrared picture data, after completing acquisition, infrared picture data and visible images data input picture cache module, data communication module is responsible for the command information between dsp chip and image information is transmitted and the communication of host computer and rear end servo-control system, target tracking module receives host computer instruction, reads the infrared picture data cached and visible images data, and the output phase answers handling result, that is target bearing information, finally, target position information is passed into rear end servo-control system.The present invention is easily achieved, and can carry out real-time modeling method processing in airborne photoelectric platform immediately after image is collected, precision is high, stability is strong, output delay is low, can replace the artificial intelligent control realized to servo-drive system.
Description
Technical field
The present invention relates to intelligent video process fields, and in particular to be a kind of real-time target in airborne photoelectric platform with
Track system and method.
Background technology
Target following refers to provide the position of target in a certain frame image of video, and then algorithm Continuous plus goes out in subsequent frame
The task of target location.Target following technology is widely used scene.In modern Airborne photoelectric platform, Target Tracking System
It plays an important role.Conventional on-board photoelectric platform does not have Target Tracking System, by the fortune for manually controlling photoelectric nacelle
It is dynamic, to track target, continuous observation is carried out to target.In modern Airborne photoelectric platform, intelligent Target tracking system is responsible for continuously
The location information of selected target is provided, to adjust the orientation of photoelectric nacelle, pitch angle immediately, target is made to be in video always
Picture center is observed convenient for user.Existing Target Tracking System generally first passes video data back local high-performance computer
Or server, target following processing is then carried out again, finally returns result.This processing mode video data transmission difficulty
Greatly, hardware computing capability is required high, it is impossible to meet the requirement of real-time modeling method.Some existing airborne target tracking systems
It is generally difficult to handle images above data all the way simultaneously, is unsatisfactory for requirement of real-time or in the case where meeting requirement of real-time,
It cannot be guaranteed good target following quality.
Invention content
The technology of the present invention solves the problems, such as：It overcomes the deficiencies of the prior art and provide real-time in a kind of airborne photoelectric platform
Target Tracking System and method realize there is the spies such as small, low in energy consumption based on the hardware platform of fpga chip plus dsp chip
Point is directly installed on airborne photoelectric platform rear end, directly carries out real-time modeling method processing at the scene, can be with extremely low delay
Target location continuously is provided, instructs the ray machine servo-control system follow-up motion of rear end.
The adopted technical solution is that：A kind of real-time modeling method system in airborne photoelectric platform, packet
Include image capture module, image decoder module, data communication module and target following die combination decision vision block.For multinuclear
Dsp chip has the design feature of multiple independent kernels, carries out parallel optimization, realizes stable and accurate real-time modeling method.
The combination decision target tracking algorism and algorithm that the main innovation of the present invention is to use in target tracking module is at specific place
The scheduling managed on device is realized.
Described image acquisition module includes：Visible light image sensor and infrared image sensor, it is seen that light image senses
Device is installed with infrared image sensor with optical axis；Optical axis calibrator error is within 2 pixels；Infrared picture data and visible images
Data are transmitted using SDI agreements；Infrared image sensor acquires infrared band and visible ray respectively with visible light image sensor
The destination image data of wave band is used for target following；
Described image decoder module includes：Two SDI receive chips, SRAM array, fpga chip and in fpga chip
The infrared picture data of operation and visible images data stream algorithm, SDI receive chip and are received from image capture module
SDI infrared picture datas and visible images data be converted to parallel data and be streaming to fpga chip, in fpga chip
Infrared picture data is with visible images data stream algorithm from infrared picture data with being decoded in visible images data flow
Go out effective infrared picture data and visible images data, the data flow include valid data, Elided data, frame synchronization,
Row synchrodata, and carry out data buffer storage using SRAM array；
The data communication module includes：Serial communication chip, the data communication program realized in fpga chip, data
Signal procedure is there are two function, first, the serial communication with exterior, defeated including parsing the instruction received and coding transmission
Go out information, second is that the data interaction between fpga chip and dsp chip, including being transmitted to the infrared of dsp chip from fpga chip
Image data and visible images data, PC control instruction and the target following knot that fpga chip is transmitted to from dsp chip
Fruit；
The target tracking module includes：Multi-core DSP chip and target tracking algorism, the target run on dsp chip with
Track algorithm according to the infrared picture data that is received from data communication module and visible images data, PC control instruct into
Row target following operation, the position that Automatic solution image middle finger sets the goal, obtain target following as a result, and be transferred to data communication
Module exports；To complete instruction response in real time, data communication works with target following, and the work on multi-core DSP chip is divided
Two tasks are controlled for target following and system, wherein 0~n-1 cores complete target following task, the last one n core completes system
Control task；The target tracking algorism is used based on infrared and visible images combination decision visual target tracking algorithms,
Decision model is built respectively for visible images and infrared image, and the acquired sample of judgement is target or background, is resolved
Target location；The probability that single model misjudgment causes tracking to fail is larger, and decision is combined using two decision models
The probability of tracking failure can be greatly reduced, realize stable and accurate target following, there are multiple independences for multi-core DSP chip
The design feature of kernel carries out parallel optimization to the different task for needing operation simultaneously, realizes real-time modeling method.The present invention
The main innovation combination decision target tracking algorism and algorithm that is to use in target tracking module in par-ticular processor
Scheduling realize.
It is real based on infrared and visible images combination decision visual target tracking algorithms in the target tracking module
Existing step is as follows：
(1) according to target initial position and initial infrared picture data and visible images data, initial infrared figure is acquired
Sample characteristics are extracted, structure is determined respectively as positive negative training sample with the target image block in visible images data as data
Plan model Dv、Dir；
(2) receive the infrared picture data of a new frame with after visible images data, from target in previous frame position
Surrounding acquisition candidate samples, judge whether candidate samples are target using decision model, determine target location in a new frame；
(3) determine result which decision model provides for optimizing decision as a result, two decisions of combination according to loss function
The differentiation of model as a result, obtain final output as a result, and the decision model of sub-optimal result is provided using optimal result amendment, eliminate
The error message in the decision model of sub-optimal result is generated, the decision model of generation sub-optimal result is enable to be tracked in succeeding target
More accurately target following is provided in the process as a result, boosting algorithm robustness.
Further realize as follows in step (1)-(3)：
(1) when extracting sample characteristics, sample image block is divided into nonoverlapping zonule, respectively according to gradient direction
Gradient magnitude in statistical regions at pixel, the original feature vector C of composition one 27 calculate standard using following formula later
Change operator, then C is standardized, obtained Standardization Operator N (i, j)：
N (i, j)=(| | C (i, j) | |2+||C(i+1,j)||2+||C(i-1,j)||2+||C(i,j+1)||2+||C(i,
j-1)||2)2
Wherein C (i, j) is the image area characteristics vector of the i-th row j row, is standardized, obtained using the following formula
To final feature vector F (i, j), the feature vector of each image block collectively constitutes clarification of objective representing matrix X；
F (i, j)=max (α, C (i, j)/N (i, j))
Wherein α is an intercept term, for eliminating the excessive noise item of eigenmatrix intermediate value so that the image extracted is special
Sign being capable of more robust expression target；
(2) initial target image block is acquired, using scaling, rotation, translation, overturning, affine transformation mode, generates a collection of mesh
Logo image positive sample, is denoted as Tp, while extract at random in the background area of image some overlapped with target image it is less or not
The image block of coincidence, as negative sample Tn；The positive sample obtained using a variety of transformation is trained, and greatly enhances decision model
Type is to the robustness that accordingly converts；
(3) in new frame image, from stochastical sampling around target previous frame position, a collection of candidate samples are obtained, uniformly
Sampling obtains a collection of candidate samples, collectively constitutes candidate target sample, stochastical sampling can increase track algorithm for target with
The robustness that machine quickly moves, uniform sampling can ensure target after any direction moves, and still be able to accurately be captured；
(4) for visible images and infrared image, decision model D is built respectivelyv、Dir,
Wherein θv, θirFor model parameter, the feature of x samples.In each frame, two are calculated as a result, being denoted as R respectivelyv、
Rir：
It determines that the result which decision model provides is exported for optimizing decision as a result, being used as according to loss function, uses simultaneously
Optimizing decision result goes to correct the decision model for generating sub-optimal result, and the decision model of generation sub-optimal result is made in subsequent frames
There is better performance；A loss function can be calculated in n-th frame in each result of decisionSentence to be promoted
Disconnected accuracy using aggregated loss function, judges optimizing decision result：
WhereinFor the loss function of model D, D ∈ D herev,Dir, D*For optimal decision model, Δ n is aggregated loss letter
Several time window lengths；In object tracking process, the result of decision of two decision models is combined, selection wherein optimal result is made
To export, and using optimal result, update suboptimum decision-making model is corrected, corrects the mistake that suboptimum decision-making model running introduces in the process
False information, two decision models are cooperated, are corrected mutually, realize stable and accurate target following.
The dsp chip is the TMS320C6678 multi-core processors of TI, has 8 independent kernels that can be run parallel, point
It is not denoted as core 0-7, after data communication module receives the instruction that host computer is sent out, is transmitted to the dsp chip of target tracking module,
Dsp chip needs the instruction of real-time response host computer, after starting target following, dsp chip need to complete simultaneously instruction response, from
Video decoding module carries infrared picture data and works with visible images data and target following；These work are can be with
It independently carries out, the system features that the present invention realizes are：Devise two subtasks operated on different kernels, wherein mesh
Mark tracing task is responsible for target location resolving, and system control tasks communication control task is responsible for instruction response, result exports and red
The carrying of outer image data and visible images data, two tasks are separately operable on different kernels, system control tasks
Infrared picture data and visible images data and control instruction are provided for target following task, target following task utilizes infrared
Image data, according to control instruction solving target position, is exported to system control tasks, system control with visible images data
Task completes the passback of target following result.Running on different kernels for task will not seize mutually process resource, will not generate
Unnecessary obstruction causes time delay, so as to ensure that the real-time of target following and instruction response；Core 0-6 operational objectives with
Track task, using based on infrared and visible images combination decision visual target tracking algorithms, completion target location resolves,
Center 0 is used as main core, completes the initialization of dsp chip and the overall operation of target following task, and core 7 runs communication control
Task, response system instruction, carries infrared picture data and visible images data, output tracking result.
The step of target following task run on dsp chip in the target tracking module is：
(1) after system electrification, the initialization of dsp chip is completed, starts target following task, is ready for target location solution
It calculates；
(2) when target following task is in non-tracking state, task is in idle condition, and receives the mesh that host computer is sent out
After marking trace command, go to (3)；
(3) target initial coordinate information R is extracted from from the Command Information Flow of host computer0, trigger inside dsp chip
Enhancing memory directly accesses (EDMA) data transmission mechanism, from the infrared picture data from data communication module and visible ray figure
As extracting initial target region P in data flowv、Pir；
(4) using the infrared picture data in initial target region and visible images data, initialization based on it is infrared with
The combination decision visual target tracking algorithm of visible images for infrared and visible images, builds decision model D respectivelyv、
Dir；
(5) when n-th frame image arrives, in the tracking result R of previous framen-1Surrounding acquisition candidate samples, extract each sample
Feature, judge whether it is target using decision model, obtain the output result R of two decision-making devicesvWith Rir；
(6) optimizing decision is determined using aggregated loss function as a result, as final output Rn, while in final result neighborhood
Interior extraction sample, is updated decision model, boosting algorithm robustness；
(7) electric under system, target following task terminates.
Run on dsp chip in the target tracking module system control communication control task the step of be：
(1) system electrification after core 0 completes dsp chip initialization, starts core 7, bring into operation communication control task；
(2) when not having instruction to arrive, control task is in idle condition, right after receiving the instruction from host computer
Instruction is parsed, and effective information is passed to core 0 is specifically performed；After starting target following, core 7 is from data communication module
It is carried in real time from front-end image sensor, by infrared picture data and the visible images data of image decoder module, supplied
Target following task uses；
(3) electric under system, communication control task terminates.
Real-time modeling method method in a kind of airborne photoelectric platform of the present invention realizes that step is as follows：
(1) infrared sensor and visible light sensor in image capture module acquires the infrared picture data of same optical axis
With visible images data, image decoder module is transferred to by SDI agreements；
(2) the serial infrared image data from image capture module and visible images data flow, first pass around SDI and connect
Chip is received, becomes parallel infrared picture data and visible images data flow, is then passed to fpga chip；Figure in fpga chip
As decoding program use state machine, effective infrared image is decoded from parallel infrared picture data and visible images data flow
Data and visible images data, are stored in SRAM array；Image decoding state machine include stIDLE, stFRAME, stLINE,
StWAIT states, state count CntV, horizontal direction meter according to vertical synchronizing signal VS, data valid signal DE, vertical direction
Number CntH, picturedeep Rows, picturewide Cols signals shift；
(3) it after the data communication program in fpga chip receives target following instruction, is read from SRAM array infrared
Image data and the buffering area in visible images data to piece, notice dsp chip take infrared picture data away from buffering area
With visible images data, while recently received host computer instruction is passed into dsp chip；
(4) dsp chip receive infrared picture data and visible images data, target initial position message and start with
After track instruction, infrared image and visible images are based respectively on, extracts positive negative sample, training decision model builds decision model,
When a new frame image arrives, candidate samples are acquired near target location in previous frame image, extract each candidate samples
Feature, determine that it is target or background using decision model, obtain target location in a new frame.Finally according to target following knot
Fruit and infrared picture data are updated decision model with visible images data；
(5) target following result is transmitted to data communication module from dsp chip, after being transferred to later by data communication module
End system.
The present invention Target Tracking System compared with prior art the advantages of have：
(1) present invention is used based on infrared and visible images combination decision visual target tracking algorithms, based on infrared
With visible images, two decision models are constructed, to carry out target following.Final goal tracking knot is determined by combination decision
Fruit.Infrared image and visible images have respective Pros and Cons in different scenes, and by combination decision, algorithm can be
Target is accurately tracked in abundant scene, while during each frame arithmetic, optimal result amendment can be utilized to generate suboptimum
As a result decision model makes algorithm have good robustness in the process of running, being capable of stable and accurate carry out target following.
(2) dominant frequency of dsp chip and computing capability are limited, of the invention in the target following program on realizing dsp chip,
According to realistic objective, specific aim optimization has been carried out based on multi-core processor, has been divided into the work of dsp chip using reasonable manner
Target following task and communication control task so that two tasks will not generate conflict, cause unnecessary run time expense.
Simultaneously in operational objective tracing task, parallel processing will be capable of in algorithm and is partially distributed to perform parallel in multiple kernels,
It ensure that the real-time of system each task, realize real-time modeling method.
Description of the drawings
Fig. 1 is the system structure diagram of the present invention；
Fig. 2 is the state transition diagram of video data decoding state machine；
Fig. 3 is target following task run flow chart on dsp chip；
Fig. 4 is experiment effect figure of the present invention.
Specific embodiment
The following describes the present invention in detail with reference to the accompanying drawings and embodiments.
As shown in Figure 1, present system includes image decoder module, data communication module and target tracking module；Image
Acquisition module includes visible light image sensor, infrared image sensor, wherein visible light image sensor acquisition visible light wave
The information of section, imaging have abundant color, texture and gradient information, and infrared image sensor is imaged according to heat radiation,
When illumination condition is poor, clear display environment and object still are able to, plays the role of supplying to visible images；During installation, it is seen that
Light image and infrared image sensor require same optical axis, and optical axis calibrator error is within 2 pixels.Wherein visible light image sensor
The image data of 30 frame 1920*1080 resolution ratio of output per second, infrared image sensor 30 frame 640*512 resolution ratio of output per second
Image data.The image data of visible images and infrared image sensor is transmitted using SDI agreements.
Image decoder module include SDI receive the infrared picture data realized in chip, SRAM array and fpga chip with
Visible images data stream program, SDI receive the infrared figures of high speed serialization SDI that chip will be received from image capture module
Fpga chip is streaming to as data and visible images data are converted to parallel data, and the decoding program in fpga chip is from red
Valid data are decoded in outer image data and visible images data, and data buffer storage is carried out using SRAM array.The SDI
It receives chip and uses GS2971A 3G-SDI video decoding chips.Fpga chip uses XC7K325T, and shape is used in fpga chip
State machine realizes the decoding of infrared picture data and visible images data, state machine include stIDLE, stFRAME, stLINE,
StWAIT states, state is according to VS (vertical synchronizing signal), DE (data valid signal), CntV (vertical direction counting), CntH
(horizontal direction counting), Rows (picturedeep), Cols (picturewide) signal shift, and the original state of state machine is
StIDLE when VS and DE is 0, into stFRAME states, prepares to decode the image data of a new frame, when VS and DE is 1,
Into stLINE states, prepare to decode the image data of new a line, when it is 0 that VS, which is 1, DE, into stWAIT states, under waiting
The arrival of data line when VS and DE is 0, is again introduced into stFRAME states, prepares to decode the image data of a new frame, tool
Body state transfer case such as Fig. 2.
Data communication module is included in the data communication program and dsp chip realized in serial communication chip, fpga chip
The data communication program of realization, serial communication chip complete single-ended signal and the conversion before differential signal, fpga chip are sent out
The single-ended signal gone out switchs to differential signal and sends or the differential signal received is switched to single-ended signal transmission to fpga chip,
The data communication program realized in fpga chip is there are two function, first, the serial communication with exterior, receives including parsing
The instruction arrived and coding send output information, second is that the data interaction between fpga chip and dsp chip, including from fpga chip
It is transmitted to the infrared picture data of dsp chip and the transmission of visible images data, PC control instruction and from dsp chip
It is transmitted to the target following result of fpga chip.The serial communication chip uses MAX3077E chips.
Target tracking module includes multi-core DSP chip and the target following program wherein realized.Multi-core DSP chip uses TI
The TMS320C6678 multi-core DSP chips of company, chip include 8 independent kernels, and each core operating frequency is 1GHz.Target
Tracking module is instructed according to the infrared picture data received from data communication module and visible images data, PC control
Target following operation is carried out, the position that Automatic solution image middle finger sets the goal obtains target following as a result, and being transferred to data and leading to
Believe module output；To complete the work such as instruction response, data communication and target following in real time, run simultaneously on multi-core DSP chip
It target following and system controls two subtasks, wherein target following task is responsible for target location resolving, communication control task
It is responsible for instruction response, result output and the carrying of infrared picture data and visible images data, two tasks to be separately operable
On different kernels.Target following program is used based on infrared and visible images combination decision visual target tracking algorithms,
For multi-core DSP chip there is the design feature of multiple independent kernels, carry out parallel optimization, realize stable and accurate real-time mesh
Mark tracking.
As shown in Figure 1, the present invention is based on infrared and visible images combination decision visual target tracking method step such as
Under：
(1) according to the infrared image and visible images received, each basic tracker of combination decision is built respectively.
The Gradient Features of target image are extracted according to the following formula first,
Gx=I (x+1, y)-I (x-1, y)
Gy=I (x, y+1)-I (x, y-1)
Wherein Gx, Gy are the gradient magnitude that position (x, y) is on x directions and y directions respectively, and I represents image.
Next according to the gradient magnitude at each pixel of Gx, Gy calculating and direction, non-overlapping copies are divided the image into
4x4 pocket, the gradient magnitude at each pixel is added up into statistics to difference by direction respectively in each region
Section.Using 9 sections in direction and the section in 18 directions in the present invention.When such as using the section in 9 directions, by gradient magnitude
Enter sections such as (0 °~40 °, 40 °~80 °, 320 °~0 °) by directional statistics.9 interval statistics results of each image block and 18th area
Between statistical result collectively constitute the original feature vector C of one 27, later using following formula normalized operator, then to C
It is standardized,
N (i, j)=(| | C (i, j) | |2+||C(i+1,j)||2+||C(i-1,j)||2+||C(i,j+1)||2+||C(i,
j-1)||2)2
Wherein C (i, j) is the image area characteristics vector of the i-th row j row, and N (i, j) is the Standardization Operator being calculated.
It is standardized using the following formula
F (i, j)=max (α, C (i, j)/N (i, j))
Wherein α is an intercept term, for eliminating the excessive noise item of eigenmatrix intermediate value so that the image extracted is special
Sign being capable of more robust expression target.F (i, j) is the feature vector finally acquired, common group of the feature vector of each image block
Into clarification of objective representing matrix X.
(2) according to target image, using modes such as scaling, rotation, translation, overturning, affine transformations, a collection of target figure is generated
The positive sample of picture, is denoted as Tp, each sample according to transformation degree, be based respectively on value for 0~1 label, with original object figure
As closer, the label value obtained by sample is bigger.Some are extracted at random in the background area of image simultaneously to overlap with target image
Less or misaligned region, as negative sample Tn, by being standardized computing cross-correlation with target image, obtain its with
The label value of the similarity of target, as negative sample.When being standardized mutual, sample image is zoomed into 16*16 pictures first
Plain size.Then it calculates as follows：
Wherein T represents initial target image sample, TnAny one negative sample is represented, two samples of ⊙ are multiplied simultaneously pixel-by-pixel
It sums it up.Conf is the similarity that this negative sample and target is calculated.
The feature and its label of each sample are extracted using the method described in step 1, training decision model obtains one
Group weight vector.Training process uses stochastic gradient descent method.
The discriminant equation of decision model is：
hθ(x)=g (θTx)
Wherein x is the feature vector that the eigenmatrix of a certain sample transforms into, and θ is the weight vector that training obtains, and g is using such as
Minor function can make the value being calculated in the section of [0,1], when h is more than a certain threshold value TconfWhen, sample is judged as mesh
Mark, otherwise, sample is considered as background.
(3) in a new frame image, from stochastical sampling around target previous frame position, a collection of sample, uniform sampling are obtained
A collection of sample is obtained, collectively constitutes candidate target sample.Stochastical sampling can increase track algorithm and target is quickly transported at random
Dynamic robustness, uniform sampling can ensure target after any direction moves, and still be able to accurately be captured.Stochastical sampling is adopted
With normal distyribution function, respectively with the x on the boundary up and down of target, centered on y-coordinate value, the seat of candidate samples is randomly generated
Scale value.It uniformly uses centered on target initial position, step-length is 2 pixels, and sliding window extracts candidate samples.Obtain candidate samples
Afterwards, sample characteristics are extracted, judge that it is background or target according to decision model.If there are multiple targets simultaneously, according to result
Position is clustered, and leaves out the larger candidate samples of deviation, and the similar candidate samples in position are done weighted average according to confidence level,
Obtain final goal position.
(4) finally, after obtaining new target location, the method described in (1) obtains a collection of positive sample and negative sample, extraction
Feature, and decision model is updated with the training method described in (2), with the robust that algorithm is kept to change target appearance
Property.
(5) for visible images and infrared image, decision model D is built respectively according to (1) to (4)v、Dir,
Wherein subscript v represents visible ray, and ir represents infrared.
For visible images and infrared image, decision model D is built respectivelyv、Dir,
Wherein θv, θirFor model parameter, the feature of x samples.In each frame, two are calculated as a result, being denoted as R respectivelyv、
Rir：
Under normal conditions, the goodness of fit of two kinds of results is higher, deviation (the school axis error for containing 2 pixels) within 3 elements.But
In certain scenes, if illumination condition is poor, background and during more uniform target temperature in image, a certain result is it is possible that larger
Drift error, at this moment, the present invention according to loss function determines optimizing decision as a result, being exported as system.Simultaneously with optimal knot
Fruit goes to correct another decision model, makes it that can have better performance in subsequent frames.Each result of decision, can in n-th frame
A loss function is enough calculatedTo promote the accuracy judged, the present invention is judged most using aggregated loss function
The excellent result of decision：
WhereinFor the loss function of model D, D ∈ D herev,Dir, wherein D*For optimal decision model, Δ n is damaged for accumulation
Lose the time window length of function.After obtaining optimal result, using optimal result position, using method restoration updating described in (4)
Suboptimum decision-making model corrects the error message introduced in its operational process.In object tracking process, two decision models are mutual
Cooperation is corrected mutually, realizes stable and accurate target following.
The method for tracking target operation of the present invention includes following operating procedure, target following task run flow chart such as Fig. 3
It is shown：
(1) after system electrification, the initialization of dsp chip is completed, starts target following task, is ready for target location solution
It calculates.
(2) when target following task is in non-tracking state, task is in idle condition.Receive the mesh that host computer is sent out
After marking trace command, go to (3).
(3) target initial coordinate information R is extracted from from the Command Information Flow of host computer0, trigger inside dsp chip
Enhancing memory directly accesses (EDMA) data transmission mechanism.From the infrared picture data from data communication module and visible ray figure
As extracting initial target region P in data flowv、Pir。
(4) using the infrared picture data in initial target region and visible images data, initialization based on it is infrared with
The combination decision visual target tracking algorithm of visible images.According to 2, for infrared and visible images, difference structure
Build decision model Dv、Dir.Its center 0 carries out the decision model D based on visible imagesvRelated operation, core 1 carried out based on red
The decision model D of outer imageirRelated operation, core 2-6 provides parallel support for the operation of algorithm on core 0-1.Including initially just
Negative sample Tp、TnAcquire, sample classification in the feature extraction of each sample and subsequent step.
(5) when n-th frame image arrives, in the tracking result R of previous framen-1Surrounding acquisition candidate samples, extract each sample
Feature, judge whether it is target using decision model.Obtain the output result R of two decision-making devicesvWith Rir。
(6) optimizing decision is determined using aggregated loss function as a result, as final output Rn.Simultaneously in final result neighborhood
Interior extraction sample, is updated decision model, boosting algorithm robustness.
(7) electric under system, real-time modeling method task terminates.
The step of communication control task run on dsp chip in target tracking module is：
(1) system electrification after core 0 completes dsp chip initialization, starts core 7, bring into operation communication control task；
(2) when not having instruction to arrive, system control tasks are in idle condition.Receive the instruction from host computer
Afterwards, instruction is parsed, effective information is passed to core 0 is specifically performed.After starting target following, core 7 leads to from data
Believe that module is carried in real time from front-end image sensor, by infrared picture data and the visible images number of image decoder module
According to for the use of target following task.
(3) electric under system, communication control task terminates.
The part of test results of the Target Tracking System of the present invention is as shown in figure 4, in Fig. 4 in the first behavior visible images
Experimental result, it can be seen that when target is by partial occlusion, remain to keep stable objects tracking.Second and the third line in Fig. 4
For the experimental result in infrared image, it can be seen that when target deforms upon, remain to keep stable objects tracking, target is complete
It is complete to block when occurring again, target can be given for change again, continue target following.
Target Tracking System in the present invention can handle infrared picture data and visible images data simultaneously, can
Reply target by partial occlusion, blocked completely by target well, and the scene that target shape generation seriously changes is kept to target
Tenacious tracking.
Above example is provided just for the sake of the description purpose of the present invention, and is not intended to limit the scope of the present invention.This
The range of invention is defined by the following claims.It the various equivalent replacements that do not depart from spirit and principles of the present invention and make and repaiies
Change, should all cover within the scope of the present invention.
Claims (7)
1. a kind of real-time modeling method system in airborne photoelectric platform, it is characterised in that：Including image capture module, image solution
Code module, data communication module and target following die combination decision vision block：
Described image acquisition module includes：Visible light image sensor and infrared image sensor, it is seen that optical image sensor with
Infrared image sensor is installed with optical axis；Infrared picture data is transmitted with visible images data using SDI agreements；Infrared image
Sensor acquires the destination image data of infrared band and visible light wave range with visible light image sensor respectively, for target following
It uses；
Described image decoder module includes：Two SDI receive chip, SRAM array, fpga chip and are run in fpga chip
Infrared picture data and visible images data stream algorithm, SDI receives the SDI that chip is received from image capture module
Infrared picture data and visible images data are converted to parallel data and are streaming to fpga chip, the infrared figure in fpga chip
As data and visible images data stream algorithm decode effectively from infrared picture data and visible images data flow
Infrared picture data and visible images data, the data flow include valid data, Elided data, frame synchronization, row and synchronize
Data, and carry out data buffer storage using SRAM array；
The data communication module includes：Serial communication chip, the data communication program realized in fpga chip, data communication
Program is there are two function, first, the serial communication with exterior, output letter is sent including parsing the instruction received and coding
Breath, second is that the data interaction between fpga chip and dsp chip, the infrared image including being transmitted to dsp chip from fpga chip
Data and visible images data, PC control instruction and the target following result that fpga chip is transmitted to from dsp chip；
The target tracking module includes：Multi-core DSP chip and target tracking algorism, the target following run on dsp chip are calculated
Method carries out mesh according to the infrared picture data received from data communication module and visible images data, PC control instruction
Mark tracking operation, the position that Automatic solution image middle finger sets the goal obtain target following as a result, and being transferred to data communication module
Output；To complete instruction response in real time, data communication works with target following, and the work on multi-core DSP chip is divided into mesh
Mark tracking and system control two tasks, wherein 0~n-1 cores complete target following task, the last one n core completes system control
Task；The target tracking algorism is used based on infrared and visible images combination decision visual target tracking algorithms, for
Visible images and infrared image build decision model respectively, and the acquired sample of judgement is target or background, solving target
Position；The probability that single model misjudgment causes tracking to fail is larger, and being combined decision using two decision models can
The probability of tracking failure is greatly reduced, realizes stable and accurate target following, there are multiple independent kernels for multi-core DSP chip
Design feature, to need simultaneously operation different task carry out parallel optimization, realize real-time modeling method.
2. the real-time modeling method system in airborne photoelectric platform according to claim 1, it is characterised in that：The target
In tracking module, based on infrared and visible images combination decision visual target tracking algorithms, realize that step is as follows：
(1) according to target initial position and initial infrared picture data and visible images data, initial infrared image number is acquired
According to the target image block in visible images data, as positive negative training sample, sample characteristics are extracted, build decision model respectively
Type Dv、Dir；
(2) receive the infrared picture data of a new frame with after visible images data, from target in previous frame around position
Candidate samples are acquired, judges whether candidate samples are target using decision model, determines target location in a new frame；
(3) determine result which decision model provides for optimizing decision as a result, two decision models of combination according to loss function
Differentiation as a result, obtain final output as a result, and the decision model of sub-optimal result is provided using optimal result amendment, eliminate and generate
Error message in the decision model of sub-optimal result enables the decision model of generation sub-optimal result to track process in succeeding target
In provide more accurately target following as a result, boosting algorithm robustness.
3. the real-time modeling method system in airborne photoelectric platform according to claim 2, it is characterised in that：The step
(1) further realize as follows in-(3)：
(1) when extracting sample characteristics, sample image block is divided into nonoverlapping zonule, is counted respectively according to gradient direction
Gradient magnitude in region at pixel, the original feature vector C of composition one 27 are calculated later using following formula normalized
Then son is standardized C, obtained Standardization Operator N (i, j)：
N (i, j)=(| | C (i, j) | |2+||C(i+1,j)||2+||C(i-1,j)||2+||C(i,j+1)||2+||C(i,j-1)|
|2)2
Wherein C (i, j) is the image area characteristics vector of the i-th row j row, is standardized, is obtained most using the following formula
Whole feature vector F (i, j), the feature vector of each image block collectively constitute clarification of objective representing matrix X；
F (i, j)=max (α, C (i, j)/N (i, j))
Wherein α is an intercept term, for eliminating the excessive noise item of eigenmatrix intermediate value so that the characteristics of image energy extracted
Enough more robust expression targets；
(2) initial target image block is acquired, using scaling, rotation, translation, overturning, affine transformation mode, generates a collection of target figure
As positive sample, it is denoted as Tp, while extract at random in the background area of image some overlapped with target image it is less or misaligned
Image block, as negative sample Tn；The positive sample obtained using a variety of transformation is trained, and greatly enhances decision model pair
The robustness accordingly converted；
(3) in new frame image, from stochastical sampling around target previous frame position, a collection of candidate samples, uniform sampling are obtained
A collection of candidate samples are obtained, collectively constitute candidate target sample, it is fast at random for target that stochastical sampling can increase track algorithm
The robustness of speed movement, uniform sampling can ensure target after any direction moves, still be able to accurately be captured；
(4) for visible images and infrared image, decision model D is built respectivelyv、Dir,
Wherein θv, θirFor model parameter, the feature of x samples in each frame, is calculated two as a result, being denoted as R respectivelyv、Rir：
Result which decision model provides is determined as optimizing decision according to loss function as a result, as output, while with optimal
The result of decision goes to correct the decision model for generating sub-optimal result, makes the decision model of generation sub-optimal result can have in subsequent frames more
Good performance；A loss function can be calculated in n-th frame in each result of decisionTo promote what is judged
Accuracy using aggregated loss function, judges optimizing decision result：
WhereinFor the loss function of model D, D ∈ Dv,Dir, D*For optimal decision model, Δ n is the time of aggregated loss function
Window length；In object tracking process, combine two decision models the result of decision, select wherein optimal result as export,
And using optimal result, amendment update suboptimum decision-making model corrects the error message introduced during suboptimum decision-making model running,
Two decision models are cooperated, are corrected mutually, realize stable and accurate target following.
4. the real-time modeling method system in a kind of airborne photoelectric platform according to claim 1, it is characterised in that：It is described
Dsp chip is the TMS320C6678 multi-core processors of TI, has 8 independent kernels that can be run parallel, is denoted as core 0-7 respectively,
After data communication module receives the instruction that host computer is sent out, the dsp chip of target tracking module is transmitted to, dsp chip needs
The instruction of real-time response host computer, after starting target following, dsp chip needs to complete instruction response simultaneously, decodes mould from video
Block carries infrared picture data and works with visible images data and target following；It devises two and operates in different kernels
On subtask, wherein target following task is responsible for target location resolving, and system control tasks communication control task is responsible for instruction
Response, result output and the carrying of infrared picture data and visible images data, two tasks are separately operable in different
On core, system control tasks provide infrared picture data and visible images data and control instruction, mesh for target following task
Tracing task is marked using infrared picture data and visible images data according to control instruction solving target position, is exported to system
Control task, system control tasks complete the passback of target following result；Core 0-6 operational objective tracing tasks, using based on infrared
It with the combination decision visual target tracking algorithm of visible images, completes target location and resolves, center 0 is used as main core, completes
The initialization of dsp chip and the overall operation of target following task, core 7 run communication control task, and response system instructs,
Carry infrared picture data and visible images data, output tracking result.
5. the real-time modeling method system in a kind of airborne photoelectric platform according to claim 1, it is characterised in that：It is described
The step of target following task run on dsp chip in target tracking module is：
(1) after system electrification, the initialization of dsp chip is completed, starts target following task, is ready for target location resolving；
(2) when target following task is in non-tracking state, task is in idle condition, receive target that host computer sends out with
After track instruction, go to (3)；
(3) target initial coordinate information R is extracted from from the Command Information Flow of host computer0, trigger the enhancing inside dsp chip
Memory directly accesses (EDMA) data transmission mechanism, from the infrared picture data from data communication module and visible images number
According to extraction initial target region P in streamv、Pir；
(4) using the infrared picture data in initial target region and visible images data, initialization based on it is infrared with it is visible
The combination decision visual target tracking algorithm of light image for infrared and visible images, builds decision model D respectivelyv、Dir；
(5) when n-th frame image arrives, in the tracking result R of previous framen-1Surrounding acquisition candidate samples, the spy for extracting each sample
Sign, judges whether it is target using decision model, obtains the output result R of two decision-making devicesvWith Rir；
(6) optimizing decision is determined using aggregated loss function as a result, as final output Rn, while carried in final result neighborhood
Sampling originally, is updated decision model, boosting algorithm robustness.
6. the real-time modeling method system in a kind of airborne photoelectric platform according to claim 1, it is characterised in that：It is described
Run on dsp chip in target tracking module system control communication control task the step of be：
(1) system electrification after core 0 completes dsp chip initialization, starts core 7, bring into operation communication control task；
(2) when not having instruction to arrive, control task is in idle condition, after receiving the instruction from host computer, to instruction
It is parsed, effective information is passed to core 0 is specifically performed；After starting target following, core 7 is real-time from data communication module
It carries from front-end image sensor, by infrared picture data and the visible images data of image decoder module, for target
Tracing task uses.
7. a kind of real-time modeling method method in airborne photoelectric platform, which is characterized in that realize that step is as follows：
(1) infrared sensor and visible light sensor in image capture module, acquire the infrared picture data of same optical axis with can
See light image data, image decoder module is transferred to by SDI agreements；
(2) the serial infrared image data from image capture module and visible images data flow first pass around SDI and receive core
Piece becomes parallel infrared picture data and visible images data flow, is then passed to fpga chip；Image solution in fpga chip
Coded program use state machine decodes effective infrared picture data from parallel infrared picture data and visible images data flow
With visible images data, it is stored in SRAM array；Image decoding state machine includes stIDLE, stFRAME, stLINE, stWAIT
State, state counts CntV according to vertical synchronizing signal VS, data valid signal DE, vertical direction, horizontal direction counts CntH,
Picturedeep Rows, picturewide Cols signals shift；
(3) after the data communication program in fpga chip receives target following instruction, infrared image is read from SRAM array
Data and the buffering area in visible images data to piece, notice dsp chip take infrared picture data away from buffering area and can
See light image data, while recently received host computer instruction is passed into dsp chip；
(4) dsp chip receives infrared picture data and refers to visible images data, target initial position message and start-up trace
After order, infrared image and visible images are based respectively on, extracts positive negative sample, training decision model builds decision model, newly
When one frame image arrives, candidate samples are acquired near target location in previous frame image, extract the spy of each candidate samples
Sign, determines that it is target or background using decision model, obtains target location in a new frame.Finally according to target following result and
Infrared picture data is updated decision model with visible images data；
(5) target following result is transmitted to data communication module from dsp chip, is transferred to rear end system by data communication module later
System.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711422365.3A CN108154523B (en) | 2017-12-25 | 2017-12-25 | A kind of real-time modeling method system and method in airborne photoelectric platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711422365.3A CN108154523B (en) | 2017-12-25 | 2017-12-25 | A kind of real-time modeling method system and method in airborne photoelectric platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108154523A true CN108154523A (en) | 2018-06-12 |
| CN108154523B CN108154523B (en) | 2018-12-18 |
Family
ID=62465881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711422365.3A Active CN108154523B (en) | 2017-12-25 | 2017-12-25 | A kind of real-time modeling method system and method in airborne photoelectric platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108154523B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109978910A (en) * | 2019-02-21 | 2019-07-05 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Target tracking system and method based on DSP |
| CN110084822A (en) * | 2019-05-05 | 2019-08-02 | 中国人民解放军战略支援部队航天工程大学 | A kind of target acquisition real time processing system and method towards the in-orbit application of satellite |
| CN110570441A (en) * | 2019-09-16 | 2019-12-13 | 广州波视信息科技股份有限公司 | Ultra-high definition low-delay video control method and system |
| CN110794713A (en) * | 2019-12-04 | 2020-02-14 | 中国航天空气动力技术研究院 | Reconnaissance type unmanned aerial vehicle photoelectric load simulation training system |
| CN111383231A (en) * | 2018-12-28 | 2020-07-07 | 成都皓图智能科技有限责任公司 | Image segmentation method, device and system based on 3D image |
| CN112991200A (en) * | 2021-02-08 | 2021-06-18 | 中国科学院西安光学精密机械研究所 | Method and device for adaptively enhancing infrared image |
| CN113395448A (en) * | 2021-06-15 | 2021-09-14 | 西安视成航空科技有限公司 | Airborne pod image searching, tracking and processing system |
| CN113709399A (en) * | 2021-08-31 | 2021-11-26 | 中国电子科技集团公司第五十八研究所 | Visual target tracking system based on DSP + FPGA |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080177507A1 (en) * | 2006-10-10 | 2008-07-24 | Mian Zahid F | Sensor data processing using dsp and fpga |
| CN102314693A (en) * | 2011-07-27 | 2012-01-11 | 中国科学院长春光学精密机械与物理研究所 | Dual mode video target identification tracking system |
-
2017
- 2017-12-25 CN CN201711422365.3A patent/CN108154523B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080177507A1 (en) * | 2006-10-10 | 2008-07-24 | Mian Zahid F | Sensor data processing using dsp and fpga |
| CN102314693A (en) * | 2011-07-27 | 2012-01-11 | 中国科学院长春光学精密机械与物理研究所 | Dual mode video target identification tracking system |
Non-Patent Citations (4)
| Title |
|---|
| HONG ZHANG 等: "A TWO-CHANNEL DETECTION AND TRACK SYSTEM DESIGN BASED ON FPGA AND DSP", 《4TH INTERNATIONAL CONFERENCE ON SOFTWARE TECHNOLOGY AND ENGINEERING (ICSTE 2012)》 * |
| 刘德全: "《空中目标ISAR实时成像技术研究》", 30 November 2014, 银川：阳光出版社 * |
| 孙略: "《视频技术基础》", 31 October 2013, 北京：世界图书北京出版公司 * |
| 王诗琴 等: "基于TMS320C6678平台的粒子滤波目标跟踪设计与仿真", 《传感器世界》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111383231A (en) * | 2018-12-28 | 2020-07-07 | 成都皓图智能科技有限责任公司 | Image segmentation method, device and system based on 3D image |
| CN111383231B (en) * | 2018-12-28 | 2023-10-27 | 成都皓图智能科技有限责任公司 | Image segmentation method, device and system based on 3D image |
| CN109978910A (en) * | 2019-02-21 | 2019-07-05 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Target tracking system and method based on DSP |
| CN110084822A (en) * | 2019-05-05 | 2019-08-02 | 中国人民解放军战略支援部队航天工程大学 | A kind of target acquisition real time processing system and method towards the in-orbit application of satellite |
| CN110570441A (en) * | 2019-09-16 | 2019-12-13 | 广州波视信息科技股份有限公司 | Ultra-high definition low-delay video control method and system |
| CN110794713A (en) * | 2019-12-04 | 2020-02-14 | 中国航天空气动力技术研究院 | Reconnaissance type unmanned aerial vehicle photoelectric load simulation training system |
| CN112991200A (en) * | 2021-02-08 | 2021-06-18 | 中国科学院西安光学精密机械研究所 | Method and device for adaptively enhancing infrared image |
| CN112991200B (en) * | 2021-02-08 | 2024-02-02 | 中国科学院西安光学精密机械研究所 | Method and device for adaptively enhancing infrared image |
| CN113395448A (en) * | 2021-06-15 | 2021-09-14 | 西安视成航空科技有限公司 | Airborne pod image searching, tracking and processing system |
| CN113395448B (en) * | 2021-06-15 | 2023-02-21 | 西安视成航空科技有限公司 | Airborne pod image searching, tracking and processing system |
| CN113709399A (en) * | 2021-08-31 | 2021-11-26 | 中国电子科技集团公司第五十八研究所 | Visual target tracking system based on DSP + FPGA |
| CN113709399B (en) * | 2021-08-31 | 2024-03-08 | 中国电子科技集团公司第五十八研究所 | Visual target tracking system based on DSP+FPGA |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108154523B (en) | 2018-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108154523B (en) | A kind of real-time modeling method system and method in airborne photoelectric platform | |
| US11055521B2 (en) | Real-time gesture recognition method and apparatus | |
| US11954904B2 (en) | Real-time gesture recognition method and apparatus | |
| CN105631870B (en) | Star chart registration based on NoC frameworks and target trajectory extracting method and device | |
| US10346198B1 (en) | Data processing architecture for improved data flow | |
| CN110533041B (en) | Regression-based multi-scale scene text detection method | |
| CN112131927B (en) | Sow delivery time prediction system based on gesture conversion characteristics in late gestation period | |
| CN110555408A (en) | Single-camera real-time three-dimensional human body posture detection method based on self-adaptive mapping relation | |
| CN111104960A (en) | Sign language identification method based on millimeter wave radar and machine vision | |
| CN112668492A (en) | Behavior identification method for self-supervised learning and skeletal information | |
| CN112365578A (en) | Three-dimensional human body model reconstruction system and method based on double cameras | |
| Zhang et al. | Gesture recognition system based on improved YOLO v3 | |
| Li et al. | IIE-SegNet: Deep semantic segmentation network with enhanced boundary based on image information entropy | |
| Tian et al. | High-speed tiny tennis ball detection based on deep convolutional neural networks | |
| CN112884803A (en) | Real-time intelligent monitoring target detection method and device based on DSP | |
| Li et al. | Lightweight particle filter for robust visual tracking | |
| Lv et al. | A feedback-driven DNN inference acceleration system for edge-assisted video analytics | |
| Han et al. | Cigarette Detection Algorithm Based on Improved Faster R-CNN | |
| Fan et al. | Toward few-shot domain adaptation with perturbation-invariant representation and transferable prototypes | |
| Cereda et al. | Improving the generalization capability of dnns for ultra-low power autonomous nano-uavs | |
| CN112215174A (en) | Sanitation vehicle state analysis method based on computer vision | |
| Li et al. | Complexity and accuracy analysis of convolutional neural networks in scene text detection | |
| CN114356076B (en) | Gesture control method and system | |
| CN115147922A (en) | Monocular pedestrian detection method, system, equipment and medium based on embedded platform | |
| Cui et al. | Game Image Detection and Application Based on Improved YOLOv5 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |