CN106530312A - Real-time image matting system based on low-power embedded system - Google Patents
Real-time image matting system based on low-power embedded system Download PDFInfo
- Publication number
- CN106530312A CN106530312A CN201610978143.9A CN201610978143A CN106530312A CN 106530312 A CN106530312 A CN 106530312A CN 201610978143 A CN201610978143 A CN 201610978143A CN 106530312 A CN106530312 A CN 106530312A
- Authority
- CN
- China
- Prior art keywords
- foreground
- video
- data
- mask
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Abstract
The invention discloses a real-time image matting system based on a low-power embedded system, and belongs to the technical field of image processing. A blue and green curtain real-time image matting system is provided. An ARM and other serial processing SoC processors +FPGA parallel processing system is used to solve the problem of poor image matting performance in a conventional embedded system. An image preprocessing unit preprocesses a foreground video source, and a foreground mask extraction unit differentiates whether foreground video signals of a RGB format generated by the image preprocessing unit are the mask information of an effective foreground portion through a point-to-point comparing algorithm. A video format repacking unit receives the differentiating result of the foreground mask extraction unit and conducts format packaging again. The image preprocessing unit, the foreground mask extraction unit and the video format repacking unit are realized by FPGA, and SoC processing unit decodes video signals sent by the video format repacking unit and finally completes image matting video signal output.
Description
Technical field
The present invention relates to a kind of method that blue, green curtain scratches picture, especially a kind of to be based on low-power-consumption embedded system real-time image scratching
Method, belongs to technical field of image processing.
Background technology
In modern society, as video technique develops to digitized, multimedia direction, demand of the people to visual sensory
Improve constantly, it is front that the Video Object Extraction technology with virtual studio, Campus TV Station as representative has shown wide application
Scape.Virtual studio is the real-time synthesis for being digitized the character activities image that camera live shoots, and makes personage with void
Intend background synchronously to change, the fusion flawless so as to realize both, to obtain perfect synthesis picture.Scratch as technology
It is widely used in the occasions such as news hookup, film trick.There is the embedded of dedicated custom with application system to function, reliability etc.
Complete in system scratch as function with it is existing completed with general purpose computer scratch as integration technology in have a share.
Traditionally, real-time image scratching function is realized using the high-end processors in x86.Real-time image scratching system on market,
All it is this mode without exception.And in low-power-consumption embedded system, also there is no the precedent of Implementation of Embedded System.And it is real-time
Scratch as FPGA is used alone and conventional processors are not used, then there is underaction.For example, during prospect, background show
From different sources, both possibly real-time videos of local input, the video or picture of local hard drive real-time decoding, it is also possible to
The far-end video of network real-time decoding.The video of correspondence decoding, the various SoC of embedded ARM easily can be tackled by accelerator,
Can solution SoC H.264/H.265 be very common in real time on the market；And realize that H.264/H.265 decoding is not using FPGA
The FPGA that also can not find now built-in H.264/H.265 stone on the market of reality.
Real-time video to be completed is scratched as function, either traditional CPU or the FPGA of parallel processing of serial process, all without
Method is provided separately perfect result.
It is typical to scratch as algorithm background：Per frame foreground and background data, Memory Buffer are respectively put into；Subsequently sentence
The whether transparent judgment principle of disconnected foreground pixel data is to judge whether R, G, B belong to " background colour " respectively, if it is right
Answer pixel to take background color output, export if not foreground color is then removed.Look very simple, but modern processors lack hardly possible
To be competent at " simple operation " of such flood tide.Due to using pipeline system, therefore per the multiple conditional judgment of pixel, make flowing water
Line optimizes complete failure；And read repeatedly prospect buffer, background buffer, write output buffer, make again DDR2, DDR3,
Mono- class internal memories of DDR4 cannot use the optimal way such as burst is read, burst is write again, therefore whole system efficiency is very low.SMPTE is fixed
The full HD video of justice, resolution is 1920*1080 pixels, even if on x86, not using asm and multimedia special instruction,
Cannot realize that the so big HD video of real-time processing data amount scratches picture.
Common processor, either x86, ARM, or PowerPC, are work in series modes.High-performance x86 processor
Although real-time image scratching can be processed, its power consumption is very big, is correspondingly embedded in the common ARM of formula system, Power PC Processor then
Substantially cannot realize.
The content of the invention
Therefore, for the above-mentioned deficiency of prior art, the present invention is all providing a kind of based on low-power-consumption embedded system reality
When scratch picture method, using the system of mono- class serial general processor+FPGA parallel processings of ARM, to solve current embedded system
Scratch in system as the not enough problem of performance.
The present invention provide based on low-power-consumption embedded system real-time image scratching method, methods described is：
Image pre-processing unit carries out pretreatment to foreground video source, and foreground video source is decomposed into by image pre-processing unit
Parallel uncorrected data is carried out YCbCr and turns RGB color conversion by the parallel uncorrected data of the embedded method of synchronization；
Image pre-processing unit is generated foreground mask extraction unit the foreground video signal of rgb format, according to pointwise pair
The algorithm of ratio distinguish be whether effective foreground part mask information；
Pointwise contrasts algorithm specifically, SoC processing units transmit the threshold values of a stingy picture, in blue curtain video source situation
Under, G channel datas are added threshold values as G components, R channel datas is added threshold values as R component, when channel B data are less than G
When component or channel B data are less than R component, then the pixel is mask prospect as effective foreground point, the otherwise pixel
Point；
Video format weight encapsulation unit receives the differentiation result of foreground mask extraction unit, records the of each effective row
One coordinate for effective foreground point and last effective foreground point of appearance occur, and this coordinate and mask information are regarded with prospect
The form encapsulation again of frequency signal；
Image pre-processing unit, foreground mask extraction unit and video format weight encapsulation unit is realized by FPGA processor；
The decoding video signal that video format weight encapsulation unit is transmitted by SoC processing units, and to extracting
Each effective row before and after mark coordinate interval according to mask information, distinguish each pixel be foreground data to be stored also
Background data, and to interval outside the whole storage background data of effective pixel points, the video signal for most completing to scratch picture at last is defeated
Go out.
Further, in video format weight encapsulation unit by coordinate points and mask information with foreground video signal again
Form is encapsulated specifically, coordinate points and mask information are embedded into foreground video signal effective by diminution script horizontal blanking section length
The front end of picture point and the rear end for starting header, the row data form after encapsulation are followed successively by end header (EAV), blanking zone, open
Coordinate, mask information area, effective image before and after beginning header (SAV), valid data.
The beneficial effects of the present invention is：The present invention provides a kind of side based on low-power-consumption embedded system real-time image scratching
Method, using the framework of FPGA+CPU, completes the abstraction function of foreground mask at FPGA ends, is judged according to mask information at ARM ends
Storage foreground video data or background video data, the pointwise contrast algorithm adopted when extracting to mask make use of well
FPGA parallel abilities, and ARM only needs to process the coordinate interval of mask, largely reduces performance consumption, can be with
Accomplish real-time image scratching.Compared with prior art, the method that the present invention is provided breaks traditions embedded based on serial processing mode
Formula process chip can not be completed the restriction of real-time image scratching, and context of methods is compensate in existing embedded system, by as ARM,
The SoC processors based on serial processing mode such as PowerPC, with concurrent operation FPGA cooperations, are finally completed low-power consumption
The technological gap of embedded system real-time image scratching, makes the design of system more flexible and changeable.
Description of the drawings
Fig. 1 is schematic flow sheet of the present invention based on the method for low-power-consumption embedded system real-time image scratching；
Fig. 2 is the row data form schematic diagram after video format of the present invention is encapsulated again.
Specific embodiment
Below in conjunction with the accompanying drawings the specific embodiment of the present invention is illustrated：
As shown in figure 1, the present invention provides a kind of based on low-power-consumption embedded system real-time image scratching method, methods described is：
Image pre-processing unit 2 carries out pretreatment to foreground video source 1, and image pre-processing unit 2 is by 1 point of foreground video source
The parallel uncorrected data for embedding the method for synchronization is solved, and parallel uncorrected data is carried out into YCbCr and turn RGB color conversion；
Image pre-processing unit 2 is generated the foreground video signal of rgb format according to pointwise pair by foreground mask extraction unit 3
The algorithm of ratio distinguish be whether effective foreground part mask information；
Whether it is effective foreground part that video format weight encapsulation unit 4 will be distinguished from foreground mask extraction unit 3 good
Mask information, records first seat for effective foreground point and last effective foreground point of appearance occur of each effective row
Mark, and this coordinate and mask information are encapsulated with foreground video signal again form；
The decoding video signal that video format weight encapsulation unit 4 is transmitted by SoC processing units 5, and to extracting
Before and after each effective row for arriving, mark coordinate is interval according to mask information, and it is foreground data to be stored to distinguish each pixel
Or background data, and to interval outside the whole storage background data of effective pixel points, most complete at last to scratch the video signal of picture
Output 7.
In foreground mask extraction unit 3, pointwise contrast algorithm is the threshold values that SoC processing units transmit a stingy picture,
In the case of blue curtain video source, using G channel datas plus threshold values as G components, using R channel datas plus threshold values as R component, when
When channel B data are less than G components or channel B data less than R component, then the pixel is used as effective foreground point, the otherwise picture
Vegetarian refreshments is mask foreground point.
As preferred embodiment, by coordinate points and mask information and foreground video in video format weight encapsulation unit 4
Signal form encapsulation again, reduces script horizontal blanking section length, and coordinate points and mask information are embedded into foreground video signal has
The front end of effect picture point and the rear end for starting header, the row data form after encapsulation are followed successively by end header (EAV) 8, blanking zone
9th, header (SAV) 10, coordinate/mask information area 11, effective image 12 are started.
Image pre-processing unit 2, foreground mask extraction unit 3, video format weight encapsulation unit 4 by FPGA realize, due to
Can accomplish without the frame of video that prestores, and without plug-in RAM, resource consumption is also few, can be from high performance-price ratio logic
The suitable FPGA of resource, SoC processing units 5 are realized by ARM, be with the addition of coordinate interval and ARM operands to be processed are dropped significantly
It is low, and can be input into from local video, far-end network transmission, internal stationary picture etc. obtains required background video source, accomplishes
Flexible real-time image scratching.
As shown in Fig. 2 in each effective row format, according to the form of embedded synchronous video signal, by coordinate points and cover
Code information is embedded into the front end and the rear end for starting header of foreground video signal effectively figure picture point, and accordingly reduces script horizontal blanking
Section length, makes script row general tempo number constant.In the YCbCr data forms that finite data bit wide is 16, Y passages are arrived
The value of highest order is negated for a time high position, and C-channel is also to negate the value of highest order for a time high position in the same manner, and do so can be avoided out
There is the situation of header in the flag information for now transmitting, and so originally can transmit 14 in one clock node of blanking zone has criterion
Will data, for convenience process of rear end SoC improve the demand of arithmetic speed, are fixed as 1010 by Y and C-channel high 4, low
Four transmit effective marker data, and the common effective marker data for transmitting 8 in video resolution are by such a clock node
In the case of SMPTE standards 1080P, before and after secured transmission of payload data, coordinate needs two pixels, transmits the mask information of a line
242 pixels are needed, totally 272 timeticks points can use due to script blanking zone, Gu blanking zone is reduced into into 30 clocks
Beat, additional 242 timeticks carry flag information in being used for transmitting a line.
Pointwise contrast algorithm used in foreground mask extraction unit 3 mainly apply three primary colories in blueness in RGB
In color standard, blue channel numerical value is big, and green passage and red channel data very little, so forms larger numerical difference, and human body
The red passage numerical value of the colour of skin is big, and the numerical value of blue channel and green passage is little, and so under the background of blue curtain, personage has very big contrast, holds
Easily detained from when scratching as threshold values is set to 20 to 50 interval in actual application, stingy as effect out is all well and good.Phase
For the data to tri- passages of RGB are all contrasted, it is better that this algorithm is realized, to scratching as effect need not reach
In the case of ultimate attainment, using pointwise contrast algorithm it is deducted come image can both meet most Man's Demands, again can be with
Reduce the cost of product.
As the transmission of data is compound BT1120 standards, so only needing to the number using standard in SoC processing units
The collecting work that just can complete foreground data is operated according to acquisition mode plus the mask of collection blanking zone.
And the operation for scratching picture is carried out inside SoC processing units, SoC processing units can be from collection in addition all the way
Unrestricted choice in the middle of the video file of video input signals, fixed video file or playback, picture group have been woven with than larger
Motility.
Picture composition is only left in work in SoC processing units, and the work of picture composition is for each pixel list
It is reciprocity for unit, step is as follows：
(1) mask of 8 bit is extended to into 8 bytes；
(2) foreground data is carried out and operation with mask；
(3) background data is carried out and operation with the inverted value of mask；
(4) by the results added of upper two computings, obtain the image value scratched.
As computing length above is 8 byte-aligneds, and all it is the logical operationss value of standard, is very suitable for SIMD
Class instruction is optimized, the NEON instructions of such as ARM.
No any branch in the middle of the processing procedure of whole algorithm, can obtain in the middle of the instruction pipeline of ARM enough
Process the performance of 1080P30 real-time operations.Meanwhile, the symmetry in structure causes the ARM CPU of multinuclear divide many parallel
Road carries out piecemeal process to the data of same two field picture simultaneously.
The above is the preferred embodiment of the present invention, it is noted that for those skilled in the art
For, on the premise of without departing from principle of the present invention, some improvements and modifications can also be made, these improvements and modifications
Should be regarded as protection scope of the present invention.
Claims (4)
- It is 1. a kind of to be based on low-power-consumption embedded system real-time image scratching method, it is characterised in that methods described is,Image pre-processing unit carries out pretreatment to foreground video source, and foreground video source is decomposed into embedded by image pre-processing unit Parallel uncorrected data is carried out YCbCr and turns RGB color conversion by the parallel uncorrected data of the method for synchronization；Image pre-processing unit is generated foreground mask extraction unit the foreground video signal of rgb format, is contrasted according to pointwise Algorithm distinguish be whether effective foreground part mask information；Pointwise contrasts algorithm specifically, SoC processing units transmit the threshold values of a stingy picture, in the case of blue curtain video source, G channel datas are added threshold values as G components, R channel datas is added threshold values as R component, when channel B data are less than G point When amount or channel B data are less than R component, then the pixel is mask foreground point as effective foreground point, the otherwise pixel；Video format weight encapsulation unit receives the differentiation result of foreground mask extraction unit, record each effective row first There is the coordinate of effective foreground point and last effective foreground point of appearance, and this coordinate and mask information are believed with foreground video Number again form encapsulation；Image pre-processing unit, foreground mask extraction unit and video format weight encapsulation unit is realized by FPGA processor；The decoding video signal that video format weight encapsulation unit is transmitted by SoC processing units, and it is every to what is extracted Before and after one effective row, mark coordinate is interval according to mask information, and it is foreground data to be stored or the back of the body to distinguish each pixel Scape data, and to interval outside the whole storage background data of effective pixel points, most complete at last to scratch the video signal output of picture.
- 2. low-power-consumption embedded system real-time image scratching method is based on as claimed in claim 1, it is characterised in that the video lattice Coordinate points and mask information are encapsulated with foreground video signal again form specifically, reducing row originally in formula weight encapsulation unit and disappeared Coordinate points and mask information are embedded into the front end of foreground video signal effectively figure picture point and are started after header by hidden section length End, the row data form after encapsulation are followed successively by end header EAV, blanking zone, start header SAV, coordinate before and after valid data, cover The code information area, effective image.
- 3. low-power-consumption embedded system real-time image scratching method is based on as claimed in claim 1, it is characterised in that at the SoC Reason unit completes to scratch as in, and the work of picture composition for the process step of each pixel cell is：The mask of 8 bit is extended to 8 bytes by step 1；Step 2 foreground data is carried out and operation with mask；Step 3 background data is carried out and operation with the inverted value of mask；Step 2 and the results added of step 3 computing are obtained the image value scratched by step 4.
- 4. low-power-consumption embedded system real-time image scratching method is based on as claimed in claim 1, it is characterised in that the pointwise pair Than, in algorithm, the threshold values value of the stingy picture that SoC processing units are transmitted is 20 to 50.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610978143.9A CN106530312B (en) | 2016-11-05 | 2016-11-05 | One kind being based on low-power-consumption embedded system real-time image scratching method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610978143.9A CN106530312B (en) | 2016-11-05 | 2016-11-05 | One kind being based on low-power-consumption embedded system real-time image scratching method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106530312A true CN106530312A (en) | 2017-03-22 |
| CN106530312B CN106530312B (en) | 2019-03-12 |
Family
ID=58349671
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610978143.9A Active CN106530312B (en) | 2016-11-05 | 2016-11-05 | One kind being based on low-power-consumption embedded system real-time image scratching method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106530312B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108616724A (en) * | 2018-05-08 | 2018-10-02 | 南京齿贝犀科技有限公司 | A kind of radio image collecting transmission platform based on FPGA and ARM structures |
| CN110390225A (en) * | 2018-04-16 | 2019-10-29 | 广州番禺职业技术学院 | Automobile video frequency inspection system |
| CN111563862A (en) * | 2020-07-16 | 2020-08-21 | 北京文香信息技术有限公司 | Method, device, storage medium and equipment for constructing virtual video based on FPGA |
| WO2022062417A1 (en) * | 2020-09-22 | 2022-03-31 | 北京沃东天骏信息技术有限公司 | Method for embedding image in video, and method and apparatus for acquiring planar prediction model |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202261667U (en) * | 2011-08-29 | 2012-05-30 | 三缘汇科技(天津)有限公司 | Interactive entertainment set top box device for KTV and family |
| CN103137094A (en) * | 2012-12-04 | 2013-06-05 | 江苏清投视讯科技有限公司 | Split joint image processor used for image matting |
| CN104469089A (en) * | 2014-12-23 | 2015-03-25 | 山东建筑大学 | Multimedia interaction teaching system and teaching method |
| CN105184787A (en) * | 2015-08-31 | 2015-12-23 | 广州市幸福网络技术有限公司 | Identification camera capable of automatically carrying out portrait cutout and method thereof |
| US20160180199A1 (en) * | 2011-11-07 | 2016-06-23 | General Electric Company | Automatic surveillance video matting using a shape prior |
| US20160203587A1 (en) * | 2015-01-14 | 2016-07-14 | Thomson Licensing | Method and apparatus for color correction in an alpha matting process |
-
2016
- 2016-11-05 CN CN201610978143.9A patent/CN106530312B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202261667U (en) * | 2011-08-29 | 2012-05-30 | 三缘汇科技(天津)有限公司 | Interactive entertainment set top box device for KTV and family |
| US20160180199A1 (en) * | 2011-11-07 | 2016-06-23 | General Electric Company | Automatic surveillance video matting using a shape prior |
| CN103137094A (en) * | 2012-12-04 | 2013-06-05 | 江苏清投视讯科技有限公司 | Split joint image processor used for image matting |
| CN104469089A (en) * | 2014-12-23 | 2015-03-25 | 山东建筑大学 | Multimedia interaction teaching system and teaching method |
| US20160203587A1 (en) * | 2015-01-14 | 2016-07-14 | Thomson Licensing | Method and apparatus for color correction in an alpha matting process |
| CN105184787A (en) * | 2015-08-31 | 2015-12-23 | 广州市幸福网络技术有限公司 | Identification camera capable of automatically carrying out portrait cutout and method thereof |
Non-Patent Citations (2)
| Title |
|---|
| 沈洋等: "交互式前景抠图技术综述", 《计算机辅助设计与图形学学报》 * |
| 韩柳等: "基于DSP抠像处理的KTV视频合成机顶盒设计与开发", 《天津职业技术师范大学学报》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110390225A (en) * | 2018-04-16 | 2019-10-29 | 广州番禺职业技术学院 | Automobile video frequency inspection system |
| CN108616724A (en) * | 2018-05-08 | 2018-10-02 | 南京齿贝犀科技有限公司 | A kind of radio image collecting transmission platform based on FPGA and ARM structures |
| CN111563862A (en) * | 2020-07-16 | 2020-08-21 | 北京文香信息技术有限公司 | Method, device, storage medium and equipment for constructing virtual video based on FPGA |
| CN111563862B (en) * | 2020-07-16 | 2020-10-27 | 北京文香信息技术有限公司 | Method, device, storage medium and equipment for constructing virtual video based on FPGA |
| WO2022062417A1 (en) * | 2020-09-22 | 2022-03-31 | 北京沃东天骏信息技术有限公司 | Method for embedding image in video, and method and apparatus for acquiring planar prediction model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106530312B (en) | 2019-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104200470B (en) | Blue screen image-matting method | |
| CN106530312A (en) | Real-time image matting system based on low-power embedded system | |
| CN108965740B (en) | Real-time video face changing method, device, equipment and storage medium | |
| KR101931466B1 (en) | Image tone adjustment using local tone curve computation | |
| CN103475826A (en) | Video matting and synthesis method | |
| CN106506950A (en) | A kind of image processing method and device | |
| CN108322706B (en) | A kind of high-definition signal processing unit and its method application | |
| CN105430368A (en) | Two-viewpoint stereo image synthesizing method and system | |
| CN106899878A (en) | A kind of adjustable video and graph compound method and system of transparency based on OMAP chips | |
| CN101651810A (en) | Device and method for processing interlaced line-crossing stereoscopic composite video signals | |
| CN109688382A (en) | A kind of underwater picture processing system of underwater robot | |
| CN108093175B (en) | A kind of adaptive defogging method of real-time high-definition video and device | |
| CN105450942A (en) | Method and device for character superimposition of video image | |
| WO2022111717A1 (en) | Image processing method and apparatus, and electronic device | |
| CN108307163B (en) | Image processing method and device, computer installation and readable storage medium storing program for executing | |
| CN109658488B (en) | Method for accelerating decoding of camera video stream through programmable GPU in virtual-real fusion system | |
| CN208424534U (en) | The cartridge assemblies and imager of imager | |
| CN106559714A (en) | A kind of extraction method of key frame towards digital video copyright protection | |
| CN103413331B (en) | A kind of support edits the high resolution video image content sparse expression method propagated | |
| CN103236038B (en) | Haze image Quick demisting processing components | |
| CN109978787B (en) | Image processing method based on biological visual computing model | |
| CN104463774B (en) | A kind of three tunnel image co-registration processor design methods based on DM642 | |
| CN203243467U (en) | Apparatus of realizing tiled display of video images by using parallel processing | |
| He et al. | FPGA-based high definition image processing system | |
| CN115002304A (en) | Video image resolution self-adaptive conversion device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |