CN107707874A - Fisheye camera video correction and transmission system and method - Google Patents
Fisheye camera video correction and transmission system and method Download PDFInfo
- Publication number
- CN107707874A CN107707874A CN201710843424.8A CN201710843424A CN107707874A CN 107707874 A CN107707874 A CN 107707874A CN 201710843424 A CN201710843424 A CN 201710843424A CN 107707874 A CN107707874 A CN 107707874A
- Authority
- CN
- China
- Prior art keywords
- video
- correction
- image
- camera
- plug
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Abstract
The present invention relates to the collection of video, correction, coding and transmission, to propose a kind of flexible, real-time fisheye camera video correction and Transmission system.The present invention, fisheye camera video correction and transmission method, step are as follows：The first step：Image is obtained using camera module CSI cameras；Second step：It is V4L2 drive plugs using pipeline first plug-in unit, to obtain CSI camera video streams；Acquired video flowing is delivered to correction plug-in unit, and correction plug-in unit is corrected to original image, obtains undeformed image；Video flowing is again by H.264 packed card；Video flowing is finally delivered to transmission plug-in unit；3rd step：RTSP server is built, and RTP video flowings are mounted in RTSP server；4th step：Client carries out video reception by VLC in LAN and played.Present invention is mainly applied to Video processing occasion.
Description
Technical field
The present invention relates to the collection of video, correction, coding and transmission, i.e. fisheye camera video correction and Transmission system.
Background technology
The angle of visual field of traditional cameras is smaller, can cause to be difficult to the requirement for meeting people.With the increasingly renewal of technology,
Omni-vision system [1] occurs.Omni-vision system includes fish eye lens system, more image mosaic systems and reflected refraction system
System.
Camera lens of the angle of visual field more than 120 ° is referred to as fish eye lens in engineering.Although this big feature of the fish-eye visual field
The many fields being widely used in life are allowed it to, but because the image of its shooting has serious distortion so that
People seem very uncomfortable.If we need the particular content using these fault images, it is necessary to by these images
Correct into the image for meeting people's custom.By fish eye lens shoot image revert to meet the mankind viewing image we be referred to as
[2] are corrected for flake.
Document [3] proposes a kind of complete fisheye camera handling process, designs and proposes bad in given FPGA
The processing framework of point correction algorithm and flake correction algorithm, and it is added into the handling process of whole camera.Document [4] extracts
Core operator --- the dot product of flake video correction, analyzes its memory access mode, it was found that memory access performance and resolution ratio
Limitation with linear relationship and buffer memory capacity causes performance is interim to decline.According to this rule, it is proposed that four pixels are simultaneously
Row processing can be optimal the conclusion of performance.Document [5] uses the spherical projection model of fish eye images, proposes that one kind is based on fish
The real-time antidote of eyelens spherical projection.
[1] Li Xi beautiful jades refractive-reflective all vision system theoretical researches and optimization design [D] University Of Tianjin .2007.
[2] a kind of fish eye lens antidote [J] computers based on spherical perspective projection constraint of Ying Xianghua, Hu Zhanyi
Journal, 2003,26 (12):1702-1708.
[3] research [D] the Tianjin University of Technology of the double fish eye images distortion correction algorithms of Sun Pei, 2015.
[4] design of the embedded flake video correction schemes of Wang Chen and realization [D] Southeast China University, 2015.
[5] Lv Yaowen, Zhe, real-time antidote [J] the Jilin University journal of the flat flakes video image distortion of Xu Xi are pacified:
Edition, 2017,55 (1):103-108.
The content of the invention
For overcome the deficiencies in the prior art, the present invention is directed to propose a kind of flexible, real-time fisheye camera video correction
And Transmission system.The technical solution adopted by the present invention is fisheye camera video correction and transmission method, and step is as follows：
The first step：Image is obtained using camera module CSI cameras；
Second step：Utilize the multimedia framework Gstreamer structure streamlines pipeline that increases income：First of pipeline
Plug-in unit is V4L2 drive plugs, to obtain CSI camera video streams；Acquired video flowing is delivered to correction plug-in unit, correction
Plug-in unit is corrected to original image, obtains undeformed image；Video flowing by H.264 packed card, is converted to H.264 video again
Stream, is easy to the transmission of video；Video flowing is finally delivered to transmission plug-in unit, its role is that video flowing is packaged as RTP
Bag；
3rd step：Using Gstreamer real time streaming transport protocol RTSP module, RTSP server is built, and RTP is regarded
Frequency sagging is downloaded in RTSP server；
4th step：Client carries out video reception by VLC in LAN and played.
What video flowing was corrected is comprised the concrete steps that, camera is demarcated, and obtains distortion parameter, is joined based on distortion
Number, the parallelization of correction algorithm is realized using graphics processor GPU, reaches the real-time Transmission of flake video.
Correction further comprises the following steps that：
(1) fisheye camera is imaged, the mistake for exactly projecting to the point on three dimensions on two dimensional surface by coordinate transform
Journey, if world coordinate system is Pw(X, Y, Z), fish-eye camera coordinates system are PM(X, Y, Z), because image passes through fish
Distorted after glasses head, it is therefore desirable to analyze the conversion process between world coordinate system and image coordinate system；
(2) camera is radial distortion, and radial distortion is exactly along the distortion of lens radius directional spreding, the mathematics of distortion
Model is described with the first few items of the taylor series expansion around principal point principle point, using k1、k2And k3I.e.
First three items, regulation formula are as follows：
x0=x (1+k1r2+k2r4+k3r6)
y0=y (1+k1r2+k2r4+k3r6)
In above formula, (x0,y0) be fault image in home position, (x, y) be correction after new position, r is fault image
The length at middle pixel range image center, the mapping relations of coordinate points can be completed by formula 1.K can be determined1、k2With
k3, i.e. distortion parameter；
(3) GPU realizes that coordinate maps
After obtaining distortion parameter, coordinate mapping of the original image to image after correction is obtained, GPU realizes Parallel Implementation coordinate
Conversion, has reached the purpose of correction, the image after being corrected.
The features of the present invention and beneficial effect are：
The system uses Nvidia Jetson TK1 to realize platform for hardware, Nvidia Jetson TK1 be the whole world first
Embedded supercomputer, the floating-point operation of the gigabit of lines per second 326 can be entered.In addition, using Gstreamer programming frameworks,
Gstreamer employs very flexible architecture in design, and provides many predefined medium process modules,
Therefore the difficulty of the system design can be greatly simplified.By establishing Pipeline, four modules are packaged into Gstreamer and inserted
Part, then plug-in unit is called to carry out flake video correction and transmission process, treatment effeciency height.Called by plug-in unit,
Ensure while carrying out real-time correction to flake video, the flexibility that the system of adding uses.
Brief description of the drawings：
Hard Pipeline streamlines schematic diagram is recorded a video and transmitted to Fig. 1 fisheye cameras.
Association between Fig. 2 element.
Fig. 3 RTSP server Establishing process.
Embodiment
The invention belongs to field of image transmission, is a kind of novel collection for realizing video, correction, encoding and decoding and transmission
System.The present invention proposes that one kind using Nvidia Jetson TK1 as platform is realized, using Gstreamer multimedia frameworks, is built
Vertical Pipeline, standard is packaged into by video acquisition module, rectification module, coding module and transport module respectively
Gstreamer plug-in units, then call plug-in unit to carry out flake video correction and the system of transmission respectively.
The system corrected and transmitted the invention particularly relates to a kind of video image obtained to fisheye camera
Design, it includes OV4689 cameras module (CSI interface), rectification module (using GPU parallel processings), H.264 encoding and decoding
Module, Ethernet transport module etc..
In the implementation of flake video correction, built-in field tradition typically uses application-specific IC
(Application Specific Integrated Circuit, ASIC) design method realizes flake video correction.The party
Although method can realize higher performance, the R＆D cycle is longer, and design is just difficult to update and upgrade once solidifying,
And do not possess portability.It is an object of the invention to provide a kind of flexible, real-time fisheye camera video correction and transmission
System.The system is to be based on Nvidia Jetson TK1 platforms, realizes the correction and transmission of real-time flake video.
Technical scheme is specifically：
1 general frame
The system uses Gstreamer Video processing frameworks, establishes Pipeline, by collection, correction, coding and transmission four
Individual module is packaged into the Gstreamer plug-in units of standard respectively.By V4L2 plug-in units (collection) gather video flowing, then video flowing according to
It is secondary by correct plug-in unit, coding plug-in unit, transmission plug-in unit.Fig. 1 is Pipeline schematic diagrames.Specific handling process is as follows：
The first step：CSI cameras are connected with TX1 development boards so that circuit board check that equipment.
Second step：In TK1 development platforms, pipeline is built using Gstreamer.Pipeline first plug-in unit
It is V4L2 drive plugs, to obtain video flowing；Acquired video flowing is delivered to correction plug-in unit, and correction plug-in unit is to original image
Correction, obtains undeformed image；Video flowing by H.264 packed card, is converted to H.264 video flowing, is easy to video again
Transmission；Video flowing is finally delivered to transmission plug-in unit, its role is that video flowing is packaged as RTP bags.
3rd step：In TK1 development platforms, using Gstreamer RTSP module, RTSP server is built, and RTP
Video flowing is mounted in RTSP server.
4th step：In LAN, client host carries out video reception by VLC and played.
2 establish Pipeline based on Gstreamer frameworks
Gstreamer is the multimedia framework of increasing income (framework) for building Stream Media Application, is substantially also one
Individual storehouse and the set of plug-in unit, Gstreamer storehouses are installed and establish pipeline according to the framework.
In this project, acquisition module, rectification module, coding module and the transport module of video will be all encapsulated into
Gstreamer standard plug-in unit forms, are easy to pipeline to call.
In Gstreamer frameworks, Media Stream passes through a series of intermediary element, and sink is flowed to from source element
element.The element of these interactions constitutes a complete pipeline.Association between element is as shown in Figure 2.
3 software module specific design schemes
3.1 rectification module
Camera is demarcated, obtains distortion parameter.Based on distortion parameter, realized and corrected using GPU (graphics processor)
The parallelization of algorithm, reach the real-time Transmission of flake video.
Rectification module concrete scheme is as follows：
(1) fisheye camera is imaged, the mistake for exactly projecting to the point on three dimensions on two dimensional surface by coordinate transform
Journey.If world coordinate system is Pw(X, Y, Z), fish-eye camera coordinates system are PM(X, Y, Z), the image coordinate of imaging plane
It is for PI(u,v).Because image is distorted after fish eye lens, it is therefore desirable to analyze world coordinate system and image
Conversion process between coordinate system.
(2) world coordinate system P herew(X, Y, Z) and image coordinate system PI(u, v) is, it is known that solve 3 equations of the above simultaneously
, can be in the hope of spin matrix R and translation matrix T using spin matrix R orthogonality.Herein, multinomial coefficient is camera
Inner parameter, spin matrix R, translation matrix T are the external parameter of camera.So far the demarcation of video camera is completed, and obtains flake
The camera parameter of camera.
(3) GPU (graphics processor) realizes that coordinate maps
After obtaining distortion parameter, coordinate mapping of the original image to image after correction can be obtained.Using GPU (at figure
Reason device) Parallel Implementation coordinate transform is realized, reach the purpose of correction, the image after being corrected.
Correction further comprises the following steps that：
(1) fisheye camera is imaged, the mistake for exactly projecting to the point on three dimensions on two dimensional surface by coordinate transform
Journey, if world coordinate system is Pw(X, Y, Z), fish-eye camera coordinates system are PM(X, Y, Z), because image passes through fish
Distorted after glasses head, it is therefore desirable to analyze the conversion process between world coordinate system and image coordinate system；
(2) camera is radial distortion, and radial distortion is exactly along the distortion of lens radius directional spreding, the mathematics of distortion
Model is described with the first few items of the taylor series expansion around principal point principle point, using k1、k2And k3I.e.
First three items, regulation formula are as follows：
x0=x (1+k1r2+k2r4+k3r6)
y0=y (1+k1r2+k2r4+k3r6)
In above formula, (x0,y0) be fault image in home position, (x, y) be correction after new position, r is fault image
The length at middle pixel range image center, the mapping relations of coordinate points can be completed by formula 1.K can be determined1、k2With
k3, i.e. distortion parameter；
3.2H.264 encoding and decoding plug-in units
Nvidia Tegra K1 development boards support multiple hardwares code encoding/decoding mode, including H.264 encoding and decoding.Using stone
The video corrected to flake carries out H.264 encoding and decoding.
3.3 Ethernets transmit
RTSP can enable media player control the transmission of media stream, and it is worked in a manner of client server,
RTSP does not transmit data in itself, but controls multimedia transmission.
Gstreamer has the RTSP streaming modules of official, and RTSP server can be built using the modularity function, realizes
The transmission of video flowing.RTSP server Establishing process is as shown in Figure 3.The factory of video to be sent is at video early stage in Fig. 3
The whole pipeline of reason, including video acquisition, video correction and video compress.
After RTSP server is started working, video flowing can be received by vlc in client.
Claims (3)
1. a kind of fisheye camera video correction and transmission method, it is characterized in that, step is as follows：
The first step：Image is obtained using camera module CSI cameras；
Second step：Utilize the multimedia framework Gstreamer structure streamlines pipeline that increases income：Pipeline first plug-in unit
It is V4L2 drive plugs, to obtain CSI camera video streams；Acquired video flowing is delivered to correction plug-in unit, corrects plug-in unit
Original image is corrected, obtains undeformed image；Video flowing by H.264 packed card, is converted to H.264 video flowing again,
It is easy to the transmission of video；Video flowing is finally delivered to transmission plug-in unit, its role is that video flowing is packaged as RTP bags；
3rd step：Using Gstreamer real time streaming transport protocol RTSP module, RTSP server is built, and RTP video flowings
It is mounted in RTSP server；
4th step：Client carries out video reception by VLC in LAN and played.
2. fisheye camera video correction as claimed in claim 1 and transmission method, it is characterized in that, video flowing is corrected
Comprise the concrete steps that, camera is demarcated, obtain distortion parameter, based on distortion parameter, realized and corrected using graphics processor GPU
The parallelization of algorithm, reach the real-time Transmission of flake video.
3. fisheye camera video correction as claimed in claim 1 and transmission method, it is characterized in that, the further specific step of correction
It is rapid as follows：
(1) fisheye camera is imaged, the process for exactly projecting to the point on three dimensions on two dimensional surface by coordinate transform, if
World coordinate system is Pw(X, Y, Z), fish-eye camera coordinates system are PM(X, Y, Z), because image passes through fish eye lens
Distorted afterwards, it is therefore desirable to analyze the conversion process between world coordinate system and image coordinate system；
(2) camera is radial distortion, and radial distortion is exactly along the distortion of lens radius directional spreding, the mathematical modeling of distortion
It is described with the first few items of the taylor series expansion around principal point principle point, using k1、k2And k3I.e. first three
, regulation formula is as follows：
x0=x (1+k1r2+k2r4+k3r6)
y0=y (1+k1r2+k2r4+k3r6)
In above formula, (x0,y0) be fault image in home position, (x, y) be correction after new position, r be fault image in picture
The length at vegetarian refreshments range image center, the mapping relations of coordinate points can be completed by formula 1.K can be determined1、k2And k3,
That is distortion parameter；
(3) GPU realizes that coordinate maps
After obtaining distortion parameter, coordinate mapping of the original image to image after correction is obtained, GPU realizes that Parallel Implementation coordinate becomes
Change, reached the purpose of correction, the image after being corrected.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710843424.8A CN107707874A (en) | 2017-09-18 | 2017-09-18 | Fisheye camera video correction and transmission system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710843424.8A CN107707874A (en) | 2017-09-18 | 2017-09-18 | Fisheye camera video correction and transmission system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107707874A true CN107707874A (en) | 2018-02-16 |
Family
ID=61172840
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710843424.8A Pending CN107707874A (en) | 2017-09-18 | 2017-09-18 | Fisheye camera video correction and transmission system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107707874A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111294580A (en) * | 2020-03-12 | 2020-06-16 | 佳都新太科技股份有限公司 | Camera video projection method, device and equipment based on GPU and storage medium |
| CN113393381A (en) * | 2021-07-08 | 2021-09-14 | 哈尔滨工业大学（深圳） | Pipeline inner wall image generation method and device and terminal equipment |
| CN115272124A (en) * | 2022-08-04 | 2022-11-01 | 北京斯年智驾科技有限公司 | Distorted image correction method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130208083A1 (en) * | 2012-02-15 | 2013-08-15 | City University Of Hong Kong | Panoramic stereo catadioptric imaging |
| CN105631844A (en) * | 2014-10-28 | 2016-06-01 | 上海大学 | Image camera calibration method |
| CN106060481A (en) * | 2016-07-18 | 2016-10-26 | 北京奇虎科技有限公司 | Video collection method and device of pan-tilt-zoom camera |
| CN106210525A (en) * | 2016-07-29 | 2016-12-07 | 乐视控股（北京）有限公司 | For realizing camera and the method for net cast |
| CN106231274A (en) * | 2016-09-23 | 2016-12-14 | 国网上海市电力公司 | Electric power website communication power supply monitoring system based on ARM and monitoring method |
| CN205812165U (en) * | 2016-06-30 | 2016-12-14 | 张先江 | A kind of live box of panoramic video |
-
2017
- 2017-09-18 CN CN201710843424.8A patent/CN107707874A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130208083A1 (en) * | 2012-02-15 | 2013-08-15 | City University Of Hong Kong | Panoramic stereo catadioptric imaging |
| CN105631844A (en) * | 2014-10-28 | 2016-06-01 | 上海大学 | Image camera calibration method |
| CN205812165U (en) * | 2016-06-30 | 2016-12-14 | 张先江 | A kind of live box of panoramic video |
| CN106060481A (en) * | 2016-07-18 | 2016-10-26 | 北京奇虎科技有限公司 | Video collection method and device of pan-tilt-zoom camera |
| CN106210525A (en) * | 2016-07-29 | 2016-12-07 | 乐视控股（北京）有限公司 | For realizing camera and the method for net cast |
| CN106231274A (en) * | 2016-09-23 | 2016-12-14 | 国网上海市电力公司 | Electric power website communication power supply monitoring system based on ARM and monitoring method |
Non-Patent Citations (2)
| Title |
|---|
| 公维理: "基于嵌入式CPU-GPU的高清鱼眼视频实时校正系统", 《计算机工程与应用》 * |
| 汪晨: "嵌入式鱼眼视频矫正方案的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111294580A (en) * | 2020-03-12 | 2020-06-16 | 佳都新太科技股份有限公司 | Camera video projection method, device and equipment based on GPU and storage medium |
| CN113393381A (en) * | 2021-07-08 | 2021-09-14 | 哈尔滨工业大学（深圳） | Pipeline inner wall image generation method and device and terminal equipment |
| CN113393381B (en) * | 2021-07-08 | 2022-11-01 | 哈尔滨工业大学（深圳） | Pipeline inner wall image generation method and device and terminal equipment |
| CN115272124A (en) * | 2022-08-04 | 2022-11-01 | 北京斯年智驾科技有限公司 | Distorted image correction method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11580697B2 (en) | Systems and methods for reconstruction and rendering of viewpoint-adaptive three-dimensional (3D) personas | |
| CN104301677B (en) | The method and device monitored towards the panoramic video of large scene | |
| CN107707874A (en) | Fisheye camera video correction and transmission system and method | |
| US7035453B2 (en) | Method and apparatus for parallel multi-view point video capturing and compression | |
| CN204090039U (en) | Integration large scene panoramic video monitoring device | |
| CN108848354B (en) | VR content camera system and working method thereof | |
| CN102256061A (en) | Two-dimensional and three-dimensional hybrid video stabilizing method | |
| CN112365407B (en) | Panoramic stitching method for camera with configurable visual angle | |
| CN103581570A (en) | Large-size screen splice system and method based on multi-media communication | |
| CN106204488A (en) | The video defogging method that a kind of OpenCL accelerates | |
| CN106780310A (en) | A kind of perspective view construction method and device | |
| CN104581075B (en) | panoramic video processing system and method based on heterogeneous platform | |
| CN105357513A (en) | Eye correction method for single camera in session video | |
| CN105959562A (en) | Method and device for obtaining panoramic photographing data and portable panoramic photographing equipment | |
| CN111083311A (en) | Synchronization system and method for airborne multi-channel video and POS data of unmanned aerial vehicle | |
| WO2019216707A1 (en) | Method and apparatus for processing three dimensional object image using point cloud data | |
| CN104363421B (en) | The method and apparatus for realizing Multi-angle camera monitoring effect | |
| CN104883481A (en) | Double-spectrum infrared thermal imaging ball machine | |
| CN106851197A (en) | Unmanned plane transmits image method and system, unmanned plane and receiving device | |
| CN104683773A (en) | Video high-speed transmission method using unmanned aerial vehicle | |
| JP3550046B2 (en) | Camera image transmission method and camera image transmission system | |
| CN106101530A (en) | A kind of method that high-speed adaptability night vision image strengthens | |
| CN106101489A (en) | Template matching monitor video mist elimination system based on cloud platform and defogging method thereof | |
| Liu et al. | Remote-sensing image compression using priori-information and feature registration | |
| CN214959901U (en) | Video image processing device based on AI chip and CPU |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180216 |