CN106454102A - Method for realizing full-hardware high-speed smart camera - Google Patents

Abstract

The invention provides a method for realizing a full-hardware high-speed smart camera. The method comprises the following steps: video encoding and video stream uploading, wherein the video encoding is video full I frame encoding realized in FPGA, the process is to perform DCT operation, quantization operation and entropy encoding operation on collected image data in sequence, and to output encoded image data at last; and the video stream uploading is to embed simplified kilomega network TCP/IP protocol stacks of the following 5 states that are only retained in the FPGA: CLOSED, SYN-RCVD, SYN-SEND, LISTEN and ESTABLISHED. According to the method provided by the invention, in the implementation method of completely packaging front end sensor driving, image acquisition, video stream encoding and kilomega network TCP/IP transmission in a hardware system, due to the architecture of full-hardware, the instantaneity is strong, the reliability is high, the design difficulty is simplified, and the design cost and the failure rate are reduced.

Description

A kind of implementation method of devices at full hardware high-speed intelligent camera

Technical field

The present invention relates to high speed imaging field, specially a kind of implementation method of devices at full hardware high-speed intelligent camera.

Background technology

High-speed intelligent camera system can be widely applied to the fields such as research, monitoring, public safety.Complete smart camera net Network, needs to have the correlation techniques such as high speed image acquisition, image coding and decoding, high speed network transmission.Realization side more general at present Method, is the acquisition realizing front-end image using FPGA, then realizes the encoding and decoding of image with High Performance DSP and express network passes Defeated.

This implementation complex structure, kernel processor chip is more；Existing its integrated functionality of image procossing dsp chip More, but high-speed intelligent camera often only needs to use code segment function therein so as to function is superfluous；Dsp system needs Substantial amounts of outer internal memory supports the operation of its systems soft ware, and cost is not low, and hardware design difficulty is big；And software systems operation is in real time Property not, it postpones to mismatch with the acquisition of front end high speed image.

Content of the invention

In order to solve the above problems, the invention provides a kind of implementation method of devices at full hardware high-speed intelligent camera, including regarding Frequency coding and video flowing upload, and described Video coding is the video full I frame coding realized inside FPGA, and its process is to gather To view data sequentially pass through DCT computing, quantization operations and entropy code computing, the view data after last exports coding；Institute State video flowing upload for the internal only reservation CLOSED of embedded FPGA, SYN-RCVD, SYN-SEND, LISTEN, The simplification kilomega network ICP/IP protocol stack of ESTABLISHED5 kind state, simplifies and realizes difficulty in FPGA.

Preferably, the DCT computing of described Video coding employs the detached fast algorithm of ranks, and formula is：

DCT computing is discrete cosine transform (Discrete Cosine Transform), and conventional two-dimensional dct transform is public Formula is：

（K=0,1 ... 7；L=0,1 ... 7）；

Wherein：X (m, n) is two-dimentional DC.

This formula needs to consume substantial amounts of multiplication resources, and operation time longer it is impossible to meet the reality of high-speed intelligent camera When property require, therefore according in FPGA can high-speed parallel computing feature, one-dimensional transform is carried out to it, finally gives above-mentioned one Dimension expression formula, this one-dimensional operating structure and typical Fourier transformation closely, can utilize Fourier transformation completely In quick butterfly computation realize.

Preferably, the DCT computing of described Video coding is realized by using the quick butterfly computation in Fourier transformation.

Preferably, the DCT computing module of described Video coding uses finite state machine, first stores input image data View data is then calculated by input data register successively through butterfly processing element 1, one-dimensional DCT and butterfly processing element 2 Process, and result of calculation is stored in 32 result of calculation registers, finally by the view data of 32 result of calculation registers Exported by the data register that is sequentially outputted to from top to bottom.

Preferably, the DCT computing module of described Video coding by data from input data register to 32 result of calculations The calculating process of register carries out four subitem timing.

, based on unidirectional upload, video flowing speed is fast, intensive for the network transmission of camera, otherwise receives the requirement of networking command Not high.Therefore original numerous and diverse ICP/IP protocol stack targetedly can be simplified, only realizing camera needs as client The function of wanting, cancels FIN-WAIT-1, FIN-WAIT-2, CLOSING, TIME-WAIT, CLOSE-WAIT, LAST-ACK state, Wait, interrupt expense to reduce network, simplifying FPGA and realize program, but do not affect normal TCP/IP transfer process.Preferably, Described video flowing uploads and by the idiographic flow of kilomega network ICP/IP protocol stack is：

When a. setting up connection, client sends SYN bag to server, and enters SYN-SEND state, and waiting for server is true Recognize；

B. server receives SYN bag it is necessary to confirm the SYN of client, oneself also sends a SYN bag, i.e. SYN+ACK simultaneously Bag, now server entrance SYN-RCVD state；

If c. SYN-SEND state is not confirmed, return CLOSED state；If SYN-RCVD state is super in transmission When, then can return to CLOSED state；

D. client receives the SYN+ACK bag of server, sends to server and confirms bag ACK, and this bag is sent, client End server enters ESTABLISHED state；

During e.ESTABLISHED state, client and server start to transmit data；

F. after client and server transmissioning data terminate, client sends FIN or RST bag, and server sends RST bag, Simultaneously close off connection, return CLOSED state.

Preferably, the entirety of the implementation method of devices at full hardware high-speed intelligent camera is realized process and is：

1). IMAQ：The front end sensors of devices at full hardware high-speed intelligent camera are in the driving of front end sensors drive module Under carry out IMAQ, the sensor resolution that the present invention selects is 3360 × 2496,16 road LVDS outputs, and design achieves Under big resolution ratio, frame per second reaches 104 frames, and every road LVDS output linear velocity reaches 600Mbps；

2). Image Acquisition：Serioparallel exchange is carried out in image collection module to the image collecting, therefrom extracts complete A frame image data, and add upper frame synchronization and line synchronising signal；

3). Video coding：To the view data video full I frame coding after conversion it is achieved that the H264 of not frame losing encodes, and Compression ratio can real-time adjustment；

4). image buffer storage：The view data cache that Video coding is terminated in DDR3,

5). video flowing uploads：Eventually through the kilomega network ICP/IP protocol stack simplifying, video flowing is uploaded to terminal.

Preferably, described front end sensors drive module, image collection module, Video coding and kilomega network ICP/IP protocol Stack is all encapsulated in the hardware system of devices at full hardware high-speed intelligent camera.

In the present invention, all algorithms are realized all in FPGA, therefore FPGA from when it is necessary to there be enough logical resources, And difference IO will can support the linear velocity of 600Mbps, final selection is altera corp Cyclone4 series EP4CE115F23I7 chip.

The present invention is directed to the transmission feature of image/video stream, highly integrated, high-speed parallel computing using fpga chip Feature, according to the functional requirement of high-speed intelligent camera, realizes video full I frame coding inside FPGA, simplifies original loaded down with trivial details thousand Million net ICP/IP protocols, thus carrying out image procossing dsp chip in cropping the high-speed intelligent camera of prior art, eliminate numerous Miscellaneous DSP interlock circuit.The video full I frame coding realized inside FPGA, has carried out one to the two-dimensional dct transform of prior art Dimension conversion process, obtains the operating structure closely with typical Fourier transformation, it is possible to use fast in Fourier transformation Fast butterfly computation realizes the H264 coding it is achieved that not frame losing, and compression ratio can real-time adjustment.The kilomega network TCP/IP association simplifying View stack is that the network transmission according to high-speed intelligent camera requires, and general uploaded videos flow data amount is big and continuous, and transmitting order to lower levels leads to The few feature of track data amount, the kilomega network ICP/IP protocol stack that design simplifies, only retain so as to have high speed package, transmission Function, without the support of software systems, can be completely embedded into independent operating in FPGA program.Front end sensors are driven by the present invention Dynamic, Image Acquisition, video stream encryption, kilomega network TCP/IP transmission are entirely encapsulated in the method realize in hardware system, due to base In the framework of devices at full hardware, real-time, reliability is high, simplifies design difficulty, reduces design cost and fault rate.

Brief description

By reading the detailed description of hereafter preferred embodiment, various other advantages and benefit are common for this area Technical staff will be clear from understanding.Accompanying drawing is only used for illustrating the purpose of preferred embodiment, and is not considered as to the present invention Restriction.And in whole accompanying drawing, it is denoted by the same reference numerals identical part.In the accompanying drawings：

Fig. 1 is the coding structure block diagram of the present invention.

Fig. 2 is the DCT computing module of the present invention.

Fig. 3 be the present invention the simplification of TCP/IP state before and after comparison diagram.

Fig. 4 is the finite state machine schematic diagram of the TCP/IP that the present invention simplifies.

Fig. 5 is the overall architecture schematic diagram of the present invention.

Fig. 6 is the resource schematic diagram that takies in FPGA of whole camera programm of the present invention.

Specific embodiment

It is more fully described the illustrative embodiments of the disclosure below with reference to accompanying drawings.Although showing this public affairs in accompanying drawing The illustrative embodiments opened are it being understood, however, that may be realized in various forms the disclosure and the reality that should not illustrated here The mode of applying is limited.On the contrary, these embodiments are provided to be able to be best understood from the disclosure, and can be by this public affairs What the scope opened was complete conveys to those skilled in the art.

According to the embodiment of the present invention, the implementation method of a kind of devices at full hardware high-speed intelligent camera that the present invention provides, bag Include Video coding and video flowing uploads, as shown in figure 1, described Video coding is the video full I frame coding realized inside FPGA, Its process is the view data collecting to be sequentially passed through DCT computing, quantization operations and entropy code computing, after last exports coding View data；As shown in figure 3, described video flowing uploads only reservation CLOSED, SYN-RCVD, the SYN- within embedded FPGA The simplification kilomega network ICP/IP protocol stack of SEND, LISTEN, ESTABLISHED5 kind state, simplifies and realizes difficulty in FPGA.

The DCT computing of described Video coding employs the detached fast algorithm of ranks, and formula is：

DCT is discrete cosine transform (Discrete Cosine Transform), and conventional two-dimensional dct transform formula is：

(k=0,1 ... 7；L=0,1 ... 7)；

WhereinX (m, n) is two-dimentional DC.

This formula needs to consume substantial amounts of multiplication resources, and operation time longer it is impossible to meet the reality of high-speed intelligent camera When property require, therefore according in FPGA can high-speed parallel computing feature, one-dimensional transform is carried out to it, finally gives above-mentioned one Dimension expression formula, this one-dimensional operation structure and typical Fourier transformation closely, completely can be using in Fourier transformations Quick butterfly computation realize.

As shown in Figure 2, the DCT computing of Video coding of the present invention is by using the quick butterfly in Fourier transformation Computing is realized, and the DCT computing module of described Video coding uses finite state machine, first input data stored input data and post Data is then processed through butterfly processing element 1, one-dimensional DCT and butterfly processing element 2 calculating, and will calculate by storage successively Result is stored in 32 result of calculation registers, finally that the data of 32 result of calculation registers is defeated by order from top to bottom Go out and exported to data register.

The DCT computing module of described Video coding by data from input data register to 32 result of calculation registers Calculating process carries out four subitem timing.

, based on unidirectional upload, video flowing speed is fast, intensive for the network transmission of camera, otherwise receives the requirement of networking command Not high.Therefore original numerous and diverse ICP/IP protocol stack targetedly can be simplified, only realizing camera needs as client The function of wanting, as shown in figure 3, cancel FIN-WAIT-1, FIN-WAIT-2, CLOSING, TIME-WAIT, CLOSE-WAIT, LAST-ACK state, waits, interrupts expense to reduce network, simplifying FPGA and realize program, but not affecting normal TCP/IP and pass Defeated flow process.As shown in figure 4, described video flowing uploads by the idiographic flow of kilomega network ICP/IP protocol stack being：

When a. setting up connection, client sends SYN bag to server, and enters SYN-SEND state, and waiting for server is true Recognize；

B. server receives SYN bag it is necessary to confirm the SYN of client, oneself also sends a SYN bag, i.e. SYN+ACK simultaneously Bag, now server entrance SYN-RCVD state；

If c. SYN-SEND state is not confirmed, return CLOSED state；If SYN-RCVD state is super in transmission When, then can return to CLOSED state；

D. client receives the SYN+ACK bag of server, sends to server and confirms bag ACK, and this bag is sent, client End server enters ESTABLISHED state；

During e.ESTABLISHED state, client and server start to transmit data；

F. after client and server transmissioning data terminate, client sends FIN or RST bag, and server sends RST bag, Simultaneously close off connection, return CLOSED state.

A kind of entirety of the implementation method of devices at full hardware high-speed intelligent camera that the present invention provides is realized process and is：

1). IMAQ：The front end sensors of devices at full hardware high-speed intelligent camera are in the driving of front end sensors drive module Under carry out IMAQ, the sensor resolution that the present invention selects is 3360 × 2496,16 road LVDS outputs, and design achieves Under big resolution ratio, frame per second reaches 104 frames, and every road LVDS output linear velocity reaches 600Mbps；

2). Image Acquisition：Serioparallel exchange is carried out in image collection module to the image collecting, therefrom extracts complete A frame image data, and add upper frame synchronization and line synchronising signal；

3). Video coding：To the view data video full I frame coding after conversion it is achieved that the H264 of not frame losing encodes, and Compression ratio can real-time adjustment；

4). image buffer storage：The view data cache that Video coding is terminated in DDR3,

5). video flowing uploads：Eventually through the kilomega network ICP/IP protocol stack simplifying, video flowing is uploaded to terminal.

Described front end sensors drive module, image collection module, Video coding and kilomega network ICP/IP protocol stack all quilts It is encapsulated in the hardware system of devices at full hardware high-speed intelligent camera.

In the present invention, all algorithms are realized all in FPGA, therefore FPGA from when it is necessary to there be enough logical resources, And difference IO will can support the linear velocity of 600Mbps, final selection is altera corp Cyclone4 series EP4CE115F23I7 chip, takies resource as shown in Figure 6.

The present invention is directed to the transmission feature of image/video stream, highly integrated, high-speed parallel computing using fpga chip Feature, according to the functional requirement of high-speed intelligent camera, realizes video full I frame coding inside FPGA, simplifies original loaded down with trivial details thousand Million net ICP/IP protocols, thus carrying out image procossing dsp chip in cropping the high-speed intelligent camera of prior art, eliminate numerous Miscellaneous DSP interlock circuit.

The video full I frame coding realized inside FPGA, has been carried out at one-dimensional transform to the two-dimensional dct transform of prior art Reason, obtains the operating structure closely with typical Fourier transformation, it is possible to use the quick butterfly fortune in Fourier transformation Calculate and realize the H264 coding it is achieved that not frame losing, and compression ratio can real-time adjustment.

The kilomega network ICP/IP protocol stack simplifying is that the network transmission according to high-speed intelligent camera requires, general uploaded videos Flow data amount is big and continuous, the few feature of transmitting order to lower levels channel data amount, the kilomega network ICP/IP protocol stack that design simplifies, and only protects Stay so as to have high speed package, the function of transmission, without the support of software systems, can be completely embedded in FPGA program independent Run.

Front end sensors are driven by the present invention, Image Acquisition, video stream encryption, kilomega network TCP/IP transmission be entirely encapsulated in Realize in a piece of low cost EP4CE115F23I7, take resource as shown in Figure 6.Finally achieve 8,000,000 pixels, 102 frames per second High-speed video images, carry out standard 1:After 8 full I frame compression codings, given upper by the gigabit TCP/IP network transmission of standard Machine.

The above, the only present invention preferably specific embodiment, but protection scope of the present invention is not limited thereto, Any those familiar with the art the invention discloses technical scope in, the change or replacement that can readily occur in, All should be included within the scope of the present invention.Therefore, protection scope of the present invention should described with the protection model of claim Enclose and be defined.

Claims (7)

1. a kind of implementation method of devices at full hardware high-speed intelligent camera, upload including Video coding and video flowing it is characterised in that：Institute Stating Video coding is the video full I frame coding realized inside FPGA, and its process is to sequentially pass through the view data collecting DCT computing, quantization operations and entropy code computing, the view data after last exports coding；It is embedded that described video flowing uploads The simplification gigabit only retaining CLOSED, SYN-RCVD, SYN-SEND, LISTEN, ESTABLISHED5 kind state within FPGA Net ICP/IP protocol stack.

2. as claimed in claim 1 a kind of implementation method of devices at full hardware high-speed intelligent camera it is characterised in that：Described video is compiled The DCT computing of code employs the detached fast algorithm of ranks, and formula is：

$X(k,l)=\left\{\begin{array}{cc}\frac{1}{8}{\mathrm{\Σ}}_{p=1}^7\{F(p,r)+F(p,s)\}& p=1,3,5,7;l=0,2,4,6\\ \frac{1}{8}{\mathrm{\Σ}}_{p=1}^7\{G(p,r)+G(p,s)\}& p=1,3,5,7;l=1,3,5,7\end{array}\right.;$

$F(p,r)={\mathrm{\Σ}}_{m=0}^7\{x\[m,n(p;a)\]+x\[m,n(p;b)\]\}cos\left\{\frac{(2m+1r\mathrm{\π})}{16}\right\};$

$F(p,s)={\mathrm{\Σ}}_{m=0}^7\{x\[m,n(p;a)\]+x\[m,n(p;b)\]\}cos\left\{\frac{(2m+1s\mathrm{\π})}{16}\right\};$

$G(p,r)={\mathrm{\Σ}}_{m=0}^7{(-1)}^{q(p;m)}\{x\[m,n(p;a)\]+x\[m,n(p;b)\]\}cos\left\{\frac{(2m+1r\mathrm{\π})}{16}\right\};$

$G(p,s)={\mathrm{\Σ}}_{m=0}^7{(-1)}^{q(p;m)}\{x\[m,n(p;a)\]+x\[m,n(p;b)\]\}cos\left\{\frac{(2m+1s\mathrm{\π})}{16}\right\}.$

3. as claimed in claim 2 a kind of implementation method of devices at full hardware high-speed intelligent camera it is characterised in that：Described video is compiled The DCT computing of code is realized by using the quick butterfly computation in Fourier transformation.

4. as claims 1 to 3 a kind of any one described devices at full hardware high-speed intelligent camera implementation method, its feature exists In：The DCT computing module of described Video coding uses finite state machine, first input image data is stored input data and deposits View data is then processed through butterfly processing element 1, one-dimensional DCT and butterfly processing element 2 calculating, and will count by device successively Calculate result and be stored in 32 result of calculation registers, finally the view data of 32 result of calculation registers is pressed from top to bottom It is sequentially outputted to data register to be exported.

5. as claimed in claim 4 a kind of implementation method of devices at full hardware high-speed intelligent camera it is characterised in that：Described video is compiled View data is carried out four from input data register to the calculating process of 32 result of calculation registers by the DCT computing module of code Subitem timing.

6. as claimed in claim 1 a kind of implementation method of devices at full hardware high-speed intelligent camera it is characterised in that：Described video flowing Upload and by the idiographic flow of kilomega network ICP/IP protocol stack be：

When a. setting up connection, client sends SYN bag to server, and enters SYN-SEND state, and waiting for server confirms；

B. server receives SYN bag it is necessary to confirm the SYN of client, oneself also sends a SYN bag simultaneously, i.e. SYN+ACK bag, Now server enters SYN-RCVD state；

If c. SYN-SEND state is not confirmed, return CLOSED state；If SYN-RCVD state is in transmission time-out, CLOSED state can be returned to；

D. client receives the SYN+ACK bag of server, sends to server and confirms bag ACK, this bag is sent, client with Server enters ESTABLISHED state；

During e.ESTABLISHED state, client and server start to transmit data；

F. after client and server transmissioning data terminate, client sends FIN or RST bag, and server sends RST bag, simultaneously Close and connect, return CLOSED state.

7. as claimed in claim 1 a kind of implementation method of devices at full hardware high-speed intelligent camera it is characterised in that：It is integrally realized Process：

1). IMAQ：The front end sensors of devices at full hardware high-speed intelligent camera are entered under the driving of front end sensors drive module Row IMAQ；

2). Image Acquisition：Serioparallel exchange is carried out in image collection module to the image collecting, therefrom extracts complete one Frame image data, and add upper frame synchronization and line synchronising signal；

3). Video coding：To the view data video full I frame coding after conversion；

4). image buffer storage：The view data cache that Video coding is terminated in DDR3,

5). video flowing uploads：Eventually through the kilomega network ICP/IP protocol stack simplifying, video flowing is uploaded to terminal.