CN108616722B - Embedded high-definition video acquisition and data stream transmission system - Google Patents
Embedded high-definition video acquisition and data stream transmission system Download PDFInfo
- Publication number
- CN108616722B CN108616722B CN201810346509.XA CN201810346509A CN108616722B CN 108616722 B CN108616722 B CN 108616722B CN 201810346509 A CN201810346509 A CN 201810346509A CN 108616722 B CN108616722 B CN 108616722B
- Authority
- CN
- China
- Prior art keywords
- data
- image
- data packet
- video
- video stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/23614—Multiplexing of additional data and video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
- H04N21/6437—Real-time Transport Protocol [RTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8166—Monomedia components thereof involving executable data, e.g. software
- H04N21/8193—Monomedia components thereof involving executable data, e.g. software dedicated tools, e.g. video decoder software or IPMP tool
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Abstract
The invention discloses an embedded high-definition video acquisition and data stream transmission system, which is based on a Samsung S5P6818 core board, adopts an OV5640 camera integrated board card to acquire high-definition image data, adopts a multimedia hardware codec integrated in the core board to compress and encode images, extracts image characteristics by using an algorithm processing algorithm, and issues real-time video and characteristic parameter data streams to a local area network through a streaming media server built on the board. The remote PC client is designed and realized to receive video stream data and image characteristic parameter data, and the functions of historical query and the like are supported through decoding, displaying and local storage.
Description
Technical Field
The invention relates to the field of real-time video monitoring and analysis, in particular to an embedded high-definition video acquisition and data stream transmission system.
Background
With the wide application of machine vision in the industrial field, the development and use of a monitoring system to replace the existing manual observation mode in the industrial field and the all-weather online monitoring become possible. On one hand, the monitoring system acquires a real-time image of the area through the image acquisition device, and sends the image to a remote terminal for storage and viewing through a certain treatment process to replace a manual on-duty monitoring mode; on the other hand, an image processing algorithm and a working condition recognition algorithm are used for extracting and respectively distinguishing image characteristics of specified objects in the industrial field, and the actual state of the industrial field is judged according to the extraction result.
The existing monitoring system generally adopts the video stream obtained by collecting, and respectively adopts different transmission channels to respectively send data to the designated equipment after corresponding processing. With the expansion of the monitoring area in the industrial field, the number of monitoring devices is increased, and the management of a remote monitoring management terminal on the field monitoring devices is complicated, on one hand, different data use different sending channels, the terminal needs to designate a corresponding receiving thread for receiving and processing, and the performance requirement is increased due to the increase of the receiving channels; on the other hand, the wired connection mode of the monitoring device is not favorable for large-scale use due to the fact that a plurality of physical limitations are stored in the device deployment and movement.
In summary, the demand of distributed monitoring and data integration is taken as a driving force, and the existing embedded technology and machine vision technology are combined to research and design an embedded high-definition video acquisition and data stream transmission system, so that the functions of network distributed monitoring, analysis, guidance and the like are supported, and the development cost is reduced on the premise of meeting the performance demand of the system.
Disclosure of Invention
In view of the above-mentioned demand for distributed monitoring intelligence, based on the existing monitoring technology, the main object of the present invention is to develop and design an embedded high-definition video acquisition and data stream transmission system, so as to implement the functions of high-definition image acquisition, analysis, encoding and transmission on a platform with low hardware complexity.
In order to achieve the technical purpose, the technical scheme of the invention is as follows:
an embedded high-definition video acquisition and data stream transmission system comprises a field monitoring terminal deployed on the field and a remote management terminal deployed at a far end, wherein the remote management terminal is connected to the field monitoring terminal through network communication;
the image acquisition module comprises a camera integrated board card and is used for realizing the real-time acquisition of on-site images; the image processing and analyzing module extracts image characteristic data from the acquired image to obtain image characteristic data; the image compression coding module performs compression coding on the image to form a video stream; the video streaming media server module respectively packs the image characteristic data and the compressed and coded video stream, and establishes a streaming media server to send the data to a network; the terminal management module receives the image characteristic data and the video stream data packet, separates the content of the data packet, decodes and stores the content, and sends an image to the display device for display;
the image processing and analyzing module, the image compression and encoding module and the video streaming media server module are all integrated on the same integrated circuit board and form an integral embedded processing terminal.
According to the embedded flotation high-definition video stream acquisition and transmission system, the image acquisition module acquires high-definition color image data, and construction of original video data is completed on the basis.
The embedded high-definition video stream acquisition and transmission system is characterized in that the image processing and analysis module preprocesses one frame of image in original video data through a main thread, creates a thread pool containing a plurality of sub-threads by using a multithreading technology, then starts the sub-threads in the thread pool to respectively execute different image characteristic extraction tasks, stops any sub-thread after finishing corresponding characteristic parameter extraction results until all sub-threads finish characteristic parameter extraction, and sends the main thread to the video stream media server module for converging and data-packaging the extraction results of all sub-threads and then continuously processing the next frame of image.
According to the embedded high-definition video stream acquisition and transmission system, the main thread carries out preprocessing on an original image, including carrying out graying, binaryzation and watershed segmentation on the original image.
In the embedded high-definition video stream acquisition and transmission system, the image compression coding module performs cutting and compression coding processing on the acquired original video stream to obtain an H.264 video stream after compression coding, and the H.264 video stream is stored in a buffer area frame by frame for a video streaming media server module to reserve.
The embedded high-definition video stream acquisition and transmission system is characterized in that a video stream media server is built on a field monitoring terminal by a video stream media server module based on an RTP/RTCP protocol, namely, video streams and image characteristic data which are subjected to compression coding are packaged and encapsulated in advance according to transmission requirements, then the RTP/RTCP protocol is adopted for loading and completing transmission, and control feedback of a remote management terminal to network video streams is received.
The embedded high-definition video stream acquisition and transmission system is characterized in that the video stream and the image characteristic data which are subjected to compression coding are packaged and encapsulated, mixed video stream and image characteristic data are packaged and encapsulated, an expansion flag bit X of an RTP protocol data message is used as a mark for distinguishing in a data packet message which is packaged and encapsulated, two kinds of mixed data packets are loaded and sent through an established streaming media server, and a receiving end separates, frames and sequences the data packets according to the expansion flag bit after receiving the data packets.
In the embedded high-definition video stream acquisition and transmission system, the receiving end receives the data packets and then separates, frames and sequences the data packets according to the extended flag bits, and the method comprises the following steps:
when the size of a received RTP data packet is 12+3+ N bytes, the data packet is image characteristic parameter data, the data of N bytes after the data packet is read and is respectively stored in corresponding characteristic parameter structure bodies; otherwise, the data packet is a video stream data packet and is subjected to framing processing;
determining the relative sequence between image frames according to the time stamp in the RTP video stream data packet, and determining the sequence of the image fragments according to the sequence number;
constructing three double-buffer linked lists for storing and sequencing video stream data packets, setting a timestamp parameter for each linked list to represent a timestamp for organizing frames of the linked lists, wherein the three framing linked lists are divided into an idle linked list and a framing linked list, after a frame of image is grouped by the framing linked list, sending the image to a decoding thread for decoding the image, then putting the framing linked list into the idle linked list, and circularly using the three linked lists;
in the framing process, when the timestamp in the received RTP data packet is earlier than the current frame, the data packet is discarded as overtime processing, and the next RTP data packet is continuously received; when the received RTP data packet timestamp belongs to the current framing timestamp, inserting the RTP data packet timestamp into a correct position in a linked list by using a sorting algorithm; when the timestamp of the received RTP data packet is later than the current framing timestamp, namely the RTP data packet of the next frame image arrives, the earlier timestamp in all the framing linked lists is called, frame data of all elements of the earlier timestamp are spliced and copied to a decoding data buffer area, then the currently received RTP data packet is taken as a first element to be stored in the linked list, other elements are deleted, and the timestamp of the framing linked list is modified.
According to the embedded high-definition video stream acquisition and transmission system, the remote management terminal controls and feeds back the network video stream to control the field monitoring terminal to play the network video stream, wherein the playing operation comprises playing, stopping, pausing and adjusting video parameters.
According to the embedded high-definition video stream acquisition and transmission system, the terminal management module independently runs on a remote PC and is used for receiving and decoding stream media data packets, storing the stream media data packets locally, displaying the stream media data packets on an interface in real time, receiving image characteristic data, storing a local database and updating an interface chart.
The invention has the technical effects that: the image acquisition equipment adopted by the monitoring system supports image resolution and can support high-resolution and high-frame-rate output images, and the formed original image provides more details for subsequent image processing; the monitoring system selects a proper processor platform, and utilizes an internal integrated hardware codec to compress and encode the high-definition image, so that the video stream service efficiency is provided on the premise of ensuring the system performance; the monitoring system distributes the coded video stream to a network through a streaming media server, each monitoring terminal in the distributed network serves as an independent server to provide streaming media service, supports the acquisition and storage of the point-to-point video stream of a client, and supports the control of the client on the pause, the playing, the recording and the like of the real-time video stream; the client management software supports the functions of management, data storage, display and historical query of the industrial field monitoring terminal, and realizes remote feedback control of the embedded monitoring platform.
Drawings
FIG. 1 is a hardware overview design framework of the present invention;
FIG. 2 is a software overall design framework of the present invention;
FIG. 3 is a flowchart of an image capture process of the embedded processing terminal of the present invention;
FIG. 4 is a flowchart of the embedded processing terminal image compression hardware encoding of the present invention;
FIG. 5 is a flow chart of the image feature parameter extraction and network transmission of the embedded processing terminal according to the present invention;
FIG. 6 is a flow chart of the construction of the embedded processing terminal video streaming media server according to the present invention;
fig. 7 is a schematic diagram of remote PC client video stream reception and decoding of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
The overall hardware design framework of the invention is shown in fig. 1, and the system can be divided into an embedded processing terminal and a PC client from different deployment positions. The high-definition image acquisition device of the embedded processing terminal is composed of an OV5640 camera integrated board card. The embedded processing terminal is formed by expanding a peripheral circuit based on a Samsung S5P6818 multi-core processor and is divided into a high-definition image acquisition module, a memory module and a peripheral interface module on hardware according to different functions of each part. The peripheral interface module mainly comprises a power supply module, a debugging interface module, an HDMI display module and a network transmission module. The PC client is an additional module of the system and is used for checking monitoring videos in real time in an industrial field and remotely managing the embedded processing terminal.
An image acquisition module set interface of the embedded processing terminal is realized by using a DVP interface, an OV5640 camera integrated board card is connected to acquire RGB high-definition images, FIMC integrated in a core board is used as a camera controller to transmit the RGB images of the OV5640 camera, and an IIC bus is used for transmitting camera control signals. The power supply module is implemented by a power management chip TW6037, and is configured to provide different output voltages for modules of the embedded processing terminal. The memory module includes 2GB DDR3SDRAM memory and 16GB eMMC memory. The debugging interface module consists of a JTAG interface and a UART interface. The network transmission module consists of an Ethernet hub and an RJ45 interface. The HDMI display module is composed of a microHDMI type interface and is used for being connected with the HDMI display to display the monitoring video.
Based on the hardware platform system, the software composition architecture of the functions of the invention is shown in fig. 2. And completing the construction of embedded platform basic system software and the design of a driver of corresponding peripheral equipment, and providing a universal calling interface for an application layer by using a universal architecture. The application layer software mainly comprises a high-definition image acquisition module, an image characteristic parameter extraction and network transmission module, a video compression coding module, a video streaming media server building module and the like. The high-definition image acquisition mainly depends on driving an OV5640 high-definition camera to acquire industrial field images and obtain an original video data stream. The image characteristic parameter extraction module extracts characteristic parameters by operating a specific image characteristic extraction algorithm. And after the characteristic parameters of one frame of image are extracted, filling and packaging the image according to the packaged class. The video compression coding module performs H264 compression on the original color video through image format conversion and video coding by calling a multimedia hardware codec MFC integrated with the Samsung S5P6618, and reduces the data volume of video units so as to be suitable for network transmission. The video streaming media server is constructed based on an open-source real-time streaming media transmission open-source third-party library JRTPLIB, and a lightweight RTP/RTCP streaming media server side is realized. The video streaming media obtains H264 video stream and image characteristic data obtained by encoding through a multimedia hardware codec, and sends the data stream to a target remote client through operations such as packetization, encapsulation and the like.
The remote PC monitoring software is mainly used for monitoring and managing and mainly comprises a real-time data stream receiving module, a data packet processing module and a remote management module. The remote PC monitoring software is developed and designed based on a cross-platform C + + image user interface Qt development framework. The real-time data stream processing module receives data stream sent by a streaming media server on the embedded processing terminal, and separates a video stream data packet and an image characteristic data packet of the received data packet according to header information. And the video stream is decoded by calling an open source audio and video decoder library FFMPEG, and is synchronously displayed on the monitoring terminal software in real time and locally stored. The image characteristic data is stored based on the Mysql database construction, and the functions of local storage and synchronous display of data icons are realized. The remote management module supports historical data query and remote feedback control of the embedded processing terminal.
The image acquisition module of the embedded processing terminal completes the drive design of the OV5640 image acquisition board card, the image acquisition work is realized based on a V4l2 framework, and the realization steps are shown in FIG. 3 and specifically comprise the following steps:
(1) open OV5640 Camera device File/dev/video 0 using the open () function;
(2) using a VIDIOC _ QUERY AP command to inquire information such as a driver and a bus supported by an OV5640 camera, and confirming the normal operation of the OV5640 equipment related driver;
(3) acquiring a video format supported by an OV5640 camera by using a VIDIOC _ G _ FMT command;
(4) setting the acquisition resolution of an OV5640 camera to 1920 multiplied by 1080 by using a VIDIOC _ S _ FMT command, wherein the format is YUYV format for subsequent H264 coding;
(5) applying for three image frame buffers by using a VIDIOC _ REQBUFS command, and mapping the frame buffer buffers to a user space by using a mmap () function according to the frame buffer addresses and the offset thereof obtained by applying, so that a user can directly read image data from a memory in the user space;
(6) and putting all the frame buffers obtained by the previous application into video input buffers by using a VIDIOC _ QBUF command, and starting video acquisition of the OV5640 camera by using a VIDIOC _ STREAMON command. Meanwhile, putting the file descriptor of the OV5640 device into a monitoring file descriptor set;
(7) when the image frame is readable and the image acquisition program is not stopped, the image frame is taken out by using a VIDIOC _ DQBUF command and copied into a designated buffer, and the subsequent video compression coding and image characteristic parameter extraction of a user are respectively carried out. Then, the frame image frame buffer is put into the input video buffer again by using a VIDIOC _ QBUF command, and the step is called circularly before the image acquisition program is not stopped, so that the video stream data is continuously read from the OV5640 camera equipment.
(8) When the image acquisition program is finished, stopping the video stream acquisition of the OV5640 camera by using a VIDIOC _ STREAMOFF command, and releasing all the memory allocated previously to avoid memory leakage. And finally closing the camera equipment by using a close () function.
The image compression coding module of the embedded processing terminal realizes H264 hardware coding of real-time video stream based on a multimedia hardware codec MFC integrated in a Samsung S5P6818 core board. The calling of the hardware codec is integrated into a V4L2 framework, and the specific implementation is as shown in fig. 4, and the steps are as follows:
(1) the FIMC converter is turned on and initialized. And opening the FIMC device file by using an open () function, and setting the mapping types of the image data input and output addresses of the FIMC, wherein the input end is set as IO _ MMAP, and the output end is set as IO _ USERPTR. And finally, setting the sizes of the image frames of the input end and the output end of the FIMC.
(2) The MFC encoder is turned on and initialized. Similar to the FIMC, the MFC device file is opened using the open () function, and the MFC image data input-output address map types are all set to IO _ MMAP. Then the MFC input/output image frame size, the coding type and the frame rate are set. The target video stream format required here is h.264, so the MFC encoder type is set to V4L2_ PIX _ FMT _ H264. SUBSCRIBE to V4L2_ EVENT _ EOS service EVENTs using the video _ SUBSCRIBE _ EVENT command over the ioctl interface.
(3) And distributing buffer areas for each module, wherein the buffer areas comprise a Camera-FIMC buffer area, a FIMC-MFC buffer area and an MFC output buffer area, and after the buffer area distribution is completed, each buffer area is mounted on a module object appointed by a band and used for input and output in the image format conversion and coding process.
(4) H.264 coding of the original image. The states of the file descriptors of the camera device, the FIMC device and the MFC device are monitored by using a poll () function, when new data is readable on the device, an ioctl () interface function is called, and the data is read from an output queue of the device and stored in a designated buffer by using a vidooc _ DQBUF command. When the state of the next processing device is idle, the VIDIOC _ QBUF command is used for placing the buffer area which is stored with the data to be processed into the input queue of the next device for the next data processing. If the equipment is in a busy state, waiting is carried out, and if the equipment is in a termination state, the conversion and coding work is finished. By the method, reading, processing and outputting of the original data by the FIMC and reading, encoding and outputting of the format conversion image by the MFC are realized. And circulating the step before the stop signal is not sent by the encoding thread or the processing equipment to realize H.264 encoding of the original video stream.
(5) And when the coding thread receives the stop signal, calling a close () function to close the FIMC and the MFC equipment, and releasing the memory resource allocated by the coding thread to avoid memory leakage.
The image processing and analyzing module of the embedded processing terminal is used as one of the core functions designed by the system. The system adopts a multi-thread design idea to design an extraction process of the image characteristic parameters, and the specific implementation flow is shown in fig. 5. The image parameters are divided into different subtasks to be executed concurrently according to the difference of the image parameters. After a frame of image is processed by a plurality of subtasks, the feature data obtained by each subtask are merged and packaged, processed by a streaming media server and sent to a remote client.
The system carries out secondary development based on the third-party library JRTPLIB, and realizes the construction of the streaming media server on the embedded processing terminal. The implementation flow is shown in fig. 6, and the specific steps are as follows:
(1) creation and initialization of RTP sessions
Calling an RTPSession class in a JRTPLIB library to instantiate a transmission object for representing the RTP session, and calling a member function Create () function of the instantiated object to initialize the transmission object, wherein the member function Create () function includes transmission parameters such as a port number for specifying data transmission and other session parameters. The RTPSession class provides automatic processing of RTCP data packets, and after initialization of the RTPSession instantiation object is completed, two port numbers of the system are occupied, wherein one port number is used for data transmission, and the other port number is used as an RTCP protocol. In the data transmission process, the server side periodically sends SR packets and receives RR packets of the client side, and parameters such as transmission rate are dynamically adjusted according to packet information.
(2) Setting appropriate time stamp
And setting the RTP protocol timestamp meeting the requirement by calling a SetTimeStamp () function in the RTPSession class. Due to the limitation of MTU in network transmission, each frame of data may be sent in packets according to its length. In order for the receiving end to recover each frame of video image after receiving the RTP packet, the receiving end must identify, sequence and reassemble the data frames according to the timestamp in the header of the RTP packet.
(3) Adding and deleting RTP destination addresses
The RTPSession class supports managing a list of target addresses. And acquiring the IP and the port number of the remote client according to the connection with the remote client, adding the IP and the port number of the remote client into a target address list by calling a member function AddDestination () of a session object, and managing the IP and the port number by related member functions in the class. All destination addresses are stored in a hash table named destination, and the destination addresses are read from the hash table when video stream data is sent.
(4) Circularly sending data packets by using SendPacket () function
And calling a member function SendPacket () of the session object to send RTP data packets to all target addresses. The member function firstly calls another class RTPPacketBuilder in the JRTPLIB library, and encapsulates data to be sent into an RTP data packet through the member function BuildPacket () of the class. And calling a member function SendRTPData () of the RTPTransmiter class to read a target address, and calling a sendto () function to send an RTP data packet to the network. And circularly calling the member function to complete the real-time sending task of the data packet.
(5) Ending the session and destroying all resources
When the streaming media server is not needed to be used any more, the streaming media service thread needs to be ended, and resources occupied by the thread are destroyed, so that memory leakage is avoided. Wherein, the RTP session is ended by using a member function BYEDesroy () function in the RTPSession class.
The data packet sent by the streaming media server comprises a compressed and coded video stream and image characteristic parameter data obtained by image processing. In the process of sending a video stream data packet, because the amount of image data of a frame after compression coding is possibly large, the length of the data packet may exceed the MTU limit of 1400 bytes in network transmission, and the length of the data packet needs to be controlled when an RTP data packet is encapsulated; for the image characteristic parameter data packet, the size is much smaller than the limit of MTU, and a specific format needs to be specified for the receiving end to identify from the received data packet. Assuming that the image feature data size is fixed to N bytes, the present invention sets the first three bytes of the buffer to 0xFF by designating a transmission buffer to which N +3 bytes are allocated, and the remaining buffer is used for image feature data padding. In the process of sending by the streaming media server, the image characteristic buffer area and the video stream buffer area are inquired in turn and circularly, and when the inquired buffer area is not empty, the buffer area is emptied after being sent.
For the transmission of the video stream, corresponding processing needs to be performed according to the size of a single frame image, and the processing flow is as shown in the figure, and the specific steps are as follows:
(1) reading a NAL unit in a video stream buffer, wherein each NAL unit consists of a start code 0001 and NALU unit data content. The length of the data content of each NAL unit is calculated as the total length of the NAL minus its start code.
(2) Judging whether the data content length of each NAL unit is larger than 1388 bytes, if the length is smaller than 1388 bytes, performing packing processing according to the NAL unit mode, namely removing the start code of each NALU unit, and taking the remaining NALU data content (including NALU header and NALU data) as the payload of an RTP data packet; and when the length is more than 1388 bytes, packaging in an FU-A mode. Firstly, the length of data content of NALU is obtained, and fragmentation is carried out by using 1388 bytes of payload of each RTP data packet as a basic unit, wherein NALU data content of which the last fragmentation is less than 1388 bytes is used as the payload of one RTP data packet. And adding RTP data packet messages to each fragment in sequence to complete encapsulation.
(3) And setting parameters M, PT and timestamp. The parameter M takes one bit in the RTP header, marking the end of a video frame for a video stream. The parameter PT is used to specify the type of payload in the RTP packet and is set to 96 according to the standard. The timestamp is used for reflecting the sampling time of the first octet of the RTP message, and the receiver calculates delay and jitter according to the received timestamp and performs synchronous control. For NAL unit packet type, parameter M is set to 1, and timestamp enters its increment timestamp _ inc in SendPacket () function; for the FU-a type, all slices are set to 0 emutexcept for the last slice of the NAL unit set to 1. Although one NAL unit is subjected to FU-A slicing, the data is reassembled into one NAL unit after the receiving end receives the data, so the timestamp value of all FU-A slicing packets of the same NAL unit is the same.
(4) And after the NAL unit is encapsulated with an RTP data packet, the encapsulated data packet is sent by calling a specified network sending function.
(5) And before the streaming media service thread is not stopped, circularly repeating (2) the following process of the step to process the next frame image in the buffer area.
And finishing the data packet sending processing of the video stream buffer zone.
The remote PC client terminal is used as the auxiliary management software of the invention, and after an operator starts the monitoring software at the remote terminal, the monitoring software sends a connection request to the embedded processing terminal of the industrial field. And after the connection with the embedded processing terminal is successful, sending the port number of the video streaming media receiving end. After receiving the data packet, the streaming media data receiving module separates the data packet and decodes the video stream, and performs playing, screen capturing, recording, pausing and other playing operations on the data packet through an autonomously designed player. And after the image characteristic parameter data module is separated, drawing a real-time curve on a chart corresponding to a software interface, and storing the characteristic data into a Mysql database. And finally, the monitoring software supports the functions of inquiring historical characteristics and searching historical images through screening conditions through a historical data inquiry module.
The remote PC client side of the present invention transplants the JRTPLIB library on the windows system through the streaming media data packet receiving module, and similarly calls the function in the library to construct the session object in the embedded side, so as to realize the receiving of the data packet, and the receiving flow is shown in fig. 7, and the specific steps are as follows:
(1) initializing receiver RTP session instance objects
Similar to the construction of a streaming media server on an embedded processing terminal, a receiving terminal transmission session object is instantiated by calling an RTPSession class, and member functions in the instantiated object are called to initialize the receiving terminal transmission session object.
(2) Setting a reception mode and an address list
The JRTPLIB library defines three reception modes for specifying reception options for data from different senders by a receiving end. The program sets the receiving mode to RECEIVEMODE _ ACCEPSOME mode by calling the member function SetReceiveMode () of the session object, so that the client only receives the RTP data packet of the specified embedded terminal, and calls the AddToAcceptList () member function to add the IP address of the embedded processing terminal, thereby completing the setting of the receiving mode and the address list.
(3) Reception of RTP data packets
The JRTPLIB supports the simultaneous reception of RTP data packets of a plurality of data sources, and the RTP data packets of each data source are respectively stored in an independent receiving cache. According to the setting of the receiving mode in step (2), the receiving end designed herein can receive RTP packets of multiple data sources. The program first calls the member function Poll () in the session object to receive the RTP packet or RTCP packet, and then traverses all data sources by circularly calling its member functions GotoFirstSourceWithData () and GotoNextSourceWithData (). After the data source is successfully acquired, calling a member function BeginDataAccess () to lock the data source, calling a GetNextPacket () member function to acquire the next RTP data packet of the current specified data source, and after framing processing of the RTP data packet is completed, calling a DeletePacket () function to delete the RTP data packet and release the space resources occupied by the RTP data packet. And finally, calling an EndDataAccess () member function to unlock the data source. In this way, the reception of one complete RTP packet is completed.
(4) Separation, framing and sequencing of RTP packets
Each RTP packet is routed independently in network transmission, and the sequence of arriving at the receiving end cannot be guaranteed to be consistent with the sending sequence. Especially for NAL data units sent by packet processing, the NAL data units are directly assembled and restored according to the receiving sequence without processing, and the accuracy of the data cannot be ensured. Meanwhile, the RTP packet includes a video stream packet and a packet of image characteristic parameter data, and needs to be separated according to a setting. Therefore, the RTP packets need to be separated, framed, and ordered after being received. Considering the requirement of real-time performance, the program sets 3 frames of buffer images, and ensures that when the first frame of image is decoded, the second frame of image is received completely and the third frame of image is receiving.
The framing of the RTP packet is based on the timestamp and the sequence number in the RTP packet message. The time stamp of the frame image represents the acquisition time of the frame image, and the time stamps of all RTP data packets of the same frame image are the same. The sequence numbers indicate the relative positions of the slices in the complete image frame after image segmentation. Therefore, the relative order between the image frames is determined by means of the time stamps in the RTP data packets, and the ordering of the image slices is determined according to the sequence numbers.
According to the above analysis, the present program utilizes the flexibility of the linked lists in terms of data movement to construct three double-buffer linked lists for storing and sorting packets, and sets a timestamp parameter for each linked list to represent the timestamp of its organization frame. The three frame-group linked lists are divided into an idle linked list and a frame-group linked list, after a frame of image is grouped by the frame-group linked list, the frame-group linked list is sent to a decoding thread by a program to be decoded, then the frame-group linked list is put into the idle linked list, and the three linked lists are recycled.
When the size of a received RTP data packet is 12+3+ N bytes, namely the data packet is image characteristic parameter data, reading the data of the N bytes after the data packet, and respectively storing the data in corresponding characteristic parameter structure bodies; otherwise, the data packet is a video stream data packet, and framing processing is carried out on the data packet.
In the framing process, when the timestamp in the received RTP data packet is earlier than the current frame, the data packet is discarded as overtime processing, and the next RTP data packet is continuously received; when the received RTP data packet timestamp belongs to the current framing timestamp, inserting the RTP data packet timestamp into a correct position in a linked list by using a sorting algorithm; when the received RTP data packet time stamp is later than the current framing time stamp, the RTP data packet of the next frame image arrives. At this time, one of the frame-group lists with the earlier time stamp is called, and the frame data of all elements of the frame-group list is spliced and copied to a decoding data buffer area. Then, the RTP data packet received at present is taken as a first element to be stored in the linked list, other elements are deleted, and the timestamp of the framing linked list is modified. At this point, the framing process of the RTP packet is completed.
(5) End of RTP session
When the image receiving thread does not receive image data any more, before exiting the thread, calling a BYEDesroy () function in the session object to end the session, destroying system resources allocated by the thread and avoiding memory leakage.
The image decoding part of the remote PC client side decodes and zooms image frames based on an open-source audio and video decoding library FFMPEG. The specific steps for decoding the video frame are as follows:
(1) the register avcodec _ register _ all () function registers a codec. This function must be called before the decoder is defined and initialized in order to use the codec, parser, hardware accelerator, etc. in the FFMPEG decoding library.
(2) The registered decoder is found based on the specified encoder ID. The program finds the encoder of H264 by calling the avcodec _ find _ decoder () function, and the ID of H264 in the FFMPEG library is AV _ CODEC _ ID _ H264.
(3) The encoder is initialized. An AVCodecContext object is defined for describing the decoder context, which contains important parameter information required by the decoder, including the media stream TYPE and the decoded picture format, which are defined here as AVMEDIA _ TYPE _ VIDEO and AV _ PIX _ FMT _ YUV420P, respectively. The avcodec _ open2() function is then called to complete the build of the AVCodecContext object.
(4) And allocating memory for the decoded frame and the image conversion frame. Memory space is allocated to the decoded frame and the image conversion frame by calling an av _ frame _ alloc () function, and the memory space is used for storing the decoded and image-scaled image data respectively.
(5) Decoding of video frames and image scaling. After finding and obtaining the decoder, the program loops to check whether a new group of frames is generated in the buffer. And when the buffer is not empty, reading a video frame, decoding and locally storing the video frame by calling an avcodec _ decode _ video2() function, wherein the format of the decoded image is YUV 420P. And then calling a sws _ scale () function to perform format conversion and image scaling on the decoded image, and obtaining an RGB24 format image for interface implementation. After each video frame finishes decoding and scaling work, releasing a buffer area occupied by the video frame to serve as a storage space of the next video frame. This step is looped until the decode thread is terminated.
(6) Freeing up decoder and system resources. The program closes the decoder in use by calling the avcodec _ close () function, and releases the system resource allocated to the thread by calling the memory release function provided in the FFMPEG decoding library, and finally ends the receiving thread.
The Chart display and data query module of the remote PC client calls the QtChart module library API function provided by Qt in a program to be packaged into a Chart type Chart _ Builder which is designed autonomously, and packages the functions of Chart creation, destruction, parameter setting, data refreshing and the like into member functions. After the chart display thread is established, a chart control is placed on the main interface according to requirements, corresponding chart class instantiation objects are established in the thread, relevant setting is completed through a function interface provided by each instantiation object, and then the chart display thread waits for a characteristic data receiving thread to receive characteristic data. And after a complete characteristic data packet is obtained once, giving different parameters of the data to different chart objects, updating the next data by using a data refreshing interface, and refreshing the UI interface for display. The client sets two screening conditions for the data query function on the main interface, namely screening according to time and screening according to characteristics. As shown in the following figure, the screening results are respectively embodied in a graph mode and a curve trend graph mode, and the functional options of exporting the data to an Excel form are supported, so that subsequent statistics and local viewing are facilitated.
Claims (7)
1. An embedded high-definition video acquisition and data stream transmission system is characterized by comprising an on-site monitoring terminal deployed on the site and a remote management terminal deployed at a far end, wherein the remote management terminal is connected to the on-site monitoring terminal through network communication;
the image acquisition module comprises a camera integrated board card and is used for realizing the real-time acquisition of on-site images; the image processing and analyzing module extracts image characteristic data from the acquired image to obtain image characteristic data; the image compression coding module performs compression coding on the image to form a video stream; the video streaming media server module respectively packs the image characteristic data and the compressed and coded video stream, and establishes a streaming media server to send the data to a network; the terminal management module receives the image characteristic data and the video stream data packet, separates the content of the data packet, decodes and stores the content, and sends an image to the display device for display;
the image processing and analyzing module, the image compression and encoding module and the video streaming media server module are all integrated on the same integrated circuit board and form an integral embedded processing terminal;
the video streaming media server module builds a video streaming media server on a field monitoring terminal based on an RTP/RTCP protocol, namely, the video streaming media server module packs and encapsulates video stream and image characteristic data which are compressed and coded in advance according to transmission requirements, then adopts the RTP/RTCP protocol to load and complete sending, and receives control feedback of a remote management terminal to a network video stream;
the video stream and the image characteristic data which are compressed and coded are packaged and encapsulated, mixed video stream and image characteristic data are packaged and encapsulated, an expansion flag bit X of an RTP protocol data message is used as a mark to distinguish in a data packet message which is packaged and encapsulated, two kinds of mixed data packets are loaded and sent through an established streaming media server, and a receiving end separates, frames and sequences the data packets according to the expansion flag bit after receiving the data packets;
the receiving end separates, frames and sequences the data packet according to the extended flag bit after receiving the data packet, and the method comprises the following steps:
when the size of a received RTP data packet is 12+3+ N bytes, the data packet is image characteristic parameter data, the data of N bytes after the data packet is read and is respectively stored in corresponding characteristic parameter structure bodies; otherwise, the data packet is a video stream data packet and is subjected to framing processing;
determining the relative sequence between image frames according to the time stamp in the RTP video stream data packet, and determining the sequence of the image fragments according to the sequence number;
constructing three double-buffer linked lists for storing and sequencing video stream data packets, setting a timestamp parameter for each linked list to represent a timestamp for organizing frames of the linked lists, wherein the three framing linked lists are divided into an idle linked list and a framing linked list, after a frame of image is grouped by the framing linked list, sending the image to a decoding thread for decoding the image, then putting the framing linked list into the idle linked list, and circularly using the three linked lists;
in the framing process, when the timestamp in the received RTP data packet is earlier than the current frame, the data packet is discarded as overtime processing, and the next RTP data packet is continuously received; when the received RTP data packet timestamp belongs to the current framing timestamp, inserting the RTP data packet timestamp into a correct position in a linked list by using a sorting algorithm; when the timestamp of the received RTP data packet is later than the current framing timestamp, namely the RTP data packet of the next frame image arrives, the earlier timestamp in all the framing linked lists is called, frame data of all elements of the earlier timestamp are spliced and copied to a decoding data buffer area, then the currently received RTP data packet is taken as a first element to be stored in the linked list, other elements are deleted, and the timestamp of the framing linked list is modified.
2. The embedded high definition video capture and data streaming system of claim 1, wherein the image capture module obtains high definition color image data and completes construction of original video data based thereon.
3. The embedded high-definition video acquisition and data stream transmission system according to claim 1, wherein the image processing analysis module preprocesses a frame of image in original video data through a main thread, creates a thread pool containing a plurality of sub-threads by using a multithreading technology, then starts the sub-threads in the thread pool to respectively execute different image feature extraction tasks, stops any sub-thread after completing a corresponding feature parameter extraction result until all sub-threads complete feature parameter extraction, converges and packages the extraction results of all sub-threads by the main thread, sends the collection results to the video stream media server module, and then continues to process the next frame of image.
4. The embedded high definition video capture and data stream transmission system of claim 3, wherein the main thread pre-processes the original image including graying, binarization and watershed segmentation.
5. The system according to claim 1, wherein the image compression and encoding module performs clipping and compression and encoding on the acquired original video stream to obtain an h.264 video stream, and stores the h.264 video stream in the buffer area frame by frame for the video streaming media server module to use.
6. The embedded high-definition video collection and data stream transmission system according to claim 1, wherein the control feedback of the remote management terminal on the network video stream is to control the on-site monitoring terminal to perform playing operation of the network video stream, including playing, terminating, pausing and adjusting video parameters.
7. The embedded high-definition video acquisition and data stream transmission system according to claim 1, wherein the terminal management module runs independently on a remote PC, and is used for receiving, decoding, storing locally and displaying interface real-time of streaming media data packets, receiving image characteristic data, storing local database and updating interface chart.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810346509.XA CN108616722B (en) | 2018-04-18 | 2018-04-18 | Embedded high-definition video acquisition and data stream transmission system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810346509.XA CN108616722B (en) | 2018-04-18 | 2018-04-18 | Embedded high-definition video acquisition and data stream transmission system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108616722A CN108616722A (en) | 2018-10-02 |
| CN108616722B true CN108616722B (en) | 2021-02-26 |
Family
ID=63660310
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810346509.XA Active CN108616722B (en) | 2018-04-18 | 2018-04-18 | Embedded high-definition video acquisition and data stream transmission system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108616722B (en) |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109729284A (en) * | 2018-12-17 | 2019-05-07 | 惠州Tcl移动通信有限公司 | Image processing method, intelligent terminal and the storage device of intelligent terminal |
| CN111343415A (en) * | 2018-12-18 | 2020-06-26 | 杭州海康威视数字技术股份有限公司 | Data transmission method and device |
| CN110062296B (en) * | 2019-04-09 | 2021-06-29 | 南昌嘉研科技有限公司 | Real-time streaming media player and architecture method thereof |
| CN110351533A (en) * | 2019-08-12 | 2019-10-18 | 桂林电子科技大学 | The network transmission monitoring system and its implementation of video image processing based on Linux |
| CN110545449A (en) * | 2019-08-30 | 2019-12-06 | 华中科技大学 | Industrial field data transmission method based on streaming media |
| CN112468887A (en) * | 2019-09-06 | 2021-03-09 | 杭州海康微影传感科技有限公司 | Thermal imaging data transmission method and device and thermal imaging equipment |
| CN110650307A (en) * | 2019-10-30 | 2020-01-03 | 广州河东科技有限公司 | QT-based audio and video plug flow method, device, equipment and storage medium |
| CN110650308A (en) * | 2019-10-30 | 2020-01-03 | 广州河东科技有限公司 | QT-based audio and video stream pulling method, device, equipment and storage medium |
| CN111125978A (en) * | 2019-12-31 | 2020-05-08 | 国微集团(深圳)有限公司 | Simulation data processing method and system |
| CN111800388A (en) * | 2020-06-09 | 2020-10-20 | 盐城网之易传媒有限公司 | Media information processing method and media information processing device |
| CN112565679A (en) * | 2020-11-03 | 2021-03-26 | 合肥工业大学 | Monitoring method and monitoring device based on machine vision |
| CN112486487B (en) * | 2020-11-30 | 2023-05-23 | 中船重工（武汉）凌久电子有限责任公司 | SoC NVR display system-based rapid development method |
| CN112487034A (en) * | 2020-12-01 | 2021-03-12 | 上海思客琦智能装备科技股份有限公司 | Double-queue asynchronous image processing method and device |
| CN113473106A (en) * | 2021-06-18 | 2021-10-01 | 青岛小鸟看看科技有限公司 | Image transmission method, image display and processing device, and image transmission system |
| CN113709495B (en) * | 2021-08-24 | 2023-11-24 | 天津津航计算技术研究所 | Synchronous decoding method based on PCIE buffer mechanism |
| CN113709518B (en) * | 2021-08-24 | 2023-11-28 | 天津津航计算技术研究所 | Real-time video transmission mode design method based on RTSP protocol |
| CN113703948A (en) * | 2021-09-03 | 2021-11-26 | 四川宇德中创信息科技有限公司 | Test paper splitting system and splitting method thereof |
| CN114584538B (en) * | 2022-03-01 | 2024-03-22 | 北京智慧星光信息技术股份有限公司 | Mobile streaming media data transmission method, device and storage medium |
| CN114363583A (en) * | 2022-03-17 | 2022-04-15 | 长沙金维信息技术有限公司 | Linux-based embedded video monitoring system construction method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101771599A (en) * | 2008-12-26 | 2010-07-07 | 中国移动通信集团公司 | Method and device for receiving RTP data packets |
| CN102547239A (en) * | 2011-12-29 | 2012-07-04 | 浙江工业大学 | Android mobile terminal-based video monitoring system |
| CN104980682A (en) * | 2015-07-02 | 2015-10-14 | 湖北贝力思智能科技有限公司 | Intelligent dynamic high-definition video transmission system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100518114C (en) * | 2006-05-29 | 2009-07-22 | 腾讯科技（深圳）有限公司 | Point-to-point data transmission method and system |
| CN101984638B (en) * | 2010-11-10 | 2013-05-15 | 河海大学常州校区 | Storage agent system used for cross-IPv4 and IPv6-network processing and method thereof |
| CN103457874A (en) * | 2012-05-28 | 2013-12-18 | 华为技术有限公司 | Data package scheduling method and data package scheduling device |
| CN104835212A (en) * | 2015-05-19 | 2015-08-12 | 湖南大麓管道工程有限公司 | Pipeline patrol, maintenance and checking method |
-
2018
- 2018-04-18 CN CN201810346509.XA patent/CN108616722B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101771599A (en) * | 2008-12-26 | 2010-07-07 | 中国移动通信集团公司 | Method and device for receiving RTP data packets |
| CN102547239A (en) * | 2011-12-29 | 2012-07-04 | 浙江工业大学 | Android mobile terminal-based video monitoring system |
| CN104980682A (en) * | 2015-07-02 | 2015-10-14 | 湖北贝力思智能科技有限公司 | Intelligent dynamic high-definition video transmission system |
Non-Patent Citations (2)
| Title |
|---|
| 嵌入式网络视频监控系统的研究与实现;徐箐;《中国优秀硕士学位论文全文数据库 信息科技辑》;20110415(第04期);第I139-259页 * |
| 嵌入式高清泡沫图像监控系统的研究与设计;胡峻;《中国优秀硕士学位论文全文数据库 工程科技Ⅰ辑》;20130315(第03期);第B021-670页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108616722A (en) | 2018-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108616722B (en) | Embedded high-definition video acquisition and data stream transmission system | |
| US10924783B2 (en) | Video coding method, system and server | |
| CN110022297B (en) | High-definition video live broadcast system | |
| US10223796B2 (en) | Adaptive video streaming | |
| JP3862321B2 (en) | Server and control method thereof | |
| US8843983B2 (en) | Video decomposition and recomposition | |
| CN113423018A (en) | Game data processing method, device and storage medium | |
| CN109922319B (en) | RTSP (real time streaming protocol) multi-video-stream parallel preprocessing method based on multi-core CPU (central processing unit) | |
| US20090016622A1 (en) | Image transmitting apparatus, image transmitting method, receiving apparatus, and image transmitting system | |
| CN104243920A (en) | Image stitching method and device based on basic stream video data packaging | |
| CN112653913A (en) | Screen projection method and device, client device and readable storage medium | |
| US20160352798A1 (en) | Systems and methods for capture and streaming of video | |
| CN112601096A (en) | Video decoding method, device, equipment and readable storage medium | |
| US20140196102A1 (en) | Method for transmitting video signals from an application on a server over an ip network to a client device | |
| US20100049832A1 (en) | Computer program product, a system and a method for providing video content to a target system | |
| CN106412518A (en) | Wireless video transmission system based on TD-LTE emergency communication | |
| CN108156414B (en) | WebRTC data encoding method, equipment and computer readable storage medium | |
| CN108124183B (en) | Method for synchronously acquiring video and audio to perform one-to-many video and audio streaming | |
| CN110062296B (en) | Real-time streaming media player and architecture method thereof | |
| US20170041364A1 (en) | System and Method for Multiplexed Video Stream Decoding in Web Browser | |
| CN113950825A (en) | Data transmission method, data processing method, movable platform, terminal equipment and chip | |
| CN110798700A (en) | Video processing method, video processing device, storage medium and electronic equipment | |
| CN103916671A (en) | OSD dynamic overlaying method and device | |
| CN106331591B (en) | The method for synchronously switching and device of monitored picture | |
| Xueya et al. | Development of wireless mobile video surveillance on windows mobile using DirectShow technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |