CN114727070A - Camera real-time video data processing method and device - Google Patents
Camera real-time video data processing method and device Download PDFInfo
- Publication number
- CN114727070A CN114727070A CN202210504413.8A CN202210504413A CN114727070A CN 114727070 A CN114727070 A CN 114727070A CN 202210504413 A CN202210504413 A CN 202210504413A CN 114727070 A CN114727070 A CN 114727070A
- Authority
- CN
- China
- Prior art keywords
- data
- video data
- camera
- real
- component
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims description 8
- 238000012545 processing Methods 0.000 claims abstract description 65
- 238000000034 method Methods 0.000 claims abstract description 52
- 230000008569 process Effects 0.000 claims abstract description 27
- 230000003993 interaction Effects 0.000 claims abstract description 16
- 230000006870 function Effects 0.000 claims abstract description 7
- 238000004891 communication Methods 0.000 claims description 20
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000011161 development Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 238000013461 design Methods 0.000 abstract description 4
- 238000001514 detection method Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
Abstract
The invention discloses a method and a device for processing real-time video data of a camera, which are characterized in that a streaming pipeline design is carried out by a gstreamer streaming media frame, the real-time video data of the camera is collected, a connection shared memory is created, the video data format is converted into a format required by output by configuring a corresponding plug-in according to the streaming mode format output by the camera, finally, an application program interaction assembly is connected to store a video data frame to a shared memory annular buffer area, other application processes acquire the real-time video data from the shared memory, the multi-process interaction of the real-time video data is realized, the problem of resource exclusivity of starting the camera is solved, compared with the method that an opencv library function is directly called to carry out video collection processing, and the collection efficiency of a system is also improved to a certain extent.
Description
Technical Field
The invention relates to the technical field of video processing, in particular to a method and a device for processing real-time video data of a camera.
Background
The MIPI-CSI interface camera is widely applied to a plurality of embedded products such as a vehicle traveling data recorder, a law enforcement instrument and a evidence obtaining system, multi-process video image acquisition processing is needed in some application scenes, but the MIPI-CSI interface camera cannot be started by different application programs at the same time, namely a certain camera is started in one process, the same camera cannot be started in the other process, and camera resources are monopolized.
The Gstreamer multimedia framework can be used for engineering development through a code direct encapsulation command, adopts a plug-in implementation mode, has good reusability of program codes, can flexibly cut plug-ins according to business requirements, has good expansibility, and is suitable for application fields such as embedded type and mobile terminals. The Gstreamer can realize a series of media solutions such as collection, encoding, decoding, rendering, filtering and the like. In an embedded image processing device system, sometimes, multiple processes are required to process real-time video data, one process starts a camera (for example, a camera of an MIPI-CSI2 interface adopted in this patent), resources are monopolized, multiple application processes that need to acquire video data cannot acquire video data directly, an udp network is usually adopted to send an RTP stream, or a server stream pushing mode (rtsp stream, rtmp stream) is adopted to realize real-time video data interaction of multiple processes, an existing patent document CN202011554999.6 discloses a high-concurrency video processing method and system for sharing data by multiple processes, and an existing patent document CN110891178A discloses a method and device for video real-time rendering, which all need to process through a coding and decoding process. However, for the embedded image processing device with low power consumption, the above methods all involve the encoding and decoding process and the micro server push stream loading, which increases the power consumption of the system, and especially causes the system to be overloaded and run slowly when multiple videos are processed.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a method and a device for processing real-time video data of a camera.
In order to achieve the purpose, the invention adopts the following technical scheme:
a camera real-time video data processing method comprises the following steps:
s1, according to the number of the configured cameras, starting corresponding number of threads to manage each path of real-time video;
s2, constructing a gstreamer pipeline in each thread to be responsible for the conversion processing of each path of video data;
s3, at least one video source component is included in the gstreamer pipeline, and the video data output by the camera is acquired through the video source component;
s4, the gstreamer pipeline at least comprises a reasoning component, and the reasoning component converts the video data into a video frame format required to be processed to obtain video frame data;
s5, the gstreamer pipeline comprises an application program interaction component, and the application program interaction component puts the video data frame into a ring-shaped shared memory area;
and S6, connecting a plurality of application processes needing to acquire video data to the annular shared memory area, and reading the video frame data from the annular shared memory area for processing.
Further, the step S2 further includes: each component attribute in the gstreamer pipeline is implemented by a standard GObject object method.
Further, the video source component is V4L2 src.
Further, the application program interaction component is an applink component.
Further, the step of transferring the sampled data of the appink component to the ring-shaped shared memory area includes:
s01, setting the emit-scales attribute of the appsink component in each gstreamer pipeline as "true";
s02, connecting the new-sample sampling signal of the applink component;
s03, storing the received video frame data in a buffer area of the constructed annular shared memory area in a message callback function, and setting the segment name of the video frame data;
and S04, acquiring real-time video frame data for processing in a plurality of application processes needing to acquire video data according to the corresponding annular shared memory area connected by the segment name.
The utility model provides a real-time video data processing apparatus of camera, real-time video data processing apparatus of camera is used for realizing the hardware device of real-time video data processing method of camera, includes camera group, data main control unit and display control screen, camera group with data main control unit connects, data main control unit with display control screen connects, main control unit includes data processing module, communication module and portable storage medium, data main control unit passes through communication module is connected with the customer end that needs to acquire video data, the external portable storage medium that removes of data processing module, communication module pass through usb with data processing module carries out data transmission.
Further, the camera group is an MIPI camera group.
Furthermore, the communication module is a 4G/WIFI module.
Furthermore, the cameras in the camera group are connected with the data processing module through usb, the communication module is fixedly connected with the data processing module through cylindrical firmware, the communication module is located above the data processing module, and the data processing module is an Nvdia Jeson series development board.
The invention has the beneficial effects that:
the invention relates to a method and a device for processing real-time video data of a camera, which are characterized in that a streaming pipeline design is carried out by a gstreamer streaming media frame, the real-time video data of the camera is collected, a connection shared memory is created, the video data format is converted into a format required by output by configuring a corresponding plug-in according to the streaming mode format output by the camera, finally, an application program interaction assembly is connected to store a video data frame to a shared memory annular buffer area, and a plurality of application processes needing to acquire the video data acquire the real-time video data from the shared memory, thereby realizing the multi-process interaction of the real-time video data and solving the problem of resource exclusivity of the opening of the camera. Meanwhile, compared with directly calling opencv library functions to carry out video acquisition processing, the acquisition efficiency of a system formed on the basis of the method and the device is improved to a certain extent.
Drawings
FIG. 1 is a schematic diagram illustrating a design of a real-time video multiprocess data interaction interface based on a gstreamer framework according to the present invention;
FIG. 2 is a schematic flow chart illustrating the transfer of sampled data from an appink device to the ring-shaped shared memory region according to the present invention;
FIG. 3 is a schematic diagram illustrating a connection mode of modules in a camera real-time video data processing apparatus according to the present invention;
FIG. 4 is a schematic structural diagram of a camera real-time video data processing apparatus according to an embodiment of the present invention;
reference numerals: 1. a data processing module; 2. a camera group; 3. and the 4G/wifi module.
Detailed Description
The invention will be further described with reference to the accompanying drawings and the detailed description below:
as shown in fig. 1, a method for processing real-time video data of a camera includes the following steps:
s1, according to the number of the configured cameras, starting a corresponding number of threads to manage each path of real-time video;
s2, constructing a streamer Pipeline (Pipeline) in each thread to be responsible for the conversion processing of each path of video data; each component attribute in the gstreamer pipeline is realized by a standard GObject object method, and the method of using GObject can inquire, set and acquire the value of the attribute for the gstreamer.
S3, at least one video source component is included in the gstreamer pipeline, and the video data output by the camera is acquired through the video source component; the video source component is V4L2 src.
S4, the gstreamer pipeline at least comprises a reasoning component, and the reasoning component converts the video data into a video frame format required to be processed to obtain video frame data;
s5, the gstreamer pipeline comprises an application program interaction component, and the application program interaction component puts the video data frame into a ring-shaped shared memory area; the application program interaction component is an applink component.
And S6, connecting a plurality of application processes needing to acquire video data to the annular shared memory area, and reading the video frame data from the annular shared memory area for processing.
Further, as shown in fig. 2, an interactive implementation method of video data and a shared memory of a gstreamer pipeline (taking an applink component (application program interaction component) as an example), where the step of transferring the sampled data of the applink component to the ring-shaped shared memory area includes:
s01, setting the emit-scales attribute of the appsink component in each gstreamer pipeline as "true";
s02, connecting the new-sample sampling signal of the applink component;
s03, storing the received video frame data in a buffer area of the constructed annular shared memory area in a message callback function, and setting the segment name of the video frame data;
and S04, acquiring real-time video frame data for processing in a plurality of application processes needing to acquire video data according to the corresponding annular shared memory area connected by the segment name.
As shown in fig. 3, a camera real-time video data processing device is a hardware device for implementing a camera real-time video data processing method, and includes a camera group 2, a data main control unit and a display control screen, where the camera group 2 is connected to the data main control unit, the data main control unit is connected to the display control screen, and the main control unit includes a data processing module 1, a communication module and a removable storage medium. The data master control unit is connected with a client end needing to acquire video data through the communication module, the data processing module is externally connected with a mobile storage medium, and the communication module performs data transmission with the data processing module through the usb.
The camera group 2 is used for outputting a real-time video, and the camera group 2 at least comprises one camera; the data processing module 1 is configured to obtain video data, convert the video data into a video frame format of a video to be processed, obtain video frame data, and further place the video frame data in the ring-shaped shared memory area. The removable storage medium is used for storing video frame data; the communication module can transmit the video frame processing data to an application process of the client; the communication module can transmit the video frame processing data to one or more application processes; the display control screen is used as a display and control terminal of the data processing module and can be configured or not.
Further, the camera group 2 is an MIPI camera group.
Further, the communication module is a 4G/WIFI module 3.
Further, the cameras in the camera group 2 are connected with the data processing module through usb, the communication module is fixedly connected with the data processing module 1 through cylindrical firmware, the communication module is located above the data processing module 1, and the data processing module is an Nvdia Jeson series development board.
As shown in FIG. 4, the multi-channel camera 2 is connected to the data processing module 1 through the usb, the 4G/wifi module 3 is fixed with the data processing module 1 through the cylindrical firmware, data are transmitted through the usb interface, and the data processing module 1 can be externally connected with a mobile storage medium.
The technical scheme of the invention is further explained by taking the double-process interactive application of the local video and the target object detection application as an example:
in the embodiment, the data processing module 1 adopts an Nvdia Jeson series Jetson Nano development board, is matched with an Ubuntu 18.04.2linux operating system, is configured with two VEYE-MIPI-327 model cameras, outputs an image resolution of 1920 x 1080, and realizes startup of a local video and target object detection application, and the target object detection adopts a deep learning framework based on Darknet.
Two threads are started, a gstreamer pipeline is constructed in each thread and is responsible for conversion processing of each video data, a V4L2src component is adopted in the pipeline to obtain UYVY mode data output by an MIPI-CSI2 camera, an nvvidconv video converter component is adopted to convert video frames into a 420I mode, pipeline stream data are distributed through a tee component, and two src pad distribution threads are created through a queue 2:
and one path of the video stream is encoded into h264 format through an encoder component omxh264enc, a matriskamuxc component encapsulates and converts the received data into a matriskal stream, and finally the pipeline stream is connected with a fileink component to write the pipeline stream into a file for video storage. And the other path continues to load a video format conversion component nvvidconv, converts video stream data from a 420I format into a BGRX format, loads a video converter video color space conversion component, converts received BGRX format data into a BGR format, finally connects an applink component to place video data frames in the BGR format into a ring-shaped shared memory area, and connects a darknet target detection application process to the created ring-shaped shared memory area to acquire image data from the ring-shaped shared memory area for vehicle target detection.
In an open-source Darknet target detection original frame, video data are obtained by calling a camera video acquisition function through an opencv software library, the two paths of video target detection processing frame rates are about 6 frames, a gstreamer pipeline is constructed to store the video data to a shared memory, a target detection process obtains video frame data from the shared memory, the two paths of video target detection processing frame rates can reach about 8 frames, and the system processing efficiency is improved to a certain extent.
In the embodiment, real-time video data of the camera is transmitted to the annular shared memory area through constructing a pipeline design based on a gstreamer, and is interacted with other application processes, so that the problem of exclusivity of the camera is solved, local video of a video in an independent process and target identification of a vehicle object are realized, and in the application process of vehicle target identification, compared with the method for directly calling an opencv video function to obtain an image frame, the method provided by the invention has the advantages that the video acquisition efficiency is improved to a certain extent, and the operation efficiency of the target identification process is improved.
Various other modifications and changes may be made by those skilled in the art based on the above-described technical solutions and concepts, and all such modifications and changes should fall within the scope of the claims of the present invention.
Claims (10)
1. A camera real-time video data processing method is characterized by comprising the following steps:
s1, according to the number of the configured cameras, starting a corresponding number of threads to manage each path of real-time video;
s2, constructing a gstreamer pipeline in each thread to be responsible for the conversion processing of each path of video data;
s3, at least one video source component is included in the gstreamer pipeline, and the video data output by the camera is acquired through the video source component;
s4, the gstreamer pipeline at least comprises a reasoning component, and the reasoning component converts the video data into a video frame format required to be processed to obtain video frame data;
s5, the gstreamer pipeline comprises an application program interaction component, and the application program interaction component puts the video data frame into a ring-shaped shared memory area;
and S6, connecting a plurality of application processes needing to acquire video data to the annular shared memory area, and reading the video frame data from the annular shared memory area for processing.
2. The method as claimed in claim 1, wherein the video data output from the camera is in a stream mode format, and the video frame format is in a BGR format.
3. The method for processing the real-time video data of the camera according to claim 1, wherein the step S2 further includes: each component attribute in the gstreamer pipeline is implemented by a standard GObject object method.
4. The method of claim 1, wherein the video source component is V4L2 src.
5. The method for processing real-time video data of a camera according to claim 1, wherein the application program interaction component is an applink component.
6. The method of claim 5, wherein the step of transferring the sampled data of the applink component to the ring-shaped shared memory area comprises:
s01, setting the emit-scales attribute of the appsink component in each gstreamer pipeline as "true";
s02, connecting the new-sample sampling signal of the applink component;
s03, storing the received video frame data in a buffer area of the constructed annular shared memory area in a message callback function, and setting the segment name of the video frame data;
and S04, acquiring real-time video frame data for processing in a plurality of application processes needing to acquire video data according to the corresponding annular shared memory area connected by the segment name.
7. The camera real-time video data processing device is characterized in that the camera real-time video data processing device is used for realizing a hardware device of the camera real-time video data processing method in claim 1, and comprises a camera group, a data main control unit and a display control screen, wherein the camera group is connected with the data main control unit, the data main control unit is connected with the display control screen, the main control unit comprises a data processing module, a communication module and a movable storage medium, the data main control unit is connected with a client side needing to acquire video data through the communication module, the data processing module is externally connected with the movable storage medium, and the communication module carries out data transmission with the data processing module through usb.
8. The device of claim 7, wherein the camera group is a MIPI camera group.
9. The device for processing real-time video data of a camera according to claim 7, wherein the communication module is a 4G/WIFI module.
10. The device for processing the real-time video data of the cameras according to claim 7, wherein the cameras in the camera group are connected with the data processing module through usb, the communication module is fixedly connected with the data processing module through cylindrical firmware, the communication module is located above the data processing module, and the data processing module is an Nvdia Jeson series development board.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210504413.8A CN114727070A (en) | 2022-05-10 | 2022-05-10 | Camera real-time video data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210504413.8A CN114727070A (en) | 2022-05-10 | 2022-05-10 | Camera real-time video data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114727070A true CN114727070A (en) | 2022-07-08 |
Family
ID=82231528
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210504413.8A Pending CN114727070A (en) | 2022-05-10 | 2022-05-10 | Camera real-time video data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114727070A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858081A (en) * | 2023-02-24 | 2023-03-28 | 深圳市东微智能科技股份有限公司 | UI (user interface) media resource display method, device and equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9387813B1 (en) * | 2012-03-21 | 2016-07-12 | Road-Iq, Llc | Device, system and method for aggregating networks and serving data from those networks to computers |
| CN109194698A (en) * | 2018-11-01 | 2019-01-11 | 浩云科技股份有限公司 | A kind of Real-time Video Processing System and method based on GStreamer frame |
| CN110493626A (en) * | 2019-09-10 | 2019-11-22 | 海信集团有限公司 | Video data handling procedure and device |
| CN112099970A (en) * | 2020-09-04 | 2020-12-18 | 中国第一汽车股份有限公司 | Video data processing method, device, equipment and storage medium |
| CN112528961A (en) * | 2020-12-28 | 2021-03-19 | 山东巍然智能科技有限公司 | Video analysis method based on Jetson Nano |
| CN112804410A (en) * | 2019-11-14 | 2021-05-14 | 西安诺瓦星云科技股份有限公司 | Multi-display-screen synchronous display method and device, video processing equipment and storage medium |
-
2022
- 2022-05-10 CN CN202210504413.8A patent/CN114727070A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9387813B1 (en) * | 2012-03-21 | 2016-07-12 | Road-Iq, Llc | Device, system and method for aggregating networks and serving data from those networks to computers |
| CN109194698A (en) * | 2018-11-01 | 2019-01-11 | 浩云科技股份有限公司 | A kind of Real-time Video Processing System and method based on GStreamer frame |
| CN110493626A (en) * | 2019-09-10 | 2019-11-22 | 海信集团有限公司 | Video data handling procedure and device |
| CN112804410A (en) * | 2019-11-14 | 2021-05-14 | 西安诺瓦星云科技股份有限公司 | Multi-display-screen synchronous display method and device, video processing equipment and storage medium |
| CN112099970A (en) * | 2020-09-04 | 2020-12-18 | 中国第一汽车股份有限公司 | Video data processing method, device, equipment and storage medium |
| CN112528961A (en) * | 2020-12-28 | 2021-03-19 | 山东巍然智能科技有限公司 | Video analysis method based on Jetson Nano |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858081A (en) * | 2023-02-24 | 2023-03-28 | 深圳市东微智能科技股份有限公司 | UI (user interface) media resource display method, device and equipment and storage medium |
| CN115858081B (en) * | 2023-02-24 | 2023-06-16 | 深圳市东微智能科技股份有限公司 | UI (user interface) media resource display method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101500128B (en) | Method and apparatus for loading additional information on display image of network camera device terminal | |
| US20090228620A1 (en) | Digital Adapter | |
| CN110366005A (en) | Sending method and device, the display methods and device of analog composite video signal | |
| CN114727070A (en) | Camera real-time video data processing method and device | |
| CN110958431A (en) | Multi-channel video compression post-transmission system and method | |
| CN102802022A (en) | Coding and decoding system and method | |
| CN206117878U (en) | Intelligent video analysis device, equipment and video monitor system | |
| US20160142461A1 (en) | Method and device for transmission of multimedia data | |
| CN203279065U (en) | Multi-screen display NVR system | |
| CN104883518A (en) | Low-cost small-sized thermal infrared imager | |
| US20210037208A1 (en) | Compositing video signals and stripping composite video signal | |
| CN113438460A (en) | Lossless AR video acquisition and transmission method, device and system | |
| CN101184177A (en) | Digital television picture catching method and system | |
| CN103974087A (en) | Video image file compressing system, client and method | |
| CN210381033U (en) | Low-power-consumption video imaging device | |
| CN106851134B (en) | Method, device and system for transmitting image data | |
| CN113612938A (en) | Multi-type adaptive resolution image conversion method and device | |
| CN114037646A (en) | Intelligent image detection method, system, readable medium and equipment based on Internet of things | |
| CN115225881B (en) | Data transmission method, device, equipment and storage medium | |
| CN113347385A (en) | Video stream transmission method, device, equipment and medium | |
| CN202059481U (en) | Digital high definition video camera based on general hardware | |
| CN104010221A (en) | Digital billboard playing system, instant monitoring system and instant monitoring method thereof | |
| CN211047082U (en) | High-speed download equipment of surveillance video | |
| CN118138703A (en) | Video data stream soft coding method, device, equipment and medium | |
| CN102082949A (en) | Device integrating various back-end functions of high-definition video monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |