CN113658210A - Front-end real-time target tracking method based on Jetson NX platform - Google Patents
Front-end real-time target tracking method based on Jetson NX platform Download PDFInfo
- Publication number
- CN113658210A CN113658210A CN202111025186.2A CN202111025186A CN113658210A CN 113658210 A CN113658210 A CN 113658210A CN 202111025186 A CN202111025186 A CN 202111025186A CN 113658210 A CN113658210 A CN 113658210A
- Authority
- CN
- China
- Prior art keywords
- jetson
- platform
- real
- tracking
- method based
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4482—Procedural
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Abstract
The invention provides a front-end real-time target tracking method based on a JetsonnX platform, which comprises the following steps of 1) storing a trained end-to-end off-line model on a host or a server; 2) converting the pth file of the tracking model into a pt file special for C compiling on a Jetson NX; 3) compiling pt files with C on Jetson NX; 4) reading a real-time camera data stream on a Jetson NX; 5) appointing a target to be tracked in a first frame, and then running a model by a program c; 6) continuously jumping to the fourth step for continuous execution; 7) human exit and/or automatic termination of the tracking process. The invention has the advantages that: under the condition that only one GPU is provided and the video memory is low, front-end (or local video) real-time tracking is carried out on the camera, and a mainstream tracking model is deployed on mobile terminal equipment, so that the real-time performance of the tracking model based on deep learning applied to edge equipment (embedded type) is greatly improved, and the power consumption is remarkably reduced.
Description
Technical Field
The invention relates to the field of target tracking, in particular to a front-end real-time target tracking method based on a Jetson NX platform.
Background
Since the deep learning models that we are mainstream are directly stored on the host or the server, the deep learning models are often calculated by the host or the server when testing the speed. When the deep learning model is deployed on the edge device (embedded type), because the CPU and GPU resources of the mobile edge device (embedded type) are limited and the consumption of computing resources is very large, the preprocessing time before tracking and the processing time in the tracking process are both longer, and the real-time performance cannot reach the speed of the host or the server during testing. The method of directly deploying deep learning models to mobile edge devices (embedded) is not desirable.
In the prior art, the NVIDIA Jetson NX suite is configured with 8GB 128-bit LPDDR4 and 64GB of eMMC memory; the Jetson NX suite is provided with a large number of abundant external interfaces, and the tracking device mainly uses a CSI interface and a small number of USB interfaces. Specifically, a CSI interface of a Jetson NX suite converts an input CSI signal stream into a depepsstream stream; and related video stream signals are subjected to video initialization processing and video target tracking through a six-core ARM CPU and 384 GPU of CUDA cores in a Jetson NX suite. In the tracking operation process, the processing speed of Jetson NX is low, and the real-time requirement cannot be met.
Therefore, a front-end real-time target tracking method based on the Jetson NX platform is needed to meet the real-time requirement in the industry.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a front-end real-time target tracking method based on a Jetson NX platform, which is used for alleviating the pressure of limitation of Jetson NX resources and can perform real-time target tracking of a camera under the condition of only one GPU.
In order to solve the technical problems, the technical scheme provided by the invention is as follows: a front end (or local video) real-time tracking method based on a Jetson NX platform and a Linux system is characterized in that: the steps for implementing the tracking method are as follows:
1) storing the trained end-to-end off-line model on the host or the server;
2) converting a pth file of the tracking model into a pt file special for C compiling on a Jetson NX platform;
3) compiling pt files on a Jetson NX platform by using C;
4) reading a real-time camera data stream on a Jetson NX platform;
5) appointing a target to be tracked in a first frame, and then running a model by a program c;
6) continuing jumping to the step 4) for continuous execution;
7) and terminating the tracking process, including manual exit and automatic termination.
As an improvement, the offline model training in the step 1) does not need the participation of edge equipment.
As an improvement, the pth file conversion in step 2) is performed on an edge device.
As an improvement, the triggering condition of the automatic termination mode in step 7) is that if the camera data stream read in step 4) is not a local video, the tracking process will automatically terminate at the last frame of the video.
As an improvement, the pth file conversion in the step 2) is performed by using a convert program.
Compared with the prior art, the invention has the advantages that: under the condition of only one GPU, a first frame of a video stream acquired by a camera is used as a first frame of target tracking at the front end (or a local video), when a user specifies an object to be tracked in the first frame, the camera can track the object and continuously execute a tracking process until the user selects to terminate the process, a mainstream tracking model is deployed on mobile terminal equipment, the real-time performance of the tracking model based on deep learning applied to edge equipment (embedded type) is greatly improved, and the power consumption is reduced.
Drawings
Fig. 1 is a schematic diagram of a tracking working state of a front-end (or local video) real-time tracking method based on Jetson NX in the present invention.
Fig. 2 is a schematic flow chart of a front-end (or local video) real-time tracking method based on Jetson NX in the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
With reference to fig. 1-2, the application discloses a front-end real-time target tracking method based on a Jetson NX platform, wherein a camera end is adopted by a real-time tracking system on the premise that only one GPU is provided and the real-time tracking system is completely offline, and the camera end is embedded into the Jetson NX platform.
And transplanting a specific network model (tracking model) to a C compiler of a Jetson NX platform development board in a camera end to accelerate tracking reasoning.
As shown in fig. 2, the front-end real-time target tracking method based on the Jetson NX platform of the present invention includes the following steps:
1) storing the trained end-to-end offline model on the host or the server, specifically, transplanting the stored tracking algorithm to a Jetson NX platform, and performing different processing analysis on the algorithm result according to the application scene;
2) converting the pth file of the tracking model into a pth file special for C compiling by using a convert program on a Jetson NX platform, and particularly dividing a complete pth file into different pth files according to an internal structure during conversion;
3) compiling pt files on a Jetson NX platform by using C, and specifically, running a convert program in 2) and C in the same environment;
4) reading a real-time camera data stream on a Jetson NX platform, acquiring a frame of image from a micro-lens, preprocessing the image, sending the preprocessed image into a tracking network for prediction, and changing the original image into an image required by the tracking network by changing the width, height and channel number of the image;
5) appointing a first frame of target to be tracked, then operating the model by the program c, manually selecting the target to be tracked by a user according to the first frame read by the camera, and tracking the position in the picture according to the input picture;
6) and continuing to jump to the fourth step for continuous execution, outputting an inference result, wherein the output result refers to the position information of the target, the position information comprises four corner points (x1, y1), (x2, y2), (x3, y3), (x4, y4) of a rectangular frame and a segmentation graph, the position information is analyzed and transformed, and the segmentation graph is drawn on an original picture, so that a final tracking result is obtained.
7) And terminating the tracking process, namely manually exiting and automatically terminating, wherein if the read video is not the camera data stream but the local video, the tracking process does not need to be manually terminated, and the tracking process can be automatically terminated at the last frame of the video.
The offline model training in the step 1) does not need the participation of edge equipment.
The pth file conversion in the step 2) is performed on an edge device.
The working principle of the invention is as follows: on the basis of installing a micro-lens and a GPU on a Jetson NX platform development board, performing front-end (or local video) real-time tracking on a camera, using a first frame of a video stream acquired by a camera as a first frame of target tracking, and when a user specifies an object to be tracked in the first frame, the camera can track the object and continuously execute a tracking process until the user selects to terminate the process.
The present invention and its embodiments have been described above, and the description is not intended to be limiting, and the drawings are only one embodiment of the present invention, and the actual structure is not limited thereto. In summary, those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (5)
1. A front-end real-time target tracking method based on a Jetson NX platform is characterized by comprising the following steps: the steps for implementing the tracking method are as follows:
1) storing the trained end-to-end off-line model on the host or the server;
2) converting a pth file of the tracking model into a pt file special for C compiling on a Jetson NX platform;
3) compiling pt files on a Jetson NX platform by using C;
4) reading a real-time camera data stream on a Jetson NX platform;
5) appointing a target to be tracked in a first frame, and then running a model by a program C;
6) continuing jumping to the step 4) for continuous execution;
7) and terminating the tracking process, including manual exit and automatic termination.
2. The front-end real-time target tracking method based on the Jetson NX platform as claimed in claim 1, wherein: the offline model training in the step 1) does not need the participation of edge equipment.
3. The front-end real-time target tracking method based on the Jetson NX platform as claimed in claim 1, wherein: the pth file conversion in the step 2) is performed on an edge device.
4. The front-end real-time target tracking method based on the Jetson NX platform as claimed in claim 1, wherein: the triggering condition of the automatic termination mode in the step 7) is that the tracking process is automatically terminated at the last frame of the video if the local video is not the camera data stream read in the step 4).
5. The front-end real-time target tracking method based on the Jetson NX platform as claimed in claim 1, wherein: and the pth file conversion in the step 2) is carried out by adopting a convert program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111025186.2A CN113658210A (en) | 2021-09-02 | 2021-09-02 | Front-end real-time target tracking method based on Jetson NX platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111025186.2A CN113658210A (en) | 2021-09-02 | 2021-09-02 | Front-end real-time target tracking method based on Jetson NX platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113658210A true CN113658210A (en) | 2021-11-16 |
Family
ID=78493461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111025186.2A Pending CN113658210A (en) | 2021-09-02 | 2021-09-02 | Front-end real-time target tracking method based on Jetson NX platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113658210A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108269269A (en) * | 2016-12-30 | 2018-07-10 | 纳恩博（北京）科技有限公司 | Method for tracking target and device |
| CN110532883A (en) * | 2019-07-30 | 2019-12-03 | 平安科技（深圳）有限公司 | On-line tracking is improved using off-line tracking algorithm |
| CN111783974A (en) * | 2020-08-12 | 2020-10-16 | 成都佳华物链云科技有限公司 | Model construction and image processing method and device, hardware platform and storage medium |
| CN112711423A (en) * | 2021-01-18 | 2021-04-27 | 深圳中兴网信科技有限公司 | Engine construction method, intrusion detection method, electronic device and readable storage medium |
-
2021
- 2021-09-02 CN CN202111025186.2A patent/CN113658210A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108269269A (en) * | 2016-12-30 | 2018-07-10 | 纳恩博（北京）科技有限公司 | Method for tracking target and device |
| CN110532883A (en) * | 2019-07-30 | 2019-12-03 | 平安科技（深圳）有限公司 | On-line tracking is improved using off-line tracking algorithm |
| CN111783974A (en) * | 2020-08-12 | 2020-10-16 | 成都佳华物链云科技有限公司 | Model construction and image processing method and device, hardware platform and storage medium |
| CN112711423A (en) * | 2021-01-18 | 2021-04-27 | 深圳中兴网信科技有限公司 | Engine construction method, intrusion detection method, electronic device and readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 黄理广: "基于 Jetson Xavier 的手扶电梯智能视频监控系统设计及实现", 《中国优秀硕士学位论文全文数据库（工程科技Ⅱ辑）》, no. 2021, pages 2 - 6 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105144118B (en) | Using test and analysis | |
| CN109871800B (en) | Human body posture estimation method and device and storage medium | |
| CN111124713B (en) | Equipment system function calling method, device, terminal equipment and storage medium | |
| JP7273129B2 (en) | Lane detection method, device, electronic device, storage medium and vehicle | |
| TWI678614B (en) | Test system | |
| US20230281515A1 (en) | Distributed learning model for fog computing | |
| CN107045475B (en) | Test method and device | |
| CN111226226A (en) | Motion-based object detection method, object detection device and electronic equipment | |
| CN111527501A (en) | Chip adaptation determining method and related product | |
| CN112416775A (en) | Software automation testing method and device based on artificial intelligence and electronic equipment | |
| CN111859630B (en) | Image compression simulation verification method, device, equipment and readable storage medium | |
| CN113658210A (en) | Front-end real-time target tracking method based on Jetson NX platform | |
| Pandey et al. | An automation framework for benchmarking and optimizing performance of remote desktops in the cloud | |
| CN112416301A (en) | Deep learning model development method and device and computer readable storage medium | |
| Nieto et al. | High performance image acquisition and processing architecture for fast plant system controllers based on FPGA and GPU | |
| CN113554550B (en) | Training method and device for image processing model, electronic equipment and storage medium | |
| CN113411425B (en) | Video super-division model construction processing method, device, computer equipment and medium | |
| CN112686901A (en) | US-CT image segmentation method and device based on deep neural network | |
| CN113760431B (en) | Application control method and device, electronic equipment and readable storage medium | |
| CN113947747B (en) | Method, device and equipment for processing monitoring image of vehicle | |
| CN114051110B (en) | Video generation method, device, electronic equipment and storage medium | |
| TWI786430B (en) | Device and method for optimizing model conversion of deep learning model, and storage medium | |
| US20220358929A1 (en) | Voice activity detection method and apparatus, electronic device and storage medium | |
| CN110992360B (en) | Equipment performance test method and device and electronic equipment | |
| CN111176925B (en) | Equipment performance test method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |