CN114879673A - Visual navigation device based on TinyML technology - Google Patents

Visual navigation device based on TinyML technology Download PDF

Info

Publication number
CN114879673A
CN114879673A CN202210506580.6A CN202210506580A CN114879673A CN 114879673 A CN114879673 A CN 114879673A CN 202210506580 A CN202210506580 A CN 202210506580A CN 114879673 A CN114879673 A CN 114879673A
Authority
CN
China
Prior art keywords
tinyml
visual navigation
navigation device
device based
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210506580.6A
Other languages
Chinese (zh)
Inventor
胡佳
李锐
王洪添
朱翔宇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Inspur Science Research Institute Co Ltd
Original Assignee
Shandong Inspur Science Research Institute Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Inspur Science Research Institute Co Ltd filed Critical Shandong Inspur Science Research Institute Co Ltd
Priority to CN202210506580.6A priority Critical patent/CN114879673A/en
Publication of CN114879673A publication Critical patent/CN114879673A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05DSYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02Control of position or course in two dimensions
    • G05D1/021Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/20Image preprocessing
    • G06V10/26Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/20Image preprocessing
    • G06V10/30Noise filtering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/94Hardware or software architectures specially adapted for image or video understanding
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Electromagnetism (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Instructional Devices (AREA)
  • Navigation (AREA)

Abstract

A visual navigation device based on a TinyML technology is a visual navigation device and a method based on micro machine learning (TinyML), an SOC (field programmable gate array + ARM architecture) processor is adopted on hardware, certain advantages are achieved when huge image data in a visual system are processed, the TinyML is deployed to an ARM embedded application platform, and secondary processing is performed on the data processed by the FPGA for the first time on the ARM platform. The system can perform navigation work such as real-time visual image acquisition, image processing, subsequent programming and path planning on an unknown environment. The device can work in any environment, and has the advantages of small error, strong anti-interference performance and strong real-time performance.

Description

Visual navigation device based on TinyML technology
Technical Field
The invention relates to the field of computer vision, in particular to a visual navigation device based on a TinyML technology.
Background
The development of computer hardware promotes the continuous upgrade of computer image processing capability and computer related technologies, the existing vision sensor has the advantages of wide signal detection range, complete target information, quick acquisition of environmental information and the like, and meanwhile, the vision system has autonomy and passivity, can normally work in places where radio signals cannot reach and has a very good navigation effect. The visual navigation system has wide application in the fields of mobile communication networks, power systems, rail transit, financial systems, smart cities, aerospace, national defense systems and the like. Meanwhile, with the development of machine learning, TinyML becomes an emerging development subject, and can realize a machine learning algorithm with low resource consumption and low power consumption on a microcontroller with limited resources.
In conventional visual navigation generic devices, high performance based computer frameworks are mostly employed. Therefore, the visual navigation system is huge in size and high in power consumption, and meanwhile, a CPU processor is adopted in the traditional visual navigation system, and the processing speed of the CPU is obviously not capable of following the current use in the face of binocular visual images with very large data volume.
Disclosure of Invention
In order to overcome the defects of the technology, the invention provides the visual navigation device based on the TinyML technology, which has the advantages of small error, strong anti-interference performance and strong real-time performance.
The technical scheme adopted by the invention for overcoming the technical problems is as follows:
a visual navigation device based on TinyML technology comprises a data acquisition unit and a visual navigation unit, wherein the data acquisition unit comprises:
the binocular camera is arranged on two sides of the intelligent terminal and used for acquiring real-time image data;
the IMU is arranged in the intelligent terminal and is used for measuring the turning angle and the course indication of a user in real time;
the visual navigation unit comprises:
the communication module is used for carrying out information interaction on information acquired by the binocular camera and the IMU and the visual navigation unit;
the SOC is used for processing information acquired by the binocular camera and the IMU into an environment map and position information;
and the memory unit is used for caching the data processed by the SOC.
Preferably, the SOC includes an FPGA chip and an ARM chip.
Preferably, the communication module is a WIFI wireless communication module.
Furthermore, the FPGA chip receives signals collected by the data collection unit from the communication module, the signals are subjected to data caching and clock domain crossing signal synchronization through a one-step clock dual-port FIFO, and the processed data are subjected to image noise reduction and filtering enhancement and then are identified by using an edge detection image identification algorithm.
Further, the ARM chip receives the image identified by the FPGA chip through a high-speed AMBA bus protocol, and carries out TinyML identification and mapping algorithm on the image to obtain a map of an unknown environment and position information of a user.
Preferably, the memory cell is a DDR4 memory cell.
Further, a display is included for displaying a map of the unknown environment and location information of the user.
The invention has the beneficial effects that: a visual navigation device and method based on micro machine learning (TinyML) adopts an SOC (field programmable gate array + ARM architecture) processor on hardware, has certain advantages when processing huge image data in a visual system, deploys the TinyML to an ARM embedded application platform, and carries out secondary processing on the data processed by the FPGA for the first time on the ARM platform. The system can perform navigation work such as real-time visual image acquisition, image processing, subsequent programming and path planning on an unknown environment. The device can work in any environment, and has the advantages of small error, strong anti-interference performance and strong real-time performance.
Drawings
FIG. 1 is a block diagram of the present invention;
FIG. 2 is a flow chart of information processing according to the present invention.
Detailed Description
The invention will be further explained with reference to fig. 1 and 2.
A visual navigation device based on TinyML technology comprises a data acquisition unit and a visual navigation unit, wherein the data acquisition unit comprises: the binocular camera is arranged on two sides of the intelligent terminal and used for acquiring real-time image data; the IMU is arranged in the intelligent terminal and is used for measuring the turning angle and the course indication of a user in real time; the visual navigation unit comprises: the communication module is used for carrying out information interaction on information acquired by the binocular camera and the IMU and the visual navigation unit; the SOC is used for processing information acquired by the binocular camera and the IMU into an environment map and position information; and the memory unit is used for caching the data processed by the SOC.
A visual navigation device and method based on micro machine learning (TinyML) adopts an SOC (field programmable gate array + ARM architecture) processor on hardware, has certain advantages when processing huge image data in a visual system, deploys the TinyML to an ARM embedded application platform, and carries out secondary processing on the data processed by the FPGA for the first time on the ARM platform. The system can perform navigation work such as real-time visual image acquisition, image processing, subsequent programming and path planning on an unknown environment. The device can work in any environment, and has the advantages of small error, strong anti-interference performance and strong real-time performance.
An IMU is typically composed of three parts, a gyroscope, an accelerometer, and a magnetometer. The gyroscope is internally provided with a gyroscope, and the axis of the gyroscope is always parallel to the initial direction due to the gyroscopic effect, so that the actual direction can be calculated through the deviation from the initial direction. The accelerometer is used for detecting the magnitude and direction of the acceleration received by the unmanned aerial vehicle, and the magnetometer is used for testing the magnetic field intensity and the direction. The earth's magnetic field is a weak magnetic field having a strength of about 0.5 to 0.6 gauss, which gradually increases from the equator toward the poles. The IMU is mainly used to measure the yaw direction and rotational angular velocity of the user, convert these information into digital signals, and transmit the signals to the communication module.
Preferably, the SOC is composed of an FPGA chip and an ARM chip. The communication module is a WIFI wireless communication module. The memory unit is a DDR4 memory unit, and the DDR4 memory unit has high-speed data read-write capability and can meet the requirements of the design.
Furthermore, the FPGA chip receives signals collected by the data collection unit from the communication module, the signals are subjected to data caching and clock domain crossing signal synchronization through a one-step clock dual-port FIFO, and the processed data are subjected to image noise reduction and filtering enhancement and then are identified by using an edge detection image identification algorithm. The ARM chip receives the image identified by the FPGA chip through a high-speed AMBA bus protocol, and carries out TinyML identification and mapping algorithm on the image to obtain a map of an unknown environment and position information of a user. The invention adopts an SOC (FPGA + ARM architecture) processor to run a TinyML software stack. In the face of very much image data, the invention adopts the method of 'area conversion speed' in FPGA to process, namely, a large number of logic gate arrays in FPGA are used for processing the bottom layer (combined image filtering) and the middle layer (image enhancement, image segmentation and image identification) of the visual image. The logic gate arrays in the FPGA all run in parallel, so the processing speed of the basic image information is far better than that of a CPU. In the invention, the realization of human-computer interaction and various machine learning algorithms is required, so the functions of map construction, path planning, human-computer interaction and the like are realized by adopting the ARM. Traditionally, machine learning models are always deployed in resource-rich environments, where data is sent from a local device to a cloud for processing. There are currently concerns about privacy, latency, storage, and energy efficiency for this approach. Since TinyML models can be embedded into microcontrollers, they are not resource intensive. This approach is the most efficient and cost effective way to inject artificial intelligence into the visual navigation device.
Further, a display is included for displaying a map of the unknown environment and location information of the user.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. The visual navigation device based on the TinyML technology is characterized by comprising a data acquisition unit and a visual navigation unit, wherein the data acquisition unit comprises:
the binocular camera is arranged on two sides of the intelligent terminal and used for acquiring real-time image data;
the IMU is arranged in the intelligent terminal and is used for measuring the turning angle and the course indication of a user in real time;
the visual navigation unit comprises:
the communication module is used for carrying out information interaction on information acquired by the binocular camera and the IMU and the visual navigation unit;
the SOC is used for processing information acquired by the binocular camera and the IMU into an environment map and position information;
and the memory unit is used for caching the data processed by the SOC.
2. A visual navigation device based on TinyML technology according to claim 1, characterized in that: the SOC is composed of an FPGA chip and an ARM chip.
3. A visual navigation device based on TinyML technology according to claim 1, characterized in that: the communication module is a WIFI wireless communication module.
4. A visual navigation device based on TinyML technology according to claim 2, characterized in that: the FPGA chip receives signals collected by the data collection unit from the communication module, the signals are subjected to data caching and clock domain crossing signal synchronization through a one-step clock dual-port FIFO, and the processed data are subjected to image noise reduction and filtering enhancement and then are identified by using an edge detection image identification algorithm.
5. A visual navigation device based on TinyML technology as claimed in claim 4, wherein: the ARM chip receives the image identified by the FPGA chip through a high-speed AMBA bus protocol, and carries out TinyML identification and mapping algorithm on the image to obtain a map of an unknown environment and position information of a user.
6. A visual navigation device based on TinyML technology according to claim 1, characterized in that: the memory cell is a DDR4 memory cell.
7. A visual navigation device based on TinyML technology as claimed in claim 5, wherein: a display is also included for displaying a map of the unknown environment and location information of the user.
CN202210506580.6A 2022-05-11 2022-05-11 Visual navigation device based on TinyML technology Pending CN114879673A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210506580.6A CN114879673A (en) 2022-05-11 2022-05-11 Visual navigation device based on TinyML technology

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210506580.6A CN114879673A (en) 2022-05-11 2022-05-11 Visual navigation device based on TinyML technology

Publications (1)

Publication Number Publication Date
CN114879673A true CN114879673A (en) 2022-08-09

Family

ID=82676291

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210506580.6A Pending CN114879673A (en) 2022-05-11 2022-05-11 Visual navigation device based on TinyML technology

Country Status (1)

Country Link
CN (1) CN114879673A (en)

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102411535A (en) * 2011-08-02 2012-04-11 上海交通大学 Navigating-SoC (System On Chip) simulating, verifying and debugging platform
CN103116175A (en) * 2013-01-18 2013-05-22 东南大学 Embedded type navigation information processor based on DSP (digital signal processor) and FPGA (field programmable gata array)
CN103914071A (en) * 2014-04-02 2014-07-09 中国农业大学 Visual navigation path recognition system of grain combine harvester
CN108594851A (en) * 2015-10-22 2018-09-28 飞智控(天津)科技有限公司 A kind of autonomous obstacle detection system of unmanned plane based on binocular vision, method and unmanned plane
CN108733064A (en) * 2017-04-18 2018-11-02 中交遥感载荷(北京)科技有限公司 A kind of the vision positioning obstacle avoidance system and its method of unmanned plane
CN110196053A (en) * 2019-06-13 2019-09-03 内蒙古大学 A kind of real-time field robot vision navigation method and system based on FPGA
US20200125961A1 (en) * 2018-10-19 2020-04-23 Oracle International Corporation Mini-machine learning
CN111343379A (en) * 2020-02-25 2020-06-26 华南理工大学 FPGA-based high-speed data acquisition device and method
CN111524177A (en) * 2020-04-16 2020-08-11 华中科技大学 Micro-miniature high-speed binocular stereoscopic vision system of robot
CN212012915U (en) * 2020-02-25 2020-11-24 华南理工大学 Binocular image and IMU data high-speed acquisition device
US20210133607A1 (en) * 2019-10-31 2021-05-06 Shoreline Iot, Inc. Systems and methods for self-learning artificial intelligence of things (aiot) devices and services
CN113391695A (en) * 2021-06-11 2021-09-14 山东浪潮科学研究院有限公司 Low-power-consumption face recognition method based on TinyML
CN113407583A (en) * 2021-06-24 2021-09-17 山东浪潮科学研究院有限公司 Method for emotion analysis and food recommendation based on AIot and TinyML technology
US20210319894A1 (en) * 2020-04-08 2021-10-14 CareBand Inc. Wearable electronic device and system using low-power cellular telecommunication protocols

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102411535A (en) * 2011-08-02 2012-04-11 上海交通大学 Navigating-SoC (System On Chip) simulating, verifying and debugging platform
CN103116175A (en) * 2013-01-18 2013-05-22 东南大学 Embedded type navigation information processor based on DSP (digital signal processor) and FPGA (field programmable gata array)
CN103914071A (en) * 2014-04-02 2014-07-09 中国农业大学 Visual navigation path recognition system of grain combine harvester
CN108594851A (en) * 2015-10-22 2018-09-28 飞智控(天津)科技有限公司 A kind of autonomous obstacle detection system of unmanned plane based on binocular vision, method and unmanned plane
CN108733064A (en) * 2017-04-18 2018-11-02 中交遥感载荷(北京)科技有限公司 A kind of the vision positioning obstacle avoidance system and its method of unmanned plane
US20200125961A1 (en) * 2018-10-19 2020-04-23 Oracle International Corporation Mini-machine learning
CN110196053A (en) * 2019-06-13 2019-09-03 内蒙古大学 A kind of real-time field robot vision navigation method and system based on FPGA
US20210133607A1 (en) * 2019-10-31 2021-05-06 Shoreline Iot, Inc. Systems and methods for self-learning artificial intelligence of things (aiot) devices and services
CN111343379A (en) * 2020-02-25 2020-06-26 华南理工大学 FPGA-based high-speed data acquisition device and method
CN212012915U (en) * 2020-02-25 2020-11-24 华南理工大学 Binocular image and IMU data high-speed acquisition device
US20210319894A1 (en) * 2020-04-08 2021-10-14 CareBand Inc. Wearable electronic device and system using low-power cellular telecommunication protocols
CN111524177A (en) * 2020-04-16 2020-08-11 华中科技大学 Micro-miniature high-speed binocular stereoscopic vision system of robot
CN113391695A (en) * 2021-06-11 2021-09-14 山东浪潮科学研究院有限公司 Low-power-consumption face recognition method based on TinyML
CN113407583A (en) * 2021-06-24 2021-09-17 山东浪潮科学研究院有限公司 Method for emotion analysis and food recommendation based on AIot and TinyML technology

Similar Documents

Publication Publication Date Title
CN109435955B (en) Performance evaluation method, device and equipment for automatic driving system and storage medium
CN109931945B (en) AR navigation method, device, equipment and storage medium
WO2018217663A1 (en) Image capture direction recognition method and server, surveillance method and system and image capture device
US20230394756A1 (en) Three-dimensional reconstruction method, three-dimensional reconstruction apparatus and storage medium
CN110276972A (en) A kind of object cognitive method and system based on car networking
CN111125283B (en) Electronic map construction method and device, computer equipment and storage medium
CN112967392A (en) Large-scale park mapping and positioning method based on multi-sensor contact
CN113807470B (en) Vehicle driving state determination method and related device
CN111694012A (en) Three-dimensional terrain online generation method and system based on airborne laser radar
EP4206731A1 (en) Target tracking method and device
CN103033836A (en) Vehicle navigation pointing device and method
Wang et al. Remote sensing monitoring method based on BDS-based maritime joint positioning model
CN115585818A (en) Map construction method and device, electronic equipment and storage medium
CN111121755B (en) Multi-sensor fusion positioning method, device, equipment and storage medium
CN110515110A (en) Method, apparatus, equipment and the computer readable storage medium of data assessment
CN116539053A (en) Real-time global point cloud map construction method, system, equipment and storage medium
Cunha et al. Hardware-accelerated data decoding and reconstruction for automotive LiDAR sensors
CN111652276B (en) All-weather portable multifunctional bionic positioning and attitude-determining viewing system and method
CN114879673A (en) Visual navigation device based on TinyML technology
CN211878189U (en) Ground-air cooperative vision navigation device based on improved map optimization SLAM
CN103335645B (en) The pinpoint image acquiring method in space towards landscape planning
CN113715817B (en) Vehicle control method, vehicle control device, computer equipment and storage medium
CN214492786U (en) Vehicle-mounted automatic driving hardware circuit and automatic driving automobile
CN106443798A (en) Road underground data acquisition and fusion method and system
Chen et al. Fully Onboard Single Pedestrian Tracking On Nano-UAV Platform

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination