WO2022151560A1 - 基于移动可穿戴计算和快速深度神经网络的智慧盲杖 - Google Patents

基于移动可穿戴计算和快速深度神经网络的智慧盲杖 Download PDF

Info

Publication number
WO2022151560A1
WO2022151560A1 PCT/CN2021/076403 CN2021076403W WO2022151560A1 WO 2022151560 A1 WO2022151560 A1 WO 2022151560A1 CN 2021076403 W CN2021076403 W CN 2021076403W WO 2022151560 A1 WO2022151560 A1 WO 2022151560A1
Authority
WO
WIPO (PCT)
Prior art keywords
obstacle
blind
button
vibration
ground
Prior art date
Application number
PCT/CN2021/076403
Other languages
English (en)
French (fr)
Inventor
何坚
张丞
刘新远
苏予涵
魏鑫
王子源
Original Assignee
北京工业大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京工业大学 filed Critical 北京工业大学
Publication of WO2022151560A1 publication Critical patent/WO2022151560A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61H—PHYSICAL THERAPY APPARATUS, e.g. DEVICES FOR LOCATING OR STIMULATING REFLEX POINTS IN THE BODY; ARTIFICIAL RESPIRATION; MASSAGE; BATHING DEVICES FOR SPECIAL THERAPEUTIC OR HYGIENIC PURPOSES OR SPECIFIC PARTS OF THE BODY
    • A61H3/00—Appliances for aiding patients or disabled persons to walk about
    • A61H3/06—Walking aids for blind persons
    • A61H3/061—Walking aids for blind persons with electronic detecting or guiding means
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61H—PHYSICAL THERAPY APPARATUS, e.g. DEVICES FOR LOCATING OR STIMULATING REFLEX POINTS IN THE BODY; ARTIFICIAL RESPIRATION; MASSAGE; BATHING DEVICES FOR SPECIAL THERAPEUTIC OR HYGIENIC PURPOSES OR SPECIFIC PARTS OF THE BODY
    • A61H3/00—Appliances for aiding patients or disabled persons to walk about
    • A61H3/06—Walking aids for blind persons
    • A61H3/068—Sticks for blind persons
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks

Definitions

  • the invention belongs to the field of electronic information, and is an auxiliary device based on mobile Internet and wearable computing technology. It mainly relies on the interaction between a smart blind stick and a mobile smart handheld device to provide voice- or tactile-based feedback for the visually impaired when walking outdoors, so as to provide feedback for the visually impaired. Assist the blind to avoid obstacles.
  • blind sticks as a necessity for visually impaired people, can sense whether there are obstacles in front of them by touching the ground.
  • the low-priced products only have the function of voice help.
  • the blind sticks integrated with infrared or ultrasonic waves for obstacle detection are very expensive. Therefore, the function of the low-priced smart blind stick is too single, and the high-priced smart blind stick faces the problems of high power consumption and large size.
  • Wearable computing provides users with computing power in a natural wearable form, making people pay more attention to product functions and ignore the appearance of computing devices.
  • Mobile computing enables computers or other information intelligent terminal equipment to realize the sharing of data and computing resources in the wireless network environment, and it is increasingly integrated into everyone's work and life with the development of mobile communication, Internet, distributed computing and other technologies.
  • Short-range wireless communication technology is a bridge that organically integrates the two.
  • the most commonly used short-range wireless communication technologies are wifi, Bluetooth, and Zigbee.
  • Bluetooth allows mobile devices to establish connections within a short distance to realize data transmission and exchange.
  • bluetooth has the characteristics of low power consumption, low radiation, low cost, and low latency. Therefore, the present invention adopts Bluetooth to realize the communication between the blind stick and the mobile smart device.
  • the blind stick includes: the rod body and the handle of the blind stick.
  • the handle of the stick is provided with a power supply and a main control module;
  • the front side of the handle is provided with a lighting lamp;
  • the inner side of the handle (that is, where the thumb is in use) is provided with a button disk, and the button disk includes One switch button and five function buttons.
  • the main control module integrates a CC2640 low-power bluetooth microprocessor, a vibration motor, a buzzer, and the lighting and button panel control links and interface connections.
  • the invention designs a disc-type button group, and a certain distance is reserved between the buttons to prevent accidental touch.
  • the button disc is mounted on the inside of the cane handle, allowing the visually impaired to perform all operations with just their thumb.
  • the buttons have Braille to represent different functions. When a visually impaired person touches a button, the cane performs that function immediately.
  • the button in the center of the disc is a switch. When in the off state, it will automatically turn on after long-pressing for 3s and give feedback to the blind by vibrating quickly. At the same time, try to connect with the mobile smart device. After the connection is successful, the device will give the user voice feedback or the blind cane.
  • the handle vibration feedback prompts "connection successful”. When it is in the power-on state, it will automatically shut down after long pressing the switch button for 3s, and it will be fed back by the vibration of the blind stick.
  • the Bluetooth module will go to sleep after 3 minutes to save energy. Touch the switch button in the sleep state, the Bluetooth module will try to connect with the device again and restart the timing.
  • Light button In low light conditions, the light assists the visually impaired to travel at night.
  • Sound and lighted call for help button In case of distress, the visually impaired can press the call for help button to activate the buzzer to sound for help, and at the same time, the lights are constantly switched on and off to form a tracer to inform rescuers of their location.
  • the present invention sends a request to the connected mobile smart device through Bluetooth, and after the device receives it, it broadcasts the current position, orientation and time to the visually impaired through its loudspeaker.
  • Obstacle detection switch lightly touch this button, the present invention sends a request to the mobile smart device to turn on or off the obstacle detection function of the mobile smart device.
  • Traffic light detection switch touch this button, the present invention sends a request to the mobile smart device to turn on or off the traffic light detection function of the mobile smart device.
  • 3Vibration/voice feedback mode can be switched according to user needs
  • Feedback is a key element of human-computer interaction.
  • the interference caused by the present invention adopts the default vibration feedback.
  • the advantage of vibration feedback is that it is simple and fast. After the touch of a button, the necessary information is fed back to the visually impaired through several preset vibration modes.
  • the present invention adopts the following modes:
  • Obstacle warning According to the distance value of the obstacle in front returned by the mobile smart device, it will vibrate at different frequencies. The smaller the distance, the faster the vibration frequency. The vibration continues until there is no obstacle ahead.
  • the system uses components and functional modules such as GPS/Beidou, compass, clock, Bluetooth and other built-in mobile smart devices to develop programs that run on mobile smart devices. Provide voice broadcasts of the current location, orientation and time of visually impaired persons to assist visually impaired persons to travel.
  • components and functional modules such as GPS/Beidou, compass, clock, Bluetooth and other built-in mobile smart devices to develop programs that run on mobile smart devices.
  • the system uses the built-in camera, barometer and other components and functional modules of mobile smart devices to develop obstacle monitoring programs based on thresholds and regional growth.
  • the mobile smart device runs the program and performs vision-based obstacle detection, and provides voice or vibration alerts for the visually impaired when an obstacle is detected.
  • the system uses the built-in camera of the mobile smart device to develop a traffic light detection and recognition program based on Yolo v4.
  • the traffic light detection program on the mobile smart device will detect and recognize the traffic lights, and inform the visually impaired of the detection results by voice.
  • the beneficial effects of the present invention are: adopting a novel structural design, enabling the present invention to be operated with one finger; automatically connecting to the mobile intelligent device and efficiently communicating with it; at the same time, broadcasting voice information, obstacle detection, Functions such as traffic light detection are transferred to mobile smart devices, which not only reduces the power consumption of the blind stick, but also helps the blind to use the blind stick and mobile smart devices to the maximum extent and without obstacles while traveling. Enhance the user experience for the blind.
  • Fig. 1 network environment where the present invention is located
  • FIG. 3 is a schematic diagram of the internal structure of the handle part of the present invention.
  • Figure 4 is a flow chart of the present invention for booting and connecting equipment
  • Figure 6 is a flow chart of the interaction between the present invention and the mobile smart device
  • Figure 8 Schematic diagram of camera coordinate system and world coordinate system
  • Fig. 1 is a network environment where the present invention is located, in the figure: 1, the handle of the blind stick; 2, the smart phone; 3, the smart glasses; 4, the wearable device; 5, the tablet computer.
  • a bluetooth connection can be established with a designated mobile smart device.
  • the mobile smart device referred to in the present invention includes: smart phone 2; smart glasses 3; wearable device 4; tablet computer 5.
  • FIG. 2 is a schematic diagram of the system structure of the present invention, as shown in the figure: when the present invention is in the power-on state, touch the switch 11, the system generates a pairing interrupt request, and the CC2640 low-power bluetooth microprocessor 6 tries to pair with the surrounding mobile smart devices .
  • the switch 11 Long press the switch 11 for 3s, the system generates a shutdown interrupt request, and the CC2640 Bluetooth low energy microprocessor 6 enters the shutdown state.
  • the function button 12 is touched, the system generates an interrupt request for the corresponding function, and the CC2640 low-power bluetooth microprocessor 6 processes the function after receiving the request.
  • CC2640 low-power bluetooth microprocessor 6 directly controls vibration motor 7; lighting 8; clock 9; buzzer 10.
  • a dormancy interrupt request is generated 3 minutes after the clock 9 arrives.
  • the CC2640 low-power bluetooth microprocessor 6 After receiving the dormancy interrupt, the CC2640 low-power bluetooth microprocessor 6 returns the clock to 0 and enters the dormant state.
  • the power module is composed of the charging controller 13 and the battery 14 .
  • the charging controller 13 adopts the MAX1555 charging controller
  • the battery 14 adopts the LP2992 lithium battery control chip, so that the module can be powered by a 3-5V lithium battery or a USB interface circuit, and can charge the lithium battery through the USB interface.
  • FIG. 3 is a schematic diagram of the internal structure of the pole handle part of the present invention, the vibration motor 6; the vibration motor 7; the lighting lamp 8; the buzzer 10; the switch 11; sexual connection.
  • the light 8 is installed on the front of the pole handle 1; the switch 11 and the function button 12 are installed in the middle of the inner side of the pole handle 1, which is convenient for the visually impaired to operate with one finger; the buzzer 10 and the CC2640 low-power Bluetooth microprocessor 6 are arranged on the pole.
  • the middle of the handle 1 is at the rear; the vibration motor 7 is arranged at the rear of the handle 1, below the palm of the visually impaired; the charging controller 13 and the battery 14 are installed at the rear of the handle 1.
  • the CC2640 low-power Bluetooth microprocessor 6 enters the sleep state, and the vibration motor 7 vibrates rapidly and briefly for 5 times. Each vibration lasts for 0.1s, and the interval is 0.1s to prompt pairing fail. While in sleep mode, touch the switch button to manually send a signal to the CC2640 Bluetooth Low Energy Microprocessor 6 to try pairing again. In the power-on state, press and hold the power button for 3s, and the vibration motor 7 will vibrate twice in a row for 0.4s with an interval of 0.1s to indicate that it has been turned off.
  • CC2640 low-power Bluetooth microprocessor 6 Send corresponding function instructions to the mobile smart device, the device receives and parses the instructions, executes the corresponding function and returns the operation result. After the blind stick receives the operation result, the vibration motor 7 will vibrate for 3 times, the duration of each vibration is 0.2s, and the interval of 0.1s indicates that the operation is successful, and it passes 5 times of rapid and short vibration, and each vibration lasts for 0.1s, and the interval of 0.1s indicates that the operation failed. .
  • CC2640 low-power Bluetooth microprocessor 6 sends the command F-INFO to the device, and the device receives and parses the command and reads the current The location, orientation, and time information of the device are broadcast through the loudspeaker.
  • Touch the obstacle detection switch, CC2640 low-power Bluetooth microprocessor 6 sends the command F-OBSTICLE to the device, the device receives and parses the command to turn on or off the obstacle detection function, and returns the operation result to the blind stick, the blind stick vibrates Feedback the result of the operation.
  • CC2640 low-power Bluetooth microprocessor 6 Touch the traffic light detection switch, CC2640 low-power Bluetooth microprocessor 6 sends the command F-TRAFFICLIGHT to the device, and the device turns on or off the traffic light detection function after receiving and parsing the command. And return the operation result to the blind stick, and the blind stick vibrates to feedback the operation result.
  • the operation result code R-SUCC indicates that the operation is successful; R-FAIL indicates that the operation fails.
  • the blind stick After the blind stick is powered on and successfully connected to the mobile smart device, it enters the sleep state.
  • the device detects an obstacle ahead, it sends an obstacle information code to the blind stick, and the blind stick wakes up after receiving the information, and then parses the obstacle information ahead.
  • the obstacle information code rules are as follows: OBS-NONE, indicating that there is no obstacle ahead; OBS-2, indicating that there is an obstacle 2 meters ahead; OBS-1, indicating that there is an obstacle 1 meter ahead. If the information code indicates that there is an obstacle ahead, the blind stick will select different vibration frequencies to provide continuous vibration feedback to the user according to the distance information of the obstacle.
  • vibration frequency of the blind stick is 5Hz; if there is an obstacle 1 meter ahead, the vibration frequency of the blind stick is 10Hz. If the received obstacle information indicates that there is no obstacle ahead, the blind stick stops vibrating and enters the sleep state.
  • the obstacle detection algorithm consists of four parts, namely threshold segmentation, coordinate system transformation, ground detection based on RANSAC and obstacle detection based on ground area growth.
  • the operation process of the obstacle detection algorithm is as follows: After the obstacle detection function is enabled, the mobile app accesses the camera of the current device, and obtains and processes the depth map data of the scene ahead in real time.
  • the threshold segmentation algorithm Since the depth map reflects the distance information from the camera to the object, refer to formula (1), set the image depth threshold ⁇ to 2000, the unit mm, and filter out the depth map data other than the effective perceptual distance, That is, objects that are too far away are ignored, and only data within the threshold ⁇ is processed, avoiding a large amount of invalid data to participate in subsequent calculations, which is beneficial to the real-time performance of the algorithm.
  • i and j represent the row and column subscripts of the pixel in the depth map, and a represents the depth value of the pixel.
  • the present invention fuses accelerometer and gyroscope data, generates depth map data as point cloud data under the camera coordinate system O c -X c Y c Z c , and then converts it into world coordinates Point cloud data under O w -X w Y w Z w to correct the direction.
  • the coordinate system conversion method is as follows.
  • the attitude angle is calculated using the acceleration sensor data and gyroscope data of the current device.
  • the rotation sequence selected for attitude calculation is ZYX.
  • the initial moment of the camera coordinate system coincides with the world coordinate system, and then rotates around its own Z, Y, and X axes in turn, and rotates around the Z axis by an angle ⁇ and around the Y axis by an angle ⁇ .
  • the accelerometer is placed horizontally, that is, the Z-axis is vertically upward, the Z-axis can read a value of 1g (g is the acceleration of gravity), and the X-axis and Y-axis read 0 in both directions.
  • the gravitational acceleration When the accelerometer rotates to a certain attitude, the gravitational acceleration will produce corresponding components on the 3 axes of acceleration. Denote the three values read by the accelerometer as [a x , a y , a z ]. According to formula (2), the angle ⁇ and the angle ⁇ can be obtained. Since the gravitational acceleration felt when rotating around the Z axis is constant, the ⁇ angle cannot be calculated only by the accelerometer.
  • the gyroscope of the device uses the gyroscope of the device to measure the angular velocity of rotation around the three axes, denoted as [g x , g y , g z ], and the angle can be obtained by integrating the angular velocity.
  • the attitude angles of the camera at time t-1 are ⁇ t-1 , ⁇ t-1 , ⁇ t-1
  • the attitude angles at time t are ⁇ gyro , ⁇ gyro , ⁇ gyro
  • ⁇ , ⁇ , and ⁇ are the corresponding attitude angle changes.
  • the change amount of the attitude angle can be obtained by integrating the angular velocity and the sampling time period, ⁇ t is the sampling time period, and the magnitude is 0.02s.
  • the angular velocity in formula (4) The world coordinate system is used as the reference system, and the angular velocity [g x , g y , g z ] read by the gyroscope at time t is based on the camera coordinate system, so the angular velocity measured by the gyroscope is converted according to formula (5).
  • the acceleration sensor can calculate the ⁇ and ⁇ angles at the time of rest.
  • the acceleration caused by the movement in a short time is inaccurate, and the average value in a long time is accurate, that is, the acceleration data has high frequency noise.
  • the gyroscope is only sensitive to the attitude change during rotation, and if there is an error in itself, the gyroscope data will have low-frequency noise. Therefore, the present invention uses the complementary filter to combine the attitude angles calculated by the two to perform complementary fusion, as shown in formula (6).
  • the present invention is set to 0.98 (6)
  • the camera attitude angle is used to convert the data in the camera coordinate system O c -X c Y c Z c to the data in the world coordinate system O w -X w Y w Z w .
  • the coordinates of the point P c in the camera coordinate system are (x c , y c , z c )
  • the attitude angle is ( ⁇ , ⁇ , ⁇ ).
  • Formula (7) represents the rotation process of the point P c , and the coordinates are converted from the camera coordinate system to the world coordinate system through three rotation matrices R 1 , R 2 , and R 3 . Perform this operation on all points in the point cloud data to complete the coordinate system transformation.
  • the three rotation matrices are:
  • Equation (8) describes the mathematical model of a plane in three-dimensional space.
  • the RANSAC algorithm uses an iterative method to fit the parameters A, B, C, and D in formula (8) of the plane mathematical model. The specific steps are as follows:
  • the appropriate number of iterations n is selected; in the RANSAC algorithm, the values of the number of iterations n and the threshold ⁇ directly affect the accuracy of the detection results. If the number of iterations n is too large, the running time will be too long, and if the value is too small, the plane cannot be fitted well. Therefore, the present invention selects the appropriate iteration number k according to formula (10). Among them, p is the probability of selecting a normal point at least once, ⁇ is the ratio of abnormal points in the point cloud, and s is the number of points selected in each iteration.
  • d i > ⁇ this point is considered as an outlier point and is not extracted as a ground point cloud; otherwise, it is a valid value point and is extracted as a ground point cloud.
  • the improvement of the present invention to the RANSAC algorithm is as follows: First, after fitting the plane parameters in step 2, the inclination angle ⁇ of the plane is calculated according to formula (12).
  • the point cloud data belonging to this plane in the world coordinate system O w -X w Y w Z w should have zero inclination angle with the ground. If the inclination angle of the fitted plane is greater than 30°, the plane is not the ground plane, it should be culled and the next steps should be stopped. This method can effectively avoid the interference of inclined surfaces such as walls in the scene, and reduce unnecessary iterative calculations.
  • the present invention calculates the height of the camera from the ground according to formula (13) according to the tilt angle ⁇ of the camera to the ground and the depth depth from the ground (as shown in FIG. 9 ). Then, the heights of all points in the point cloud are obtained, and the height difference with the ground is calculated. The points with height difference greater than 200 (mm) are removed and the RANSAC algorithm is no longer executed.
  • the present invention divides the data collected by the camera into four areas, A, B, C, and D, as shown in FIG. 10 .
  • A is the normal traffic area
  • B is the left environmental area
  • C is the right environmental area
  • D is the far area ahead.
  • Area A is located in front of the user and contains the most abundant ground information. Therefore, the present invention regards area C as a region of interest (ROI), and only generates point clouds for the depth map data of the ROI.
  • the RANSAC algorithm performs ground fitting and maps the fitted ground point cloud to the depth map.
  • the improved RANSAC algorithm can roughly fit the ground plane, but it is insufficient for image edge processing. And as mentioned above, in order to avoid excessive computation, the present invention only generates point clouds for the ROI region (A region) and fits the ground. However, when the walking direction of the blind person changes, it is also necessary to perform supplementary detection on the ground in areas B and C on the left and right sides. Therefore, the present invention uses a region growing algorithm to find the remaining ground. According to the calculation formula (4) of the rotation angle of the gyroscope, the current walking direction of the blind person is judged.
  • a positive ⁇ value means that the blind person is turning to the left at this time, and the obstacle information in the left area B needs to be detected;
  • a negative ⁇ value means that the blind person is turning to the right at this time, and the obstacle information in the right area C needs to be detected.
  • the role of the region growing algorithm is to merge points that meet the growth rules together to form a flat region. Its operation process is:
  • step 2 Repeat step 2 until ground area growth is performed for each seed point in the set S.
  • each point in the neighborhood must be judged according to the specified growth rules.
  • the ground pixel value in the depth map is only related to the pixels in the four neighborhoods, and the depth values of the left and right neighborhoods and the upper and lower neighborhoods of the ground pixels change in different ways, the traditional method is not suitable for the scene in which the present invention is located. Therefore, a method is proposed. How to select three neighborhoods. For the seed point G i , firstly, in the unit of row, select the left and right neighbor pixel points in the same row with it, and add the points that satisfy the growth rule to the set R i . When the growth of elements in the same row is completed, the upper neighboring pixel points of all seed points in the row are selected, and the points that satisfy the growth rules among these points are added to the set R i .
  • P ij does not belong to any growth region, i.e. (m ⁇ [1,s] and m ⁇ i).
  • the app sends the obstacle information code to the blind stick. If the number of obstacle pixels in an area is less than 10,000, it is considered that there are no obstacles in this area, and the app sends the code of no obstacle ahead to the blind stick.

Landscapes

  • Health & Medical Sciences (AREA)
  • Epidemiology (AREA)
  • Pain & Pain Management (AREA)
  • Physical Education & Sports Medicine (AREA)
  • Rehabilitation Therapy (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • Rehabilitation Tools (AREA)

Abstract

基于移动可穿戴计算的智慧盲杖属于电子信息领域，是一种基于可穿戴计算技术的盲人辅助装置。装置自身具有照明、发声发光呼救功能并可通过按钮控制已连接的移动智能设备(2,3,4,5)上功能的启用或关闭。移动智能设备(2,3,4,5)上的功能有：使用阈值分割、RANSAC和区域生长算法实现障碍物检测功能；语音播报当前时间、位置信息功能；红绿灯检测功能。本智慧盲杖具有单指操作的特性；自动连接移动智能设备(2,3,4,5)，并与其高效通信；配套的APP上具有语音信息播报、障碍物检测、红绿灯检测功能；采用振动的方式向盲人迅速反馈，增强其使用体验。

Description

[根据细则26改正04.03.2021]　基于移动可穿戴计算和快速深度神经网络的智慧盲杖 技术领域
本发明属于电子信息领域，是一种基于移动互联网和可穿戴计算技术的辅助装置，主要借助智慧盲杖与移动智能手持设备的交互，为视障人士室外行走提供基于语音或触觉的反馈，以辅助盲人避障。
背景技术
在我国，视力障碍人士逾1730万，居世界第一。盲杖作为视障人士出行必需品，能够通过触碰地表来感知前方是否有障碍物。目前市场上的智能盲杖种类繁多，价格低廉的产品仅有语音求助功能，然而集成红外线或超声波进行障碍物探测的盲杖价格又十分高昂。所以造成了价格低的智能盲杖功能过于单一，高价智能盲杖又面临着功耗高、体积大等问题。
可穿戴计算以自然的穿戴形式为用户提供计算能力，让人们更加关注于产品功能而忽略计算设备的外观形态。移动计算使得计算机或其他信息智能终端设备在无线网络环境下实现数据及计算资源的共享，其随着移动通信、互联网、分布式计算等技术的发展日益融入到大家的工作生活中。短距离无线通信技术是将二者有机融合的桥梁，目前最常用的短距离无线通信技术有wifi、蓝牙、Zigbee。其中，蓝牙可以让移动设备在短距离内建立连接，实现数据传输和交换。与wifi和zigbee相比，蓝牙具备低功耗、低辐射、低成本、低延时等特点。因此，本发明采用蓝牙实现盲杖与移动智能设备间的通讯。
发明内容
为实现移动智能设备与盲杖通讯，并通过移动智能设备和盲杖辅助视障人士出行的目标，本发明借助可穿戴计算、移动计算和蓝牙技术的优势，提出的基于移动可穿戴计算的智慧盲杖包括：盲杖的杆体和杖柄。其中所述杖柄中设置有电源、主控模块；所述杖柄前侧设置有照明灯；所述杖柄内侧(即使用中大拇指所处位置)设置有按钮盘，所述按钮盘包括一个开关按钮以及五个功能按钮。所述主控模块集成了CC2640低功耗蓝牙微处理器、振动马达、蜂鸣器以及所述照明灯、按钮盘控制链路与接口连接。
本发明的特征如下：
①单指操作
本发明设计了一种圆盘式按钮组，按钮之间保留一定距离以防误触。按钮盘安装在盲杖杖柄的内侧，视障人士仅通过大拇指即可完成所有操作。按钮上带有盲文，代表不同功能。视障人士轻触某按钮后，盲杖会立即执行该功能。
②按钮功能
开关按钮：
圆盘中心的按钮为开关，处于关机状态时，长按3s后自动开机并迅速以振动方式向盲人反馈，同时尝试与移动智能设备连接，连接成功后由设备向用户语音反馈或由盲杖杖柄振动反馈提示“连接成功”。处于开机状态时，长按3s开关按钮后自动关机，并由盲杖振动反馈。
如遇配对不成功或用户无需连接设备等情况，蓝牙模块在3分钟后会进入休眠状态以节省能源。在休眠状态中轻触开关按钮，蓝牙模块重新尝试与设备连接并重新开始计时。
照明灯按钮：在光线较暗的情况下，照明灯辅助视力障碍人士夜间出行。
发声、发光呼救按钮：如遇险情，视障人士可按下呼救按钮，启动蜂鸣器发声呼救，同时照明灯不断开关形成曳光向救援人员告知其所处位置。
信息播报按钮：轻触该按钮，本发明由蓝牙向已连接的移动智能设备发送请求，设备接收后通过其扩音器向视障人士语音播报当前位置、朝向、时间。
障碍物检测开关：轻触该按钮，本发明向移动智能设备发送请求，打开或关闭移动智能设备的障碍物检测功能。
红绿灯检测开关：轻触该按钮，本发明向移动智能设备发送请求，打开或关闭移动智能设备的红绿灯检测功能。
③振动/语音反馈模式根据用户需求切换
反馈是人机交互的关键要素。本发明在设计过程中考虑到视障人士在户外行走时，双耳需密切关注周围环境中的声音，进而判断车辆的行驶方向、红绿灯起止时间等等，为了减少来自语音反馈给盲人判断环境状态带来的干扰，本发明采用默认振动反馈。振动反馈的优势在于简单迅速，轻触按钮后通过预设的几种振动模式向视障人士反馈必要信息。
本发明采用如下模式：
1)开机：1次连续长振动，持续时间0.8s
2)任意功能操作成功：3次短振动，每次振动持续时间0.2s，间隔0.1s。
3)任意功能操作失败：5次快速短促振动，每次振动持续0.1s，间隔0.1s。
4)障碍物警示：根据移动智能设备回传的前方障碍物距离数值，进行不同频率的振动。距离越小，振动频率越快。振动持续至前方无障碍物时停止。
5)关机：2次中等时间振动，每次振动持续时间0.4s，间隔0.1s。
在移动设备app中，视障人士可根据需要选择使用语音反馈或振动反馈。
④基于移动智能设备的盲人位置、朝向和时间信息语音播报
系统利用移动智能设备内置的GPS/北斗、指南针、时钟、蓝牙等元器件和功能模块，开发运行于移动智能设备的程序，通过接收智慧盲杖上“信息播报功能”的请求，为视障人士提供语音播报视障人士当前所处位置、朝向和时间，辅助视障人士出行。
⑤基于移动智能设备的障碍物检测报警
系统利用移动智能设备内置的摄像头、气压计等元器件和功能模块，开发基于阈值和区域成长的障碍物监测程序。当用户打开智慧盲杖上“障碍物检测开关”，移动智能设备运行该程序并进行基于视觉的障碍物检测，当检测到障碍物时为视障人士提供语音或振动告警。
⑥基于移动智能设备的红绿灯监测
系统利用移动智能设备内置的摄像头开发基于Yolo v4的红绿灯检测识别程序。当用户打开智慧盲杖上“红绿灯检测开关”，移动智能设备上的红绿灯检测程序将进行红绿灯的检测识别，并将检测结果通过语音告知视障人士。
与现有技术相比，本发明的有益效果是：采用新型的结构设计，使得本发明能够单指操作；自动连接移动智能设备，并与其高效通信；同时，将语音信息播报、障碍物检测、红绿灯检测等功能转移至移动智能设备上，既降低了盲杖的功耗，又能帮助盲人在行进中最大化、无障碍地利用盲杖和移动智能设备；采用振动的方式向盲人迅速反馈，增强盲人的使用体验。
附图说明
图1本发明所处网络环境
图2本发明的系统结构示意图
图3本发明杖柄部分的内部结构示意图
图4本发明开机及连接设备流程图
图5本发明功能执行流程图
图6本发明与移动智能设备交互流程图
图7障碍物检测功能执行流程
图8相机坐标系与世界坐标系示意图
图9摄像机高度示意图
图10区域划分和地面检测结果
具体实施方式
图1为本发明所处网络环境，图中：1、盲杖杖柄；2、智能手机；3、智能眼镜；4、可穿戴设备；5、平板电脑。如图所示，盲杖杖柄1中的蓝牙模块开启后可与指定移动智能设备建立蓝牙连接，本发明所指的移动智能设备包括：智能手机2；智能眼镜3；可穿戴设备4； 平板电脑5。
图2为本发明的系统结构示意图，如图所示：本发明处于开机状态时，轻触开关11，系统产生配对中断请求，CC2640低功耗蓝牙微处理器6尝试与周围的移动智能设备配对。长按3s开关11，系统产生关机中断请求，CC2640低功耗蓝牙微处理器6进入关闭状态。轻触功能按钮12，系统产生对应功能的中断请求，CC2640低功耗蓝牙微处理器6接收请求后进行该功能的处理。CC2640低功耗蓝牙微处理器6直接控制振动马达7；照明灯8；时钟9；蜂鸣器10。时钟9到达3分钟后产生休眠中断请求，CC2640低功耗蓝牙微处理器6接收休眠中断后将时钟归0，并进入休眠状态。电源模块由充电控制器13；电池14构成。充电控制器13采用MAX1555充电控制器，电池14采用LP2992锂电池控制芯片，使模块既能通过3～5V锂电池供电，也可以通过USB接口电路供电，并可通过USB接口为锂电池充电。
图3为本发明杖柄部分的内部结构示意图，所述振动马达6；振动马达7；照明灯8；蜂鸣器10；开关11；功能按钮12均与CC2640低功耗蓝牙微处理器6电性连接。照明灯8安装在杖柄1前部；开关11和功能按钮12安装在杖柄1内侧中部，方便视障人士单指操作；蜂鸣器10和CC2640低功耗蓝牙微处理器6设置在杖柄1中部偏后；振动马达7设置在杖柄1偏后部，位于视障人士手掌下方；充电控制器13和电池14安装在杖柄1后部。
本发明开关机及连接设备的过程：
如图4所示当盲杖处于关机状态时，长按3s开关按钮开机，振动马达5通过0.8s的长振动提示已开机，并启动时钟。随后唤醒CC2640低功耗蓝牙微处理器6，自动进入配对模式，尝试与周围的移动智能设备配对。连接成功后通过移动智能设备语音提示“连接成功”，时钟归0。若连接不成功，且未接收到休眠中断，CC2640低功耗蓝牙微处理器6继续尝试配对。时钟到达3分钟后发送休眠中断请求并归0，CC2640低功耗蓝牙微处理器6进入休眠状态，振动马达7以5次快速短促振动，每次振动持续0.1s，间隔0.1s的方式提示配对失败。处于休眠模式时，轻触开关按钮，手动向CC2640低功耗蓝牙微处理器6发送信号，再次尝试配对。开机状态下，长按3s开关机按钮，振动马达7通过连续两次0.4s，间隔0.1s的振动提示已关机。
本发明功能的执行流程：
如图5所示，盲杖开机后，轻触照明灯开关，产生照明灯中断请求，CC2640低功耗蓝牙微处理器6接收请求后开启照明灯8，再次轻触照明灯按钮关闭照明灯。轻触呼救功能按钮，产生呼救中断请求，CC2640低功耗蓝牙微处理器6接收请求后开启蜂鸣器10，并不断开关照明灯8，再次轻触呼救功能按钮关闭蜂鸣器10和照明灯8。连接移动智能设备后，轻触语音播报当前位置、朝向、时间按钮、障碍物检测开关、红绿灯检测开关，分别产生语音播报中断、 障碍物检测中断、红绿灯检测中断，CC2640低功耗蓝牙微处理器6向移动智能设备发送相应功能指令，设备接收并解析指令，执行对应功能并回传操作结果。盲杖接收操作结果后振动马达7通过3次短振动，每次振动持续时间0.2s，间隔0.1s提示操作成功，通过5次快速短促振动，每次振动持续0.1s，间隔0.1s提示操作失败。
本发明与移动智能设备交互流程：
如图6所示，盲杖开机并与移动智能设备连接成功后，轻触信息播报按钮，CC2640低功耗蓝牙微处理器6向设备发送指令F-INFO，设备接收并解析指令后读取当前设备的位置、朝向、时间信息，并通过扩音器播报。轻触障碍物检测开关，CC2640低功耗蓝牙微处理器6向设备发送指令F-OBSTICLE，设备接收并解析指令后打开或关闭障碍物检测功能，并向盲杖回传操作结果，盲杖振动反馈操作结果。轻触红绿灯检测开关，CC2640低功耗蓝牙微处理器6向设备发送指令F-TRAFFICLIGHT，设备接收并解析指令后打开或关闭红绿灯检测功能。并向盲杖回传操作结果，盲杖振动反馈操作结果。其中，操作结果代码R-SUCC表示操作成功；R-FAIL表示操作失败。
本发明障碍物检测功能执行流程：
如图7所示，盲杖开机并与移动智能设备连接成功后，进入休眠状态。当设备检测到前方出现障碍物时，向盲杖发送障碍物信息代码，盲杖接收到信息后被唤醒，随即解析前方障碍物信息。其中，障碍物信息代码规则如下：OBS-NONE，表示前方无障碍物；OBS-2，表示前方2米有障碍物；OBS-1，表示前方1米有障碍物。若信息代码表示前方有障碍物，盲杖则根据障碍物的距离信息选择不同的振动频率向用户持续振动反馈。振动频率选取：前方2米出现障碍物，盲杖振动频率为5Hz；前方1米出现障碍物，盲杖振动频率为10Hz。若接收的障碍物信息表示前方无障碍物，盲杖停止振动，进入休眠状态。
障碍物检测算法：
障碍物检测算法4部分组成，分别是阈值分割、坐标系转换、基于RANSAC的地面检测和基于地面区域生长的障碍物检测。障碍物检测算法运行流程如下：障碍物检测功能开启后，移动端app访问当前设备的摄像机，实时获取并处理前方场景的深度图数据。
首先运行阈值分割算法，由于深度图反映的是摄像头到物体的距离信息，因此参考公式(1)，将图像深度阈值σ设置为2000，单位mm，滤除有效感知距离之外的深度图数据，即忽略距离过远的物体，只处理在阈值σ范围内的数据，避免大量无效数据参与后续计算，有利于算法的实时性。公式(1)中i、j表示像素点在深度图中的行、列下标，a表示该像素点的深度值。
Figure PCTCN2021076403-appb-000001
第二，如图8所示，本发明融合加速度计以及陀螺仪数据，将深度图数据生成为相机坐标 系O c-X cY cZ c下的点云数据，再将其转换成世界坐标系O w-X wY wZ w下的点云数据，以矫正方向。坐标系转换方式如下所述。
首先利用当前设备的加速度传感器数据以及陀螺仪数据计算姿态角度。姿态解算选用的旋转顺序为ZYX，摄像头坐标系初始时刻与世界坐标系重合，然后依次绕自己的Z、Y、X轴进行旋转，绕着Z轴旋转角度γ、绕着Y轴旋转角度β、绕着X轴旋转角度α。当加速度计水平放置，即Z轴竖直向上时，Z轴可以读到1g的数值(g为重力加速度)，X轴和Y轴两个方向读到0。当加速度计旋转一定的姿态时，重力加速度会在加速度的3个轴上产生相应的分量。将加速度计读到的3个值记作[a x,a y,a z]。根据公式(2)可以求出角度α和角度β，由于绕Z轴旋转时，感受到的重力加速度是不变的，因此仅靠加速度计无法计算γ角。
Figure PCTCN2021076403-appb-000002
接着使用设备的陀螺仪测量绕3个轴转动的角速度，记作[g x,g y,g z]，对角速度积分可以得到角度。如公式(3)所示，摄像头在t-1时刻时的姿态角度为α t-1、β t-1、γ t-1，t时刻的姿态角为α gyro、β gyro、γ gyro，摄像头由t-1时刻到t时刻经历了3次旋转，Δα、Δβ、Δγ为对应的姿态角度变化量。参考公式(4)，姿态角度的变化量可通过角速度与采样时间周期积分得到，Δt为采样时间周期，大小为0.02s。注：公式(4)中的角速度
Figure PCTCN2021076403-appb-000003
以世界坐标系为参考系，而陀螺仪在第t时刻读出的角速度[g x,g y,g z]以相机坐标系为参考系，故依据公式(5)将陀螺仪测量的角速度转换为姿态更新需要用到的角速度
Figure PCTCN2021076403-appb-000004
Figure PCTCN2021076403-appb-000005
Figure PCTCN2021076403-appb-000006
Figure PCTCN2021076403-appb-000007
根据上面的分析，加速度传感器在静止时刻，可以计算出α和β角，在短时间内因为运动造成的加速度是不准确的，在长时间内的平均值是准确的，即加速度数据存在高频噪声。陀螺仪仅对转动时的姿态变化敏感，且其若本身存在误差，使得陀螺仪数据存在低频噪声。因此，本发明使用互补滤波器结合两者计算的姿态角进行互补融合，如公式(6)所示。
Figure PCTCN2021076403-appb-000008
其中，k为比例系数，
本发明设置为0.98        (6)
最后，利用相机姿态角实现将相机坐标系O c-X cY cZ c中数据向世界坐标系O w-X wY wZ w数据的转换。假定相机坐标系统中点P c的坐标为(x c,y c,z c)，姿态角为(α,β,γ)。公式(7)表示点P c的旋转过程，通过3个旋转矩阵R 1、R 2、R 3将坐标又相机坐标系下转换为世界坐标系下。对点云数据中的所点进行此操作即可完成坐标系转换。
Figure PCTCN2021076403-appb-000009
三个旋转矩阵分别为：
Figure PCTCN2021076403-appb-000010
Figure PCTCN2021076403-appb-000011
第三，如图9所示，为了拍摄到前方距离身体较近的障碍物，需稍向下倾斜摄像头，但由于身前的地面与摄像头距离太近，利用阈值分割无法简单地将地面和障碍物分开，所以还需要对这种情况进行单独的地面去除操作。对此，本发明采用改进后的随机样本一致性算法(random sample consensus，RANSAC)进行地面检测并滤除。公式(8)描述了三维空间中一个平面的数学模型。RANSAC算法采用迭代方式拟合平面数学模型公式(8)中的参数A、B、C、D，其具体步骤如下：
①从世界坐标系O w-X wY wZ w下的点云数据中随机选取一个集合的点，用于计算公式(8)中A、B、C、D的最初取值。
②依据公式(9)计算出点云中每个点(x i,y i,z i)到该平面的距离d i。
③设置距离阈值δ，如果d i<δ，那么点(x i,y i,z i)属于平面；否则，该点不属于平面。
④重复步骤②～④，迭代n次，比较并选取含有效数据点最多的平面作为最终结果；
Ax+By+Cz＝D        (8)
Figure PCTCN2021076403-appb-000012
据公式(10)来选择合适迭代次数n；在RANSAC算法中，迭代次数n和阈值δ的取值直接影响检测结果的精度。迭代次数n取值过大会导致运行时间过长，取值过小又不能较好的拟合平面。故本发明依据公式(10)来选择合适迭代次数k。其中，p为至少一次选取到正常点的概率，η为点云异常点的比率，s为每次迭代选取点的数量。此外，若阈值δ取值过小会漏掉属于地面的点云，而δ取值过大则会添加异常点。对此，本文按照公式(11)计算点到平面距离的 标准差σ，并取2σ作为阈值(即δ＝2σ)。当d i>δ时，此点被认为是异常值点，不当作地面点云提取；反之，为有效值点，被当作地面点云提取出来。
Figure PCTCN2021076403-appb-000013
Figure PCTCN2021076403-appb-000014
其中，
Figure PCTCN2021076403-appb-000015
本发明对RANSAC算法的改进如下：首先，在步骤②拟合出平面参数后，按照公式(12)求解该平面的倾斜角度θ。世界坐标系O w-X wY wZ w下属于该平面的点云数据，其与地面间的倾斜角应该为零。如果拟合平面的倾斜角度大于30°，则该平面不是地平面，应剔除该平面并停止执行后续步骤。这一方法可有效避免场景中墙壁等倾斜面的干扰，减少了不必要的迭代计算。其次，本发明依据摄像机对地倾斜角度θ和距离地面的深度depth，按照公式(13)计算摄像机距离地面的高度height(如图9所示)。随后求出点云中所有的点的高度，并计算与地面的高度差，将高度差大于200(mm)的点清除不再执行RANSAC算法。
Figure PCTCN2021076403-appb-000016
height＝cosθ×depth      (13)
如果对整个RGB-D图像的点云都采用RANSAC算法进行平面拟合，计算量过大难以满足实时性要求。对此，本发明将摄像机采集的数据划分为A、B、C、D四个区域，如图10所示。其中，A为正常通行区域，B为左侧环境区域，C为右侧环境区域，D为前方较远区域。A区域位于使用者身前，包含最为丰富的地面信息，所以本发明将C区域视为感兴趣区域(Region of Interest，ROI)，只对ROI的深度图数据生成点云，采用上述改进后的RANSAC算法进行地面拟合，并将拟合出来的地面点云映射到深度图中。
第四，采用改进后的RANSAC算法可以大致拟合出地平面，但其对图像边缘处理存在不足。并且如前所述，为了避免计算量过大，本发明只对ROI区域(A区域)生成点云并拟合地面。但当盲人行走方向发生变化时，还需要对左右两侧B、C区域的地面进行补充检测，因此本发明采用区域生长算法查找剩余地面。根据陀螺仪旋转角度的计算公式(4)，判断当前盲人的行走方向。Δγ数值为正代表此时盲人正在转向左方，左侧区域B的障碍物信息需要检测；Δγ数值为负代表此时盲人正在转向右方，右侧区域C的障碍物信息需要检测。
区域生长算法的作用是将符合生长规则的点合并到一起，进而形成平面区域。其运行流程为：
①将RANSAC算法拟合到的地面像素点作为种子点，将所有的种子点存储在集合S＝{G 1,G 2,…,G s}中。
②从集合S中取出种子点G i(i＝1,2,…,s)，以G i为中心点进行地面区域生长，将种子点G i的生长区域集合计作R i，将点G i的上、左、右邻域点记作P ij(j＝1,2,3)。以下文 定义的生长规则为标准，将P ij(j＝1,2,3)中满足生长规则的邻域点添加到集合R i中。
③重复步骤②，直到为集合S中的每个种子点都进行地面区域生长。
④最后将所有种子点G i(i＝1,2,…,s)所产生的生长区域R i合并，得到总地面区域R g，如公式(14)所示，s为集合S中点的个数，∪表示取并集。
Figure PCTCN2021076403-appb-000017
区域生长一般有8邻域和4邻域两种邻域选取方式，一旦确定邻域选取方式，须对邻域中每个点按照规定的生长规则进行判断。考虑到深度图中地面像素值仅与4邻域像素存在联系，且地面像素的左右邻域和上下邻域深度值变化方式不同，传统的方式不适用于本发明所处场景，故提出一种三邻域的选取方式。对于种子点G i，首先以行为单位，选取与其处于同一行的左右邻域像素点，将满足生长规则的点添加到集合R i中。当同一行元素生长结束后，再选取该行所有种子点的上邻域像素点，将这些点中满足生长规则的点添加到集合R i中。
本发明规定的生长规则如下。S＝{G 1,G 2,…,G s}为种子点集合，S中种子点个数为s。对于任一种子点G i(i＝1,2,…,s)的上、左、右邻域点P ij(j＝1,2,3)，若满足如下条件则添加到种子点G i的生长区域R i中：
(1)P ij不属于任何生长区域，即
Figure PCTCN2021076403-appb-000018
(m∈[1,s]且m≠i)。
(2)通常地面左右邻域像素的深度值之差不超过15(mm)，地面的下邻域像素到上邻域像素深度值会呈现递增态势。若P ij满足公式(15)，则将其添加到R i中。其中δ h为增长阈值，设为100(mm)，h G为种子点G i的深度值，h为点P ij的深度值。
(3)邻域像素点的选取必须在B、C区域，超出范围就会停止生长。
Figure PCTCN2021076403-appb-000019
最后，将上述算法提取出的地面像素清除，留下的像素均为障碍物。
如果检测结果中A、B、C中某一区域出现障碍物像素数量超过10000个，就视这个区域存在障碍物，app向盲杖发送障碍物信息代码。如果一个区域障碍物像素数量低于10000，则视这个区域不存在障碍物，app向盲杖发送前方无障碍物代码。

Claims (6)

  1. 基于移动可穿戴计算的智慧盲杖，其特征在于：盲杖的杆体和杖柄；其中所述杖柄中设置有电源、主控模块；所述杖柄前侧设置有照明灯；所述杖柄内侧设置有按钮盘，所述按钮盘包括一个开关按钮以及五个功能按钮；所述主控模块集成了CC2640低功耗蓝牙微处理器、振动马达、蜂鸣器、照明灯及按钮盘控制链路。
  2. 应用如权利要求1所述的基于移动可穿戴计算的智慧盲杖的方法，其特征在于：
    ①单指操作
    按钮盘安装在盲杖杖柄的内侧，视障人士仅通过大拇指即可完成所有操作；按钮上带有盲文，代表不同功能；视障人士轻触某按钮后，盲杖会立即执行该功能；
    ②按钮功能
    开关按钮：
    圆盘中心的按钮为开关，处于关机状态时，长按3s后自动开机并迅速以振动方式向盲人反馈，同时尝试与移动智能设备连接，连接成功后由设备向用户语音反馈或由盲杖杖柄振动反馈提示“连接成功”；处于开机状态时，长按3s开关按钮后自动关机，并由盲杖振动反馈；
    如遇配对不成功或用户无需连接设备，蓝牙模块在3分钟后会进入休眠状态以节省能源；在休眠状态中轻触开关按钮，蓝牙模块重新尝试与设备连接并重新开始计时；
    照明灯按钮：在光线较暗的情况下，照明灯辅助视力障碍人士夜间出行；
    发声、发光呼救按钮：如遇险情，视障人士可按下呼救按钮，启动蜂鸣器发声呼救，同时照明灯不断开关形成曳光向救援人员告知其所处位置；
    信息播报按钮：轻触该按钮，由蓝牙向已连接的移动智能设备发送请求，设备接收后通过其扩音器向视障人士语音播报当前位置、朝向、时间；
    障碍物检测开关：轻触该按钮，向移动智能设备发送请求，打开或关闭移动智能设备的障碍物检测功能；
    红绿灯检测开关：轻触该按钮，移动智能设备发送请求，打开或关闭移动智能设备的红绿灯检测功能；
    ③振动/语音反馈模式根据用户需求切换
    默认振动反馈；振动反馈的优势在于简单迅速，轻触按钮后通过预设的几种振动模式向视障人士反馈必要信息；
    ④基于移动智能设备的盲人位置、朝向和时间信息语音播报
    系统利用移动智能设备内置的GPS/北斗、指南针、时钟、蓝牙等元器件和功能模块，开发运行于移动智能设备的程序，通过接收智慧盲杖上“信息播报功能”的请求，为视障人士提供语音播报视障人士当前所处位置、朝向和时间，辅助视障人士出行；
    ⑤基于移动智能设备的障碍物检测报警
    系统利用移动智能设备内置的摄像头、气压计等元器件和功能模块，开发基于阈值和区域成长的障碍物监测程序；当用户打开智慧盲杖上“障碍物检测开关”，移动智能设备运行该程序并进行基于视觉的障碍物检测，当检测到障碍物时为视障人士提供语音或振动告警；
    ⑥基于移动智能设备的红绿灯监测
    系统利用移动智能设备内置的摄像头开发基于Yolo v4的红绿灯检测识别程序；当用户打开智慧盲杖上“红绿灯检测开关”，移动智能设备上的红绿灯检测程序将进行红绿灯的检测识别，并将检测结果通过语音告知视障人士。
  3. 应用如权利要求1所述的基于移动可穿戴计算的智慧盲杖的方法，其特征在于，反馈具体采用如下模式：
    1)开机：1次连续长振动，持续时间0.8s
    2)任意功能操作成功：3次短振动，每次振动持续时间0.2s，间隔0.1s；
    3)任意功能操作失败：5次快速短促振动，每次振动持续0.1s，间隔0.1s；
    4)障碍物警示：根据移动智能设备回传的前方障碍物距离数值，进行不同频率的振动；距离越小，振动频率越快；振动持续至前方无障碍物时停止；
    5)关机：2次中等时间振动，每次振动持续时间0.4s，间隔0.1s。
  4. 应用如权利要求1所述的基于移动可穿戴计算的智慧盲杖的方法，其特征在于，开机状态时，轻触开关，系统产生配对中断请求，CC2640低功耗蓝牙微处理器尝试与周围的移动智能设备配对；长按3s开关，系统产生关机中断请求，CC2640低功耗蓝牙微处理器进入关闭状态；轻触功能按钮，系统产生对应功能的中断请求，CC2640低功耗蓝牙微处理器接收请求后进行该功能的处理；CC2640低功耗蓝牙微处理器直接控制振动马达；照明灯；时钟和蜂鸣器10；时钟到达3分钟后产生休眠中断请求，CC2640低功耗蓝牙微处理器6接收休眠中断后将时钟归0，并进入休眠状态；电源模块由充电控制器和电池14 构成；
    当盲杖处于关机状态时，长按3s开关按钮开机，振动马达5通过0.8s的长振动提示已开机，并启动时钟；随后唤醒CC2640低功耗蓝牙微处理器，自动进入配对模式，尝试与周围的移动智能设备配对；连接成功后通过移动智能设备语音提示“连接成功”，时钟归0；若连接不成功，且未接收到休眠中断，CC2640低功耗蓝牙微处理器继续尝试配对；时钟到达3分钟后发送休眠中断请求并归0，CC2640低功耗蓝牙微处理器进入休眠状态，振动马达以5次快速短促振动，每次振动持续0.1s，间隔0.1s的方式提示配对失败；处于休眠模式时，轻触开关按钮，手动向CC2640低功耗蓝牙微处理器发送信号，再次尝试配对；开机状态下，长按3s开关机按钮，振动马达通过连续两次0.4s，间隔0.1s的振动提示已关机；
    盲杖开机后，轻触照明灯开关，产生照明灯中断请求，CC2640低功耗蓝牙微处理器接收请求后开启照明灯，再次轻触照明灯按钮关闭照明灯；轻触呼救功能按钮，产生呼救中断请求，CC2640低功耗蓝牙微处理器接收请求后开启蜂鸣器，并不断开关照明灯，再次轻触呼救功能按钮关闭蜂鸣器和照明灯；连接移动智能设备后，轻触语音播报当前位置、朝向、时间按钮、障碍物检测开关、红绿灯检测开关，分别产生语音播报中断、障碍物检测中断、红绿灯检测中断，CC2640低功耗蓝牙微处理器向移动智能设备发送相应功能指令，设备接收并解析指令，执行对应功能并回传操作结果；盲杖接收操作结果后振动马达通过3次短振动，每次振动持续时间0.2s，间隔0.1s提示操作成功，通过5次振动，每次振动持续0.1s，间隔0.1s提示操作失败；
    盲杖开机并与移动智能设备连接成功后，轻触信息播报按钮，CC2640低功耗蓝牙微处理器向设备发送指令F-INFO，设备接收并解析指令后读取当前设备的位置、朝向、时间信息，并通过扩音器播报；轻触障碍物检测开关，CC2640低功耗蓝牙微处理器向设备发送指令F-OBSTICLE，设备接收并解析指令后打开或关闭障碍物检测功能，并向盲杖回传操作结果，盲杖振动反馈操作结果；轻触红绿灯检测开关，CC2640低功耗蓝牙微处理器向设备发送指令F-TRAFFICLIGHT，设备接收并解析指令后打开或关闭红绿灯检测功能；并向盲杖回传操作结果，盲杖振动反馈操作结果；其中，操作结果代码R-SUCC表示操作成功；R-FAIL表示操作失败。
  5. 应用如权利要求1所述的基于移动可穿戴计算的智慧盲杖的方法，其特征在于，障碍物检测功能执行流程：
    盲杖开机并与移动智能设备连接成功后，进入休眠状态；当设备检测到前方出现障碍物时，向盲杖发送障碍物信息代码，盲杖接收到信息后被唤醒，随即解析前方障碍物信息； 其中，障碍物信息代码规则如下：OBS-NONE，表示前方无障碍物；OBS-2，表示前方2米有障碍物；OBS-1，表示前方1米有障碍物；若信息代码表示前方有障碍物，盲杖则根据障碍物的距离信息选择不同的振动频率向用户持续振动反馈；振动频率选取：前方2米出现障碍物，盲杖振动频率为5Hz；前方1米出现障碍物，盲杖振动频率为10Hz；若接收的障碍物信息表示前方无障碍物，盲杖停止振动，进入休眠状态。
  6. 应用如权利要求1所述的基于移动可穿戴计算的智慧盲杖的方法，其特征在于，障碍物检测算法包括：是阈值分割、坐标系转换、基于RANSAC的地面检测和基于地面区域生长的障碍物检测；
    障碍物检测算法运行流程如下：障碍物检测功能开启后，移动端app访问当前设备的摄像机，实时获取并处理前方场景的深度图数据；
    首先运行阈值分割算法，由于深度图反映的是摄像头到物体的距离信息，因此参考公式(1)，将图像深度阈值σ设置为2000，单位mm，滤除有效感知距离之外的深度图数据，即忽略距离过远的物体，只处理在阈值σ范围内的数据；公式(1)中i、j表示像素点在深度图中的行、列下标，a表示该像素点的深度值；
    Figure PCTCN2021076403-appb-100001
    第二，融合加速度计以及陀螺仪数据，将深度图数据生成为相机坐标系O c-X cY cZ c下的点云数据，再将其转换成世界坐标系O w-X wY wZ w下的点云数据，以矫正方向；坐标系转换方式如下所述；
    首先利用当前设备的加速度传感器数据以及陀螺仪数据计算姿态角度；姿态解算选用的旋转顺序为ZYX，摄像头坐标系初始时刻与世界坐标系重合，然后依次绕自己的Z、Y、X轴进行旋转，绕着Z轴旋转角度γ、绕着Y轴旋转角度β、绕着X轴旋转角度α；当加速度计水平放置，即Z轴竖直向上时，Z轴可以读到1g的数值，g为重力加速度；X轴和Y轴两个方向读到0；当加速度计旋转一定的姿态时，重力加速度会在加速度的3个轴上产生相应的分量；将加速度计读到的3个值记作[a x,a y,a z]；根据公式(2)求出角度α和角度β，由于绕Z轴旋转时，感受到的重力加速度是不变的，因此仅靠加速度计无法计算γ角；
    Figure PCTCN2021076403-appb-100002
    接着使用设备的陀螺仪测量绕3个轴转动的角速度，记作[g x,g y,g z]，对角速度积分可以得到角度；如公式(3)所示，摄像头在t-1时刻时的姿态角度为α t-1、β t-1、γ t-1，t时刻的姿态角为α gyro、β gyro、γ gyro，摄像头由t-1时刻到t时刻经历了3次旋转，Δα、Δβ、Δγ为对应的姿态角度变化量；参考公式(4)，姿态角度的变化量通过角速度与采样时间周期积分 得到，Δt为采样时间周期，大小为0.02s；
    公式(4)中的角速度
    Figure PCTCN2021076403-appb-100003
    以世界坐标系为参考系，而陀螺仪在第t时刻读出的角速度[g x,g y,g z]以相机坐标系为参考系，故依据公式(5)将陀螺仪测量的角速度转换为姿态更新需要用到的角速度
    Figure PCTCN2021076403-appb-100004
    Figure PCTCN2021076403-appb-100005
    Figure PCTCN2021076403-appb-100006
    Figure PCTCN2021076403-appb-100007
    根据上面的分析，加速度传感器在静止时刻，计算出α和β角，在短时间内因为运动造成的加速度是不准确的，在长时间内的平均值是准确的，即加速度数据存在高频噪声；陀螺仪仅对转动时的姿态变化敏感，且其若本身存在误差，使得陀螺仪数据存在低频噪声；使用互补滤波器结合两者计算的姿态角进行互补融合，如公式(6)所示；
    Figure PCTCN2021076403-appb-100008
    其中，k为比例系数，
    k设置为0.98   (6)
    最后，利用相机姿态角实现将相机坐标系O c-X cY cZ c中数据向世界坐标系O w-X wY wZ w数据的转换；假定相机坐标系统中点P c的坐标为(x c,y c,z c)，姿态角为(α,β,γ)；公式(7)表示点P c的旋转过程，通过3个旋转矩阵R 1、R 2、R 3将坐标又相机坐标系下转换为世界坐标系下；对点云数据中的所点进行此操作即完成坐标系转换；
    Figure PCTCN2021076403-appb-100009
    三个旋转矩阵分别为：
    Figure PCTCN2021076403-appb-100010
    Figure PCTCN2021076403-appb-100011
    第三，采用改进后的随机样本一致性算法(random sample consensus，RANSAC)进行地面检测并滤除；公式(8)描述了三维空间中一个平面的数学模型；RANSAC算法采用迭代 方式拟合平面数学模型公式(8)中的参数A、B、C、D，其具体步骤如下：
    ①从世界坐标系O w-X wY wZ w下的点云数据中随机选取一个集合的点，用于计算公式(8)中A、B、C、D的最初取值；
    ②依据公式(9)计算出点云中每个点(x i,y i,z i)到该平面的距离d i；
    ③设置距离阈值δ，如果d i<δ，那么点(x i,y i,z i)属于平面；否则，该点不属于平面；
    ④重复步骤②～④，迭代n次，比较并选取含有效数据点最多的平面作为最终结果；
    Ax+By+Cz＝D  (8)
    Figure PCTCN2021076403-appb-100012
    据公式(10)来选择合适迭代次数n；其中，p为至少一次选取到正常点的概率，η为点云异常点的比率，s为每次迭代选取点的数量；按照公式(11)计算点到平面距离的标准差σ，并取2σ作为阈值即δ＝2σ；当d i>δ时，此点被认为是异常值点，不当作地面点云提取；反之，为有效值点，被当作地面点云提取出来；
    Figure PCTCN2021076403-appb-100013
    Figure PCTCN2021076403-appb-100014
    其中，
    Figure PCTCN2021076403-appb-100015
    首先，在步骤②拟合出平面参数后，按照公式(12)求解该平面的倾斜角度θ；世界坐标系O w-X wY wZ w下属于该平面的点云数据，其与地面间的倾斜角应该为零；如果拟合平面的倾斜角度大于30°，则该平面不是地平面，应剔除该平面并停止执行后续步骤；
    其次，依据摄像机对地倾斜角度θ和距离地面的深度depth，按照公式(13)计算摄像机距离地面的高度height；随后求出点云中所有的点的高度，并计算与地面的高度差，将高度差大于200(mm)的点清除不再执行RANSAC算法；
    Figure PCTCN2021076403-appb-100016
    height＝cosθ×depth  (13)
    将摄像机采集的数据划分为A、B、C、D四个区域，如图10所示；其中，A为正常通行区域，B为左侧环境区域，C为右侧环境区域，D为前方较远区域；A区域位于使用者身前，包含最为丰富的地面信息，将C区域视为感兴趣区域ROI，只对ROI的深度图数据生成点云，采用上述改进后的RANSAC算法进行地面拟合，并将拟合出来的地面点云映射到深度图中；
    第四，采用区域生长算法查找剩余地面；根据陀螺仪旋转角度的计算公式(4)，判断当前盲人的行走方向；Δγ数值为正代表此时盲人正在转向左方，左侧区域B的障碍物信息 需要检测；Δγ数值为负代表此时盲人正在转向右方，右侧区域C的障碍物信息需要检测；
    区域生长算法的作用是将符合生长规则的点合并到一起，进而形成平面区域；其运行流程为：
    (一)将RANSAC算法拟合到的地面像素点作为种子点，将所有的种子点存储在集合S＝{G 1,G 2,…,G s}中；
    (二)从集合S中取出种子点G i(i＝1,2,…,s)，以G i为中心点进行地面区域生长，将种子点G i的生长区域集合计作R i，将点G i的上、左、右邻域点记作P ij(j＝1,2,3)；以下文定义的生长规则为标准，将P ij(j＝1,2,3)中满足生长规则的邻域点添加到集合R i中；
    (三)重复步骤(二)，直到为集合S中的每个种子点都进行地面区域生长；
    (四)最后将所有种子点G i(i＝1,2,…,s)所产生的生长区域R i合并，得到总地面区域R g，如公式(14)所示，s为集合S中点的个数，∪表示取并集；
    Figure PCTCN2021076403-appb-100017
    考虑到深度图中地面像素值仅与4邻域像素存在联系，且地面像素的左右邻域和上下邻域深度值变化方式不同，提出一种三邻域的选取方式；对于种子点G i，首先以行为单位，选取与其处于同一行的左右邻域像素点，将满足生长规则的点添加到集合R i中；当同一行元素生长结束后，再选取该行所有种子点的上邻域像素点，将这些点中满足生长规则的点添加到集合R i中；
    规定的生长规则如下；S＝{G 1,G 2,…,G s}为种子点集合，S中种子点个数为s；对于任一种子点G i(i＝1,2,…,s)的上、左、右邻域点P ij(j＝1,2,3)，若满足如下条件则添加到种子点G i的生长区域R i中：
    (1)P ij不属于任何生长区域，即
    Figure PCTCN2021076403-appb-100018
    (m∈[1,s]且m≠i)；
    (2)若P ij满足公式(15)，则将其添加到R i中；其中δ h为增长阈值，设为100(mm)，h G为种子点G i的深度值，h为点P ij的深度值；
    (3)邻域像素点的选取必须在B、C区域，超出范围就会停止生长；
    Figure PCTCN2021076403-appb-100019
    将提取出的地面像素清除，留下的像素均为障碍物；
    如果检测结果中A、B、C中某一区域出现障碍物像素数量超过10000个，就视这个区域存在障碍物，app向盲杖发送障碍物信息代码；如果一个区域障碍物像素数量低于10000，则视这个区域不存在障碍物，app向盲杖发送前方无障碍物代码。
PCT/CN2021/076403 2021-01-14 2021-02-09 基于移动可穿戴计算和快速深度神经网络的智慧盲杖 WO2022151560A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110058304.3 2021-01-14
CN202110058304.3A CN112891162B (zh) 2021-01-14 2021-01-14 基于移动可穿戴计算的智慧盲杖

Publications (1)

Publication Number Publication Date
WO2022151560A1 true WO2022151560A1 (zh) 2022-07-21

Family

ID=76113964

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/076403 WO2022151560A1 (zh) 2021-01-14 2021-02-09 基于移动可穿戴计算和快速深度神经网络的智慧盲杖

Country Status (2)

Country Link
CN (1) CN112891162B (zh)
WO (1) WO2022151560A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115218918A (zh) * 2022-09-20 2022-10-21 上海仙工智能科技有限公司 一种智能导盲方法及导盲设备

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114177017A (zh) * 2021-12-29 2022-03-15 东北农业大学 一种盲人出行辅助系统
WO2024005243A1 (ko) * 2022-07-01 2024-01-04 엘지전자 주식회사 시각 장애인용 보행 보조 시스템 및 그 방법

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN204319215U (zh) * 2014-12-04 2015-05-13 天津朗誉科技发展有限公司 一种盲人手杖
CN105232295A (zh) * 2015-10-16 2016-01-13 北京机械设备研究所 一种用于避障的智能导引系统
WO2016199991A1 (ko) * 2015-06-08 2016-12-15 (주)아이온 시각 장애인 및 노약자를 위한 지능형 보행 보조 장치
EP3109668A1 (fr) * 2015-06-25 2016-12-28 Gosense Appareil à main pour un utilisateur déficient visuel
KR101779718B1 (ko) * 2016-08-03 2017-09-19 울산대학교 산학협력단 시각장애인을 위한 안전 지팡이 장치 및 그 제어 방법
WO2018056571A1 (ko) * 2016-09-20 2018-03-29 박제상 사용자별 커스텀 키보드 매핑이 가능한 블루투스 IoT 키보드
CN208598763U (zh) * 2017-07-27 2019-03-15 西南大学 一种盲人用多功能手杖
CN110575371A (zh) * 2019-10-22 2019-12-17 大连民族大学 一种智能导盲手杖及控制方法
CN112076071A (zh) * 2020-09-01 2020-12-15 孙松 一种可定位导航的盲人手杖

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003093454A (ja) * 2001-09-21 2003-04-02 Ricoh Co Ltd 視覚障害者用杖
CN106618980A (zh) * 2016-11-05 2017-05-10 深圳市前海安测信息技术有限公司 可穿戴智能导盲系统
CN211410027U (zh) * 2019-08-29 2020-09-04 北京阳光智源科技有限公司 一种可穿戴激光测距盲杖模组

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN204319215U (zh) * 2014-12-04 2015-05-13 天津朗誉科技发展有限公司 一种盲人手杖
WO2016199991A1 (ko) * 2015-06-08 2016-12-15 (주)아이온 시각 장애인 및 노약자를 위한 지능형 보행 보조 장치
EP3109668A1 (fr) * 2015-06-25 2016-12-28 Gosense Appareil à main pour un utilisateur déficient visuel
CN105232295A (zh) * 2015-10-16 2016-01-13 北京机械设备研究所 一种用于避障的智能导引系统
KR101779718B1 (ko) * 2016-08-03 2017-09-19 울산대학교 산학협력단 시각장애인을 위한 안전 지팡이 장치 및 그 제어 방법
WO2018056571A1 (ko) * 2016-09-20 2018-03-29 박제상 사용자별 커스텀 키보드 매핑이 가능한 블루투스 IoT 키보드
CN208598763U (zh) * 2017-07-27 2019-03-15 西南大学 一种盲人用多功能手杖
CN110575371A (zh) * 2019-10-22 2019-12-17 大连民族大学 一种智能导盲手杖及控制方法
CN112076071A (zh) * 2020-09-01 2020-12-15 孙松 一种可定位导航的盲人手杖

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115218918A (zh) * 2022-09-20 2022-10-21 上海仙工智能科技有限公司 一种智能导盲方法及导盲设备
CN115218918B (zh) * 2022-09-20 2022-12-27 上海仙工智能科技有限公司 一种智能导盲方法及导盲设备

Also Published As

Publication number Publication date
CN112891162B (zh) 2022-11-22
CN112891162A (zh) 2021-06-04

Similar Documents

Publication Publication Date Title
WO2022151560A1 (zh) 基于移动可穿戴计算和快速深度神经网络的智慧盲杖
CN104949707B (zh) 基于信息推送的运动监测设备及方法
CN205494329U (zh) 智能自救导盲拐杖
CN108509037B (zh) 一种信息显示方法及移动终端
CN111126182B (zh) 车道线检测方法、装置、电子设备及存储介质
CN105214296B (zh) 一种获得运动信息的方法
CN103677267A (zh) 移动终端及其唤醒方法、装置
CN105938396A (zh) 一种音乐播放器的控制系统及方法
CN105278672A (zh) 具有运动特征化电路的电子设备
CN110095128B (zh) 获取缺失道路情报的方法、装置、设备及存储介质
CN110970003A (zh) 屏幕亮度调整方法、装置、电子设备及存储介质
CN109059929B (zh) 导航方法、装置、穿戴式设备及存储介质
WO2018045569A1 (zh) 一种控制移动终端屏幕的方法及装置
CN106547204A (zh) 一种智能手表及其亮显屏幕的方法
CN107566627B (zh) 用户不良使用习惯辅助提醒系统及方法
CN113393580B (zh) 一种地图道路绘制方法、装置及相关产品
CN114543313B (zh) 空调控制方法、服务器、空调及用户终端
CN109040968A (zh) 路况提醒方法、移动终端及计算机可读存储介质
CN115686264B (zh) 一种防误触方法及设备
CN110632773A (zh) 一种智能近视防控眼镜
CN105282409A (zh) 相机装置及相机装置控制方法
CN110197159B (zh) 指纹采集方法及终端
CN203694029U (zh) 一种超声测距盲人眼镜
CN113160260A (zh) 一种头眼双通道智能人机交互系统及运行方法
TW201303652A (zh) 三維輸入感測系統

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21918759

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21918759

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 15/01/2024)