CN114387535A

CN114387535A - Multi-mode identification system and blind person glasses

Info

Publication number: CN114387535A
Application number: CN202111487864.7A
Authority: CN
Inventors: 但雨芳; 陶剑文; 潘婕; 周亚峰; 史凯凯; 陈乾; 沈兴垚; 吴云凯; 程嘉威; 崔青雨
Original assignee: Ningbo Polytechnic
Current assignee: Ningbo Polytechnic
Priority date: 2021-12-08
Filing date: 2021-12-08
Publication date: 2022-04-22

Abstract

The invention belongs to the technical field of computer deep learning, and provides a multi-mode identification system and blind person glasses, which comprise: the obstacle avoidance module is used for acquiring point cloud information of the obstacles through the binocular, identifying the types of the obstacles through a 3D point cloud technology and acquiring position information of the obstacles, and then transmitting an obstacle avoidance decision scheme to the blind in a voice broadcast mode; the goods identification module is used for identifying the commodity labels on the supermarket shelves through an optical character recognition technology; the currency recognition module is used for recognizing the currency of each denomination through the trained classification model; and the photographing module is used for photographing people and objects in real time through the binocular head. The invention has the advantages that common scene modes in the life of the blind are integrated into a whole to form a multi-mode identification system, and the multi-mode identification system is applied to the glasses, thereby providing great convenience for the life and the trip of the blind.

Description

Multi-mode identification system and blind person glasses

Technical Field

The invention relates to the technical field of computer deep learning, in particular to a multi-mode recognition system and blind person glasses.

Background

The blind people are special vulnerable groups, and the living quality and the living condition of the blind people are key problems concerned by governments and society in China for a long time. According to the published data of the Chinese association of the blind people, as long as 2019, about 1700 blind people exist in China, and the blind people live in the world first, and the living and the working of the blind people become a serious social problem.

The blind people suffer from a lot of inconvenience in daily life due to physical defects. For example: crossing roads, going up and down steps, recognizing characters and the like.

Currently, the existing blind obstacle avoidance glasses on the market mainly comprise the following 3 types:

(1) the ultrasonic blind obstacle avoidance glasses obtain the distance between obstacles through ultrasonic reflection and convert the distance into an audio signal, and have the main defects that: is easily interfered by environmental factors such as temperature, wind direction, noise and the like;

(2) the blind-person obstacle-avoiding glasses have the main defects that data are acquired through the double cameras, information is processed through the plug-in equipment, and then water drop sound is used as prompt sound: the obtained data has narrow visual angle and unclear prompt tone;

(3) brain-computer interface blind person keeps away barrier glasses, it needs to carry out electrode sheetmetal implantation on the skull, realizes black and white formation of image through the electric pulse signal that obtains. The main defects are as follows: the operation is uncertain and expensive.

Disclosure of Invention

The invention aims to provide a multi-mode identification system and blind glasses, which are used for solving the problem that corresponding information cannot be identified under various scenes of a blind;

in order to achieve the purpose, the invention adopts the technical scheme that:

a multiple pattern recognition system comprising:

the obstacle avoidance module is used for acquiring point cloud information of the obstacles through the binocular, identifying the types of the obstacles through a 3D point cloud technology and acquiring position information of the obstacles, and then transmitting an obstacle avoidance decision scheme to the blind in a voice broadcast mode;

the goods identification module is used for identifying the commodity labels on the supermarket shelves through an optical character recognition technology;

the currency recognition module is used for recognizing the currency of each denomination through the trained classification model;

and the photographing module is used for photographing people and objects in real time through the binocular head.

Further, keep away barrier module includes:

the point cloud information acquisition unit is used for acquiring 3D point cloud information around the blind person through the binocular;

the system comprises a point cloud information processing unit, a blind person detection unit and a blind person detection unit, wherein the point cloud information processing unit is used for acquiring position information of an obstacle from 3D point cloud information around the blind person through a 3D point cloud technology and identifying the type of the obstacle, and the position information comprises azimuth information and distance information of the obstacle;

and the decision unit is used for generating a corresponding obstacle avoidance decision scheme according to the position information of the obstacle and transmitting the obstacle avoidance decision scheme to the blind in a voice broadcast mode.

Further, the step of identifying the obstacle category in the processed obstacle point cloud information through a 3D point cloud technology includes:

mapping 3D point cloud information around the blind person onto a two-dimensional image and rasterizing;

obtaining point cloud information of the obstacle from the rasterized image by using a clustering algorithm;

and identifying the obstacle according to the obtained obstacle point cloud information through a preset object identification model.

Further, the step of establishing the preset object recognition model includes:

acquiring image data of various obstacles in a preset mode, and establishing a corresponding data set;

carrying out data preprocessing on the established data set;

training a first preset object recognition model by the preprocessed data set;

transferring the parameters of the trained first preset object recognition model to a second preset object recognition model;

training a second preset object recognition model by the preprocessed data set in a training and freezing mode to obtain a trained object recognition model;

and performing model optimization on the trained object recognition model to obtain a preset object recognition model.

Further, the step of performing model optimization on the trained object recognition model includes:

finding the optimal parameters of the object identification model through a grid search algorithm;

removing redundant suggestion boxes input into the classifier by a non-maximum suppression algorithm;

and (3) attenuating the learning rate of the model by a cosine annealing algorithm to make the model tend to an optimal solution.

Further, the method for acquiring the position information of the barrier from the point cloud information around the blind by the 3D point cloud technology comprises the following steps: and acquiring the direction information and the distance information of the identified obstacle through the infrared structured light emitted by the binocular head.

Further, the step of identifying the currency of each denomination through the classification model comprises:

acquiring image data of the currency in a preset mode, and establishing a currency data set;

pre-processing the proposed currency data set, the pre-processing comprising tagging image data of the currency;

and training a preset classification model through the preprocessed currency data set to obtain a trained classification model.

The present invention also aims to provide a pair of glasses for the blind, comprising:

the frame glasses are provided with binocular lenses and a control chip electrically connected with the binocular lenses, and the control chip comprises the multi-mode identification system as claimed in claim 1;

the side of the frame glasses is provided with a plurality of buttons which are electrically connected with the control chip.

Compared with the prior art, the invention at least comprises the following beneficial effects:

(1) the invention integrates the common scene modes in the life of the blind, forms a multi-mode identification system and provides great convenience for the life and the trip of the blind.

(2) The method can identify corresponding information in multiple scenes, identify the type of the current obstacle through deep learning and obtain the position and distance data of the obstacle through infrared structured light detection sent by an infrared transmitter in an obstacle avoidance mode, then generate a corresponding obstacle avoidance decision scheme according to the information of the obstacle, and finally remind the obstacle information and the obstacle avoidance decision scheme around the blind through a voice broadcast scheme.

(3) The commodity label on the supermarket shelf is accurately identified in the supermarket mode, and information such as commodity names and prices can be accurately provided for the blind. In the photographing mode, people and objects around can be photographed and captured in real time. And the trained classification model can accurately identify the currency information corresponding to the face value, so that the blind can accurately distinguish the currencies with different face values when purchasing the currency information, and the risk of being cheated is reduced.

(4) The blind person can be provided with different types of help by switching the mode to different modes through the buttons under different scene modes.

(5) The multi-mode identification system is applied to the glasses for the blind, so that the blind can identify objects under different application scenes only by wearing special glasses, and great convenience is brought to the daily life of the blind.

Drawings

FIG. 1 is a block diagram of a multi-pattern recognition system of the present invention;

fig. 2 is a schematic frame diagram of an obstacle avoidance module according to an embodiment of the present invention;

FIG. 3 is a flow chart illustrating the identification of obstacle classes according to an embodiment of the present invention;

fig. 4 is a flowchart of establishing the preset object recognition model according to a first embodiment of the present invention;

FIG. 5 is a flowchart illustrating model optimization of a trained object recognition model according to a first embodiment of the present invention;

FIG. 6 is a flow chart illustrating the process of identifying currency for each denomination via a classification model according to one embodiment of the present invention.

Detailed Description

It should be noted that all the directional indicators (such as up, down, left, right, front, and rear … …) in the embodiment of the present invention are only used to explain the relative position relationship between the components, the movement situation, etc. in a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indicator is changed accordingly.

Moreover, descriptions of the present invention as relating to "first," "second," "a," etc. are for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicit ly indicating a number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

In the present invention, unless otherwise expressly stated or limited, the terms "connected," "secured," and the like are to be construed broadly, and for example, "secured" may be a fixed connection, a removable connection, or an integral part; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or they may be connected internally or in any other suitable relationship, unless expressly stated otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.

In addition, the technical solutions in the embodiments of the present invention may be combined with each other, but it must be based on the realization of those skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination of technical solutions should not be considered to exist, and is not within the protection scope of the present invention.

The following are specific embodiments of the present invention, and the technical solutions of the present invention will be further described with reference to the drawings, but the present invention is not limited to these embodiments.

Example one

As shown in fig. 1, the present invention provides a multi-pattern recognition system, comprising:

and the obstacle avoidance module is used for acquiring point cloud information of the obstacles through the binocular, identifying the types of the obstacles through a 3D point cloud technology, acquiring position information of the obstacles, and transmitting an obstacle avoidance decision scheme to the blind in a voice broadcast mode.

And the goods identification module is used for identifying the commodity labels on the supermarket shelves through an optical character recognition technology.

And the currency recognition module is used for recognizing the currency of each denomination through the trained classification model.

Specifically, as shown in fig. 2, the obstacle avoidance module includes:

in the obstacle avoidance state, the most important information given to the blind is the type and position of the obstacle and corresponding distance information, so that the method can accurately judge the corresponding type of the obstacle by combining a 3D point cloud technology with deep learning.

Specifically, as shown in fig. 3, the step of identifying the obstacle category in the processed obstacle point cloud information by using the 3D point cloud technology includes:

s1, mapping the 3D point cloud information around the blind person onto a two-dimensional image and rasterizing;

after the 3D point cloud information is acquired, the point cloud information is subjected to color-based visual segmentation and is divided into a plane area and a height area, wherein the plane area is used for grid map generation, and the height area is used for obstacle establishment.

S2, obtaining point cloud information of the obstacle from the rasterized image by using a clustering algorithm;

and deleting point cloud information which does not belong to the obstacle from the image through a K-means clustering algorithm, and determining the edge area of the obstacle.

And S3, recognizing the obstacle through the preset object recognition model on the obtained obstacle point cloud information.

And identifying the point cloud information of the obstacle through the trained deep learning network model so as to obtain the corresponding obstacle type.

As shown in fig. 4, the step of establishing the preset object recognition model includes:

a1, acquiring image data of various obstacles in a preset mode, and establishing a corresponding data set;

the invention obtains image data of various obstacles in a photographing and network downloading mode, and establishes corresponding data sets according to categories, wherein the data sets comprise a stair data set, a garbage can data set and the like.

A2, carrying out data preprocessing on the established data set;

the invention carries out a series of data preprocessing such as omnibearing screening, augmentation, enhancement, labeling and the like on the acquired image data, and provides accurate high-quality data preparation for model training.

A3, transferring the parameters of the pre-trained first preset object recognition model to a second preset object recognition model;

the first preset object identification model selected by the invention is a VGG16 convolutional neural network model, and can well extract the characteristics of the obstacles, so that a good foundation is established for subsequent obstacle identification. The second preset object recognition model selected by the invention is a YOLO-V4 TINY network model, the YOLO-V4 TINY network structure is a reduced version of YOLO-V4, the model belongs to a lightweight model, the parameters are only 600 thousands, which is equivalent to one tenth of the original parameters, and the detection speed is greatly improved.

The integral network structure of YOLO-V4 TINY has 38 layers, three residual error units are used, the activation function uses LeakyReLU, the classification and regression of the target are changed into two feature layers, and a Feature Pyramid (FPN) network is used when the effective feature layers are combined. The CSPnet structure is also used, the channel segmentation is carried out on the feature extraction network, the feature layer channel output after the convolution of 3x3 is divided into two parts, the second part is taken, and the YOLO-V4 TINY network is used, so that the response time of the obstacle avoidance method is greatly reduced on the obstacle, and the blind can quickly obtain corresponding obstacle information.

A4, training a second preset object recognition model by the preprocessed data set in a training and freezing mode to obtain a trained object recognition model;

according to the invention, through a knowledge migration mode, the same network parameters are obtained from a trained VGG16 convolutional neural network model and are applied to a YOLO-V4 TINY network model, and then only part of convolutional layers of the trained YOLO-V4 TINY network model need to be frozen to train untrained convolutional layers and full-connection layers, so that the training process is greatly reduced, the learning efficiency of the model is accelerated and optimized, and zero learning is not needed like most networks.

And A5, performing model optimization on the trained object recognition model to obtain a preset object recognition model.

Specifically, as shown in fig. 5, the step of performing model optimization on the trained object recognition model includes:

and A51, finding the optimal parameters of the object recognition model through a grid search algorithm.

The grid search algorithm is an exhaustive search method for specified parameter values, the optimal learning algorithm is obtained by optimizing parameters of an estimation function through a cross validation method, the used parameters which are required to be changed in a model are manually given, the algorithm automatically runs the used parameters through an exhaustive method, the optimal parameters of a network are found, and the process of manually and continuously adjusting the parameters is reduced.

A52, removing redundant suggestion boxes input into the classifier by a non-maximum suppression algorithm.

In the process of detecting the obstacle target, a large number of candidate frames are generated at the same target position, and the candidate frames may overlap with each other, and at this time, the optimal target suggestion frame needs to be found by using non-maximum suppression, and redundant suggestion frames are eliminated, so that subsequent obstacle classification is performed better.

And A53, attenuating the learning rate of the model by a cosine annealing algorithm to lead the model to approach the optimal solution.

In the traditional training process, the learning rate is gradually reduced, so that the model gradually finds a local optimal point. In this process, because the learning rate is initially high, the model does not step into a steep local optimum point, but moves quickly to a flat local optimum point. As the learning rate gradually decreases, the model eventually converges to a better optimal point.

Since the learning rate of cosine annealing drops rapidly, the model quickly steps into a local optimum point (whether steep or not) and the model of the local optimum point is saved. And after the model is saved, the learning rate is restored to a larger value again, the current local optimal point is escaped, and a new optimal point is searched. Because the models with different local optimal points have larger diversity, the overall effect of the models can be improved, and the training cost is not increased.

After the type of the obstacle is identified, the position and distance information of the obstacle is measured, and the method for acquiring the position information of the obstacle from the point cloud information around the blind by the 3D point cloud technology comprises the following steps: and acquiring the direction information and the distance information of the identified obstacle through the infrared structured light emitted by the binocular head.

The method can identify corresponding information in multiple scenes, identify the type of the current obstacle through deep learning and obtain the position and distance data of the obstacle through infrared structured light detection sent by an infrared transmitter in an obstacle avoidance mode, then generate a corresponding obstacle avoidance decision scheme according to the information of the obstacle, and finally remind the obstacle information and the obstacle avoidance decision scheme around the blind through a voice broadcast scheme.

The invention also has a currency mode, a supermarket mode and a photographing mode while providing an obstacle avoidance mode.

Under a supermarket mode, the commodity labels on the supermarket shelf can be accurately identified through an optical character recognition technology (OCR technology), and information such as commodity names and prices can be accurately provided for the blind.

In the currency mode, as shown in fig. 6, the step of identifying the currency of each denomination through the classification model includes:

b1, acquiring image data of the currency in a preset mode, and establishing a currency data set;

b2, preprocessing the suggested currency data set, wherein the preprocessing comprises marking the image data of the currency;

and B3, training a preset classification model through the preprocessed currency data set to obtain a trained classification model.

B4, the trained classification model can accurately identify the currency information corresponding to the face value, thereby helping the blind to accurately distinguish the currencies with different face values when purchasing, and reducing the risk of being cheated.

In the photographing mode, people and objects around can be photographed and captured in real time.

The invention integrates the common scene modes in the life of the blind, forms a multi-mode identification system and provides great convenience for the life and the trip of the blind.

Example two

The invention provides a pair of glasses for the blind, which comprises:

the frame glasses are provided with binocular lenses and control chips electrically connected with the binocular lenses, and the control chips comprise the multi-mode identification system. The side of the frame glasses is provided with a plurality of buttons which are electrically connected with the control chip.

Each button represents a mode, and the mode is switched to different modes through the buttons under different scene modes, so that different types of help are provided for the blind.

The multi-mode identification system is applied to the glasses for the blind, so that the blind can identify objects under different application scenes only by wearing special glasses, and great convenience is brought to the daily life of the blind.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.

Claims

1. A multimodal identification system, comprising:

2. The system of claim 1, wherein the obstacle avoidance module comprises:

3. The multimodal recognition system of claim 2, wherein the step of recognizing the obstacle category in the processed obstacle point cloud information by means of a 3D point cloud technique comprises:

4. A multimodal identification system as claimed in claim 3, wherein the step of establishing the predetermined object recognition model comprises:

carrying out data preprocessing on the established data set;

training a first preset object recognition model by the preprocessed data set;

5. The system of claim 4, wherein the step of model optimizing the trained object recognition model comprises:

6. The multi-mode recognition system of claim 2, wherein the method for obtaining the position information of the obstacle from the point cloud information around the blind by the 3D point cloud technology comprises: and acquiring the direction information and the distance information of the identified obstacle through the infrared structured light emitted by the binocular head.

7. A multimodal identification system as claimed in claim 1 wherein the step of identifying the currency of each denomination by a classification model comprises:

8. Glasses for the blind, characterized in that it comprises: