KR102291296B1 - Walking assistance system for the visually-impaired person - Google Patents

Abstract

Disclosed is a walking assistance system for visually impaired persons. According to the present invention, the walking assistance system for visually impaired persons comprises a user terminal performing walking assistance for a visually impaired person and an earphone outputting walking assistance performed in the user terminal as voice. The user terminal is configured to divide and recognize objects including roads and obstacles by a semantic segmentation algorithm and analyze the recognized objects in real-time by an object detection algorithm to detect the obstacles. Accordingly, the depth of an image is recognized by using a visual difference of a dual camera image and the image for each pixel is analyzed by using the semantic segmentation algorithm and the object detection algorithm, thereby providing an effect of distinguishing a sidewalk from a road and accurately detecting obstacles. In addition, the types of obstacles are specified and increased by the object detection algorithm, thereby providing an effect of accurately recognizing the obstacles and perfectly assisting the visually impaired person in walking. In addition, learning is performed on a walkable area by using images acquired by using smartphones of a plurality of visually impaired persons and stored street view images, thereby providing an effect of constructing accurate data on the walkable area for each location.

Description

WALKING ASSISTANCE SYSTEM FOR THE VISUALLY-IMPAIRED PERSON

The present invention relates to a walking assistance system, and more particularly, to a walking assistance system for the visually impaired.

Although various means for the walking safety of the visually impaired have been developed, there is no safe means for assisting the visually impaired to walk.

There is a need for a method in which blind people can easily use walking aids, but it is difficult to travel long distances alone without a guide dog or a helper.

In addition to detecting and responding to suddenly approaching obstacles, it is also necessary to distinguish whether the object on the image is an obstacle or a sidewalk, and to accurately identify the walkable area.

As such, the development of safer and more accurate means for the visually impaired is required.

Laid-Open Patent Publication No. 10-2021-0014405 Registered Patent Publication No. 10-1929875

An object of the present invention is to provide a walking assistance system for the visually impaired.

According to an aspect of the present invention described above, a system for assisting the visually impaired walking includes: a user terminal for performing walking assistance for the visually impaired; It may be configured to include an earphone for outputting a voice for walking assistance performed in the user terminal.

Here, the user terminal divides and recognizes objects including roads and obstacles by a semantic segmentation algorithm, and analyzes the recognized objects in real time by an object detection algorithm to detect obstacles. can be configured.

And it may be configured to further include a walking assistance guide device attached to the body or belongings of the visually impaired to output a road guide associated with the walking assistance performed by the user terminal.

Meanwhile, the user terminal may be configured to perform walking assistance by analyzing an obstacle position using a dual camera module.

According to the above-described walking assistance system for the visually impaired, the depth of the image is determined using the visual difference of the dual camera image, and the image is analyzed for each pixel using a semantic segmentation algorithm and an object detection algorithm. This has the effect of distinguishing the sidewalk from the road and accurately detecting obstacles. In addition, by specifying and increasing the types of obstacles by the object detection algorithm, there is an effect of accurately recognizing obstacles and perfectly assisting the visually impaired.

In addition, by performing learning on the walkable area using images using smart phones of various visually impaired people and street view images secured in advance, it has the effect of enabling more accurate data construction on the walkable area for each location. .

1 is a block diagram of a walking assistance system for the visually impaired according to an embodiment of the present invention.
2 is a schematic diagram of a semantic segmentation algorithm according to an embodiment of the present invention.
3 is a schematic diagram of an object detection algorithm according to an embodiment of the present invention.
4 is an exemplary diagram of an image representing an obstacle detection process according to an embodiment of the present invention.
5 is an exemplary diagram of an obstacle distance detection algorithm using parallax of dual camera images according to an embodiment of the present invention.
6 is an exemplary view of an image illustrating a process of calculating a distance to an obstacle according to an embodiment of the present invention.
7 is a schematic diagram of a process of outputting as text by applying a weight to a distance to an obstacle according to an embodiment of the present invention.
8 is an exemplary diagram of a deep learning image and a depth image according to an embodiment of the present invention.
9 is an exemplary view of an image showing a complementary process of deep learning and depth analysis according to an embodiment of the present invention.
10 is an exemplary diagram of an image of a walkable area according to an embodiment of the present invention.
11 is an exemplary view of an image showing a proximity obstacle according to an embodiment of the present invention.
12 is a diagram illustrating an auxiliary battery connected to a user terminal.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed content for carrying out the invention. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a walking assistance system for the visually impaired according to an embodiment of the present invention, FIG. 2 is a schematic diagram of a semantic segmentation algorithm according to an embodiment of the present invention, and FIG. 3 is a diagram of the present invention It is a schematic diagram of an object detection algorithm according to an embodiment, FIG. 4 is an exemplary view of an image showing an obstacle detection process according to an embodiment of the present invention, and FIG. 5 is a dual camera according to an embodiment of the present invention It is an exemplary diagram of an obstacle distance detection algorithm using parallax of an image, FIG. 6 is an exemplary view of an image showing a process for calculating a distance to an obstacle according to an embodiment of the present invention, and FIG. 7 is an embodiment of the present invention It is a schematic diagram of a process of outputting as text by applying a weight to the distance to an obstacle according to , FIG. 8 is an exemplary view of a deep learning image and a depth image according to an embodiment of the present invention, and FIG. 9 is an embodiment of the present invention It is an exemplary view of an image showing a complementary process of deep learning and depth analysis according to an example, FIG. 10 is an exemplary view of an image for a walkable area according to an embodiment of the present invention, and FIG. 11 is an embodiment of the present invention It is an exemplary view of an image representing a proximity obstacle according to an example.

First, referring to FIG. 1 , the walking assistance system for the visually impaired according to an embodiment of the present invention includes a user terminal 100 , a walking assistance learning server 200 , a cane attachment guide device 300 , and a wheelchair attachment guide device 400 . , may be configured to include an auxiliary battery (500).

Hereinafter, a detailed configuration will be described.

The user terminal 100 may be configured to assist the visually impaired in walking.

The user terminal 100 includes a GPS module 101, a touch screen 102, a current location notification module 103, a dual camera module 104, a semantic segmentation object detection application module 105, a cell differentiation module for each frame 106 ), obstacle detection module 107, obstacle distance calculation module 108, cell location-based weighting module 109, pixel depth calculation module 110, pixel depth normalization module 111, pixel depth threshold contrast module 112 ), walkable area calculation module 113, text output module 114, continuous frame proximity obstacle detection module 115, proximity obstacle notification module 116, destination voice input module 117, voice navigation module 118 ), a three-axis sensor 119 , a posture determination module 120 , a horizontal fixation notification module 121 , and a real-time reference module 122 for a walking path by location.

Hereinafter, a detailed configuration will be described.

The GPS module 101 may be configured to receive a GPS signal and calculate current location information.

The touch screen 102 may be configured to output a moving image of the front. Here, the moving image is an image obtained by synthesizing the dual image generated by the dual camera module 104 .

The current location notification module 103 may be configured to immediately voice output the current location information calculated by the GPS module 101 when the touch screen 102 is touched by the visually impaired. When a touch is input even while guiding a route, it may be configured to immediately output current location information by voice. Directional guidance may be performed in a forward, backward, left, and right direction, or may be made in a direction for each time of the clock. For example, it may be a navigation guide, such as "Go 5m in the 3 o'clock direction."

The dual camera module 104 may be configured to generate a dual moving image of the front. Since the visually impaired must find a way while photographing the front using the user terminal 100, it may be configured to be attached to the shoulder or the like without lifting it. An attachment auxiliary device (not shown) for attachment of the user terminal 100 may be further used.

The semantic segmentation object detection application module 105 is configured to output a moving image by performing an analysis for a semantic segmentation algorithm and object detection on a stereo type dual moving image image generated by the dual camera module 104 can be FIG. 2 shows that an object is detected by applying a semantic label by performing a semantic segmentation algorithm. And referring to FIG. 3 , it shows that the semantic segmentation object detection application module 105 performs an object detection algorithm to additionally detect an object.

The frame-by-frame cell differentiation module 106 may be configured to differentiate by designating a cell for each frame of the moving image output by the semantic segmentation object detection application module 105 . For example, by designating 32 cells in each frame, the frames can be divided.

The obstacle detection module 107 may be configured to detect an obstacle in each frame in which a cell is differentiated by the frame-by-frame cell differentiation module 106 . Based on each cell, it is possible to determine where the obstacle is located. Referring to FIG. 4 , the purple region is tentatively identified as a gaitable region, and portions marked with dots are recognized as the location of a nearby obstacle during walking.

The obstacle distance calculation module 108 may be configured to calculate a distance to the obstacle when the obstacle is detected by the obstacle detection module 107 . In FIG. 5 , the distance on the image is calculated using the parallax, that is, the time difference between the dual images in the dual camera module 104 that generates the stereo type dual image, and a calibration technique is applied to compare the actual distance on the image. It may be configured to calculate a distance.

The cell location-based weight application module 109 may be configured to apply a weight based on a cell location of a cell in which an obstacle is located when the distance is calculated by the obstacle distance calculation module 108 . It may be configured to determine where the cell in which the obstacle is located is located in the entire screen, and to apply a higher weight if it is located closer to the center of the screen.

The pixel depth calculation module 110 may be configured to calculate a pixel depth of each pixel by using the time difference between the dual video images generated by the dual camera module 104 .

The pixel depth normalization module 111 may be configured to normalize the pixel depth calculated by the pixel depth calculation module 110 .

The pixel depth threshold contrast module 112 may be configured to compare the pixel depth normalized in the pixel depth normalization module 111 with a predetermined threshold.

The walkable area calculation module 113 is configured to calculate the walkable area in each frame of the current moving image image by using the obstacle detected by the obstacle detection module 107 and the distance to the obstacle calculated by the obstacle distance calculating module 108. can be configured. 8 shows an image analyzed using a semantic segmentation algorithm, an object detection algorithm, and a pixel depth, and FIG. 9 shows an image on which an obstacle analysis is performed by the semantic segmentation algorithm and object detection.

In addition, when the pixel depth of each cell is greater than the threshold as a result of the comparison of the pixel depth threshold contrast module 112, the walkable area calculation module 113 processes the corresponding pixel as a blocked cell to calculate the walkable area. can be configured. 10 illustrates an image displayed by calculating a walkable area.

The text output module 114 may be configured to output the walkable area calculated by the walkable area calculation module 113 as text. 7 shows an example of detecting an obstacle and a walkable area and outputting it as text.

The continuous frame proximity obstacle detection module 115 may be configured to determine whether the obstacle detected by the obstacle detection module 107 is continuous for a predetermined number of frames or more of the moving image image. 11 illustrates an image in which a proximity obstacle is detected.

As a result of the above determination, if more than a predetermined number of frames are continuous, the continuous frame proximity obstacle detection module 115 may be configured to detect the obstacle as a proximity obstacle.

For example, when a specific obstacle is continuous for 5 frames or more, and if the pixel area of the obstacle becomes larger, it may be detected as approaching the visually impaired.

The proximity obstacle notification module 116 may be configured to immediately output a notification of the proximity of the obstacle when the continuous frame proximity obstacle detection module 115 detects the proximity obstacle. It is possible to output a notification of the proximity of obstacles preferentially during route guidance.

The destination voice input module 117 may be configured to receive a voice input of the walking destination of the visually impaired.

The voice guidance module 118 uses the current location information calculated by the GPS module 101 and the text output from the text output module 114 to guide the way to the walking destination received voice input from the destination voice input module 117 . It may be configured to output a voice by performing .

The three-axis sensor 119 may be configured to detect the posture of the user terminal 100 .

The posture determination module 120 may be configured to determine whether the posture detected by the three-axis sensor 119 is maintained in the horizontal direction. When the user terminal 100 is maintained in the horizontal direction, it is possible to broadly grasp the surrounding obstacles or the walkable area using the dual camera module 104 . In the case of the vertical direction, it is difficult to detect surrounding obstacles and the dual camera module 104 mainly captures a large number of sky areas, so it is preferable to maintain the horizontal direction.

As a result of determining whether the posture is maintained in the horizontal direction in the posture determination module 120, the horizontal fix notification module 121 may be configured to immediately output a voice notification to maintain the posture in the horizontal direction if it is determined that the posture is not maintained in the horizontal direction. . That is, when the horizontal direction is not maintained beyond a predetermined threshold angle, a voice notification may be immediately output in preference to a voice guidance voice to maintain the horizontal direction.

The location-specific walking path real-time reference module 122 may be configured to support the walkable area calculation of the walkable area calculation module 113 by referring to the location-specific walking path-related data from the walking assistance learning server 200 in real time. Pedestrian path-related data by location includes image data collected from the user terminal 100 of various visually impaired people in the walking assistance learning server 200 and the walking area and obstacles that are more closely identified through learning of the previously secured Street View image. It may be configured to include data such as

The gait assistance learning server 200 may be configured to perform gait learning in relation to the gait assistance performed by the user terminal 100 .

The walking assistance learning server 200 includes a current location receiving module 201, an image related data receiving module 202, an image related data storage module 203, a street view database 204, a walking path learning module 205, a location It may be configured to include a star trail database 206 .

Hereinafter, a detailed configuration will be described.

The current location receiving module 201 may be configured to receive the current location information calculated by the GPS module 101 in real time.

The image related data receiving module 202 may be configured to receive the walkable area image related data from the walkable area calculation module 113 in real time.

The image related data storage module 203 may be configured to store the walkable area image related data received in real time by the image related data receiving module 202 .

The street view database 204 may be configured to pre-store a street view image related to the current location information received in real time from the current location receiving module 201 . For example, it may be a street view image obtained in advance, such as a street view image of Daum company.

The walking path learning module 205 includes data related to the walkable area image stored in the image related data storage module 203 and the current received in real time from the current location receiving module 201 among the street view images pre-stored in the street view database 204 . It may be configured to learn a corresponding walking path by referring to a street view image related to location information to generate walking path related data for each location. That is, by comparing and learning the walkable area image and the street view image collected from the user terminal 100 of various visually impaired people, it is possible to generate more accurate and detailed walking path-related data for each location.

The location-specific walking path database 206 may be configured to store location-specific walking path related data generated by the walking path learning module 205 .

The stick attachment guide device 300 may be attached to the cane of the visually impaired and may be configured to output a road guide associated with the walking assistance performed by the user terminal 100 .

Vibration motors may be respectively attached to the front, rear, left, and right regions of the handle portion to the stick attachment guide device 300 , and the direction of the front, rear, left, and right may be guided through vibration.

The wheelchair attachment guide apparatus 400 may be attached to a wheelchair of a visually impaired person and may be configured to output a road guide associated with walking assistance performed by the user terminal 100 .

Vibration motors for guiding forward, backward, left and right directions may be attached to both armrests of the wheelchair, and navigation may be made using this.

In addition to the cane attachment guide device 300 and the wheelchair attachment guide device 400, various auxiliary guide devices such as earphones may be further provided.

The auxiliary battery 500 may be configured to provide auxiliary power to the user terminal 100 . The battery consumption of the user terminal 100 may be high. And if the battery runs out while navigating, it can be very frustrating. Accordingly, it may be necessary to use the auxiliary battery 500 .

12 is a diagram illustrating an auxiliary battery connected to a user terminal.

The auxiliary battery 500 may be configured to include a battery amount detection/determination module 501 , a heat generating module 502 , and a scent diffusing module 503 .

Hereinafter, a detailed configuration will be described.

The battery amount detection/determination module 501 may be configured to detect an auxiliary battery amount in real time and compare it with a predetermined step-by-step threshold.

The heat generating module 502 may be configured to generate heat in a corresponding step when the auxiliary battery amount is determined to be less than or equal to a predetermined step threshold as a result of the determination of the battery amount detecting/determining module 501 . The lower the step, the higher the calorific value.

The heating module 502 is preferably provided at an end of the auxiliary battery 500 .

The scent diffusing module 503 may be configured to emit scent in a corresponding step when the auxiliary battery amount is determined to be less than or equal to a threshold value for each step as a result of the determination of the battery amount detecting/determining module 501 . The lower the level, the more scent diffusing may be configured.

Alternatively, as shown, the amount of the battery may be olfactorily recognized by using the injection unit 533 connected to each of a plurality of hyangaek dispensers (not shown) provided therein so that different scents are scented for each rental value for each predetermined step.

Through such heat and smell, the visually impaired can sense the remaining battery level and can cope with it.

Although not shown, it is preferable that the auxiliary battery 500 is provided with a neck strap so that it can be hung around the user's neck in a state in which the heating module 503 is disposed upward.

In this case, the neck strap is formed in a form coupled to the auxiliary battery 500 with a material such as wire, and a vibrating part is provided in the coupled part, so that the vibration is transmitted to the neck strap through the vibration of the vibrating part depending on the situation. You can also indicate the remaining amount.

Although described with reference to the above embodiments, those skilled in the art can understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the following claims. There will be.

100: user terminal
101: GPS module
102: touch screen
103: current location notification module
104: dual camera module
105: Semantic segmentation object detection application module
106: frame-by-frame cell differentiation module
107: obstacle detection module
108: obstacle distance calculation module
109: cell location-based weighting module
110: pixel depth calculation module
111: pixel depth normalization module
112: pixel depth threshold contrast module
113: walkable area calculation module
114: text output module
115: continuous frame proximity obstacle detection module
116: proximity obstacle notification module
117: destination voice input module
118: voice guidance module
119: 3-axis sensor
120: posture determination module
121: horizontal fixed notification module
122: real-time reference module for walking path by location
200: walking aid learning server
201: current location receiving module
202: image-related data receiving module
203: image related data storage module
204: Street View Database
205: walking path learning module
206: Pedestrian path database by location
300: stick attachment guide device
400: wheelchair attachment guide device
500: auxiliary battery
501: battery level detection/judgment module
502: heat module
503: scent diffusing module

Claims (3)

A user terminal for performing walking assistance for the visually impaired;
an earphone for outputting a voice for walking assistance performed by the user terminal;
a gait assistance learning server for performing gait learning in relation to gait assistance performed by the user terminal;
Comprising an auxiliary battery for providing auxiliary power to the user terminal,
The user terminal is
a GPS module for receiving a GPS signal and calculating current location information;
a touch screen that outputs a moving image of the front;
a current location notification module for immediately outputting the current location information calculated by the GPS module by voice when the touch screen is touched by the visually impaired;
a dual camera module for generating a dual moving image of the front stereo type;
a semantic segmentation algorithm and an analysis for object detection on the stereo type dual video image generated by the dual camera module, and a semantic segmentation object detection application module for outputting a moving image;
a frame-by-frame cell differentiation module for specifying and differentiating cells for each frame of the moving image output by the semantic segmentation object detection application module;
an obstacle detection module for detecting an obstacle in each frame in which cells are differentiated by the frame-by-frame cell differentiation module;
an obstacle distance calculation module for calculating a distance to the obstacle when the obstacle detection module detects an obstacle;
When the distance is calculated by the obstacle distance calculation module, a weight is applied based on the cell position of the cell in which the obstacle is located, and the position of the obstacle on the screen is determined by determining where the cell in which the obstacle is located is located on the entire screen. a cell location-based weighting module that applies a higher weight as the cell location of the cell is closer to the center of the screen;
a pixel depth calculation module for calculating a pixel depth of each pixel using the time difference of the dual moving image generated by the dual camera module;
a pixel depth normalization module for normalizing the pixel depth calculated by the pixel depth calculation module;
a pixel depth threshold comparison module for comparing the pixel depth normalized in the pixel depth normalization module with a predetermined threshold;
By using the obstacle detected by the obstacle detection module and the distance to the obstacle calculated by the obstacle distance calculating module, a walkable area is calculated in each frame of the current moving image image, and the pixel depth threshold comparison module compares the result of each cell. a walkable area calculation module for calculating a walkable area by processing the pixel as a blocked cell when the pixel depth is greater than the threshold;
a text output module for outputting the walkable area calculated by the walkable area calculation module as text;
It is determined whether the obstacle detected by the obstacle detection module is continuous for more than a predetermined number of frames of the moving image image, and as a result of the determination, the obstacle is detected as a proximity obstacle continuous frame proximity obstacle detection module;
a proximity obstacle notification module configured to immediately notify and output an obstacle proximity when the continuous frame proximity obstacle detection module detects a proximity obstacle;
a destination voice input module for receiving a voice input of the walking destination of the visually impaired;
a voice guidance module for performing voice guidance to a walking destination received voice input from the destination voice input module by using the current location information calculated by the GPS module and the text output from the text output module;
a 3-axis sensor for detecting the posture of the user terminal;
a posture determination module for determining whether the posture sensed by the three-axis sensor is maintained in a horizontal direction;
a horizontal fix notification module for immediately outputting a voice notification to maintain the posture in the horizontal direction when it is determined that the posture is not maintained in the horizontal direction as a result of the posture determination module determining whether the posture is maintained in the horizontal direction;
It is configured to include a real-time reference module for each location that supports the calculation of the walkable area of the walkable area calculation module by referring to the data related to the walking path by location from the walking assistance learning server in real time,
The walking assistance learning server,
a current location receiving module configured to receive the current location information calculated by the GPS module in real time;
an image-related data receiving module configured to receive, in real time, image-related data about a walkable area from the walkable area calculation module;
an image-related data storage module for storing the image-related data of the walkable area received in real time from the image-related data receiving module;
a street view database in which a street view image related to the current location information received in real time from the current location receiving module is stored in advance;
The corresponding walking path is determined by referring to the street view image related to the current location information received in real time from the current location receiving module among the walkable area image related data stored in the image related data storage module and the street view images stored in advance in the street view database. a walking path learning module that learns to generate walking path-related data for each location;
It is configured to include a walking path database for each location in which data related to the walking path for each location generated by the walking path learning module is stored,
The auxiliary battery is
a battery amount detection/determination module that detects the auxiliary battery amount in real time and compares it with a predetermined step-by-step threshold;
a heating module configured to generate heat for each step when the auxiliary battery amount is determined to be less than or equal to a threshold value for each step as a result of the determination of the battery amount detection/determination module;
and a scent diffusing module for diffusing scent for each step when the auxiliary battery amount is determined to be less than or equal to a threshold value for each step as a result of the determination of the battery amount detection/determination module.
delete delete

Images