WO2016047890A1

WO2016047890A1 - Walking assistance method and system, and recording medium for performing same

Info

Publication number: WO2016047890A1
Application number: PCT/KR2015/005982
Authority: WO
Inventors: 한영준; 한헌수; 린칭; 김민수; 정환익
Original assignee: 숭실대학교산학협력단
Priority date: 2014-09-26
Filing date: 2015-06-15
Publication date: 2016-03-31

Abstract

A walking assistance method and system, and a recording medium for performing the same are disclosed. The walking assistance method comprises: generating a laser profile including depth information for each point of an object which is in front of a user; extracting edges from an image acquired using a camera so as to generate an edge profile including edge information on the image; generating a multi-modal profile by merging the generated laser profile and the generated edge profile; detecting and classifying obstacles located in a short distance using the generated multi-modal profile; predicting a movement of the user using the multi-modal profile; recognizing a walking environment of the user using information on the detected and classified obstacles and information on the movement of the user; and providing a walking assistance guide service to the user according to the recognized walking environment.

Description

Walking assistance method and system, recording medium for performing the same

The present invention relates to a walking assistance method and system, and a recording medium for performing the same. More particularly, the present invention relates to a method for detecting a obstacle by combining a depth information obtained by using a laser sensor and edge information of an image obtained by using a camera. The present invention relates to a walking assistance method and system for classifying, detecting and classifying obstacles, and providing walking assistance guide information according to a situation of a current walking environment, and a recording medium for performing the same.

In general, the walking assistance method that helps the user who is visually impaired to walk detects an object in front of the user by using a laser scanner sensor and measures the distance to the detected object and provides the user to the user, or uses the camera in front of the user. Recognizes an existing object and measures the distance to the recognized object and provides it to the user.

However, in the conventional walking assistance method, the laser scanner sensor may have a high straightness and may miss a dangerous obstacle, and may not provide an outline of the obstacle because only one-dimensional distance information may be measured. Accordingly, there is a disadvantage in that accurate walking assistance information cannot be provided to the user. In addition, the camera may incorrectly recognize the shadow as an obstacle, and when detecting an obstacle in direct sunlight, an object reflecting light, a strong light source in the rear, or a low light environment, direct sunlight, an object reflecting light, or a strong rear light Due to the influence of the light source or the low light environment, the image is distorted, and obstacles cannot be detected accurately. Accordingly, there is a disadvantage in that accurate walking assistance information cannot be provided to the user.

Therefore, the obstacle information at a short distance is detected and classified through the data generated by fusing the distance information acquired by using the laser sensor and the image information acquired by the camera, the information on the detected and classified obstacle, and the walking state of the user. According to this situation, a walking assistance method for providing information on a direction and a situation in which the user proceeds is required.

In one aspect of the present invention, a laser profile including depth information and an edge profile including edge information of an image are fused to generate a multi-modal profile, and a multi-modal profile is used to detect and classify obstacles in a short distance. A walking assistance method and system for detecting and classifying obstacles in a remote location using an edge profile, and providing walking assistance information required by a user according to the walking state of the user and information on the detected and classified obstacles, and for performing the same. Provide a record carrier.

According to an aspect of the present invention, a walk assistance method includes generating a laser profile model including depth information obtained by using a laser sensor, and generating an edge profile model including edge information extracted from an image obtained by using an image sensor. Generate, generate a multimodal profile model by fusing the laser profile model and the edge profile model, detect and classify an obstacle present at a short distance from the multimodal profile model, and an obstacle present at a distance from the edge profile model. Detects and classifies the symbol, recognizes the situation in front of the user who is assisted with the walking by using the information on the detected and classified obstacle, and provides information on the direction and the situation to proceed according to the situation information in front of the user.

Generating the laser profile model may classify the laser profile according to whether the values of the depth information obtained by using the depth information are similar to distinguish the laser profile of the ground from the laser profile of the obstacle.

Categorizing the laser profile according to similarity of the obtained depth information may include calculating a difference value between the obtained depth information values, and if the calculated difference value is less than a predetermined reference value, The laser profile having the value may be classified into the same laser profile, and if the calculated difference value is equal to or greater than a predetermined reference value, the laser profile having the value of the obtained depth information may be classified into a different laser profile.

The generating of the multimodal profile model may generate the multimodal profile model by matching a coordinate point of depth information included in the laser profile model with a corresponding point of an image included in the edge profile model.

Detecting and classifying obstacles that exist at a short distance from the multi-modal profile model may include determining whether there is an edge profile connected to the peripheral eight directions of the point where the laser profile model and the edge profile model match when generating the multi-modal profile model. If the detected edge profile exists, vertical and horizontal histograms may be calculated in the region where the edge profile is detected to detect and classify obstacles existing in the near distance.

Categorizing the obstacle present in the near field may include calculating a width of the obstacle through a horizontal histogram of the laser profile model, calculating a height of the obstacle through a vertical histogram of the edge profile model, The obstacle may be classified by using the height to infer the size and shape of the obstacle.

Detecting and classifying obstacles that exist at a distance from the edge profile model includes extracting a vertical edge component from the acquired image, top-view transforming the extracted image of the vertical edge component, and performing a morphology calculation on the top-view transformed image. The labeling may be performed to classify the edge blob which is the obstacle region, obtain an ellipse from the edge blob through elliptic approximation, and calculate the direction of the ellipse to classify the obstacle according to the direction of the ellipse.

Recognizing the situation in front of the user who is assisted with the walking assistance using the information on the detected and classified obstacles, and providing information on the direction and the situation to proceed according to the situation information in front of the user, the optical flow to the obtained image A vector and an extended Kalman filter are applied to predict the movement of the user, and recognize the situation of the walking environment including the walking state and the walking direction of the user from the movement of the user and information about the detected and classified obstacles. According to the situation of the pedestrian environment, it is possible to provide information on the direction and the situation to proceed.

Providing information about the direction and the situation to proceed according to the situation of the walking environment, searching for a plurality of words corresponding to the situation of the walking environment of the user according to the situation of the walking environment of the user, By combining words, a sentence having a predetermined pattern may be generated, and a guide message including the generated sentence may be output.

The computer program may be a computer readable recording medium having recorded thereon a computer program for providing information to assist a user walking.

According to another aspect of the present invention, a walk assistance system includes a laser scanner that detects depth information of a space located in front of the camera, a camera that photographs the space located in front of the camera, and a laser profile model using depth information detected by the laser scanner. Generate an edge profile, extract edge information from an image captured by the camera, generate an edge profile model including the edge information, fuse the laser profile and the edge profile model, and generate a multimodal profile model; Detect and classify obstacles located in front of the multi-modal profile model, and detect and classify obstacles that exist in the distance from the edge profile model, and use the information on the detected and classified obstacles in front of the user who is assisted with walking. Recognize the situation before the user Depending on the situation, and a walking guide apparatus that provides information on the walking direction and the situation.

According to one aspect of the present invention, by using the multi-modal profile fusion of the laser profile obtained through the laser scanner and the edge profile obtained through the camera by detecting the obstacle located at a short distance of the obstacle as well as the position information of the obstacle It is possible to obtain the shape information can provide a more accurate walk assistance service.

In addition, as well as providing information to the obstacle located in front of the user, it is possible to provide information about the optimal walking direction required by the user by predicting the user's movement using a multi-modal profile.

1 is a diagram illustrating a walking assistance system according to an exemplary embodiment of the present invention.

FIG. 2 is a block diagram of the walking guide apparatus shown in FIG. 1.

3A and 3B illustrate an example of the multi-modal profile generator shown in FIG. 2.

4 is a diagram illustrating a method of detecting an obstacle using a multi-modal profile.

5A, 5B, 5C, 5D, and 5E are diagrams for explaining a method for detecting a remote obstacle.

6 is a diagram illustrating an example of an obstacle classification unit that classifies obstacles in a short distance.

7 is a diagram illustrating a method of elliptical approximation of an edge blob.

8A and 8B are diagrams for explaining that the shape of an obstacle at a distance when the top view is changed.

9 is a diagram illustrating a method of calculating the directionality of an ellipse.

FIG. 10 is a diagram illustrating an example of an obstacle classification unit classifying obstacles at a distance.

11 is a flowchart illustrating a control method of a walking assistance system according to an exemplary embodiment of the present invention.

12 is a flowchart illustrating a control method of a walking assistance system according to an exemplary embodiment of the present invention.

1: walk assistance system

100: laser scanner

200: camera

300: walking guidance device

DETAILED DESCRIPTION The following detailed description of the invention refers to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be embodied in other embodiments without departing from the spirit and scope of the invention in connection with one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

1 is a view showing a walking assistance system according to an embodiment of the present invention, Figure 2 is a block diagram of the walking guidance device shown in Figure 1, Figures 3a, 3b is a multi-modal profile generation shown in FIG. 4 is a diagram illustrating an example of a part, and FIG. 4 is a diagram illustrating a method of detecting an obstacle using a multi-modal profile.

The walking assistance system 1 according to an embodiment of the present invention detects and classifies obstacles in front of the user who is provided with the walking assistance guide service of the walking assistance system 1, and predicts the movement of the user to be optimal for the user. Can provide walking assistance services.

Referring to FIG. 1, a walking assistance system 1 according to an exemplary embodiment may include a laser scanner 100, a camera 200, and a walking guide device 300.

The laser scanner 100 may detect distance information on an object (eg, a ground or an obstacle) in front of the user by transmitting a laser in front of the user, that is, in front of the user. .

Specifically, the laser scanner 100 may transmit a single laser pulse in front of the user. The laser scanner 100 may detect an object in front of the user by receiving a laser returning the reflected laser pulse reflected on the surface of the object present within the sensor range of the laser scanner 100. In this case, the laser scanner 100 may calculate a distance from the laser scanner 100 to an obstacle by measuring a time when the transmitted laser pulse is reflected on the surface of the object within the sensor range of the laser scanner 100 and returning. .

The laser scanner 100 may be mounted on one side of the user's body where the movement of the user's waist or the user's walking is relatively small.

The camera 200 may be mounted at a position close to the laser scanner 100 to photograph the front of the user.

In detail, the camera 200 may capture a surrounding image at a near and far distance of the user. In this case, the short range area may be an area within 3 m from the camera 200, and the far area may be an area in the range of 3 to 15 m from the camera 200. In addition, the ranges of the near and far areas can be adjusted, set by the system manufacturer from the time of product release, and set by the user.

The walking guidance apparatus 300 may predict the obstacles in front of the user and the user's movement by using the information acquired through the laser scanner 100 and the camera 200, and the information on the obstacles in front of the user and By using the information on the movement of the user can provide an optimal walk assistance service for the current walking situation of the user.

Referring to FIG. 2, the walking guide apparatus 300 includes a laser profile generator 310, an edge profile generator 320, a multi-modal profile generator 330, an obstacle detector 340, and an obstacle classifier 350. And a guide information extractor 360.

The laser profile generator 310 may generate the laser profile by obtaining depth information using distance information between the object in front of the user acquired from the laser scanner 100 and the laser scanner 100.

In detail, the laser profile generator 310 may receive distance information on each point of the object in front of the user. The laser profile generator 310 may acquire depth information about an object in front of the user using the received distance information. In this case, the depth information includes distance information that can indicate how close the laser scanner 100 is to the object in front of the user, so that each point of the user and the object in front of the user is close to the laser scanner 100. It may be an indicator. For example, if there is a pillar standing on the ground and the ground in front of the user, the pillar standing on the ground will be closer to the user than the ground when viewed horizontally. At this time, the depth information indicates that the pillar is closer than the ground.

The laser profile generator 310 may generate a laser profile including the acquired depth information. In this case, since the laser profile includes depth information for each point of the object in front of the user, the laser profile may be a set of depth information for each point of the object in front of the user.

In addition, the laser profile generator 310 may extract location information (eg, two-dimensional coordinates (x, y)) of each point of the object in front of the user. The laser profile generator 310 may match the depth information of each point of the object in front of the user with the position information of each point of the object in front of the user. The laser profile generator 310 may generate a laser profile by using depth information of each point of the object in front of the user and information matching the location information of each point of the object in front of the user.

In addition, the laser profile generator 310 may classify the laser profile using the acquired depth information.

In detail, the laser profile generator 310 may classify the laser profile using depth information obtained to accurately distinguish objects (eg, ground and obstacles) in front of the user. In this case, the laser profile generator 310 may calculate a difference between the acquired depth information. The laser profile generator 310 may detect whether a difference value between the acquired depth information is equal to or greater than a predetermined reference value (or threshold value). The laser profile generator 310 may classify coordinate points having the acquired depth information into the same laser profile if the difference value between the acquired depth information is less than a predetermined reference value, and the difference value between the acquired depth information. If it is equal to or more than the predetermined predetermined reference value, the coordinate point having the acquired depth information can be classified into another laser profile. For example, if the depth information of the first coordinate point is 10 cm, the depth information of the second coordinate point is 10.5 cm, and the predetermined constant reference value is 1 cm, the difference between the acquired depth information is calculated to be 0.5 cm. . In this case, since the difference between the acquired depth information is less than a predetermined reference value, the first coordinate point and the second coordinate point may be included in the same laser profile (eg, the first laser profile). In addition, if the depth information of the first coordinate point is 10cm, the depth information of the second coordinate point is 12cm, and the predetermined constant reference value is 1cm, the difference between the acquired depth information is calculated as 2cm. At this time, since the difference value between the acquired depth information is equal to or greater than a predetermined reference value, the first coordinate point and the second coordinate point are different laser profiles (for example, the first coordinate point is the first laser profile, the second coordinate point). The coordinate point can be classified into a second laser profile.

The edge profile generator 320 may generate an edge profile including edge information extracted by extracting an edge from a surrounding image in front of the user acquired through the camera 200.

In detail, the edge profile generator 320 may convert the surrounding image into a gray image. The edge profile generator 320 may extract an edge by using the −1 0 1 mask on the gray image. In this case, the -1 0 1 mask may be a sobel edge, a prewitt edge, and a Roberts edge. If the mask is capable of extracting an edge from a gray image, the mask other than the above-listed masks may be used. Of course, other masks may be applied.

The edge profile generator 320 may generate an edge profile by matching the extracted edge with the location information (eg, two-dimensional coordinates (x, y)) of the extracted edge.

The multi-modal profile generator 330 may generate a multi-modal profile by fusing the laser profile generated by the laser profile generator 310 and the edge profile generated by the edge profile generator 320.

In detail, as illustrated in FIG. 3A, the multi-modal profile generator 330 uses location information of each point of the object in front of the user included in the laser profile and location information of the edge included in the edge profile. The laser profile and the edge profile can be fused. At this time, the coordinate point included in the laser profile and the coordinate point of the edge (edge) included in the edge profile is matched to detect a point corresponding to the coordinate point included in the laser profile and the coordinate point of the edge included in the edge profile. can do. The multi-modal profile generator 330 may match the laser profile and the edge profile according to the detected point as shown in FIG. 3B. In this case, the multimodal profile in which the laser profile and the edge profile are fused may include edge information and depth information on coordinate points of each edge.

The obstacle detector 340 may detect an obstacle in a short distance of the user by using the multimodal profile generated by the multimodal profile generator 330.

In detail, the obstacle detector 340 may detect an obstacle based on a point where the laser profile and the edge profile match when the multimodal profile is generated. As illustrated in FIG. 4, the obstacle detector 340 may detect whether there is an edge profile that is connected in one shape with respect to the peripheral eight directions of the point where the laser profile and the edge profile match. If there is an edge profile connected in one shape in the peripheral eight directions of the point where the laser profile and the edge profile match, the obstacle detection unit 340 is connected in one of the eight directions around the point where the laser profile and the edge profile match. Bins may be generated for regions where edge profiles exist. The obstacle detector 340 may generate histograms in the vertical and horizontal directions with respect to the region in which the bin is generated. The obstacle detector 340 detects a region having a high histogram with respect to the vertical direction of the region where the bin is generated, and detects a region having a high histogram with respect to the horizontal direction of the region where the bin is generated. can do. The obstacle detector 340 may detect a region where a high histogram value overlaps with a high histogram value in a vertical direction. At this time, the edge profile connected in one shape is more likely to be an obstacle in front of the user. Thus, the area where the histogram is high in the vertical and horizontal directions is detected in the area where the obstacle is located in one shape. Can mean. The obstacle detection unit 340 may detect an area in which an area having a high histogram value overlaps as an obstacle area and remove the background area from the obstacle area by performing clustering on the detected obstacle area. Only can be detected.

In addition, the obstacle detector 340 may determine a predetermined search order (for example, the left side of the image) when there is no edge profile connected in one shape in the peripheral eight directions of the point where the laser profile and the edge profile are matched. From the top to the bottom right, you can search for edge profiles that are connected in one shape for the surrounding eight directions of the registration points, which are then ordered: In this case, the predetermined search order may be set from the product release by the system manufacturer, and may be set by the user.

5A, 5B, 5C, 5D, and 5E are diagrams for explaining a method for detecting a remote obstacle. FIG. 6 is a diagram illustrating an example of an obstacle classification unit classifying obstacles at a short distance, and FIG. 7 is a diagram illustrating a method of elliptic approximation of an edge blob, and FIGS. 8A and 8B are remote views at the time of the top view transformation. FIG. 9 is a diagram illustrating a method of calculating an ellipse's directionality, and FIG. 10 is an example of an obstacle classification unit classifying an obstacle at a distance. The figure is shown.

The obstacle detector 340 according to an embodiment of the present invention may detect an obstacle at a far distance by using an edge profile.

In detail, as illustrated in FIGS. 5A and 5B, the obstacle detector 340 may extract only the vertical edge component of the surrounding image in front of the user from the edge profile. As illustrated in FIG. 5C, the obstacle detector 340 may convert the image obtained by extracting only the vertical edge component to the top view transform. At this time, the top view conversion of the surrounding image is because the camera attached to the user's body acquires the image information on the front side, so it is difficult to accurately detect the distance and direction to the obstacle in the distance. Accordingly, the obstacle detecting unit 340 converts the camera's point of view from the front to a top-view point of view looking down from the vertical in order to accurately detect information on the distance and direction to the obstacle at a distance. Can be. In this case, since the method of converting the front view to the top view is disclosed in Korean Patent No. 1342124, a detailed description thereof will be omitted.

Referring to FIG. 5D, the obstacle detector 340 performs an opening technique that continuously performs erosion and expansion operations to remove noise from the top-view converted image, and conversely, to connect an edge obtained from one object. Morphological operations, such as a closing technique that continuously performs expansion and erosion operations, can be performed.

On the other hand, if there is an object (eg, an obstacle) that is separated from the background in the image, the objects will be connected to the same edge. Accordingly, in order to detect an obstacle in the image, the obstacle detection unit 340 may detect an obstacle at a long distance by applying a labeling to the image from which the noise is removed and detecting edge blobs connected to the same edge as shown in FIG. 5E.

The obstacle classification unit 350 may classify the obstacle according to the size or shape of the obstacle detected in the near and far distances.

In detail, the obstacle classifying unit 350 may classify the obstacle in the short range detected using the multi-modal profile. The obstacle classification unit 350 may infer the size or shape of the obstacle by calculating the width and height of the detected obstacle using the multi-modal profile. In this case, the width and height of the obstacle may be calculated through the horizontal histogram of the laser profile included in the multi-modal profile, the height of the obstacle may be calculated through the vertical histogram of the edge profile included in the multi-modal profile. . The obstacle classification unit 350 may classify the obstacle into two or three steps as shown in FIG. 6. First, the obstacle classification unit 350 may classify the first stage by using the depth information of the obstacle. At this time, the obstacle classifying unit 350 may classify the obstacle based on the detected obstacle and the depth information included in the laser profile of the ground on the horizontal line. The obstacle classification unit 350 may classify the object as positive if the depth value of the obstacle is a positive value based on the detected information on the detected obstacle and the depth information included in the laser profile of the ground on the horizontal line. If it is, it can be classified as ground. If the depth value of obstacle is negative, it can be classified as negative. In addition, the obstacle classification unit 350 may classify the obstacle according to the shape of the obstacle. In this case, the shape of the obstacle may be inferred according to the calculated ratio of the width to the height of the obstacle. For example, a high ratio of width to height of an obstacle classified as positive means that it is wider than its height, so the shape of the obstacle is a planar like lying on the floor. Since the height is high compared to the obstacle shape may be a vertical shape such as standing on the floor. In addition, since the obstacle classified as ground means the ground literally, there is no shape to classify, and the obstacle classified as negative may be classified according to the width of the obstacle. Obstacles classified as negative can be classified as holes if their width is negligibly narrow compared to their height, and can be classified as n-curb if their width is similar to their height or greater than the width of the obstacle classified as a hole. If it is larger than the height, it can be classified as a drop-off like an inclined surface.

In addition, the obstacle classifying unit 350 may classify the obstacle in the distance detected using the edge profile. The obstacle classifying unit 350 detects an ellipse by ellipsing the edge blob detected by the obstacle detecting unit 340, and detects an obstacle at a long distance using the detected short-to-large-axis ratio of the ellipse and the direction guide of the ellipse. .

In detail, as illustrated in FIG. 7, the obstacle classifying unit 350 may ellipse an edge blob detected by the obstacle detecting unit 340. In this case, the ellipse approximation may be calculated using the moments of the pixels included in the edge blob, and the rotation rate, the long axis, and the short axis length of the ellipse may be calculated through Equation 1.

At this time,

Is the rotation rate of the ellipse,

Is the length of the long axis of the ellipse,

Is the length of the short axis of the ellipse. Also,

Is an image moment for a certain area R, which is defined by Equation 2.

Here, x and y, as shown in Figure 8a, 8b, when the viewpoint of the camera is converted from the front view to the top view view, the two-dimensional coordinates of the shape of the obstacle to the ground is stretched in the radial direction.

The obstacle classification unit 350 may detect whether the obstacle is standing on the ground or lying down by analyzing the distortion of the image generated during the top view conversion. When the image of the front view is converted to the top view, an object having a predetermined height or more may be liable to fall in the radial direction from the camera 200. Referring to FIG. 8A, when a viewpoint of an image photographing an object having a certain height, such as a bar or a person, is converted into a top view, the horizontal edge of the object is parallel to the ground so that there is no change, but the vertical edge of the object is directed toward the ground. Can increase. On the contrary, referring to FIG. 8B, when a viewpoint of an image of a low-level object, such as a flat file, is converted to a top-view point of view, an object with a low height, such as a file, is low enough to appear parallel to the ground. The conversion may not cause distortion. The obstacle classification unit 350 may analyze the screen distortion phenomenon during the top view conversion to distinguish whether the obstacle in the distance is standing or lying on the ground. Referring to FIG. 9, the obstacle classifying unit 350 may determine the angle of the straight line formed by the original position of the camera 200 and the center of the ellipse (obstacle) and the direction of the ellipse (obstacle) in the top view image through [Equation 3]. DRO (Deviation form Radial Orientation), which indicates the degree of distortion of the screen during the top view transformation, may be calculated. At this time, if the DRO value is greater than or equal to a predetermined value, it may be determined that the obstacle is standing on the ground. If the DRO value is less than the predetermined value, it may be determined that the obstacle is lying on the ground.

At this time,

Is the coordinate of the center point of the i-ellipse,

Represents coordinates of the original position of the camera 200,

Is the angle between the major axis of the i-th ellipse and the horizontal plane.

The obstacle classification unit 350 may classify the obstacle at a distance using the DRO value and the long axis to short axis ratio of the ellipse. Referring to FIG. 10, when the DRO value is greater than or equal to a predetermined value, the obstacle sorting unit 350 may classify the obstacle into a shape that is erected on the ground, that is, vertically. It can be classified as a planar, a shape lying on the side. The obstacle classification unit 350 may classify the obstacle using the long axis to short axis ratio as a next step. The obstacle sorting unit 350 may classify the long pole-to-shorten ratios of the vertically classified obstacles into poles having a long shape. It can be classified into blocks, which are shapes. In addition, the obstacle classification unit 350 may be classified as curbs representing the sidewalk block if the long axis-to-short ratio among the obstacles classified as planar is more than a predetermined predetermined ratio, and the flat pile if the long-axis-short ratio is less than the predetermined predetermined ratio. It can be classified into piles representing.

The guide information extractor 360 may extract optimal walking assistance guide information suitable for a walking environment including information about a user's movement and an obstacle in front of the current user.

In detail, the guide information extractor 360 may predict the motion of the current user by using the light flow vector. In this case, since the user's position is generally located at the bottom center of the image, the guide information extractor 360 may set a region of interest at the bottom center of the image and calculate a light flow vector of the region of interest to predict the movement of the user. . In this case, the guide information extractor 360 divides nine zones after setting the ROI, calculates an optical flow vector of each corresponding zone, and minimizes an error rate of the optical flow vectors by calculating an average of the calculated optical flow vectors.

The guide information extractor 360 may recognize the current situation of the current user through the information on the obstacle obtained through the obstacle detector 340 and the obstacle classifier 350 and the user's movement obtained using the optical flow vector. have. For example, if an obstacle is located 10m away from the user's 10 o'clock position and the user is currently walking toward 12 o'clock, the user is far from the obstacle and if the user is walking in the direction of progress, it is unlikely to collide with the obstacle. The walking state can be detected as a safe state.If an obstacle exists at a position 10m away from the user's 10 o'clock direction and the user is currently walking to the 10 o'clock direction, the user's walking is present because the obstacle exists in the user's traveling direction but is far away. Ecology can be detected in a normal state, and if there is an obstacle at a position 3m away from the user's 12 o'clock direction and the user walks in the 12 o'clock direction, the user's walking state can be detected as a dangerous state.

In addition, the guide information extractor 360 may detect information about a direction in which the user is going according to the current situation of the user.

Specifically, when the guide information extractor 360 is hit by an obstacle in front of the user when the user proceeds in the direction of progress, the user must proceed in order not to hit an obstacle in front of the user by applying an extended Kalman filter. The direction can be detected. At this time, since the Extended Kalman filter can predict the position to which the user will move after a certain time, it can predict whether the object will not hit an obstacle in front of the user if the user rotates at an angle based on the current progress direction. Using the extended Kalman filter, the guide information extractor 360 may extract optimal walking assistance guide information suitable for the current situation of the user. For example, if the user is currently proceeding at 12 o'clock and there is an obstacle at a position 5 m away from the user's 12 o'clock, the guide information extractor 360 applies the extended Kalman filter to the user's current progress direction, That is, it is possible to detect whether or not the position is similar to the obstacle position when moving 5m by turning 10 ° clockwise with respect to 12 o'clock, and if the movement is not similar to the position of obstacle when moving 5m by turning 10 ° clockwise, The optimal walking direction according to the situation can be detected in a direction rotated by 10 ° clockwise from the current direction.

The guide information extractor 360 may generate a sentence having a predetermined pattern by extracting words corresponding to the current walking environment of the user.

In detail, the guide information extractor 360 may define words corresponding to the walking state of the user. At this time, the walking state of the user is divided into three stages, a safe state, a normal state, a dangerous state, and the guide information extracting unit 360 is information about the direction to proceed, the user's operation and the position of the obstacle if the user's walking state is a safe state. You can define a word that includes, and if the user's walking state is a normal state can define a word containing information about the direction to proceed, the user's motion and the location of the obstacle, if the user's walking state is dangerous state A word including information about a direction and a user's motion may be defined. At this time, when the walking state of the user is a dangerous state, there is an obstacle that is very close to the situation, and the walking direction is likely to change suddenly, so that the information on the walking direction and the user's motion is not included without outputting information on the position of the obstacle. Define only words.

For example, if the user's walking state is safe and there is a bar-shaped obstacle 10m away from the user's current walking direction at 10 o'clock, and the current user is walking at 12 o'clock, the walking direction corresponds to the direction to proceed. The word to be defined as "current direction" or "12 o'clock", the word corresponding to the user's action can be defined as "to go the same as the present" or "to go to the average step", the obstacle of The word corresponding to the position may be defined as "rod", "10 o'clock" and "10m". In addition, when the user's walking state is a normal state and a bar-shaped obstacle exists at a position 10 m away from the user's current walking direction at 12 o'clock, and the current user is walking at 12 o'clock, the word corresponding to the direction to proceed is expanded. The direction detected by the Kalman filter, for example, "11 o'clock" may be defined, and the word corresponding to the user's motion may be defined as "quickly visible", and the word corresponding to the position of the obstacle is "bar". , “12 o'clock” and “10 m”. In addition, when the user's walking state is a dangerous state and a bar-shaped obstacle exists at a position 3 m away from the user's current walking direction at 12 o'clock, the word corresponding to the direction to proceed is a direction detected by the extended Kalman filter, for example, It can be defined as "10 o'clock" and the word corresponding to the user's motion can be defined as "quickly go" or "stop".

The guide information extractor 360 may generate a sentence having a predetermined pattern by inserting words defined according to the walking state of the user. For example, the guide information extracting unit 360 defines a word defined when the user's walking state is a safe state, that is, "current progress direction", "go to the average step", "rod", "10 o'clock direction", and "10m" In the preset sentence and say "(Bar) is in front of (10m) at (10 o'clock). (Go to the average step in the (current direction)." Can produce In addition, if the walking state of the user is a normal state, the defined words, that is, "11 o'clock", "go quickly", "bar", "12 o'clock", and "10m" are inserted into a preset sentence, and the "(bar) is added. (At 10 o'clock) at (12 o'clock). (Go as soon as possible) at (11 o'clock). " Can produce In addition, if the user's walking state is dangerous, insert the defined word, "10 o'clock" or "quickly go" into the preset sentence, and "go quickly" (10 o'clock). " Can produce

The walking assistance system 1 according to an exemplary embodiment of the present invention may output a guide message including a sentence generated from the guide information extractor 360. In this case, the walking assistance system 1 may output a guide message by using an output module (not shown) provided in the walking guide device 300, and an earphone (not shown) or a headset (not shown) mounted on a user's ear. You can output the guide message through.

Hereinafter, a control method of a walking assistance system for providing a walking assistance guide service by detecting an obstacle at a short distance according to an embodiment of the present disclosure will be described with reference to FIG. 11.

First, a laser profile including depth information about each point is generated using the distance information of each point acquired through the laser scanner 100 (410).

In addition, an edge profile including edge information is generated by extracting the edges in the vertical and horizontal directions from the surrounding image captured by the camera 200 (415).

In addition, the multi-modal profile is generated by fusing the generated laser profile and the edge profile (420).

In this case, the fusion of the laser profile and the edge profile may fuse the laser profile and the edge profile so that the edge profile corresponding to the coordinate point of the laser profile is matched with the detected detection point.

In addition, in order to detect an obstacle in the multi-modal profile, it is detected whether there is an edge profile connected in one shape in the peripheral eight directions based on the matched point of the laser point and the edge profile (425).

At this time, if there is no edge profile connected in one shape in the peripheral eight directions of the matched point, the edge profile connected in one shape in the peripheral eight directions of the point determined in the next order according to a predetermined search order It is detected whether there exists (430).

In addition, when edge profiles connected in one shape are detected 425 and 435 in the eight directions around the matched point, an obstacle is detected by calculating a histogram in an area where the edge profiles connected in one shape are detected (440). ).

At this time, the histogram is calculated in the vertical and horizontal directions to the region where the edge profile connected in one shape is detected, and the region where the histogram value is high in the vertical direction and the region where the histogram value is high in the horizontal direction overlap. It may be detected as an obstacle area, and only an obstacle except a background area may be detected in the obstacle area detected through clustering.

In addition, the width of the obstacle is calculated through the horizontal histogram of the laser profile, and the height or height of the obstacle is calculated by the vertical histogram of the edge profile to infer the size or shape of the obstacle (445).

In this case, the width-to-height ratio may be calculated using the calculated width and height of the obstacle, and the width-to-height ratio may be used to detect whether the obstacle is standing on the ground or lying on the ground. A large width-to-height ratio can be inferred into a flat shape, such as a pile or a low sidewalk block, and a small width-to-height ratio can be inferred to an elongated shape such as a rod or a person.

In addition, the obstacle is classified 450 according to the size or shape of the inferred obstacle, and the user's movement is predicted by applying the optical flow vector and the extended Kalman filter (455).

At this time, the information on the current direction of the user can be detected by using the optical flow vector, and in which direction the current user moves using the extended Kalman filter to avoid collisions with obstacles. Information can be detected.

In addition, the optimal walking direction is calculated using information on the detected and classified obstacles and information on the user's movement (460).

In operation 465, a walking assistance guide message including the calculated optimal walking direction and obstacle information is generated and output.

In this case, the walking assistance system 1 may recognize the current walking state of the user by using information about the movement and obstacles of the user, and generate a walking assistance guide message by defining words corresponding to the recognized walking state. .

Hereinafter, a control method of a walking assistance system for providing a walking assistance guide service by detecting an obstacle at a distance according to an embodiment of the present disclosure will be described with reference to FIG. 12.

First, an edge profile including edge information is generated by extracting edges in the vertical and horizontal directions from the surrounding image captured by the camera 200 (510).

In addition, in order to extract the obstacle from the edge profile, only the vertical edge component that contributes to the obstacle extraction is extracted to extract an image including only the vertical edge component (515).

Also, in order to detect whether an obstacle at a distance is standing on the ground or lying down, the image photographed from the front view is converted into the top view view (520).

In this case, since the horizontal edge is parallel to the ground, the distortion does not occur when converted to the top view, but the vertical edge may occur when the vertical edge is converted to the top view. The walking assistance system 1 of the present invention may analyze this distortion phenomenon to detect whether an obstacle is standing on the ground or lying down.

Also, in order to detect obstacles in the image converted to the top view, noise is removed that may affect the obstacle detection, and labeling is performed to detect obstacles by using the property that one object is connected to the same edge. The edge blob including the obstacle is extracted (525).

In addition, an ellipse approximation is performed on the extracted edge blob to obtain an ellipse from the pixels forming the edge blob (530).

In addition, the length of the ellipse representing the degree of change of the ellipse direction when the top view is transformed by calculating the lengths of the obtained ellipses and the length of the short axis and changing the ellipse of the ellipse due to the distortion phenomenon generated when converting to the top view point of view. Is detected (535).

The ratio of the long axis to the short axis is calculated using the calculated long axis and the short axis length, and the obstacle is classified according to the calculated long axis to short axis ratio and the directionality of the ellipse (540).

In addition, the optical flow vector and the extended Kalman filter are applied to predict the motion of the user (545).

In operation 550, an optimal walking direction is calculated using information on the detected and classified obstacles and information on the user's movement.

In operation 555, a walking assistance guide message including the calculated optimal walking direction and obstacle information is generated and output.

Such a technology for detecting an obstacle in a short distance and a long distance and predicting a user's movement to provide an optimal walk assistance service is implemented in the form of program instructions that can be implemented as an application or executed through various computer components. It can be recorded on a computer readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the computer-readable recording medium are those specially designed and configured for the present invention, and may be known and available to those skilled in the computer software arts.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the process according to the invention, and vice versa.

Although the above has been described with reference to the embodiments, those skilled in the art will understand that various modifications and changes can be made without departing from the spirit and scope of the invention as set forth in the claims below. Could be.

Claims

A laser profile model including depth information obtained using a laser sensor is generated, an edge profile model including edge information extracted from an image acquired using an image sensor is generated, and the laser profile model and the edge profile model are generated. To create a multimodal profile model,

Detect and classify obstacles that exist in the near distance from the multi-modal profile model, detect and classify obstacles that exist in the distance from the edge profile model,

The walk assistance method of recognizing the situation in front of the user receiving the walking assistance using the information on the detected and classified obstacles, and providing information on the direction and the situation to proceed according to the situation information in front of the user.
The method of claim 1,

Generating the laser profile model,

The walk assistance method of classifying a laser profile according to whether or not the values of the depth information obtained by using the depth information to distinguish the laser profile of the ground and the laser profile of the obstacle.
The method of claim 2,

Categorizing the laser profile according to similarity of the obtained depth information,

Compute a difference value between the values of the acquired depth information, and if the calculated difference value is less than a predetermined reference value, the laser profile having the value of the acquired depth information is classified into the same laser profile, and the calculated difference value is The walk assistance method of classifying a laser profile having a value of the acquired depth information into a different laser profile if it is equal to or greater than a predetermined reference value.
The method of claim 1,

Generating the multi-modal profile model,

And generating a multi-modal profile model by matching a coordinate point of depth information included in the laser profile model with a corresponding point of an image included in the edge profile model.
The method of claim 1,

Detecting and classifying obstacles in the near field from the multi-modal profile model,

When generating the multi-modal profile model, the laser profile model and the edge profile model detects whether there is a connected edge profile for the peripheral eight directions of the matched point, and if the connected edge profile exists, the edge profile is detected. The walk assistance method of detecting and classifying obstacles in the near field by calculating vertical and horizontal histograms in an area.
The method of claim 5, wherein

Categorizing the obstacles present in the short distance,

The width of the obstacle is calculated through a horizontal histogram of the laser profile model, the height of the obstacle is calculated through a vertical histogram of the edge profile model, and the size and shape of the obstacle are determined using the width and height of the obstacle. Walking assistance method for inferring and classifying the obstacle.
The method of claim 1,

Detecting and classifying obstacles that exist at a distance from the edge profile model,

Extract the vertical edge component from the obtained image, top view transform the extracted image of the vertical edge component, perform morphology calculation and labeling on the top view transformed image, and classify the edge blob which is the obstacle region, and oval The walk assistance method of obtaining an ellipse from the edge blob through approximation, calculating the direction of the ellipse, and classifying the obstacle according to the direction of the ellipse.
The method of claim 1,

Recognizing the situation in front of the user receiving the walking assistance using the information on the detected and classified obstacles, and providing information about the direction and the situation to proceed according to the situation information in front of the user,

A walking environment including a walking state and a walking direction of the user from the movement of the user and information about the detected and classified obstacles by predicting the movement of the user by applying an optical flow vector and an extended Kalman filter to the obtained image. Recognizing the situation of the walk assistance method for providing information on the direction and the situation to proceed according to the situation of the walking environment.
The method of claim 8,

Providing information on the direction and the situation to proceed according to the situation of the walking environment,

Searching for a plurality of words corresponding to the walking environment of the user according to the walking environment of the user, generating a sentence having a predetermined pattern by combining the searched plurality of words, and including the generated sentences Pedestrian assistance method that outputs a guidance message.
A computer-readable recording medium having a computer program recorded thereon for providing information for assisting walking of a user according to any one of claims 1 to 9.
A laser scanner for detecting depth information of a space located in front of the apparatus;

A camera for photographing the space located in front of the camera;

Generate a laser profile model using the depth information detected by the laser scanner, extract edge information from the image taken by the camera and generate an edge profile model including the edge information, the laser profile and the Fuse an edge profile model to generate a multimodal profile model, detect and classify obstacles located forward from the multimodal profile model, detect and classify obstacles remote from the edge profile model, and detect and classify the obstacles. And a walking guidance device that recognizes a situation in front of a user who receives walking assistance using information on an obstacle and provides information on a walking direction and a situation according to the situation in front of the user.