JP7029170B2 - Sidewalk progress support system and sidewalk progress support software - Google Patents

Description

The present invention relates to a sidewalk progress support system and sidewalk progress support software. More specifically, the present invention relates to a sidewalk progress support system and sidewalk progress support software for the vulnerable to walking such as the elderly and the visually impaired.

It is very difficult for a vulnerable person to go out, walk on the road, and reach the destination, no matter how simple the road is.

Braille blocks, guide bells, Braille signs, and white canes exist to support the walking of these vulnerable people, especially those who are visually impaired, and guide dogs are used and walking training is also conducted by walking trainers.

In addition, various techniques for supporting the walking of the vulnerable person have been proposed.
For example, Patent Document 1 describes a personal navigation system connected to the Internet and including a monitoring device and a guidance device.

The monitoring device monitors the route through which the user of the mobile phone travels through the Internet.
In addition, the guidance device has various databases inside, searches for data that guides the user of the mobile phone to the destination, and provides the data to the monitoring device.

In addition, among the databases provided by the guidance device, the map information database contains digital map information, a list of obstacles such as sidewalks and road steps in each area, telegraph pillars on the sidewalks, and flower beds. Contains data about the location.

In addition, the user of the mobile phone proceeds along the regular route while receiving voice instructions, whereby the position information at that time is sent from the mobile phone to the monitoring device.
In addition, the monitoring device checks the position information of the mobile phone, and if there is an obstacle such as a post within a range of several meters ahead, the direction of travel is temporarily changed to avoid the obstacle. Execute.

Japanese Unexamined Patent Publication No. 2004-110794

However, in the system described in Patent Document 1, although it is possible to avoid a stationary object such as a telephone pole or a curb installed in advance, there is a problem that it is difficult to avoid a moving object such as a passerby or a person driving a bicycle.

Further, regarding stationary objects, for example, construction-related installations such as cones suddenly installed on the sidewalk on the day when a pedestrian walks are not immediately reflected in the database. Therefore, in the system described in Patent Document 1, Another problem is that it is difficult to avoid such stationary objects.
Therefore, there is a need for a further system that can support the walking of the vulnerable.

The present invention has been devised in view of the above points, and an object of the present invention is to provide a sidewalk progress support system and sidewalk progress support software capable of supporting the progress of a vulnerable person on a sidewalk.

In order to achieve the above object, the sidewalk progress support system of the present invention uses an imaging unit capable of capturing at least the front of a pedestrian and a moving object in the captured image obtained by the imaging unit. An object detection unit capable of detecting a certain movable object and a stationary object, and a first line segment extending from the lower part of the captured image toward a substantially central region of the captured image in the captured image. The distance from the lower part of the captured image toward the substantially central region of the captured image and the distance from the lower portion of the captured image to the substantially central region of the captured image became shorter. A line segment detection unit capable of detecting a second line segment, and a region between the first line segment and the second line segment detected by the line segment detection unit in the captured image. A sidewalk determination unit that can determine a sidewalk, a current position information detection unit that can detect the current position information that is the position information of a pedestrian equipped with the image pickup unit, and a direction that can detect the image pickup direction of the image pickup unit. A detection unit, a distance measurement unit that can measure the distance from the image pickup unit to an object imaged by the image pickup unit based on the captured image, and a map information recording unit that can record map information that is information about a map. The position of the stationary object or the movable object detected by the object detection unit is between the imaging direction detected by the direction detecting unit and the imaging unit and the stationary object or the movable object measured by the distance measuring unit. Based on the distance, it can be calculated as a coordinate value of the dynamic coordinate system with the position of the pedestrian wearing the image pickup unit as the origin, and the current position information detected by the current position information detection unit is used as the origin. , The calculated same coordinate value is converted into the latitude and longitude of the map coordinate system included in the map information recorded by the map information recording unit, and the map coordinates of the same stationary object or the same movable object can be created. The creation unit, the detected object information recording unit capable of recording the movable object information which is information about the movable object and the stationary object information which is the information about the stationary object, the movable object or the stationary object detected by the object detection unit, and the detection. The object identification unit capable of identifying the movable object and the stationary object by collating the movable object information and the stationary object information recorded by the object information recording unit, and the same when the sidewalk determination unit determines the sidewalk. The latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to each of the first line segment and the second line segment sandwiching the sidewalk, and the first line segment and the same first line segment. The map created by the object map coordinate creation unit corresponding to a stationary object or a movable object located between the two line segments. Based on the latitude and longitude position information of the coordinates, the progressable area position information detection unit that can detect the position information of the travelable area in the same sidewalk, the voice, and the map information recorded by the map information recording unit. Based on this, the destination information determination unit that can determine the destination information that is information about the destination, the destination information determined by the destination information determination unit, and the current position detected by the current location information detection unit. A route information determination unit that can determine route information that is information about a route between a destination and a current position based on information, the route information determined by the route information determination unit, and the progressable area position. Based on the travelable area position information detected by the information detection unit, the progress that can determine the travelable route information that is the route between the destination and the current position and is the information about the route that the pedestrian can travel. It is equipped with a possible route information determination unit.

Here, the movable object or the stationary object detected by the object detection unit is collated with the movable object information and the stationary object information recorded by the detected object information recording unit, and the movable object and the stationary object can be identified by the object identification unit. , Can identify objects that actually exist at least in front of the vulnerable walking on the sidewalk.

Further, the position of the stationary object or the movable object detected by the object detection unit is based on the image pickup direction detected by the direction detection unit and the distance between the image pickup unit measured by the distance measurement unit and the stationary object or the movable object. , It is possible to calculate as the coordinate value of the dynamic coordinate system with the position of the pedestrian with the image pickup unit as the origin, and the calculated coordinate value with the current position information detected by the current position information detection unit as the origin. , An object that can create map coordinates of a stationary object or a movable object by converting it to the latitude and longitude of the map coordinate system included in the map information recorded by the map information recording unit. At least, it is possible to grasp the position of an object that actually exists in front of it in map coordinates.

Further, based on the route information determined by the route information determination unit and the travelable area position information detected by the travelable area position information detection unit, the pedestrian advances along the route between the destination and the current position. The travelable route information determination unit, which can determine the travelable route information that is information about the possible route, considers the situation on the sidewalk when actually walking, which is not included in the map information, and determines the travelable route of the sidewalk. I can decide.

Further, the sidewalk progress support system of the present invention can notify that a movable object exists when the object identification unit identifies a movable object, and can provide the progressable route information determined by the progressable route information determination unit. It can be configured to include a notification unit capable of notification.

In this case, the weak walking person can know the route information that can be advanced corresponding to the situation on the sidewalk when actually walking.

Further, in the sidewalk progress support system of the present invention, the distance measured by the distance measuring unit between the object to which the imaging direction of the imaging unit is directed and the imaging unit detected by the direction detecting unit is not more than a predetermined value. In addition, when the object detected by the object detection unit has a vertical line segment that is a line segment extending in the vertical direction by a predetermined length or more, the control to notify the notification unit or the object measured by the distance measurement unit and the image pickup unit. It is possible to have a configuration including a control unit capable of controlling to notify the content of guiding in the direction in which the distance between the object and the object is larger than a predetermined value.

In this case, a weak walking person can easily avoid a collision with an object.

Further, in the sidewalk progress support system of the present invention, the line segment detection unit further intersects the first line segment and the second line segment in the captured image, that is, the first crossing line segment and the second line segment. The intersection of the line segments can be detected, the distance measuring unit can further measure the distance between the line segments based on the captured image, and the control unit can measure the first intersection measured by the distance measuring unit. When the distance between the line segment and the second crossing line segment is larger than a predetermined value, the image pickup direction of the image pickup unit turns to the left. It is possible to control the notification unit to notify the content of guiding to the right direction change, which is to turn to the direction.

In this case, since the area between the first intersection line segment and the second intersection line segment, that is, the area intersecting the sidewalk is often a roadway, a weak walking person is made to image the condition of the roadway when crossing the roadway. be able to.

Further, in the sidewalk progress support system of the present invention, when a stationary object is identified as a braille block as a result of identification by the object identification section, the map coordinates created by the object map coordinate creation section corresponding to the braille block are used. It has a position information comparison unit that can compare the latitude and longitude position information with the current position information detected by the current position information detection unit. When specified, the meaning of the braille block connector, which is a configuration in which a plurality of braille blocks are connected to each other, can be further identified based on the stationary object information recorded by the detected object information recording unit, and the control unit can specify the position. When the position information of the Braille block and the current position information do not match as a result of comparison by the information comparison unit, the configuration can be made so that the notification unit can be controlled to notify the guidance to the position of the Braille block.

In this case, even if the weak walking person moves away from the Braille block on the sidewalk, he / she can approach the Braille block.

Further, in the sidewalk progress support system of the present invention, the distance measuring unit can irradiate a laser, and can further measure the distance from the imaging unit to the object imaged by the imaging unit based on the reflection of the irradiated laser. The control unit can be controlled to notify that there is a step in the notification unit when the continuously measured distance changes based on the reflection of the laser irradiated diagonally downward. can do.

Further, in the sidewalk progress support system of the present invention, if a stereo camera such as two eyes or three eyes can be used, the distance measuring unit is approximately from the binocular parallax of the image in the range of a short distance of about 10 to 15 m. The distance can be calculated.
However, the accuracy is not as high as that of a laser range finder, and it is difficult to determine a step, but it is possible to measure the approximate distance to a gait obstacle.

In this case, the weak walking person can avoid the step and appropriately cope with the step.

Further, the sidewalk progress support system of the present invention is based on the map coordinates of a stationary object created by the object map coordinate creation unit, the specific result of the object identification unit identifying the stationary object, and the map information recorded by the map information recording unit. Based on this, an object placement information creation unit that can create object placement information that is information about the placement of the specified object on the map, and an object placement information recording unit that can record the object placement information created by the object placement information creation unit. It can be configured to include.

In this case, it is possible to record what kind of object is actually placed on the sidewalk in which place so that it can be searched.

Further, the sidewalk progress support system of the present invention is related to the destination, which is information related to the destination information among the voice, the destination information determined by the destination information determination unit, and the map information recorded by the map information recording unit. A local destination information determination unit capable of determining local destination information, which is more local information than the destination information, is provided based on the map information, and the route information determination unit is the purpose determined by the destination information determination unit. Instead of the location information, the route information can be determined based on the local destination information determined by the local destination information determination unit and the current position information detected by the current position information detection unit.

In this case, the destination can be determined in more detail to meet the wishes of the pedestrian.

Further, in the sidewalk progress support system of the present invention, the current position information detection unit detects the travelable route information determined by the travelable route information determination unit and the travelable route of the travelable route information when the pedestrian advances. The configuration may include a progress history recording unit that can record the current position information in association with the captured image when the current position information is detected.

In this case, the actual walking of a weak walking person to the destination can be recorded and used as a reference when the same pedestrian or another pedestrian walks on the same route.

Further, the sidewalk progress support system of the present invention includes a search unit capable of searching the map information recorded by the map information recording unit and the object arrangement information recorded by the object arrangement information recording unit based on the voice, and the notification unit is provided. , The search result of the search unit can be announced.

In this case, the vulnerable person can search and know not only the information included in the map information but also what kind of object is actually arranged in the surrounding information of the place where he / she is currently located.

Further, in the sidewalk progress support system of the present invention, the image cutout portion capable of cutting out the images of the movable object and the stationary object detected by the object detection unit from the captured image, and the movable object cut out by the image cutout portion. A configuration including a teacher information recording unit capable of recording teacher information, which is information obtained by associating a cut-out image of a cut-out image and a stationary object with a specific result specified by an object specifying unit for a movable object and a stationary object. Can be.

In this case, deep learning can be performed to improve the accuracy of the artificial intelligence used when identifying the object.

Further, in the sidewalk progress support system of the present invention, the notification unit has a voice notification unit capable of announcing the content of the notification by voice and a longitudinal direction, and one side in the longitudinal direction and the opposite side to the other side. A main body that can be detachably connected to the other side located in is attached to one surface of the main body side by side in the same direction as the longitudinal direction of the main body, and when notifying, one according to the content of the notification. The unit or the whole can be configured to be at least one of the vibration notification units having a plurality of vibrating bodies capable of vibrating.

In this case, regarding the vibration notification unit, one side of the main body and the other side of the main body can be connected to form an annular body, and the main body can be attached to the pedestrian's body. It can be announced earlier than that.
Further, "one side in the longitudinal direction" and "the other side in the longitudinal direction" shall each include a predetermined range from the end portion.
Further, the voice notification unit can announce the content of the notification in more detail than the notification of the vibration notification unit.

Further, in order to achieve the above object, the sidewalk progress support software of the present invention is a sideway progress support software for supporting the progress of pedestrians on the sidewalk, and is information on an information processing device and a map. A moving object that is a moving object and a stationary object that are detected in the captured image obtained by imaging the map information recording unit that can record map information and the imaging unit that can image at least the front of the pedestrian. The image pickup unit attaches the position of a stationary object based on the imaging direction of the image pickup unit and the distance between the image pickup unit and the stationary object or the movable object measured based on the image pickup image. It can be calculated as the coordinate value of the dynamic coordinate system with the position of the pedestrian as the origin, and the current position information, which is the detected position information of the pedestrian with the imaging unit attached, is used as the origin. An object map coordinate creation unit that can create map coordinates for the same stationary object or the same movable object by converting the same coordinate values to the latitude and longitude of the map coordinate system included in the map information recorded by the map information recording unit. A detection object information recording unit capable of recording movable object information which is information about a movable object and stationary object information which is information about a stationary object, a movable object or a stationary object detected in the captured image, and the detection. By collating the movable object information and the stationary object information recorded by the object information recording unit, the movable object and the stationary object can be identified, and the image pickup detected in the captured image. A first line segment extending from the lower part of the image toward the substantially central region of the captured image, and extending from the lower part of the captured image toward the substantially central region of the captured image and extending from the lower part of the captured image to the substantially central region of the captured image. The first line segment sandwiching the same walkway when the area between the second line segment and the second line segment whose distance to the first line segment became shorter as it approached the substantially central region was determined as the sidewalk. And, it is located between the latitude and longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to each of the second line segment and the first line segment and the second line segment. Progressable area position information that can detect the position information of the progressable area in the same sidewalk based on the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to the stationary object or the movable object. Based on the detection unit, the voice, and the map information recorded by the map information recording unit, the destination information determination unit capable of determining the destination information, which is information about the destination, and the destination information determination unit Based on the determined destination information and the detected current position information, the destination and the present A route information determination unit that can determine route information that is information about a route to and from a position, the route information determined by the route information determination unit, and the travelable area detected by the travelable area position information detection unit. A means including a travelable route information determination unit capable of determining travelable route information which is a route between a destination and a current position and is information on a route that a pedestrian can travel based on the position information. It is software to function as.

Further, in order to achieve the above object, the sidewalk progress support software of the present invention is a sidewalk progress support software for supporting the progress of a pedestrian on a sidewalk, and an information processing device is used at least in front of the pedestrian. In the captured image obtained by the image pickup unit capable of capturing the image, the object detection unit capable of detecting a moving object which is a moving object and a stationary object which is a stationary object, and the same in the captured image. A first line segment extending from the lower part of the captured image toward the substantially central region of the captured image, and extending from the lower part of the captured image toward the substantially central region of the captured image and extending from the lower part of the captured image to the captured image. A line segment detection unit capable of detecting a second line segment whose distance from the first line segment becomes shorter as it approaches the substantially central region of the above, and the line segment detection unit in the captured image. A sidewalk determination unit that can determine an area between the first line segment and the second line segment as a sidewalk, and a map information recording unit that can record map information that is information about a map. The position of the stationary object or the movable object detected by the object detection unit is determined by the imaging direction of the imaging unit and the distance between the imaging unit and the stationary object or the movable object measured based on the captured image. Based on the above, it is possible to calculate as a coordinate value of the dynamic coordinate system with the position of the pedestrian equipped with the image pickup unit as the origin, and with the detected position information of the pedestrian equipped with the image pickup unit. The same coordinate value calculated with a certain current position information as the origin is converted into the latitude and longitude of the map coordinate system included in the map information recorded by the map information recording unit, and the map coordinates of the stationary object or the movable object are converted. An object map coordinate creation unit capable of creating a Alternatively, the stationary object is collated with the moving object information and the stationary object information recorded by the detected object information recording unit, and the moving object and the stationary object can be identified by the object specifying unit and the sidewalk determination unit. The latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to each of the first line segment and the second line segment sandwiching the same walkway when the sidewalk is determined, and the same first. The same sidewalk based on the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to a stationary object or a movable object located between the first line segment and the second line segment. The progressable area position information detection unit that can detect the position information of the progressable area in the inside, the voice, and the map information description. Based on the map information recorded by the recording unit, the destination information determination unit capable of determining the destination information, which is information about the destination, and the destination information determined by the destination information determination unit are detected. A route information determination unit capable of determining route information which is information on a route between a destination and a current position based on the current position information, the route information determined by the route information determination unit, and the above-mentioned route information. Based on the progressable area position information detected by the progressable area position information detection unit, the progressable route information which is the route between the destination and the current position and which is the information about the route where the pedestrian can proceed is obtained. It is software for functioning as a means including a determinable progressable route information determination unit.

Here, the object identification unit can identify an object that actually exists at least in front of a weak walking person walking on the sidewalk.
In addition, the object map coordinate creation unit can grasp the position of the object actually existing at least in front of the weak walking person walking on the sidewalk in the map coordinates.
In addition, the progressable route information determination unit can determine the progressable route of the sidewalk in consideration of the situation on the sidewalk when actually walking, which is not included in the map information.

The sidewalk progress support system according to the present invention can support the progress of a weak walking person on the sidewalk.
The sidewalk progress support software according to the present invention can support the progress of a weak walking person on the sidewalk.

It is a schematic diagram which shows an example of the structure of the sidewalk progress support system to which this invention is applied. It is a schematic diagram (a) which shows the 1st example of the captured image in the sidewalk progress support system to which this invention is applied, and the schematic diagram (b) which shows an example of the augmented reality image of the captured image shown in the schematic diagram (a). .. It is a schematic diagram which shows an example of the dynamic coordinate system and the map coordinate system in the sidewalk progress support system to which this invention is applied. It is a schematic diagram (a) which shows the 2nd example of the captured image in the sidewalk progress support system to which this invention is applied, and the schematic diagram (b) which shows an example of the augmented reality image of the captured image shown in the schematic diagram (a). .. It is a schematic diagram which shows an example of the augmented reality image of the captured image of the 3rd example in the sidewalk progress support system to which this invention is applied. It is a schematic diagram which shows an example of the augmented reality image of the captured image of the 4th example in the sidewalk progress support system to which this invention is applied. It is a schematic diagram which shows an example of the state of detecting a step by using the eyeglass device used in the sidewalk progress support system to which this invention is applied. It is a schematic diagram which shows the example which used the guide dog robot instead of the white cane in the sidewalk progress support system to which this invention was applied. It is a schematic diagram (a) which shows an example of the belt device provided in the sidewalk progress support system of this invention, and is the schematic diagram (b) which shows an example of a pedestrian wearing a belt device.

Hereinafter, embodiments of the present invention will be described with reference to the drawings for the purpose of understanding the present invention.
FIG. 1 is a schematic view showing an example of a configuration of a sidewalk progress support system to which the present invention is applied.

The sidewalk progress support system 1 of the present invention shown in FIG. 1 includes a glasses device 2 having a general shape of glasses, and a server 3 capable of communicating with the glasses device 2 via the Internet 30.
Further, the eyeglass device 2 is attached to, for example, a visually impaired pedestrian.

Further, the eyeglass device 2 has an image pickup camera 4 capable of capturing at least the front of a pedestrian.
Here, the image pickup camera is an example of the image pickup unit.

Further, the eyeglass device 2 has an image pickup image transmission unit 4A capable of transmitting an image captured by the image pickup camera 4 to the server 3 via the Internet 30.

On the other hand, the server 3 has a captured image receiving unit 4B capable of receiving the transmitted captured image.

Further, the eyeglass device 2 has an object detection unit 5.
Here, the object detection unit 5 can detect a moving object which is a moving object and a stationary object which is a stationary object in the captured image obtained by the image pickup camera 4.
Specific examples of the movable object include a passerby, a person riding a bicycle, and an animal.
Specific examples of the stationary object include a braille block, a telephone pole, a curb, a vending machine, various signs, a mailbox, a raw wall, a flower bed, a parked car, a bicycle parked, and a roadside tree. ..

Further, the glasses device 2 can transmit information about the stationary object and the moving object detected by the object detection unit 5, for example, an augmented reality image displaying the color of the object and the existing area rectangle surrounding the object to the server 3 via the Internet 30. It has a detection object transmission unit 5A.

On the other hand, the server 3 has a detection object receiving unit 5B capable of receiving information about the transmitted stationary object and moving object.

Further, the eyeglass device 2 has a line segment detection unit 6.
Here, the line segment detection unit 6 includes a first line segment extending from the lower part of the captured image toward the substantially central region of the captured image in the captured image, and the line segment detecting unit 6 from the lower portion of the captured image to the substantially central region of the captured image. It is possible to detect a second line segment whose distance from the first line segment becomes shorter as it extends toward and approaches a substantially central region of the captured image from the lower part of the captured image.

The line segment detection unit 6 can detect a first line segment and a second line segment based on a stationary object such as a curb in a captured image or a line segment of a building along a sidewalk.

Further, the line segment detection unit 6 can further detect the first crossing line segment and the second crossing line segment that intersect each of the first line segment and the second line segment in the captured image. be.

Further, the eyeglass device 2 has a sidewalk determination unit 7.
Here, the sidewalk determination unit 7 can determine the area between the first line segment and the second line segment detected by the line segment detection unit 6 as a sidewalk in the captured image.

Further, the glasses device 2 transmits information about the sidewalk determined by the sidewalk determination unit 7, for example, an augmented reality image displaying the first line segment and the second line segment sandwiching the sidewalk to the server 3 via the Internet 30. It has a possible decision sidewalk transmission unit 7A.

On the other hand, the server 3 has a determination sidewalk receiving unit 7B capable of receiving information about the transmitted sidewalk.

Further, the eyeglass device 2 has a current position information detection unit 8.
Here, the current position information detection unit 8 can detect the current position information which is the position information of the pedestrian who is equipped with the image pickup camera 4, that is, the glasses device 2 having the image pickup camera 4.
Further, the current position information is the latitude and longitude detected each time the image pickup camera 4 takes an image.

Further, the glasses device 2 has a current position information transmission unit 8A capable of transmitting the current position information of a pedestrian detected by the current position information detection unit 8 to the server 3 via the Internet 30.

On the other hand, the server 3 has a current position information receiving unit 8B capable of receiving the transmitted current position information of the pedestrian.

Further, the eyeglass device 2 has a direction detection unit 9 capable of detecting the image pickup direction of the image pickup camera 2, that is, the front direction of the pedestrian.
Here, the direction detection unit is specifically, for example, an electronic compass.

Further, the eyeglass device 2 has an image pickup direction transmission unit 9A capable of transmitting information regarding the image pickup direction detected by the direction detection unit 9 to the server 3 via the Internet 30.

On the other hand, the server 3 has an image pickup direction receiving unit 9B capable of receiving the transmitted information regarding the image pickup direction.

Further, the eyeglass device 2 has a distance measuring unit 10.
Here, the distance measuring unit 10 can measure the distance from the image pickup camera to the object captured by the image pickup camera based on the image pickup image.

Further, the distance measuring unit 10 can further measure the distance between the line segments based on the captured image.
Further, the distance measuring unit 10 can irradiate the laser, and can further measure the distance from the image pickup camera 4 to the object imaged by the image pickup camera 4 based on the reflection of the irradiated laser.

Further, the eyeglass device 2 has a measurement distance transmission unit 10A capable of transmitting information regarding the distance measured by the distance measurement unit 10 to the server 3 via the Internet 30.

On the other hand, the server 3 has a measurement distance receiving unit 10B capable of receiving information regarding the transmitted distance.

Further, the server 3 has a map information database 11 capable of recording map information which is information related to a map.
Here, the map information database is an example of the map information recording unit.

Further, the server 3 has an object map coordinate creating unit 12.
Here, the object map coordinate creation unit 12 is stationary with the image pickup direction detected by the direction detection unit 9 and the image pickup camera 4 measured by the distance measurement unit 10 for the position of the stationary object or the movable object detected by the object detection unit 5. It can be calculated as a coordinate value of a dynamic coordinate system having the position of a pedestrian on which the image pickup camera 4 is attached as an origin based on the distance between an object or a movable object.
Further, the object map coordinate creation unit 12 uses the current position information detected by the current position information detection unit 8 as the origin, and the calculated coordinate values are the latitude of the map coordinate system included in the map information recorded by the map information database 11. It can be converted to longitude to create map coordinates for stationary or moving objects.

Further, the server 3 has a detected object information database 13 capable of recording movable object information which is information about a movable object and stationary object information which is information about a stationary object.
Here, the detected object information database is an example of the detected object information recording unit.

Further, specifically, the detected object information database 13 can record numerical information, for example, size and color information of the existing area rectangle of each object in the captured image.

Further, the server 3 has an object identification unit 14.
Here, the object identification unit 14 collates the movable object or the stationary object detected by the object detection unit 5 with the movable object information and the stationary object information recorded by the detected object information database 13, and obtains the movable object and the stationary object. It is identifiable.

Further, the server 3 has a specific object transmission unit 14A capable of transmitting information about an object specified by the object identification unit 14 to the eyeglass device 2 via the Internet 30.

On the other hand, the eyeglass device 2 has a specific object receiving unit 14B capable of receiving information about the transmitted object.

Further, the server 3 has a progressable area position information detection unit 15.
Here, the progressable area position information detection unit 15 is the object map coordinate creation unit 12 corresponding to each of the first line segment and the second line segment sandwiching the sidewalk when the sidewalk determination unit 7 determines the sidewalk. The latitude / longitude position information of the map coordinates created by the object map coordinates created by the object map coordinate creating unit 12 corresponding to the stationary object or the movable object located between the first line segment and the second line segment. It is possible to detect the position information of the travelable area in the sidewalk based on the latitude / longitude position information.

Further, the server 3 has a destination information determination unit 16 capable of determining destination information, which is information about the destination, based on the voice and the map information recorded by the map information database 11.
Further, the glasses device 2 has a voice transmission unit 16A capable of transmitting voice to the server 3 via the Internet 30, and the server 3 has a voice reception unit 16B capable of receiving the transmitted voice.

Further, the server 3 has a question analysis unit 29 capable of performing voice recognition of the question voice and analyzing the question content among the voices received by the voice receiving unit 16B.

Further, the server 3 has a voice answering unit 16C capable of transmitting a voice answer, which is an answer to the question obtained by the question analysis unit 29, to the eyeglass device 2 via the Internet 30.
Further, the glasses device 2 has a response receiving unit 16D capable of receiving the transmitted voice response.

Further, the server 3 has a route information determination unit 17.
Here, the route information determination unit 17 between the destination and the current position based on the destination information determined by the destination information determination unit 16 and the current position information detected by the current position information detection unit 8. It is possible to determine route information, which is information about the sidewalk network route.

Further, the server 3 has a progressable route information determination unit 18.
Here, the progressable route information determination unit 18 has a destination and a current position based on the route information determined by the route information determination unit 17 and the progressable area position information detected by the progressable area position information detection unit 15. It is possible to determine the route information that is the route between the vehicle and the pedestrian, which is information about the route that the pedestrian can travel.

Further, the server 3 has a progressable route information transmission unit 18A capable of transmitting the progressable route information determined by the progressable route information determination unit 18 to the glasses device 2 via the Internet 30.

On the other hand, the spectacles device 2 has a progressable route information receiving unit 18B capable of receiving the transmitted progressable route information.

Further, the eyeglass device 2 has a notification unit 19.
Here, the notification unit 19 can notify that the movable object exists when the object identification unit 14 identifies the movable object, and also announces the progressable route information determined by the progressable route information determination unit 18. It is possible.
Further, the method of notification is not particularly limited, and examples thereof include voice and vibration.

Further, the eyeglass device 2 has a control unit 20.
Here, in the control unit 20, the distance measured by the distance measuring unit 10 between the object to which the image pickup direction of the image pickup camera 4 is directed and the image pickup camera 4 detected by the direction detection unit 9 is equal to or less than a predetermined value, for example, 2 m. When the object detected by the object detection unit 5 has a vertical line segment extending in a predetermined length or more, for example, 1 m or more in the vertical direction, the control for causing the notification unit 19 to notify a warning is as follows. It is possible.

Alternatively, in the control unit 20, the distance measured by the distance measuring unit 10 between the object to which the image pickup direction of the image pickup camera 4 is directed and the image pickup camera 4 detected by the direction detection unit 9 is a predetermined value or less, for example, 2 m or less. And when the object detected by the object detection unit 5 has a vertical line segment extending in the vertical direction by a predetermined length or more, for example, 1 m or more, the distance measuring unit 10 measures the object and images the image. It is possible to control to notify the content of guiding in a direction in which the distance to the camera 4 is larger than a predetermined value, for example, 2 m.

Further, the control unit 20 takes an image of the image pickup camera 4 when the distance between the first crossing line segment and the second crossing line segment measured by the distance measuring unit 10 is larger than a predetermined value, for example, about 2.5 m. It is possible to control the notification unit to notify the content of guiding the left direction change, which is the direction to the left, and the content of guiding the image pickup camera 4 to the right direction, which is the direction to the right.

Further, the control unit 20 can control the notification unit 19 to notify that a step is present when the continuously measured distance changes based on the reflection of the laser irradiated diagonally downward.

Further, the server 3 has a position information comparison unit 21.
Here, the position information comparison unit 21 is the map coordinates created by the object map coordinate creation unit 12 corresponding to the Braille block when the stationary object is specified as a Braille block as a result of the identification by the object identification unit 14. It is possible to compare the latitude / longitude position information of the above with the current position information detected by the current position information detecting unit 8.

Further, the server 3 has a comparison result transmission unit 21A capable of transmitting the comparison result of the position information comparison unit 21 to the eyeglass device 2 via the Internet 30.
On the other hand, the eyeglass device 2 has a comparison result receiving unit 21B capable of receiving the transmitted comparison result.

Further, the object identification unit 14 has a plurality of Braille characters based on the stationary object information recorded by the detected object information database 13 when the stationary object is specified as a Braille block as a result of the identification by the object identification unit 14. It is possible to specify the meaning of a Braille block connected body, which is a structure in which blocks are connected to each other.

Further, the control unit 20 can control the notification unit 19 to notify the guidance to the position of the Braille block when the position information of the Braille block and the current position information do not match as a result of the comparison by the position information comparison unit 21. be.

Further, the server 3 has an object arrangement information creating unit 22.
Here, the object arrangement information creating unit 22 has the map coordinates of the stationary object created by the object map coordinate creating unit 12, the specific result of the object specifying unit 14 identifying the stationary object, and the map recorded by the map information database 11. Based on the information, it is possible to create object placement information, which is information regarding the placement of the specified object on the map.

Further, the server 3 has an object arrangement information database 23 capable of recording the object arrangement information created by the object arrangement information creation unit 22.
Here, the object arrangement information database is an example of the object arrangement information recording unit.

Further, the server 3 has a local destination information determination unit 24.
Here, the local destination information determination unit 24 is the information related to the destination information among the voice, the destination information determined by the destination information determination unit 16, and the map information recorded by the map information database 11. It is possible to determine local destination (intermediate destination) information, which is more local information than destination information, based on the location-related map information.

Further, the route information determination unit 17 has the local destination information determined by the local destination information determination unit 24 instead of the destination information determined by the destination information determination unit 16, and the current position information detection unit 8 has detected the current location information. The route information can be determined based on the position information.

Further, the server 3 has a progress history database 25.
Here, in the progress history database 25, the current position information detection unit 8 detects the progressable route information determined by the progressable route information determination unit 18 and the progressable route of the progressable route information when the pedestrian advances. It is possible to record the current position information in association with the captured image when the current position information is detected.

Further, the server 3 has a search unit 26.
Here, the search unit 26 can search the map information recorded by the map information database 11 and the object placement information recorded by the object placement information database 23 based on the voice.

Further, the server 3 has a search result transmission unit 26A capable of transmitting the result obtained by the search unit 26 to the eyeglass device 2 via the Internet 30.

On the other hand, the glasses device 2 has a search result receiving unit 26B capable of receiving the transmitted search result.
Further, the notification unit 19 can announce the search result of the search unit 26.

Further, the eyeglass device 2 has an image cutting unit 27 capable of cutting out images of each of a movable object and a stationary object detected by the object detecting unit 5 from the captured image.
Further, the eyeglass device 2 has a cut-out image transmission unit 27A capable of transmitting the cut-out image cut out by the image cut-out unit 27 to the server 3 via the Internet 30.

On the other hand, the server 3 has a cutout image receiving unit 27B capable of receiving the transmitted cutout image.

Further, the server 3 has a teacher information database 28.
Here, the teacher information database 28 includes a cut-out image of a movable object and a cut-out image of a stationary object cut out by the image cutting unit 27, and a specific result in which the movable object and the stationary object are specified by the object specifying unit 14. It is possible to record teacher information (for example, an object name or an object ID) which is information obtained in association with.

Further, the sidewalk progress support system of the present invention does not necessarily have to be provided with a notification unit as long as the pedestrian can walk on the sidewalk according to the progressable route information. For example, the guided type with wheels that the pedestrian can walk while grasping. It is also possible to allow pedestrians to follow the progressable route information as a result of the robot traveling on the sidewalk according to the progressable route information.

It should be noted that if there is a notification unit, a weak walking person can know the route information that can be advanced corresponding to the situation on the sidewalk when actually walking, which is preferable.

Further, the sidewalk progress support system of the present invention does not necessarily have to be provided with a control unit, but if the control unit is provided, a weak walking person can easily avoid a collision with an object, which is preferable.

Further, the line segment detecting unit does not necessarily have to be able to detect the first crossing line segment and the second crossing line segment, and the distance measuring unit does not necessarily have to detect the line segment and the line segment based on the captured image. The distance between them does not have to be measurable, and the control unit does not necessarily have to measure the distance between the first intersection line segment and the second intersection line segment measured by the distance measurement unit when the distance between them is larger than a predetermined value. It may not be possible to control the notification unit to notify the content of guiding the left direction change or the right direction change.

However, the line segment detecting unit can detect the first crossing line segment and the second crossing line segment, and the distance measuring unit can measure the distance between the line segments and the line segment based on the captured image. If the control unit can be controlled in this way, the area between the first crossing line segment and the second crossing line segment, that is, the area intersecting the sidewalk is often a roadway, and therefore when crossing the roadway. It is preferable because it allows pedestrians to take an image of the condition of the road.

Further, the sidewalk progress support system of the present invention does not necessarily have to be provided with a position information comparison unit, and the control unit cannot always control the notification unit to notify the guidance to the position of the Braille block. May be good.

However, if there is a position information comparison unit and the control unit can be controlled in this way, it is preferable because even if a weak walking person moves away from the Braille block on the sidewalk, he / she can approach the Braille block.

Further, in the sidewalk progress support system of the present invention, the distance measuring unit necessarily irradiates a laser and further measures the distance from the image pickup camera, that is, the image pickup unit to the object imaged by the image pickup unit based on the reflection of the irradiated laser. It does not have to be possible, and the control unit makes the notification unit notify that there is a step when the continuously measured distance changes based on the reflection of the laser irradiated diagonally downward. It does not have to be controllable.

However, if the distance measuring unit can measure the distance in this way and the control unit can control it in this way, it is preferable that the weak walking person can avoid the step and appropriately cope with the step. ..

Further, the sidewalk progress support system of the present invention does not necessarily have to include an object arrangement information creating unit and an object arrangement information database, that is, an object arrangement information recording unit.

However, it is preferable to have an object arrangement information creating unit and an object arrangement information recording unit because it is possible to record what kind of object is actually arranged on the sidewalk in which place so that it can be searched.

Further, the sidewalk progress support system of the present invention does not necessarily have to include a local destination information determination unit, and the route information determination unit does not necessarily have a local destination information determined by the destination information determination unit. It may not be possible to determine the route information based on the local destination information determined by the destination information determination unit and the current position information detected by the current position information detection unit.

However, if there is a local destination information determination unit and the route information determination unit can determine in this way, it is preferable because the destination can be determined in more detail and the pedestrian's wishes can be met.

Further, the sidewalk progress support system of the present invention does not necessarily have to include a progress history database, that is, a progress history recording unit.
However, if there is a progress history recording unit, it is possible to record the actual walking of a pedestrian to the destination and use it as a reference when the same pedestrian or another pedestrian walks on the same route. preferable.

Further, the sidewalk progress support system of the present invention does not necessarily have to include a search unit, and the notification unit does not necessarily have to be able to notify the search result of the search unit.
However, if there is a search unit and the notification unit can notify in this way, the pedestrian can actually know what kind of object is not only the information contained in the map information but also the information around the place where he is currently located. It is preferable because it is possible to search and know whether or not is arranged.

Further, the sidewalk progress support system of the present invention does not necessarily have to include an image cutting unit and a teacher information database, that is, a teacher information recording unit.
However, it is preferable to have an image cutting unit and a teacher information recording unit because deep learning can be performed to improve the accuracy of artificial intelligence used when identifying an object.

Further, the sidewalk progress support software of the present invention for executing such a sidewalk progress support system is software for making an information processing device function as a server having the above-mentioned functions.

Further, the example shown in FIG. 1 is an example, and the functions of the eyeglass device and the server are not limited thereto.
For example, the server may have an object detection unit, a line segment detection unit, and a sidewalk determination unit. In this case, the sidewalk progress support software of the present invention for executing the sidewalk progress support system is an information processing device. Is software for further functioning as a server having the functions of an object detection unit, a line segment detection unit, and a sidewalk determination unit.

Further, for example, the server may have only a function of searching a map route and a function of accumulating information, and the glasses device may have other functions for the sidewalk progress support system of the present invention.

Next, an example of the flow of walking support for a pedestrian using the sidewalk progress support system of the present invention will be described.

The server 3 takes in the map information including the map coordinate system (global map), and the map information database 11 of the server 3 records the map information.

Further, the current position information detection unit 8 of the glasses device 2 receives signals provided from a plurality of GPS satellites constituting GPS (Global Positioning System), for example, every 0.5 to 1 second. The current position information of the image pickup camera 4 possessed by the glasses device 2, that is, the latitude and longitude, which is the current position information of the pedestrian wearing the glasses device 2 having the image pickup camera 4, is detected.

Further, the current position information transmission unit 8A of the eyeglass device 2 transmits the current position information of the pedestrian detected by the current position information detection unit 8 to the server 3 via the Internet 30.
Further, the current position information receiving unit 8B of the server 3 receives the current position information of the pedestrian from the glasses device 2.

A pedestrian aloud, for example, a place name or a building name of a place he / she wants to go to, and the voice transmission unit 16A of the glasses device 2 transmits the voice of the place name or the building name to a server via the Internet 30. Send to 3.

Further, the voice receiving unit 16B of the server 3 receives the voice of the place name or the building name of the place where the pedestrian wants to go.
Then, the destination information determination unit 16 of the server 3 determines the destination information based on the voice received by the voice reception unit 16B and the map information recorded by the map information database 11.

Further, the route information determination unit 17 of the server 3 determines the destination and the current position based on the destination information determined by the destination information determination unit 16 and the current position information detected by the current position information detection unit 8. Determine the route information between.

Further, the server 3 moves with the current position of the pedestrian as the origin, the front direction of the pedestrian as the y-axis, and the 90-degree right-hand direction and the 90-degree left-hand direction as the x-axis with respect to the front direction. Set the coordinate system (local map).

Further, the server 3 reads a CNN (Convolutional Neural Network) algorithm, which is an artificial intelligence for identifying an object that hinders walking.

FIG. 2A is a schematic view showing a first example of a captured image in a sidewalk progress support system to which the present invention is applied, and FIG. 2B is an augmented reality of the captured image shown in FIG. 2A. It is a schematic diagram which shows an example of an image.

The image pickup camera 4 captures the landscape in front of the pedestrian, for example, every 0.5 to 1 second, and obtains a frame image 41. If the frame image 41 is large, a reduced image having a width of 500 pixels and a height of 300 lines is created, for example, by a pyramid image creation method.

Further, the direction detection unit 9 detects the image pickup direction of the image pickup camera 4.
Further, the image pickup direction transmission unit 9A of the eyeglass device 2 transmits information regarding the image pickup direction detected by the direction detection unit 9 to the server 3 via the Internet 30.

Further, the image pickup direction receiving unit 9B of the server 3 receives information regarding the image pickup direction from the eyeglass device 2.

Then, the distance measuring unit 10 irradiates the laser in the left-right direction and forward of the pedestrian, for example, diagonally downward toward the road surface in the range from the pedestrian to 3 m, and horizontally in the range from the pedestrian to 10 to 15 m. The oblique irradiation laser irradiates from left to right as much as possible, and the width of the sidewalk at that distance is also estimated.

Further, the distance measuring unit 10 measures the distance from the image pickup camera 4 to an object such as a telephone pole imaged by the image pickup camera 4 based on the reflection of the irradiated horizontal laser.

Further, the measurement distance transmission unit 10A of the eyeglass device 2 transmits information regarding the distance measured by the distance measurement unit 10 to the server 3 via the Internet 30.
Further, the measurement distance receiving unit 10B of the server 3 receives information on the distance from the spectacle device 2.

In addition, a long line segment such as a tall pole or a vertical wall of a building is detected in the frame image 41 by the Hough transform.

The average angle of the inclination of the detected long line segment is calculated, and the rotation correction is performed with the center of the frame image 41 as the rotation center so that the long line segment becomes vertical.
That is, as shown in FIG. 2A, the area where the frame image 41 and the rotation-corrected image are misaligned and do not overlap is displayed in black.

This "rotation-corrected image" will be used as an input image in the subsequent processing, and this input image will be referred to as a "captured image".

Then, the captured image transmission unit 4A of the glasses device 2 transmits the captured image to the server 3 via the Internet 30.
Further, the captured image receiving unit 4B of the server 3 receives the captured image from the glasses device 2.

The captured image 40 shown in FIG. 2A is an example of an image in which the right curb 42, the left curb 43, the sidewalk 44, the telephone pole 45, the passerby 46, and the like are captured.

Next, the object detection unit 5 performs "HOG (Histogram of Oriented Gradients) person detection processing" on the captured image 40, and detects a passerby 46 which is a movable object in the captured image 40. , As shown in FIG. 2B, the person candidate existence area rectangle 51 surrounding the detected passerby 46 is displayed in the augmented reality image 47.

Then, the object identification unit 14 collates the numerical information of the detected area rectangle of the passerby 46, for example, with the condition of the person recorded by the detected object information database, for example, the range of size, and the size is within the range. Further, in order to further improve the accuracy, CNN is used to calculate the feature amount that the detected passerby 46 is a person, and if this feature amount is equal to or more than the specified value, it is considered to be a person. Identify.

Further, the object detection unit 5 is stationary based on a vertical line segment 52, which is a line segment having a length longer than a predetermined length extending in the vertical direction within a range of a vertical angle of ± 5 degrees, detected in the captured image 40 by Huff conversion. The telegraph column 45, which is an object, is detected.
That is, the object detection unit 5 detects the telegraph column 45 in which the distance between the vertical line segment 52 and the vertical line segment 52 is within a predetermined range in the captured image 40, and as shown in FIG. 2 (b). In addition, the vertical line segment 52 is displayed in the augmented reality image 47 along the direction in which the detected telegraph column 45 extends.

Then, the object identification unit 14 finds a numerical value in the width direction of the detected telegraph column 45, that is, the distance between the vertical line segment 52 and the vertical line segment 52, the length of the vertical line segment 52, and the detected object information database 13. The recorded conditions of a walking obstacle having a vertical line segment characteristic, for example, the numerical range in the width direction and the numerical range of the vertical line segment length are collated.

Further, the object identification unit 14 collates the detected concentration change in the width direction of the telegraph pole 45 with the concentration change in the width direction of the gait obstacle having the vertical line segment characteristic recorded by the detected object information database 13. ..
As a result of collation, when the numerical value in the width direction and the density change in the width direction of the detected telephone pole 45 match, the object identification unit 14 uses the detected telephone pole 45 as a walking obstacle having a vertical line component characteristic. Identify as being, or identify as a telephone pole.

In particular, if the value of the vertical line segment of the detected object is larger than the specified value in the width direction and the upper part of the detected object is located in the upper half of the captured image 40, the object identification unit 14 Identifies the detected object as a streetlight, traffic light, telephone pole, or the like.
By examining the concentration change in the width direction, it is possible to identify whether the object is a columnar object having an object only near the center line or a wall-shaped (plate-like) object in which the object continues in the left or right region of the center line.

Then, the specific object transmission unit 14A of the server 3 transmits information about the object specified by the object identification unit 14 to the eyeglass device 2 via the Internet 30.
Further, the specific object receiving unit 14B of the eyeglass device 2 receives information about the object from the server 3.

Further, although not shown, the object detection unit 5 extracts a low-saturation region, for example, a saturation value range [0, 5] pixels from the captured image 40 by a binarization process and performs a labeling process to obtain a minute amount. Blobs (lumps) and oversized blobs (lumps) are removed, and the right curb candidate existence area rectangle and the left curb candidate existence area rectangle surrounding the remaining area are displayed.

Then, the object identification unit 14 determines the numerical information of the detected right curb 42, for example, the size and saturation, and the numerical information of the detected left curb 43, for example, the size and saturation, and the detected object. The size and saturation are within the range by collating with the conditions of the low-saturation walking obstacle recorded by the information database 13, for example, the size range and the saturation range, and CNN is used to further improve the accuracy. Then, the characteristic amount that the detected right side curb 42 and the left side curb 43 are curbs is calculated, and when the characteristic amount is equal to or more than the specified value, it is specified as a curb.

Further, the line segment detection unit 6 captures images from the right inclined line segment 49 extending inclined from the lower portion of the captured image 40 toward the substantially central region of the captured image 40 and the lower portion of the captured image 40 in the captured image 40. With the left inclined line segment 50, which extends inclined toward the substantially central region of the image 40 and the distance between the lower portion of the captured image 40 and the right inclined line segment 49 becomes shorter as it approaches the substantially central region of the captured image 40. Is detected.

In general, the width of the sidewalk is often constant in the range of 20 to 30 m in the traveling direction, and when the sidewalk is imaged facing the same direction as the sidewalk extends, the alignment on the left and right sides of the sidewalk is left and right in the captured image. It is reflected as a diagonal line of.
The left and right diagonal lines appear to intersect at one point near the center of the captured image. This intersection is called the vanishing point.

Further, the right inclined line segment 49 is a line segment detected based on the right curb 42, and the left inclined line segment 50 is a line segment detected based on the left curb 43.
Further, the right inclined line segment is an example of the first line segment, and the left inclined line segment is an example of the second line segment.

Therefore, as shown in FIG. 2B, the line segment detection unit 6 displays the right inclined line segment 49 along the right curb 42 and the left inclined line segment along the left curb 43 in the augmented reality image 47. Display 50.
Further, since the right side inclined line segment 49 and the left side inclined line segment 50 intersect near the center of the augmented reality image 47 as shown in FIG. 2B, the vanishing point 48 is located near the center of the augmented reality image 47. It is displayed.

Then, the sidewalk determination unit 7 determines the area between the right side slope line segment 49 and the left side slope line segment 50 as a sidewalk.

Further, in the present invention, if the line segment detection unit can detect the first line segment and the second line segment, it is based on the detected vanishing point after detecting the vanishing point near the center of the captured image. It is also possible to detect the first line segment and the second line segment.

Further, the determination sidewalk transmission unit 7A of the glasses device 2 displays information about the sidewalk determined by the sidewalk determination unit 7, for example, an augmented reality image displaying the first line segment and the second line segment sandwiching the sidewalk on the Internet 30. It is transmitted to the server 3 via.
Further, the determination sidewalk receiving unit 7B of the server 3 receives information about the sidewalk from the glasses device 2.

FIG. 3 is a schematic diagram showing an example of a dynamic coordinate system and a map coordinate system in a sidewalk progress support system to which the present invention is applied.

The object map coordinate creation unit 12 calculates the positions of the right curb 42, the left curb 43, the telephone pole 45, and the passerby 46 detected by the object detection unit 5 as the coordinate values of the dynamic coordinate system 57.

That is, the position of the pedestrian is the origin 56, the imaging direction detected by the direction detection unit 9, that is, the front direction of the pedestrian is the y-axis 54, and the right-hand direction 90 degrees with respect to this front direction is the x positive axis. A dynamic coordinate system with a radius of about 20 to 30 m from the origin, which is set to 55, is set.
Further, whether the right curb 42, the left curb 43, the telephone pole 45, and the passerby 46 captured by the image pickup camera 4 are located on the right side or the left side of the y-axis in the front direction of the pedestrian is an captured image. It is judged from.

Further, the distance between the image pickup camera 4 and the right side edge stone 42 measured by the distance measuring unit 10, the distance between the image pickup camera 4 and the left side edge stone 43, and the distance between the image pickup camera 4 and the telegraph column 45. And the angle, and the distance and the direction angle from the origin of the dynamic coordinate system can be known based on the distance and the angle between the image pickup camera 4 and the passerby 46.
The angle referred to here is an angle at which the object can be seen from the front direction, and is a direction angle of the obstacle object obtained by adding the front direction angle and the left-right angle.

Therefore, the object map coordinate creation unit 12 can calculate the positions of the right curb 42, the left curb 43, the telephone pole 45, and the passerby 46 detected by the object detection unit 5 as the coordinate values of the dynamic coordinate system 57.

Further, since the current position information detection unit 8 detects the latitude and longitude, which is the current position information of the pedestrian, for example, every 0.5 to 1 second, that is, every time the image pickup camera 4 takes an image, the dynamic coordinate system 57 You can see the latitude and longitude of the origin.
Therefore, the object map coordinate creation unit 12 uses the current position information detected by the current position information detection unit 8 as the origin, and the coordinate values of the dynamic coordinate system 57 are included in the map information recorded by the map information database 11. Converted to the latitude and longitude of 58, the map coordinates of the right side edge stone 42, the left side edge stone 43, the telegraph pillar 45, and the passerby 46 are created.

Further, the server 3 can save the created object name and its map coordinates every time the object map coordinate creating unit 12 creates the map coordinates of the detected stationary object or movable object.

Further, the sidewalk determination unit 7 determines the area between the right side inclined line segment 49 and the left side inclined line segment 50 as a sidewalk, and the right side inclined line segment 49 is a line detected based on the right curb 42. Since the left inclined line segment 50 is a line segment detected based on the left curb 43, the region between the right curb 42 and the left curb 43 in FIG. 3 is the sidewalk region 53.

Further, the route information determined by the route information determination unit 17 is information determined based on the map information, but the telephone pole 45 and the passerby 46 are present in the sidewalk area 53, and the map information includes this information. Such information is not included, and such information is not included in the route information. Therefore, if walking navigation is performed based on such route information, the actual walking will be affected.

Therefore, the progressable area position information detection unit 15 of the server has the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit 12 corresponding to the right side inclination line 49 and the left side inclination line 50, respectively, and the right side inclination. The sidewalk area 53 is based on the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit 12 corresponding to the telegraph pillar 45 and the passerby 46 located between the line segment 49 and the left inclined line segment 50. Detects the position information of the progressable area in the area.

Then, the progressable route information determination unit 18 of the server determines the destination and the present based on the route information determined by the route information determination unit 17 and the progressable area position information detected by the progressable area position information detection unit 15. Determine the travelable route information to and from the location.
That is, the route information changed in order to avoid an object that hinders walking, such as a telephone pole or a passerby, is determined.

Further, the progressable route information transmission unit 18A of the server 3 transmits the progressable route information determined by the progressable route information determination unit 18 to the glasses device 2 via the Internet 30.

On the other hand, the progressable route information receiving unit 18B of the glasses device 2 receives the progressable route information from the server 3.

Then, the notification unit 19 of the glasses device 2 announces the travelable route information by voice. Specifically, for example, the progressable route "I created a route. The route is ○○○○." Will be explained.
Further, since the object identification unit 14 identifies a person who is a movable object, that is, a passerby 46, the existence of the person is also notified by voice.

Further, when the image pickup direction of the image pickup camera 4 detected by the direction detection unit 9 is facing the telephone pole 45, that is, when the pedestrian is facing the telephone pole 45 and the front direction of the pedestrian is facing the telephone pole 45. When the distance measured by the distance measuring unit 10 between the telephone pole 45 and the image pickup camera 4 is a predetermined value, for example, 2 m or less, the telephone pole 45 has a vertical portion 52 extending in the vertical direction by a predetermined length or more. Therefore, the control unit 20 of the glasses device 2 controls the notification unit 19 to notify the warning.
Alternatively, in this case, the distance measuring unit 10 controls to notify the content of guiding in the direction in which the measured distance becomes larger than a predetermined value.

Further, the image cutting unit 27 of the glasses device 2 cuts out the images of the right side curb 42, the left side curb 43, the telephone pole 45, and the passerby 46 detected by the object detection unit 5 from the captured image.

Then, the cut-out image transmission unit 27A of the eyeglass device 2 transmits the image cut out by the image cut-out unit 27 to the server 3 via the Internet 30.

Further, the cut-out image receiving unit 27B of the server 3 receives the cut-out images of the right side curb 42, the left side curb 43, the telephone pole 45, and the passerby 46 cut out by the image cutting unit 27 from the glasses device 2.

Then, the teacher information database 28 of the server 3 receives cutout images of the right curb 42, the left curb 43, the telephone pole 45 and the passerby 46, and the right curb 42, the left curb 43, the telephone pole 45 and the passerby 46. Is associated with the specific result specified by the object identification unit 14, and is recorded as teacher information.

Further, the object placement information creating unit 22 of the server 3 has map coordinates of the right curb 42, the left curb 43 and the telegraph pillar 45 created by the object map coordinate creating unit 12, and the object specifying unit 14 has the right curb 42 and the left curb 43. And, based on the specific result of specifying the telegraph pillar 45 and the map information recorded by the map information database 11, the object arrangement information on the map of the identified right curb 42, left curb 43 and telegraph pillar 45 is created.
Then, the object arrangement information database 23 of the server records the object arrangement information.

Further, for example, a pedestrian limits a place in Kokura Station, such as "the sea side of Kokura Station" as the building name of the place he / she wants to go, and transmits the voice to the server 3 by the voice transmission unit 16A. If so, first, the destination information determination unit 16 of the server 3 determines the destination information "Kokura Station".

Next, the local destination information determination unit 24 of the server determines the local destination information.
That is, the voice "sea side of Kokura Station" received by the voice receiving unit 16B of the server 3, the destination information "Kokura Station" determined by the destination information determination unit 16, and the map information recorded by the map information database 11. Of these, based on the destination-related map information related to the destination information "Kokura Station", that is, the map information around Kokura Station, the local destination information "Kokura Station Sea" is more local than the destination information "Kokura Station". "Side" is decided.

Further, the route information determination unit 17 is locally based on the local destination information "sea side of Kokura station" instead of the destination information "Kokura station" determined by the destination information determination unit 16 and the current position information. Determine route information between the destination and the current location.

Further, the progressable route information determination unit 18 determines the progressable route information based on the route information between the local destination and the current position determined by the route information determination unit 17 and the progressable area position information. ..

Further, since the current position information detection unit 8 detects the current position information of the pedestrian each time the image pickup camera 4 takes an image, the image pickup image transmission unit 4A captures the captured image when the current position information is detected. It can be sent to the server 3.

Then, when the pedestrian actually travels on the travelable route, the current position information detection unit 8 detects the current position information during the progress, the current position information transmission unit 8A transmits the current position information to the server 3, and the current position information is transmitted to the server 3. The captured image transmission unit 4A transmits the captured image when the current position information is detected to the server 3.

The current position information receiving unit 8B of the server 3 receives the current position information from the spectacle device 2, and the captured image receiving unit 4B of the server receives the captured image when the current position information is detected from the spectacle device 2.

Therefore, in the progress history database 25 of the server 3, the progressable route information determined by the progressable route information determination unit 18, the current position information when the pedestrian travels on the progressable route, and the current position information are detected. It is recorded in association with the captured image of the time.

Further, when a pedestrian asks, for example, "Where is this?", The voice transmission unit 16A of the glasses device 2 transmits the question voice to the server 3.
Then, the voice receiving unit 16B of the server 3 receives the question voice, and the question analysis unit 29 of the server further analyzes the question voice.

Further, the current position information of the pedestrian is detected by the current position information detection unit 8 of the spectacle device 2 every 0.5 to 1 second, for example, and the current position information transmission unit 8A of the spectacle device 2 transmits the current position information to the server 3. , The server 3 refers to the map information recorded by the map information database 11, and the voice response unit 16C of the server 3 transmits the response content, for example, "XX2-chome, near the XX restaurant." To the glasses device 2 by voice. do.

Then, the voice receiving unit 16D of the glasses device 2 receives the answer contents from the server.

Further, when a pedestrian asks, for example, "How far is it to the south side of Kokura Station?", The voice response unit 16C of the server 3 receives the current position information of the pedestrian and the determined route information. Answer content For example, "It is about 150m." Is transmitted by voice to the glasses device 2.

Further, when a pedestrian asks, for example, "Is there a vending machine nearby?", In this case, it is treated as a search, and the search unit 26 of the server 3 displays the map information recorded by the map information database 11 and an object. The object placement information recorded by the placement information database 23 is searched.

Then, the search result of the search unit 26, for example, "I'm sorry, the answer cannot be found." Is transmitted by the search result transmission unit 26A of the server 3 to the glasses device 2 by voice via the Internet 30.
Further, the search result receiving unit 26B of the glasses device 2 receives the search result, and the notification unit 19 announces the search result by voice.

4 (a) is a schematic view showing a second example of the captured image in the sidewalk progress support system to which the present invention is applied, and FIG. 4 (b) is an augmented reality of the captured image shown in FIG. 4 (a). It is a schematic diagram which shows an example of an image.

Similar to the example shown in FIG. 2A, the image pickup camera 4 captures the landscape in front of the pedestrian, for example, every 0.5 to 1 second, and obtains a frame image 61.

Further, as in the example shown in FIG. 2A, the average angle of the inclination of the detected long line segment is calculated, and the rotation is corrected with the center of the frame image 61 as the rotation center so that the long line segment becomes vertical. I do.

The captured image 60 shown in FIG. 4A is an example of an image in which a pedestrian crossing 62, a sidewalk 63, a first passerby 64, a second passerby 65, and the like are captured.

Next, the object detection unit 5 performs "HOG person detection processing" on the captured image 60, and detects the first passerby 64 and the second passerby 65, which are movable objects, in the captured image 60. As shown in FIG. 4B, the existence of the first person candidate existence area rectangular 70 surrounding the detected first passerby 64 and the existence of the second person candidate surrounding the detected second passerby 65. The area rectangle 71 is displayed in the augmented reality image 66.

Then, the object identification unit 14 determines the numerical information, for example, the size of the area rectangle in which the first passerby 64 and the second passerby 65 are detected, and the condition, for example, the size of the person recorded by the detected object information database 13. The size is within the range by collating with the range of, and further, the feature amount that the detected first passerby 64 and the second passerby 65 are people is calculated by using CNN. , If this feature amount is more than the specified value, it is identified as a person.

Further, although not shown, a vertical line segment extending in the vertical direction within a range of a vertical angle of ± 5 degrees detected in the captured image 60 by the object detection unit 5 is a line segment having a predetermined length or longer. Detects a building that is a stationary object facing the sidewalk based on.

Then, the object identification unit 14 collates the numerical information, for example, of the detected rectangular existing area of the building with the building condition, for example, the range of the size recorded by the detected object information database 13, and the size is within the range. Furthermore, in order to improve the accuracy, CNN is used to calculate the feature amount that the detected building is certainly a building, and if this feature amount is equal to or more than the specified value, it is identified as a building. do.

Further, the line segment detection unit 6 captures images from the right inclined line segment 68 extending inclined from the lower part of the captured image 60 toward the substantially central region of the captured image 60 and the lower part of the captured image 60 in the captured image 60. With the left inclined line segment 69, which extends inclined toward the substantially central region of the image 60 and the distance between the lower portion of the captured image 60 and the right inclined line segment 68 becomes shorter as it approaches the substantially central region of the captured image 60. Is detected.

Further, the right side inclined line segment 68 is a line segment detected based on the building facing the sidewalk, and the left side inclined line segment 69 is a right side inclined line segment with respect to the vanishing point existing near the center of the captured image 60. It is a line segment detected based on the same angle as the angle of the minute 68.

Therefore, as shown in FIG. 4B, the vanishing point 67 is displayed near the center of the augmented reality image 66, and the right side inclined line segment 68 is displayed along the building facing the side surface in the augmented reality image 66. The left side inclined line segment 69 having the same angle as the right side inclined line segment 68 with respect to the vanishing point 67 is displayed.

Then, the sidewalk determination unit 7 determines the area between the right side slope line segment 68 and the left side slope line segment 69 as a sidewalk.

Further, the line segment detection unit 6 further detects the first crossing line segment 72 and the second crossing line segment that intersect each of the right side inclined line segment 68 and the left side inclined line segment 69 in the captured image 60. do.
That is, the first crossing line segment 72 and the second crossing line segment are line segments detected based on the sidewalk boundary installation.
The second crossing line segment is not shown.

Further, the distance measuring unit 10 measures the distance between the first crossing line segment 72 and the second crossing line segment, that is, the road width, based on the captured image 60.

Further, in the control unit 20, the distance between the first crossing line segment 72 and the second crossing line segment measured by the distance measuring unit is a predetermined value, for example, about 2.5 m, which is a distance that one automobile can pass. When the size is larger, the content of guiding the image pickup camera 4 to the left direction change and the content of guiding the image pickup camera 4 to the right direction change to the right direction are announced. Control is performed to notify the unit 19.

Further, although not shown, the object detection unit 5 performs HSV color conversion of the captured image 60 and separates it into a hue image, a saturation image, and a brightness image, and corresponds to the yellow color of the braille block from the hue image and the saturation image. A binarization process for extracting the specified hue range and saturation range is performed, and a labeling process is performed on the result to remove minute blobs (lumps). Display a rectangle.

Then, the object identification unit 14 collates the detected hue and saturation of the Braille block with the conditions of the Braille block recorded by the detected object information database 13, for example, the hue range and the saturation range, and the hue and saturation. If is within range, identify it as a Braille block.

Further, the object identification unit 14 is a shape pattern of a Braille block connected body recorded by the detected object information database 13 in which a plurality of Braille blocks are connected to each other, for example, a triangular pyramid, a T-shape, a left branch, a right branch, or these. The meaning of the Braille block connector is specified by collating the complex type of the above with the shape pattern of the detected Braille block connector of the plurality of Braille blocks.

Then, the position information comparison unit 21 compares the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit 12 corresponding to the Braille block with the current position information detected by the current position information detection unit 8.

Further, the control unit 20 controls the notification unit 19 to notify the guidance to the position of the Braille block when the position information of the Braille block and the current position information do not match as a result of the comparison by the position information comparison unit 21.

Further, although not shown, the object detection unit 5 performs HSV color conversion of the captured image 40 or the captured image 60 and separates them into a hue image, a saturation image, and a brightness image, and the saturation image is separated into a high saturation region, for example, [[ 100, 255] A binarization process for extracting pixels is performed, and a labeling process is performed on the result to remove minute blobs (lumps) or oversized blobs, and a high-saturation walking disorder surrounding the remaining area. Display the object candidate existence area rectangle.

Then, the object identification unit 14 determines the numerical information of the detected high-saturation gait obstacle, for example, the existence area rectangle of the roadside tree, for example, the size and saturation, and the condition of the high-saturation gait obstacle, for example, the existence area recorded by the detected object information database 13. As a result of collating with the size range and saturation range of the rectangle, the size and saturation are within the range, and CNN is used to improve the accuracy, and the detected roadside tree is a tree. The feature amount is calculated, and if this feature amount is equal to or more than the specified value, it is identified as a tree.

FIG. 5 is a schematic view showing an example of an augmented reality image of the captured image of the third example in the sidewalk progress support system to which the present invention is applied.

Similar to the example shown in FIG. 2A, the image pickup camera 4 captures the landscape in front of the pedestrian, for example, every 0.5 to 1 second, and obtains a frame image 73.

Further, as in the example shown in FIG. 2A, the average angle of the inclination of the detected long line segment is calculated, and the rotation is corrected with the center of the frame image 73 as the rotation center so that the long line segment becomes vertical. I do.

The captured image at this time is an image in which the first Braille block 75, the second Braille block 76, the sidewalk 77, the pedestrian crossing 83, and the like are captured.

Next, the object detection unit 5 performs HSV color conversion of the captured image, separates it into a hue image, a saturation image, and a lightness image, and a predetermined hue range corresponding to the yellow color of the Braille block from the hue image and the saturation image and Performs binarization processing to extract the saturation range.

Then, the object detection unit 5 performs a labeling process on the result to remove the minute blobs (lumps), and then, as shown in FIG. 5, the remaining region, that is, the detected first Braille block 75 is removed. The enclosed first Braille block candidate existence area rectangle 81 and the second Braille block candidate existence area rectangle 82 surrounding the detected second Braille block 76 are displayed in the augmented reality image 74.

Then, the object identification unit 14 determines the hue and saturation of each of the detected first Braille block 75 and the second Braille block 76, and the conditions of the Braille block recorded by the detected object information database 13, such as the range and color of the hue. If the hue and saturation are within the range by collating with the range of degrees, it is specified as a Braille block.

Further, the object identification unit 14 includes a shape pattern of a Braille block connector recorded by the detected object information database 13, for example, a triangular pyramid, a T-shape, a left branch, a right branch, or a composite type thereof, and a plurality of detected firsts. The meaning of the Braille block connection is specified by collating with the shape pattern of each Braille block connection of the Braille block 75 and the second Braille block 76.

Further, although not shown, the object detection unit 5 extracts a low-saturation region, for example, a saturation value range [0, 5] pixels from the captured image by binarization processing and performs labeling processing to perform a minute blob (). Remove the lumps and blobs that are too large, and display the left sidewalk boundary installation candidate existence area rectangle that surrounds the remaining area.

Further, the object identification unit 14 collates the numerical information of the detected area rectangular of the left sidewalk boundary installation object, for example, with the range of the condition, for example, the size of the sidewalk boundary installation object recorded by the detected object information database 13. Then, the size is within the range, and CNN is used to further improve the accuracy, and the feature amount that the detected sidewalk boundary installation is certainly the sidewalk boundary installation is calculated, and this feature amount is calculated. If is greater than or equal to the specified value, it is identified as a sidewalk boundary installation.

Further, in the captured image, the line segment detection unit 6 has a right-sided inclined line segment 79 extending inclined from the lower part of the captured image toward the substantially central region of the captured image, and a substantially center of the captured image from the lower part of the captured image. The left side inclined line segment 80 is detected as the distance between the right side inclined line segment 68 and the right side inclined line segment 68 becomes shorter as the image is inclined toward the region and extends from the lower part of the captured image toward the substantially central region of the captured image.

Further, the left inclined line segment 80 is a line segment detected based on the side surface boundary installation object, and the right inclined line segment 79 is a left inclined line segment 80 with respect to a vanishing point existing near the center of the captured image. It is a line segment detected based on an angle as high as the angle of.

Therefore, as shown in FIG. 5, the vanishing point 78 is displayed near the center of the augmented reality image 74, and in the augmented reality image 74, the left side inclined line segment 80 is displayed along the sidewalk boundary installation object, and the vanishing point 78 is displayed. On the other hand, the right side inclined line segment 79 having the same angle as the angle of the left side inclined line segment 80 is displayed.

Then, the sidewalk determination unit 7 determines the area between the right side slope line segment 79 and the left side slope line segment 80 as a sidewalk.

Further, the specific object transmission unit 14A of the server 3 transmits information about the Braille block connector specified by the object identification unit 14 to the eyeglass device 2 via the Internet 30.
Further, the specific object receiving unit 14B of the eyeglass device 2 receives information about the Braille block connector from the server 3.

Then, the control unit 20 controls the notification unit 19 to notify the information about the Braille block connection body, that is, the meaning of the Braille block connection body by voice.
In the case of the Braille block connection shown in FIG. 5, the notification unit 19 specifically announces, for example, "There is a pedestrian crossing Braille block 3 m ahead."

FIG. 6 is a schematic view showing an example of an augmented reality image of the captured image of the fourth example in the sidewalk progress support system to which the present invention is applied.

Similar to the example shown in FIG. 2A, the image pickup camera 4 captures the landscape in front of the pedestrian, for example, every 0.5 to 1 second, and obtains a frame image 84.

Further, as in the example shown in FIG. 2A, the average angle of the inclination of the detected long line segment is calculated, and the rotation is corrected with the center of the frame image 84 as the rotation center so that the long line segment becomes vertical. I do.

The captured image at this time is an image obtained by capturing the sidewalk 86, the first passerby 87, the second passerby 89, and the like.

Next, the object detection unit 5 performs "HOG person detection processing" on the captured image, detects the first passerby 87 and the second passerby 89, which are movable objects, in the captured image, and FIG. As shown in the above, a first person candidate existence area rectangle 88 surrounding the detected first passerby 87 and a second person candidate existence area rectangle 90 surrounding the detected second passerby 89. It is displayed in the augmented reality image 85.

Then, the object identification unit 14 determines the numerical information, for example, the size of the area rectangle in which the first passerby 87 and the second passerby 89 are detected, and the condition, for example, the size of the person recorded by the detected object information database 13. The size is within the range by collating with the range of, and further, the feature amount that the detected first passerby 87 and the second passerby 89 are human beings is calculated by using CNN. However, if this feature amount is equal to or greater than the specified value, it is identified as a person.

Further, although not shown, the object detection unit 5 extracts a low-saturation region, for example, a saturation value range [0, 5] pixels from the captured image by binarization processing and performs labeling processing to perform a minute blob (). Remove the lumps and blobs that are too large, and display the left sidewalk boundary installation candidate existence area rectangle that surrounds the remaining area.

Further, the object identification unit 14 collates the numerical information of the detected area rectangular of the left sidewalk boundary installation object, for example, with the range of the condition, for example, the size of the sidewalk boundary installation object recorded by the detected object information database 13. Then, the size is within the range, and CNN is used to further improve the accuracy, and the feature amount that the detected sidewalk boundary installation is certainly the sidewalk boundary installation is calculated, and this feature amount is calculated. If is greater than or equal to the specified value, it is identified as a sidewalk boundary installation.

Further, in the captured image, the line segment detection unit 6 has a right-sided inclined line segment 92 extending inclined from the lower part of the captured image toward the substantially central region of the captured image, and a substantially center of the captured image from the lower part of the captured image. The left side inclined line segment 93 is detected as the distance between the right side inclined line segment 92 and the right side inclined line segment 92 becomes shorter as the image is inclined toward the region and extends from the lower part of the captured image toward the substantially central region of the captured image.

Further, the left inclined line segment 93 is a line segment detected based on the side wall boundary installation object, and the right inclined line segment 92 is the left inclined line segment 93 with respect to the vanishing point existing near the center of the captured image. It is a line segment detected based on an angle as high as the angle of.

Therefore, as shown in FIG. 6, the vanishing point 91 is displayed near the center of the augmented reality image 85, and in the augmented reality image 85, the left side inclined line segment 93 is displayed along the sidewalk boundary installation object, and the vanishing point 91 is displayed. On the other hand, the right side inclined line segment 92 having the same angle as the angle of the left side inclined line segment 93 is displayed.

Then, the sidewalk determination unit 7 determines the area between the right side slope line segment 92 and the left side slope line segment 93 as a sidewalk.

Further, since the object identification unit 14 identifies a person who is a movable object, that is, the first passerby 87 and the second passerby 89, the notification unit 19 announces the existence of the person by voice.

FIG. 7 is a schematic view showing an example of detecting a step using a spectacle device used in a sidewalk progress support system to which the present invention is applied.

The distance measuring unit 10 of the glasses device 2 constituting the sidewalk progress support system 1 of the present invention irradiates the laser 96 to capture a stationary object imaged by the image pickup camera 4 from the image pickup camera 4, that is, the image pickup camera 4 possessed by the glasses device 2. The distance from the walking surface 100 located diagonally downward can also be measured.

Further, the laser 96 is continuously irradiated while the eyeglass device 2 is operating.

Further, as shown in FIG. 7, when a step 98 lower than the walking surface 100 on which the pedestrian 95 holding the white cane 97 stands is present in front of the pedestrian 95, the walking surface is viewed from the image pickup camera 4. The distance from the image pickup camera 4 to the step surface 101 lowered by the step 98 is longer than the distance to 100.

Therefore, the control unit 20 has a step 98 in the notification unit 19 when the distance measured continuously changes based on the reflection of the laser 96 continuously irradiated diagonally downward, that is, when the distance becomes long. Controls to announce the existence.

Further, FIG. 8 is a schematic view showing an example in which a guide dog robot is used instead of the white cane in the sidewalk progress support system to which the present invention is applied.

In this case, it is conceivable that the guide dog robot 99 has an image pickup camera, a distance measuring unit that irradiates a laser, and a travelable route information receiving unit, and can automatically travel according to the received travelable route information.
Therefore, a visually impaired pedestrian can walk along the progressable path by grasping and walking the guide dog robot 99 as shown in FIG.

Further, the server 3 may be configured to be a cloud server, or the server 3 may be configured to have a cloud server different from the server 3.

Further, when a cloud server different from the server 3 exists, the server 3 is installed for each pedestrian who uses the system of the present invention.
Then, the information recorded by the progress history database 25 of each of the plurality of servers 3, that is, the travelable route information actually traveled by the pedestrian, the current position information when the pedestrian has progressed, and the image capture when the current position information is detected. Each of the plurality of servers 3 can transmit the image to the cloud server and collect a lot of information.

Further, each of the plurality of servers 3 is updated with information on walking obstacles among the objects specified by the object identification unit 14, question information (voice) received from pedestrians, and map information recorded by the map information database 11. A lot of information can be collected by sending the map update information in the case of the case to the cloud server.

That is, various groups and users such as disability support groups in various regions can collect information such as routes and walking obstacles in each region on the cloud server by using the sidewalk progress support system of the present invention. , Various groups and users in different regions can work together to collect information.
Then, based on the collected information, information such as detailed routes and gait obstacles can be prepared.

Therefore, even when a pedestrian using the sidewalk progress support system of the present invention goes from a residential area to another area, that is, a destination area for travel or work, the server 3 corresponding to the pedestrian communicates with the cloud server. Then, information such as detailed routes and pedestrian obstacles in the destination area can be used, and such a cloud cooperation function can improve the convenience and safety of the sidewalk progress support system of the present invention.

FIG. 9A is a schematic view showing an example of a belt device included in the sidewalk progress support system of the present invention, and FIG. 9B is a schematic view showing an example of a pedestrian wearing the belt device.

The sidewalk progress support system 1 of the present invention includes both or one of a voice notification unit for voice notification and a vibration notification unit for vibration notification.
Specifically, for example, the glasses device 2 has a voice notification unit and a vibration notification unit, and a belt device 102 that can be attached to the body of a pedestrian 95 or the like separately from the glasses device 2 has a vibration notification unit 103. I'm doing it.

When the vibration notification unit 103 of the belt device 102 vibrates, for example, the glasses device 2 transmits the notification content to the belt device 102, and the vibration notification unit 103 of the belt device 102 is based on the transmitted notification content. And announce by vibration.
That is, the vibration notification unit 103 of the belt device 102 vibrates when the notification unit 19 of the eyeglass device 2 makes a voice notification.
Therefore, although not shown, the belt device 102 has a receiving unit capable of receiving the notification content.

Further, the vibration notification unit 103 specifically includes a main body 103A and a plurality of vibrating bodies 103B.
Here, the main body 103A has a longitudinal direction, and one side in the longitudinal direction and the other side located on the opposite side to the one side can be detachably connected to each other.
Further, the plurality of vibrating bodies 103B are mounted side by side in the same direction as the longitudinal direction of the main body 103A on one surface of the main body 103A, and can vibrate when notified.

That is, the main body 103A of the vibration notification unit 103 may have an annular shape as shown in FIG. 9 (a) in which one side in the longitudinal direction and the other side opposite to the one side are detachably connected to each other. It is possible, and it forms an annular shape because the main body 103A is attached to the pedestrian's body, for example, the torso.

Further, when the main body 103A of the vibration notification unit 103 forms an annular shape, the main body 103A is attached to the pedestrian so that the vibrating body 103B comes into contact with the pedestrian. At this time, it is assumed that the location of the main body 103A that comes into contact with, for example, the front side of the pedestrian's body is fixed.
Further, as shown in FIG. 9A, when the main body 103A becomes annular, the vibrating body 103B is in a state of being arranged in all directions of 360 degrees.

Further, the vibration mode of the vibrating body 103B changes variously depending on the content of the notification.
Specifically, for example, in the case of a notification content that guides a pedestrian to a left turn, only the vibrating body 103B that contacts the left side of the pedestrian's torso vibrates, and the notification content that guides the pedestrian to a right turn. If so, only the vibrating body 103B that contacts the right side of the pedestrian's torso vibrates.
In addition, if the content of the notification is to stop the pedestrian from walking, for example, only the vibrating body that comes into contact with the rear side of the pedestrian's torso vibrates, causing the pedestrian to have an obstacle to walking such as a passerby, a telegraph column, or a step. For example, all the vibrating bodies vibrate at the same time as long as the content announces the existence of the object.

As shown in FIG. 9B, when a step 98 lower than the walking surface 100 on which the pedestrian 95 holding the white cane 97 stands is present in front of the pedestrian 95, the control unit 20 notifies. Control is performed to notify the unit 19 of the existence of the step 98, and the content of the notification, that is, the existence of the step 98 is transmitted from the glasses device 2 to the belt device 102, and the vibration notification unit 103 of the belt device 102 is transmitted. The notification is made by vibration based on the content of the notification, that is, all the vibrating bodies 103B vibrate at the same time.

As described above, since the sidewalk progress support system of the present invention includes an object identification unit, it is possible to identify an object that actually exists at least in front of a weak walking person walking on the sidewalk.
Further, since the sidewalk progress support system of the present invention includes an object map coordinate creation unit, it is possible to grasp the position of an object actually existing at least in front of a weak walking person walking on the sidewalk in map coordinates.
Further, since the sidewalk progress support system of the present invention includes a progressable route information determination unit, the progressable route of the sidewalk is determined in consideration of the situation on the sidewalk when actually walking, which is not included in the map information. can.

Therefore, the sidewalk progress support system of the present invention can support the progress on the sidewalk of a weak walking person.

Further, the sidewalk progress support software of the present invention also causes the information processing device to function as a means including an object identification unit, an object map coordinate creation unit, and a travelable route information determination unit, so that the information processing device can travel on the sidewalk of a weak walking person. I can help.

1 Sidewalk progress support system 2 Glasses device 3 Server 4 Image pickup camera 4A Image capture image transmission unit 4B Image capture image receiver 5 Object detection unit 5A Detection object transmission unit 5B Detection object receiver unit 6 Line segment detection unit 7 Sidewalk determination unit 7A Decision sidewalk transmission 7B Determination sidewalk receiver 8 Current position information detector 8A Current position information transmitter 8B Current position information receiver 9 Direction detector 9A Imaging direction transmitter 9B Imaging direction receiver 10 Distance measurement unit 10A Measurement distance transmitter 10B Measurement distance Receiver 11 Map information database 12 Object map coordinate creation unit 13 Detected object information database 14 Object identification unit 14A Specific object transmission unit 14B Specific object reception unit 15 Progressable area position information detection unit 16 Destination information determination unit 16A Voice transmission unit 16B Voice receiving unit 16C Voice response unit 16D Voice receiving unit 17 Route information determination unit 18 Progressable route information determination unit 18A Progressable route information transmission unit 18B Progressive route information reception unit 19 Notification unit 20 Control unit 21 Position information comparison unit 21A Comparison Result transmission unit 21B Comparison result reception unit 22 Object placement information creation unit 23 Object placement information database 24 Local destination information determination unit 25 Progress history database 26 Search unit 26A Search result transmission unit 26B Search result reception unit 27 Image cutout unit 27A Off Output image transmission unit 27B Cutout image reception unit 28 Teacher information database 29 Question analysis unit 30 Internet 40 Captured image 41 Frame image 42 Right edge stone 43 Left edge stone 44 Sidewalk 45 Telegraph pillar 46 Passerby 47 Augmented reality image 48 Disappearance point 49 Right tilt Line 50 Left inclined line 51 Person candidate existence area Rectangular 52 Vertical line 53 Sideway area 54 y-axis 55 x Positive axis 56 Origin 57 Dynamic coordinate system 58 Map coordinate system 60 Captured image 61 Frame image 62 Crossing sidewalk 63 Sidewalk 64th 1 Passerby 65 Second Passerby 66 Augmented Reality Image 67 Disappearance Point 68 Right Inclined Line 69 Left Inclined Line 70 First Person Candidate Existence Area Rectangular 71 Second Person Candidate Existence Area Rectangular 72 First Crossing Line 73 Frame image 74 Augmented reality image 75 First Braille block 76 Second Braille block 77 Sidewalk 78 Disappearance point 79 Right tilt line 80 Left tilt Line segment 81 First Braille block candidate existence area rectangle 82 Second Braille block candidate existence area rectangle 83 Crosswalk 84 Frame image 85 Augmented reality image 86 Sidewalk 87 First passerby 88 First person candidate existence area rectangle 89 Second passerby 90 Second person candidate existence area Rectangle 91 Disappearance point 92 Right sidewalk line segment 93 Left sidewalk line segment 95 Pedestrian 96 Laser 97 White cane 98 Step 99 Blind dog robot 100 Walking surface 101 Step surface 102 Belt device 103 Vibration notification unit 103A Main unit 103B Vibrating body

Claims (14)

An image pickup unit that can capture at least the front of a pedestrian,
An object detection unit capable of detecting a moving object which is a moving object and a stationary object which is a stationary object in the captured image obtained by the imaging unit.
In the captured image, a first line segment extending from the lower part of the captured image toward the substantially central region of the captured image and the same as extending from the lower portion of the captured image toward the substantially central region of the captured image. A line segment detection unit capable of detecting a second line segment whose distance from the first line segment becomes shorter as it approaches the substantially central region of the captured image from the lower part of the captured image, and a line segment detection unit.
A sidewalk determination unit that can determine the area between the first line segment and the second line segment as a sidewalk detected by the line segment detection unit in the captured image.
The current position information detection unit capable of detecting the current position information which is the position information of the pedestrian to which the image pickup unit is attached, and the current position information detection unit.
A direction detection unit capable of detecting the imaging direction of the imaging unit, and a direction detection unit.
A distance measuring unit capable of measuring the distance from the imaging unit to an object imaged by the imaging unit based on the captured image, and a distance measuring unit.
A map information recording unit that can record map information, which is information about maps,
The position of the stationary object or the movable object detected by the object detection unit is the distance between the image pickup direction detected by the direction detection unit and the image pickup unit and the stationary object or the movable object measured by the distance measurement unit. Based on the above, it can be calculated as a coordinate value of the dynamic coordinate system with the position of the pedestrian equipped with the image pickup unit as the origin, and the current position information detected by the current position information detection unit is used as the origin. Object map coordinate creation that can create map coordinates of the same stationary object or the same movable object by converting the calculated same coordinate value to the latitude and longitude of the map coordinate system included in the map information recorded by the map information recording unit. Department and
A detection object information recording unit that can record moving object information that is information about moving objects and stationary object information that is information about stationary objects, and
An object capable of identifying the movable object and the stationary object by collating the movable object or the stationary object detected by the object detection unit with the movable object information and the stationary object information recorded by the detected object information recording unit. With a specific part
The latitude / longitude position of the map coordinates created by the object map coordinate creation unit corresponding to each of the first line segment and the second line segment sandwiching the sidewalk when the sidewalk determination unit determines the sidewalk. The information and the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to the stationary object or the movable object located between the first line segment and the second line segment. Based on the progressable area position information detection unit that can detect the position information of the progressable area in the same sidewalk,
A destination information determination unit that can determine destination information, which is information about a destination, based on the voice and the map information recorded by the map information recording unit.
Based on the destination information determined by the destination information determination unit and the current position information detected by the current position information detection unit, route information which is information on a route between the destination and the current position is obtained. Determinable route information determination unit and
A pedestrian who is a route between a destination and a current position based on the route information determined by the route information determination unit and the travelable area position information detected by the travelable area position information detection unit. The progressable route information determination unit that can determine the progressable route information, which is the information about the progressable route,
A sidewalk progress support system including a notification unit capable of notifying the progressable route information determined by the progressable route information determination unit . The sidewalk progress support system according to claim 1 , wherein the notification unit can notify that a movable object exists when the object identification unit identifies a movable object. The distance measured by the distance measuring unit between the object to which the imaging direction of the imaging unit is directed and the imaging unit detected by the direction detecting unit is equal to or less than a predetermined value, and the object detecting unit is When the detected object has a vertical line segment that is a line segment extending in the vertical direction by a predetermined length or more, the control for causing the notification unit to notify a warning or the same object and the same image pickup unit measured by the same distance measurement unit. It is provided with a control unit capable of controlling to notify the content of guiding in the direction in which the distance between the distances becomes larger than a predetermined value.
The sidewalk progress support system according to claim 1 or 2. The line segment detection unit can further detect a first crossing line segment and a second crossing line segment that intersect each of the first line segment and the second line segment in the captured image. And
The distance measuring unit can further measure the distance between the line segments based on the captured image.
When the distance between the first crossing line segment and the second crossing line segment measured by the distance measuring unit is larger than a predetermined value, the control unit causes the imaging direction of the imaging unit to move to the left. The third aspect of claim 3, wherein it is possible to control the notification unit to notify the content of guiding to a left direction change, which is to face, and the content to guide to a right direction change, which is that the image pickup direction of the imaging unit is to turn to the right. Sidewalk progress support system. When the stationary object is identified as a Braille block as a result of the identification of the object identification unit, the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to the Braille block, and the present It is equipped with a position information comparison unit that can compare the current position information detected by the position information detection unit.
When the stationary object is identified as a Braille block as a result of the identification by the object identification unit, the object identification unit has a plurality of Braille characters based on the stationary object information recorded by the detection object information recording unit. It is possible to specify the meaning of the Braille block connection, which is a structure in which blocks are connected to each other.
The control unit can control the notification unit to notify the guidance to the position of the Braille block when the position information of the Braille block and the current position information do not match as a result of comparison by the position information comparison unit. The sidewalk progress support system according to item 3 or claim 4. The distance measuring unit can irradiate a laser, and can further measure the distance from the image pickup unit to an object imaged by the image pickup unit based on the reflection of the irradiated laser.
3. The control unit can control to notify that a step is present in the notification unit when the continuously measured distance changes based on the reflection of the laser irradiated diagonally downward. , The sidewalk progress support system according to claim 4 or 5. It was specified based on the map coordinates of the stationary object created by the object map coordinate creating unit, the specific result of the object specifying unit identifying the stationary object, and the map information recorded by the map information recording unit. An object placement information creation unit that can create object placement information, which is information about the placement of objects on a map,
It is provided with an object arrangement information recording unit capable of recording the object arrangement information created by the object arrangement information creation unit.
The sidewalk progress support system according to claim 1, claim 2, claim 3, claim 4, claim 5 or claim 6. The destination is based on the voice, the destination information determined by the destination information determination unit, and the destination-related map information which is the information related to the destination information among the map information recorded by the map information recording unit. It is equipped with a local destination information determination unit that can determine local destination information that is more local information than information.
The route information determination unit has the local destination information determined by the local destination information determination unit instead of the destination information determined by the destination information determination unit, and the current position detected by the current position information detection unit. The sidewalk progress support system according to claim 1, claim 2, claim 3, claim 4, claim 5, claim 6 or claim 7, wherein the route information can be determined based on the position information. The progressable route information determined by the progressable route information determination unit, the current position information detected by the current position information detection unit when a pedestrian travels along the progressable route of the progressable route information, and the current position. Claim 1, claim 2, claim 3, claim 4, claim 5, claim 6, claim 7 provided with a progress history recording unit that can record in association with the captured image when the position information is detected. Alternatively, the sidewalk progress support system according to claim 8. A search unit capable of searching the map information recorded by the map information recording unit and the object arrangement information recorded by the object arrangement information recording unit based on voice is provided.
The sidewalk progress support system according to claim 7, wherein the notification unit can notify the search result of the search unit. An image cutting unit capable of cutting out images of each of a movable object and a stationary object detected by the object detecting unit from the captured image, and an image cutting unit.
Information obtained by associating a cut-out image of a movable object and a cut-out image of a stationary object cut out by the image cutting portion with a specific result specified by the object specifying portion of the moving object and the stationary object. Claim 1, claim 2, claim 3, claim 4, claim 5, claim 6, claim 7, claim 8, claim 9 or claim provided with a teacher information recording unit capable of recording certain teacher information. The sidewalk progress support system according to claim 10. The notification unit has a voice notification unit capable of announcing the content of the notification by voice, and is detachable from one side in the longitudinal direction and the other side located on the opposite side to the one side. A plurality of vibrations that are mounted side by side in the same direction as the longitudinal direction of the main body and that can be partially or wholly vibrated depending on the content of the notification when the main body is connected and one surface of the main body. At least one of the vibration notification parts having a body
The sidewalk progress support system according to claim 1 or 2. Sidewalk progress support software to support the progress of pedestrians on the sidewalk,
Information processing equipment,
A map information recording unit that can record map information, which is information about maps,
The image pickup unit captures the positions of the moving object, which is a moving object, and the stationary object, which is a stationary object, detected in the captured image obtained by the image pickup unit that can image at least the front of the pedestrian. Dynamic coordinates with the position of the pedestrian on which the image pickup unit is attached as the origin, based on the direction and the distance between the image pickup unit and the stationary object or the movable object measured based on the image pickup image. The same coordinate value calculated with the current position information, which is the position information of the pedestrian equipped with the image pickup unit, which can be calculated as the coordinate value of the system and is detected, is used as the origin, is used as the map information recording unit. An object map coordinate creation unit that can create map coordinates of the same stationary object or the same movable object by converting it to the latitude and longitude of the map coordinate system included in the map information recorded by
A detection object information recording unit that can record moving object information that is information about moving objects and stationary object information that is information about stationary objects, and
The movable object or the stationary object can be identified by collating the movable object or the stationary object detected in the captured image with the movable object information and the stationary object information recorded by the detected object information recording unit. Object identification part and
A first line segment detected in the captured image extending from the lower part of the captured image toward the substantially central region of the captured image, and a first line segment extending from the lower portion of the captured image toward the substantially central region of the captured image. When the area between the second line segment and the second line segment, which has become shorter as it extends and approaches the substantially central region of the captured image from the lower part of the captured image, is determined as a sidewalk. The latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to each of the first line segment and the second line segment sandwiching the same sidewalk, and the first line segment. A travelable area in the sidewalk based on the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to a stationary object or a movable object located between the second line segment. Progressable area position information detection unit that can detect the position information of
A destination information determination unit that can determine destination information, which is information about a destination, based on the voice and the map information recorded by the map information recording unit.
Route information determination that can determine route information that is information about a route between a destination and a current position based on the destination information determined by the destination information determination unit and the detected current position information. Department and
A pedestrian who is a route between a destination and a current position based on the route information determined by the route information determination unit and the travelable area position information detected by the travelable area position information detection unit. The progressable route information determination unit that can determine the progressable route information, which is the information about the progressable route,
A notification unit that can notify the progressable route information determined by the progressable route information determination unit, and a notification unit that can notify the progressable route information.
Sidewalk progress support software to function as a means including. Sidewalk progress support software to support the progress of pedestrians on the sidewalk,
Information processing equipment,
An object detection unit that can detect a moving object that is a moving object and a stationary object that is a stationary object in the captured image obtained by an imaging unit that can image at least the front of a pedestrian.
In the captured image, a first line segment extending from the lower part of the captured image toward the substantially central region of the captured image and the same as extending from the lower portion of the captured image toward the substantially central region of the captured image. A line segment detection unit capable of detecting a second line segment whose distance from the first line segment becomes shorter as it approaches the substantially central region of the captured image from the lower part of the captured image, and a line segment detection unit.
A sidewalk determination unit that can determine the area between the first line segment and the second line segment as a sidewalk detected by the line segment detection unit in the captured image.
A map information recording unit that can record map information, which is information about maps,
The position of the stationary object or the movable object detected by the object detection unit is determined by the imaging direction of the imaging unit and the distance between the imaging unit and the stationary object or the movable object measured based on the captured image. Based on the above, it is possible to calculate as a coordinate value of the dynamic coordinate system with the position of the pedestrian equipped with the imaging unit as the origin, and the detected position information of the pedestrian equipped with the imaging unit is used. The same coordinate value calculated with a certain current position information as the origin is converted into the latitude and longitude of the map coordinate system included in the map information recorded by the map information recording unit, and the map coordinates of the stationary object or the movable object are converted. Object map coordinate creation unit that can create
A detection object information recording unit that can record moving object information that is information about moving objects and stationary object information that is information about stationary objects, and
An object capable of identifying the movable object and the stationary object by collating the movable object or the stationary object detected by the object detection unit with the movable object information and the stationary object information recorded by the detected object information recording unit. With a specific part
The latitude / longitude position of the map coordinates created by the object map coordinate creation unit corresponding to each of the first line segment and the second line segment sandwiching the sidewalk when the sidewalk determination unit determines the sidewalk. The information and the latitude / longitude position information of the map coordinates created by the object map coordinate creation unit corresponding to the stationary object or the movable object located between the first line segment and the second line segment. Based on the progressable area position information detection unit that can detect the position information of the progressable area in the same sidewalk,
A destination information determination unit that can determine destination information, which is information about a destination, based on the voice and the map information recorded by the map information recording unit.
Route information determination that can determine route information that is information about a route between a destination and a current position based on the destination information determined by the destination information determination unit and the detected current position information. Department and
A pedestrian who is a route between a destination and a current position based on the route information determined by the route information determination unit and the travelable area position information detected by the travelable area position information detection unit. The progressable route information determination unit that can determine the progressable route information, which is the information about the progressable route,
A notification unit that can notify the progressable route information determined by the progressable route information determination unit, and a notification unit that can notify the progressable route information.
Sidewalk progress support software to function as a means including.

Images