JP2002065721A - Device and method for supporting environmental recognition for visually handicapped - Google Patents

Abstract

PROBLEM TO BE SOLVED: To securely inform a visually handicapped of distances to, movement of, etc., various objects existing everywhere in a space spreading three- dimensionally. SOLUTION: A space in front of a user is photographed by a multiple-lens stereo camera 20 mounted on the forehead of the user. A data processor 30 receives the photographed picture, calculates the three-dimensional positions (distance picture) of the objects in the space in front of the user, identifies the objects as the ground surface or various kinds of obstacles on the basis of the distance picture and drives a tactile display device 40 which the user touches with his/her hand and a stereo headphone 50 mounted on both ears of the user by using the result of identification. The device 40 forms a recessed/ projecting picture expressing the position/height of the ground surface or obstacles by many pins 41 which are freely movable back and forth and arranged on a display surface. A stereo headphone 50 generates stereo acoustic which informs the user of the direction and the distance of the ground surface or obstacles with respect to the user.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a device for assisting a visually impaired person in recognizing a surrounding environment (for example, the presence of an obstacle or unevenness on the ground). The application of the device of the invention is mainly, but not exclusively, for assisting the visually impaired.

[0002]

2. Description of the Related Art A conventional apparatus of this type is disclosed in
97457 and JP-A-7-116208. The device described in JP-A-1-97457 is a combination of a CCD camera and a tactile display device. The CCD camera captures an image of the front of the user and outputs an electric signal representing the light intensity of each pixel to the tactile display device. The tactile display device is provided with a large number of movable pins which are two-dimensionally arranged on the display surface corresponding to each pixel, and adjusts the amount of protrusion of each pin by an electric signal of each pixel from the CCD camera. As a result, a concavo-convex pattern representing the brightness of the image captured by the CCD camera is formed on the display surface. When the user touches the uneven pattern, the user can tactilely recognize the state in front of the user.

The device described in Japanese Patent Application Laid-Open No. Hei 7-116208 is a combination of a distance measuring / temperature measuring device and a stimulating device. The distance measuring device emits infrared light from the position of the user in one directional direction, receives reflected light from an object existing in the directional direction, measures the distance from the light receiving position to the object, Further, the temperature of the object is measured from the wavelength component of the light received from the object. Then, the risk of the object is calculated from the measured distance and temperature of the object. The stimulator has two or so many retractable pins,
By protruding the pin according to the risk of the object, a stimulus corresponding to the risk of the object is given to the user. When detecting objects in various directions, the pointing direction of the distance measuring / temperature measuring device can be changed manually or automatically. For example, an object from near to far in the walking direction can be detected by changing the pointing direction far from the step forward while walking.

[0004]

Problems to be Solved by the Invention
According to the device of No. 2, the user can recognize a two-dimensional image obtained by photographing the front of the user with the camera through the uneven pattern of the tactile display device. However, this two-dimensional image represents the brightness of various objects within the field of view of the camera, and does not represent the distance to the objects. for that reason,
This two-dimensional image is not very useful for the user in determining walking safety and the like.

On the other hand, the device disclosed in Japanese Patent Application Laid-Open No. Hei 7-116208 measures the distance of an object existing in a certain directional direction, and gives a stimulus to the user according to the distance. Thereby, the user can determine the security of the pointing direction. However, what this device can detect at one time is the distance to an object existing in one direction. In other words, this device only detects the distance to an "one point" of an object existing in one direction and informs the user of the distance, and various obstacles existing everywhere in a three-dimensional spatial region extending in front of the user. It is not possible to detect the position of a stone (eg, a stone at the foot, a stair in front, a wall on the left side, a person going to the right, etc.) at a time and teach the user. For this reason, the user has difficulty grasping the situation of the entire area necessary for determining the safety of walking. Also, because of this, if the object is moving,
It is also difficult for the user to accurately grasp the state of the movement (for example, trying to cross the front from the right hand).

Accordingly, it is an object of the present invention to provide an apparatus capable of accurately informing a user of distances, movements, and the like of various objects existing everywhere in a spatial region having a three-dimensional spread.

[0007]

An environment recognition support apparatus for a visually impaired person according to the present invention uses a space monitoring means for monitoring a three-dimensional space extending in front of a user and an output signal of the space monitoring means. Calculating means for calculating three-dimensional coordinates of an object area occupied by the object in the three-dimensional space; and a drive signal for generating a tactile stimulus or sound corresponding to the three-dimensional coordinates of the object area calculated by the calculating means. A driving unit that outputs the signal; and a notifying unit that receives a driving signal from the driving unit and generates a tactile stimulus or a sound.

This environment recognition support apparatus for the visually impaired monitors a three-dimensional space extending in front of the user, calculates three-dimensional coordinates of an object area occupied by the object in the three-dimensional space, and calculates the three-dimensional coordinates of the object area. A tactile stimulus or sound corresponding to the three-dimensional coordinates of the region is provided to the user. Instead of detecting an object as a “single point” existing in one directional direction as viewed from the user as in the related art, it has spread as a “plane” as viewed from the user.
The user monitors the three-dimensional space at substantially the same time, grasps the three-dimensional coordinates of various object regions existing throughout the three-dimensional space at a glance, and gives a stimulus to the user accordingly, so that the user is present in the surroundings. The direction and distance of various objects to be performed can be grasped much more easily than before, and safety such as walking can be more easily confirmed.

In a preferred embodiment, a tactile display device that generates a concavo-convex image that can be sensed by touching by a user is used as the notification means. This tactile display device expresses at which three-dimensional position in the three-dimensional space the object is present, based on which position of the concavo-convex image and how much protrusion or dent is present.

[0010] In a preferred embodiment, a stereo sound generator that generates stereo sound that allows the user to sense the distance and direction of the sound source with the left and right ears is used as the notification means. This stereo sound generation device provides a user with stereo sound expressing in which direction and how far the object is from the user.

In a preferred embodiment, a multi-eye camera for photographing the three-dimensional space from a plurality of different positions is used as the space monitoring means. Using a plurality of images taken by the multi-lens camera, stereoscopic viewing
The three-dimensional coordinates of the object region in the three-dimensional space are calculated, and the concavo-convex image and the stereo sound are controlled accordingly. By using a multi-lens camera, the three-dimensional coordinates of an object can be calculated with high accuracy under natural lighting. When a video camera is used as a camera, the user can grasp the three-dimensional position of a moving object in real time. Can be sensed.

In a preferred embodiment, after calculating three-dimensional coordinates of an object region in a three-dimensional space, a process called environment recognition is performed based on the three-dimensional coordinates, and individual objects existing in the three-dimensional space are processed. And the three-dimensional coordinates are identified. Then, the above-mentioned uneven image and stereo sound are controlled according to the type of the identified object and the three-dimensional coordinates. For example, the environment recognition identifies whether an individual object is an obstacle (eg, a person, a car, an installation on the street, a hole, a step, a building, a wall, etc.) (eg, flat ground), and For at least an object determined to be an obstacle, the unevenness image and stereo sound are controlled so that the user can sense the direction, distance, height, and the like of the obstacle in accordance with the three-dimensional coordinates.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows an overall configuration of an embodiment of a visual impaired person environment recognition support apparatus according to the present invention.

This visually impaired person environment recognition support device includes a stereo camera 20, a data processing device 30, and an environment notification device 40 or 50.

The stereo camera 20 is used for photographing to create a range image using stereo vision, and includes a plurality of video cameras (for example, a CCD camera) arranged at different positions in substantially the same direction. ) 21. From the viewpoint of using stereo vision, the stereo camera 2
0 may have any number of video cameras as long as it is two or more, but in this embodiment, in order to obtain a highly accurate range image, nine video cameras arranged in a 3 × 3 matrix are used. Use a multi-view stereo camera with many video cameras. The stereo camera 20 is mounted on a position convenient for photographing the front of the user, for example, at the front of the hat 10 (that is, at the front of the user). The stereo camera 20 (that is, the nine video cameras 2
1) has a wide viewing angle necessary for the user to confirm walking safety. With this stereo camera 20,
With this viewing angle, it is possible to photograph a space region that spreads three-dimensionally in front of the user at one time.

The data processing device 30 includes the stereo camera 2
By inputting the latest frame images of the nine moving image data output from the nine video cameras 21 and processing these nine frame images by stereoscopic viewing,
A latest distance image of the field of view of the stereo camera 20 (that is, an image in which the distance from the position of the stereo camera 20 to the area occupied by the object in the three-dimensional space within the field of view in pixels) is formed. Thus, the data processing device 3
0 continuously and successively creates the latest distance image (that is, creates a distance moving image), and controls and drives the environment notification device 40 or 50 based on this. For this purpose, the data processing device 30 includes a distance image calculation unit 31 that calculates a distance image, and the environment notification device 40 or 5 based on the distance image.
And a control / drive unit 32 for controlling and driving 0. The control / drive unit 32 analyzes the distance image to identify the ground from the distance image, and further, an object having a height higher than the ground (an obstacle having a height higher than the ground) or a drop below the ground. Identify recesses (minus-height obstacles) and the like. The process of identifying various objects existing in the field of view of the camera in this way is referred to as “environment recognition” in this specification. The control / drive unit 32 sends a drive signal corresponding to the type of the various objects such as the ground and obstacles identified by the environment recognition (that is, whether the object is the ground or an obstacle) and the three-dimensional position, to the environment notification device 40.
Or output to 50.

The environment notification device 40 or 50 is one of the tactile display device 40 and the stereo headphone 50, or a combination of both. The tactile display device 40 is, like the tactile display device described in JP-A-1-97457, two-dimensionally arranged with a large number of movable pins 41 standing on one surface (display surface) of a base plate. The protrusion amount of each pin 41 can be adjusted independently, and the many two-dimensionally arranged pins 41 form an uneven image. The user touches and feels the uneven image formed by the pins 41 of the tactile display device 40 with a hand or the like, thereby grasping the types and positions of various objects represented by the uneven image. For example, a place where the pin 41 protrudes high means that an obstacle exists there, and a place where the pin 41 is low means that the place is a ground on which it can walk.
The tactile display device 40 can be attached to a place suitable for the user to touch the display surface, such as a hand, for example, a position of the user's waist, a position of a grip of a walking aid cane (FIG. 2, reference numeral 140), and the like. it can.

The stereo headphone 50 is a normal stereo headphone having two earphones (or speakers) 50L and 50R for giving independent sounds to the left and right ears of the user. By using the driving method, the position and the distance of the obstacle can be audibly notified to the user. One of the methods is a method using a stereo sound effect (that is, an effect of making a user audibly grasp a position and a distance of a sound source by making various differences in characteristics of sound given to left and right ears). is there. That is, the control / drive unit 32 identifies where and what obstacle is present based on the distance image, and assumes that a sound is emitted from each of the identified obstacles. This is a method of calculating sound audible to the ears and outputting the calculated sound to the left and right earphones 50L and 50R. At this time, if a specific sound is determined in advance for each type of obstacle, the type of obstacle can be easily known. Another method is to output to the stereo headphones 50 a voice message (for example, “Peer, obstacle exists 2 m ahead”) indicating the position and distance of the obstacle.

Either the tactile display device 40 or the stereo headphone 50 may be used, but if both are used in combination, the user can more easily grasp the surrounding environment.

This environment recognition support apparatus for the visually impaired person desirably has an angle sensor (or horizontal sensor) 60 for detecting the movement (particularly, rotation angle) of the stereo camera 20 in addition to the above-described configuration, and a hat 10 or the like. Can be installed at appropriate places. This is because even if the direction of the stereo camera 20 changes due to the movement of the user's head, the data processing device 3
This is so that 0 can accurately grasp it and determine the horizontal direction. As the angle sensor (or horizontal sensor) 60, for example, an angle sensor that measures a rotational acceleration by using a gyro to calculate a tilt angle, or always indicates a horizontal direction regardless of how the posture changes. A horizontal sensor that outputs a signal can be employed. When using an angle sensor, the rotation of the head is pitch (moving upward or downward) and roll (moving the head to the left or right)
And yaw (movement to rotate horizontally horizontally).
When an angle sensor is used, it is of primary importance to detect the angle of the pitch that is most frequently performed by the user among these three types of rotation, and it is more desirable to detect the angle of the roll in addition. By detecting the pitch and roll angle of the head in this manner, even if the user moves the head, the data processing device 30 can always accurately recognize the horizontal direction, and as a result, the control of the data processing device 30・ Drive unit 3
The accuracy of the environment recognition performed by the device 2 is improved, and more accurate environmental information can be transmitted to the user.

Hereinafter, the processing of generating the distance image, recognizing the environment, and driving the environment notification device 40 or 50 performed by the data processing device 30 will be described in more detail.

First, generation of a range image by stereo vision based on a moving image from the stereo camera 20 will be described. The distance image calculation unit 31 of the data processing device 30 captures, from the stereo camera 20, the latest frames (gray-scale still luminance images) of the nine moving images output from the nine video cameras 21. Range image calculator 31
Is a luminance image from one camera (hereinafter, referred to as a reference camera) 21 located at the center of the video cameras 21 arranged in a 3 × 3 matrix, and the other eight cameras 21 The eight reference images are used as reference images, and a pair is formed between each of the eight reference images and the reference image (eight pairs are formed). For each pair, the parallax of each pixel between the two luminance images is determined by a predetermined value. Ask by the way.

Here, as a method for obtaining the parallax, for example, a method disclosed in Japanese Patent Application Laid-Open No. H11-175725 can be used. The method disclosed in Japanese Patent Application Laid-Open No. H11-175725 is simply as follows. First, one pixel is selected on the reference image, and a window area of a predetermined size (for example, 3 × 3 pixels) centered on the selected pixel is extracted from the reference image. Next, a pixel (referred to as a corresponding candidate point) at a position shifted from the selected pixel by a predetermined parallax on the reference image is selected, and a window region of the same size centered on the corresponding candidate point is extracted from the reference image. Then, the similarity of the luminance pattern (for example, between pixels corresponding to positions in both window regions) between the window region of the corresponding candidate point extracted from the reference image and the window region of the selected pixel extracted from the reference image. Calculate the reciprocal of the square addition value of the luminance value difference). While sequentially changing the parallax from the minimum value to the maximum value and moving the corresponding candidate point,
For each corresponding candidate point, the calculation of the similarity between the window area of the corresponding candidate point and the window area of the selected pixel from the reference image is repeated. From the result, the corresponding candidate point with the highest similarity is selected, and the parallax corresponding to the corresponding candidate point is determined as the parallax in the selected pixel. Such determination of parallax is performed for all pixels of the reference image. From the parallax of each pixel of the reference image, the distance between the portion corresponding to each pixel of the object and the reference camera is determined on a one-to-one basis. Therefore, by calculating the parallax for all the pixels of the reference image, a distance image that represents the distance from the reference camera to the object for each pixel of the reference image is obtained as a result.

The distance image calculation unit 31 calculates a distance image for each of the eight pairs by the above method, integrates the eight distance images by a statistical method (for example, calculates an average), and obtains a result. Is output as the final distance image.

Next, the environment recognition performed by the control / drive unit 32 of the data processing device 30 will be described.

The environment recognition is basically performed, for example, in Japanese Unexamined Patent Application Publication No.
The method can be carried out using the method described in JP-A-143659. In short, this method is based on the two-dimensional coordinates of each pixel in the distance image and the distance, and calculates the three-dimensional coordinates of each pixel in a predetermined coordinate system.
Calculate the three-dimensional coordinates and use the height indicated by the three-dimensional coordinates to have a height lower than a predetermined threshold suitable for the ground (or a height between a predetermined upper threshold and a lower threshold suitable for the ground) ) And extract it as the ground. Next, a pixel group having a height that is higher than the ground by a predetermined value or more is extracted, and the pixel group is classified into a set of pixels having a short distance from each other, and each pixel set is identified as an individual obstacle. I do.

When such environment recognition is performed, the control / drive unit 32 calculates the three-dimensional coordinates of each pixel from the distance image, taking into account that the directivity of the stereo camera 20 is not horizontal. A process of converting a coordinate system based on the direction of the stereo camera 20 represented by the distance image into a coordinate system based on the horizontal direction is performed. The coordinate conversion process will be described with reference to FIG.

FIG. 2 shows the relationship between the direction of the stereo camera 20 and the ground.

In FIG. 2, a dotted line 130 indicates the horizontal direction, a dashed line 120 indicates the directivity of the stereo camera 20, and an arrow 110 indicates the directivity 12 of the stereo camera 20.
It shows a viewing angle that is evenly opened in the vertical direction around 0. Although not shown, when the user 100 is viewed from directly above, the directional direction 120 of the stereo camera 20 is directly in front of the head of the user 100, and the field of view is centered on the directional direction 120 in FIG. Open right and left at the same angle as the viewing angle 110.

In confirming the safety of walking, the user 100
The most important information is the shape of the ground 200 in the traveling direction of the user 100 and the presence or absence of an obstacle such as a hole or an object in a height range from a height slightly lower than the ground 200 to a height slightly higher than the user's head. is there. Therefore, as shown in FIG. 2, in a state where the user 100 is walking forward in a normal posture, the directional direction 120 of the stereo camera 20 is set to be inclined downward by an angle θ from the horizontal. The viewing angle 110 of the stereo camera 20
In the traveling direction of the user 100, in particular, in a distance range from near the foot important for walking to about 5 meters or 10 meters ahead, from a height somewhat lower than the ground 200 to a height slightly higher than the user's head. Height range.

Therefore, as shown in FIG.
Is normally facing forward, and the coordinate system X′-Y′-Z ′ based on the pointing direction 120 of the stereo camera 20 (hereinafter referred to as a camera coordinate system) is a coordinate system based on the horizontal direction. It is tilted forward and downward by a predetermined angle θ with respect to XYZ (hereinafter referred to as a global coordinate system). The data of the distance image created by the distance image calculation unit 31 based on the data from the stereo camera 20 is stored in a camera coordinate system X′-Y′−
Z ′. When performing environment recognition, the control / drive unit 32 calculates the three-dimensional coordinate values in the camera coordinate system X′-Y′-Z ′ of each pixel directly obtained from the distance image using the angle θ, 3 expressed in the global coordinate system XYZ
Convert to dimensional coordinate values. When the angle sensor 60 shown in FIG. 1 is provided, the control / drive unit 32
In this coordinate conversion, in addition to the angle θ, the pitch angle (rotation angle about the X ′ axis) and the roll angle (rotation angle about the Y ′ axis) of the user's head detected by the angle sensor 60 are also considered. Put in. In this way, the control / drive unit 32 calculates the three-dimensional coordinate value of each pixel expressed in the overall coordinate system XYZ based on the horizontal direction, and based on the calculated value, calculates
The ground 200 is identified by the environment recognition method of -175725, and an obstacle on the ground 200 is identified.

When performing the environment recognition method disclosed in Japanese Patent Application Laid-Open No. H11-175725, the control / drive unit 32 utilizes the height of the user 100 which is known in advance and uses a height lower than the height of the stereo camera 20 by the height of the user. , A higher threshold value that is higher by a predetermined amount and a lower threshold value that is lower by a predetermined amount are set, and a pixel group having a height between the upper threshold value and the lower threshold value is extracted. Can be identified as Normally, the ground 200 is horizontal, so that the ground 200 can be accurately identified by this method. On the other hand, when the ground 200 is a slope or a stair, the pixels extracted by this method are used only for a striped portion of the same height existing near the feet of the user 100 in the entire ground 200. Represent a small number of pixels. In that case, the entire ground 200 is assumed to be a slope or a stair along an inclined plane such that the horizontal surface is slightly rotated by an angle with the extracted striped portion as a rotation axis. . Therefore, in order to identify such an inclined ground 200, a coordinate axis extending horizontally at right angles to the striped portion is assumed, and the pixel having the lowest height at each position on the coordinate axis is determined. Pick out. The selected pixels are candidates for the inclined ground 200. Subsequently, from these selected pixels, pixels capable of forming the same inclined plane as the above-mentioned striped portion are extracted. The inclined plane formed by the pixel group extracted in this way is referred to as ground 2
00 can be identified. In this case, if a substantially continuous inclined plane is formed by the finally extracted pixel group, it can be determined that the ground 200 is a slope along the inclined plane. On the other hand, what is finally formed by the extracted pixels is not a substantially continuous inclined plane,
In the case of an inclined plane divided into a plurality of stripe-shaped portions arranged substantially in parallel at intervals, the ground 20
0 can be determined to be a staircase. Furthermore, if a horizontal plane connected to the hill or stair is detected far from the ground 200 determined to be the hill or stair, it can also be identified as the ground 200. In this way, the ground 200 can be distinguished and recognized as a horizontal ground, an uphill, a downhill, an up stair, a down stair, or the like.

After identifying the ground 200 in this way, the control
The drive unit 32 extracts pixels having a height higher than a predetermined height (a minimum height that may cause a walking obstacle, for example, 5 cm) from the ground 200, and extracts those pixels. By grouping pixels that are close to each other in distance, the pixels are classified into several pixel groups, and each pixel group is obstacles having a plus height (for example, walls, pillars, people, vehicles, installed objects, signboards, steps, etc.). Rise, etc.). further,
The control / drive unit 32 extracts a pixel having a height that is lower than the ground 200 by a predetermined depth (a minimum depth that may be determined to be a walking obstacle, for example, 5 cm). Then, those pixels are classified into several pixel groups by grouping them with pixels that are close to each other in a distance, and each pixel group is obstacles having a minus height (for example, holes, depressions, grooves, depressions of steps). Etc.).

After identifying the obstacles of the plus height and the minus height in this way, the control / drive unit 32 further divides the height of each obstacle, the plane size, the positional relationship with the ground, and the like. However, each obstacle can be further finely identified based on them. For example, for height,
The maximum and minimum heights are classified according to the height of the user, the waist of the user, the knees of the user, the ground, the depth under the ground of 10 cm, and the like.
Classification is performed by dividing by m, 50 cm, or the like. Regarding the positional relationship with the ground, classification is performed by distinguishing from the ground or in contact with the ground. Using the classification results, obstacles with a plus height, for example, those with a maximum height of more than height and a plane size of 1 m or more (such as walls, buildings and large installations), and those with a maximum height of less than height Those with a plane size of 1m or more (automobiles, guardrails, etc.), those with a maximum height of more than waist and a plane size of less than 1m (people or small installations, etc.), those with a maximum height of less than a knee and a plane size of 1m or more (E.g., stairway entrances, rising steps, low and large installations), those with a maximum height of less than knees and a plane size of less than 50m (rocks on the road, small items, etc.), and a minimum height of less than height It is determined whether the object corresponds to an object distant from the ground (a suspended obstacle, an obstacle protruding sideways from a building, or the like). Obstacles with a negative height, for example, those with a minimum height of less than 10 cm below the ground and in contact with the ground (e.g., stairs, ramps, road gutters, station platform edges, ground Judgment is made as to which of the following is applicable, such as a deep hole or the like whose minimum height is higher than 10 cm below the ground and in contact with the ground (a shallow hole or dent in the ground, a small step, etc.).

The control / drive unit 32 detects a stripe-shaped area on the surface of the ground 200 where small projections are regularly arranged, and identifies the area as a Braille stripe for guiding the visually impaired. Can also.

By performing environment recognition by the above method, for example, the following can be recognized.

Ground (including floor), table surface, inclination and unevenness of those surfaces Stairs (up and down), wall plus height obstacles (cars, people, walls, signboards, stumbles, etc.), minus Obstacles of height (holes, drop of steps at the edge of station platform, gutters, etc.) Braille stripes on roads, separation stones of roads and sidewalks Objects (people, objects, dishes, food) Spaces (in trains, workshops) Next, a process of controlling and driving the environment notification device 40 or 50 performed by the control / drive unit 32 using the result of the environment recognition will be described.

First, processing for controlling and driving the tactile display device 40 will be described. Methods for controlling and driving the tactile display device 40 include (1) a method of displaying the position and height of an object identified by environmental recognition on the tactile display device 40, and (2)
An arbitrary plane is provided in a three-dimensional space within the viewing angle of the camera,
There are two methods for displaying the presence or absence of an object in the plane on the tactile display device 40.

FIG. 3 shows an example of a method of displaying the position and height of the object identified in (1) above on the tactile display device 40.

FIG. 3A is a plan view of a place where the user 100 stands, and the user 100
Standing on the upper left, walls 300, 31 on the diagonally left and right sides
0 shows an example in which there is an ascending stair 320 in front of the front. Viewing angle 11 of stereo camera 20 at user's head
0, the ground 200 and the wall 30 in front of the user 100
0, 310 and up stairs 320 are included. Dotted line 40
The area surrounded by 0 is the X-line displayed on the tactile display device 40.
The Y coordinate area is shown.

The control / drive unit 32 selects an object (ground or obstacle) identified by the environment recognition and located in the XY coordinate area 400 to be displayed, and sends the selected object to the tactile display device 40. indicate. At this time, as shown in FIG. 3B, the X and Y coordinates of the pixel group representing each object (that is, the horizontal position of each object) are set to the position of each pin 41 on the display surface of the tactile display device 40. The maximum Z coordinate of the pixel group representing each object (that is, the maximum height of each object) is assigned to each pin 4
1 corresponds to the protrusion amount (height). At this time, if the X, Y, and Z coordinate values of each pixel can correspond to each protrusion amount of each pin 41 of the tactile display device 40 on a one-to-one basis, the X, Y, and Z coordinate values of each pixel remain unchanged. Can be used,
Actually, the number of arrangements of the pins 41 and the number of controllable protrusion amounts of the tactile display device 40 are not so large.
It is not possible to make one-to-one correspondence with the X, Y, and Z coordinate values of each pixel. Therefore, the XY coordinate area 400 to be displayed is divided by the number of rows and columns of the pins 41, divided into area blocks corresponding to the pins 41, and the maximum Z coordinate value of the portion of each object located in each area block Is determined as the maximum height of each object in each region block, and the maximum height of each block is discretized by the number of steps of the protrusion amount of the pin 41, and the maximum height thus discretized is used. The amount of protrusion of each pin 41 is controlled.

As a result, as shown in FIG. 3B, on the display surface of the tactile display device 40, the display target area 40 is displayed from directly above.
Looking at 0, an uneven pattern expressing the maximum height of each location in the area 400 by the height of each pin 41 is displayed. FIG.
In the case of (B), the height of each pin 41 can be controlled in three stages, and the pin 41 indicated by a white circle is the lowest and the ground 200
The pin 41 indicated by a hatched circle represents the first half of the ascending stairs 320 (a kind of obstacle of a positive height) at an intermediate height, and the pin 41 indicated by a black circle
Is the highest and represents the latter half of the walls 300 and 310 and the ascending stairs 320 (both are types of obstacles of a plus height). The pin 41 with an x mark corresponds to a position outside the viewing angle 110, and this is set to a medium height like the pin 41 of hatching, for example.
Attention may be given to the user that it is outside the viewing angle 110. If the number of steps of the height of the pin 41 is larger, various kinds of obstacles as described in the description of the environment recognition can be distinguished by the height of the pin, and the ground is a stair or a slope. In that case, the change in the height can be expressed by changing the height of the pin.

According to this display method, the user touches the display surface of the tactile display device 40 and feels the height of each place, so that the place where the ground is, where the obstacle exists, and the obstacle It is possible to estimate the size, height and planar shape of the vehicle, and to grasp the state of the environment in the traveling direction.

As described above, the stereo camera 2
Since the distance image obtained from the photographing result of 0 is a moving image that changes in real time, the movement of the target object also appears in real time on the display surface of the tactile display device 40 that expresses it.
For example, when the position of a wall or stairs with respect to the user changes as the user walks, or an object such as a person or a car moves by itself, the tactile display device 40 includes a protruding pin 41 corresponding to the object. Moving to another pin 41, the user can recognize the movement of those objects in real time.

4 to 7 show some other examples of a method for displaying an obstacle identified by environment recognition on the tactile display device 40. FIG.

In the example of FIG. 4, as shown in FIG.
It is assumed that there is a ground (road) 200 surrounded by walls 330 and 340 on both sides in the display target area, and a hole (obstacle having a minus height) 350 in the ground 200. In this case, as shown in FIG. 4B, in the tactile display device 40,
3, the maximum height of the walls 330, 340 and the ground (road) 200 is represented by the height of the pin 41 (pins shown by white and black circles) and holes (holes). With respect to the (obstacle of minus height) 350, the height of the pin 41 is raised above the ground or the pin 41 is vibrated (a pin shown by a hatched circle) in order to draw the user's particular attention.

In the example of FIG. 5, as shown in FIG.
In the area to be displayed, there is a ground (road) 200 surrounded by walls 330 and 340 on both sides, and an obstacle having a predetermined range of height and size that can cause a stumbling (obstruction of a plus height) in the ground 200 Object) 360 is assumed. In this case, as shown in FIG. 5B, in the tactile display device 40,
3, the maximum height of the walls 330, 340 and the ground (road) 200 is represented by the height of the pin 41 (pins shown by white and black circles), and a stumbling block is formed. As for the obstacle 360 which may be a cause, the height of the pin 41 is set higher than the ground or the pin 41 is vibrated (a pin shown by a hatched circle) in order to draw the user's particular attention.

In the example of FIG. 6, as shown in FIG.
It is assumed that there is a ground (road) 200 surrounded by walls 330 and 340 on both sides in the display target area, and a Braille stripe 370 for guiding a visually impaired person exists in the ground 200. In this case, as shown in FIG. 6B, in the tactile display device 40, the maximum height of the walls 330 and 340 and the ground (road) 200 is determined by the height of the pin 41 as in the example of FIG. 3. In addition to expressing (pins shown by white and black circles), for the Braille stripe 370, the height of the pins 41 is set higher than the ground or the pins 41 are vibrated to urge the user to particularly notice. (Pin indicated by hatched circle).

According to the display methods shown in FIGS. 4 to 6, it is possible to effectively inform the user of the position of an object that the user should particularly recognize when walking.

In the example of FIG. 7, as shown in the plan view of FIG. 7A, there is an object identified by environment recognition, such as a Braille stripe 380, in an area where the user 100 has already passed. The control / drive unit 32 also stores the position of the object such as the Braille stripe 380 in the area that has already passed, as well as the trajectory 390 that the user 100 has walked. Then, when the user 100 selects the special display mode, FIG.
As shown in (B), an object (a pin shown by a hatched circle) such as a Braille stripe 380 in the stored passing area and a locus 390 (shown by a black circle) of the user 100 are displayed on the tactile display device 40. Pin). Using this display mode, the user can do the same as just looking back and checking the trajectory that he has just walked, and is useful for determining whether or not the user is walking on the correct route.

FIGS. 8 and 9 show the display method (2) described above,
That is, an example of a method of providing an arbitrary plane in the space to be displayed and displaying the presence or absence of an object in the plane on the tactile display device 40 is shown.

FIG. 8 shows various examples of how to set a plane in a space when performing this method.

FIG. 8A shows an example in which horizontal planes 600A to 600H are set at arbitrary positions along the Z-axis of the global coordinate system in a three-dimensional space within the viewing angle of the stereo camera, and an example of the global coordinate system. Vertical plane parallel to the X axis at an arbitrary position along the Y axis 6
An example in which 01A to 601J are set is shown. Also,
FIG. 8B shows a vertical plane 602 parallel to the Y axis at an arbitrary position along the X axis of the whole coordinate system in the same three-dimensional space.
The example which sets A-602I is shown.

When one of the planes shown in FIG. 8 is set in the space, the control / drive unit 32 selects an object that intersects with the set plane from the objects identified by the environment recognition. The position of the selected object in the setting plane is displayed on the tactile display device 40. In this case, the display surface of the tactile display device 40 is regarded as the setting plane, and only the pins 41 at the position where the object is present are projected, and the pins 41 at the position where no object is present are not projected. I do. In the tactile display device 40 used in this display method, it is sufficient that the number of steps of the amount of protrusion of the pin 41 is either two or not.

Preferably, the control / drive unit 32 automatically moves the position of the set plane. For example, when a horizontal plane is set, the height position of the horizontal plane is gradually increased from the lowest position 600H shown in FIG. 8A to the highest position 600A, and when it reaches the highest position 600A. Sweeping the entire range of the display target space on the set plane is repeated at a cycle of, for example, about several seconds, such as returning to the lowest position 600H again and repeating the same operation. Then, at each position of the setting plane in this sweeping, the object existing in the setting plane is displayed on the tactile display device 40 as described above.

FIG. 9 shows a display target area 400 illustrated in FIG. 3A by using the method of sweeping the setting plane.
3 is displayed on the tactile display device 40. FIG. 9A is a display example when a horizontal plane is set at the lowest position of the ground 200, and the walls 300, 3
The pins 41 (pins indicated by black circles) in the entire area where an obstacle having a plus height such as 10 and the climbing stairs 320 are protruding, and the pins 41 in the area of the ground 200 where there is no obstacle (open) Only the pin shown in a circle) is down. FIG. 9 (B) shows that the horizontal plane has a medium height (for example,
m) is a display example when the display is raised to m), and the walls 300, 3
Only the pins 41 (pins indicated by black circles) in the region where the deep part of the ascending stairs 10 and the ascending stairs 320 exist still protrude, but the pins in the region 700 of the ascending stairs 320 in the lower part near the ascending entrance are still protruding. Is down. FIG. 9 (C)
Is a display example when the horizontal plane is raised to the highest position (for example, 2 m above the ground), and only the pin 41 (the pin shown by a black circle) in the area where the walls 300 and 310 are present is protruding. The pins in the other areas are all down.

The display of the tactile display device 40 changes in the order of FIGS. 9A, 9B, and 9C, and is repeated, for example, every several seconds. The user touches and feels the transition of this display with his / her hand, etc., so that the user can grasp the height of the object and the location of the object in the space from a position slightly lower than the ground to a position slightly higher than the user's height. can do.

FIG. 8B shows a method of sweeping the setting plane not only in the horizontal plane but also in the vertical planes 601A to 601J parallel to the X axis shown in FIG. 8A, for example. Vertical plane 602A parallel to the Y axis
602I can be similarly performed. Also,
As a method of setting the plane, not only a method of setting along the coordinate axes of the entire coordinate system as shown in FIG. 8 but also a method of setting the plane in an arbitrary direction desired by the user is possible.

Next, a process for controlling and driving the stereo headphones 50 using the result of the environment recognition will be described.

FIG. 10 shows an example of a method for controlling and driving the stereo headphones 50.

As shown in FIG. 10, for example, in the space to be displayed, there is a ground (road) 200, walls 330, 340 on both sides thereof, and a hole 350, a rock 360, etc. in the ground 200. Suppose. The control / drive unit 32 is an obstacle that hinders the user 100 from walking, that is, the wall 33.
0, 340, hole 350, and rock 36 are regarded as independent sound sources. At this time, the walls 330 and 340 and the hole 35
Each obstacle such as 0 or rock 360 is regarded as a sound source that emits a sound having a tone and a sound period peculiar to the type of each obstacle classified by the environment recognition, and the user recognizes each obstacle from the unique sound. To be able to identify the type. Also, the wall 33
For obstacles large in size like 0 and 340,
Depending on the distance from the user, multiple parts (for example,
(A part close to the user, a part slightly far from the user, a part far from the user, etc.), and each part may be regarded as an independent sound source. Then, the control / drive unit 32 considers that each sound source emits its own sound, and the sound is
The waveform, intensity, time, and the like when received by the left and right ears 101 and 102 are calculated to generate acoustic signals to be received by the left and right ears.
To the left and right earphones 50L and 50R. The user can feel what kind of obstacle is present in what direction and at what distance by the stereo sound effect heard by the left and right ears 101 and 102.
That is, an acoustic three-dimensional image of the display target space is provided to the user using the stereo sound effect.

As another method, in addition to the stereophonic effect, the principle that dolphins and bats recognize the surrounding environment by ultrasonic waves can be used. That is, instead of considering each of the obstacles 330, 340, 350, and 360 as a sound source, the user 100 gives a certain sound to a certain period (for example, 2
３3 seconds), the user 100
Sound produced by each of the obstacles 330, 340, 350, 36
It simulates how it will sound when reflected at 0 and received by the left and right ears 101, 102 of the user 100. FIG. 11 shows the control / drive unit 3 when performing this method.
2 shows the flow of processing.

As shown in FIG. 11, it is assumed that after the environment is recognized based on the distance image of the target space to identify each obstacle (step S1), a virtual sound wave is emitted from the user to the target space. (S2), the intensity of the received sound in the left and right ears when the sound wave is reflected by each obstacle and received by the left and right ears of the user, the delay time from the sound wave emission to the sound reception, and the like are calculated (S3). ). At this time, if the frequency and tone of the reflected sound are different for each type of obstacle,
The user can identify the type of each obstacle from the reflected sound. Further, in the calculation of the delay time from the sound wave emission to the sound reception, the sound speed is set to be lower than the true sound speed, so that even when the distance is small, the user's ear has a clear sound reception time. Make them feel the difference. After calculating the sound reception of the left and right ears, the sound is output to the earphones of the left and right ears (S4).

According to this method, the distance of the obstacle appears not only as the intensity of the audible sound but also as a difference in audible time, so that the user can more clearly grasp the distance to the obstacle.

FIG. 12 shows a modification of the shape of the display surface of the tactile display device 40.

The display surface of the tactile display device 40 may be a simple quadrangle, but may be a fan-like shape corresponding to the shape of the hand 800 as shown in FIG. Further, such a shape close to a fan shape is close to the planar shape of the field of view of the stereo camera, so that the state of the space in the field of view can be expressed without waste.

FIG. 13 shows a modification of the stereo camera.

In the stereo camera shown in FIG. 13A, a plurality of video cameras 910 are attached to the front part corresponding to the forehead of the hair band 900. In the stereo camera of FIG. 13B, a plurality of video cameras 910 are attached to the front frame of the glasses 920. In the stereo camera of FIG. 13C, a plurality of video cameras 910 are attached to the tip of a flashlight-type stick 930 held by hand. Thus, various configurations can be adopted for the stereo camera.

Although the representative embodiment of the present invention has been described above, the above embodiment is merely an example for describing the present invention, and is not intended to limit the present invention to the above embodiment. Therefore, the present invention can be implemented in various modes other than the above-described embodiment.

For example, using a tactile control device capable of expressing only a binary value indicating whether each pin is out or not, the control / drive unit performs the simplest environment recognition for identifying only the ground or an obstacle, and The pin of the tactile sensation control device may be pinned out only at the location where the obstacle exists, so that only the obstacle may be simply notified to the user.

Also, a system for determining the position of the user, such as GPS, and a map system which shows the inside of the building and the terrain on the ground in a very detailed and accurate manner are added, so that the user can walk anywhere on the map in any direction. Is detected with high accuracy, and this detection result is combined with the result of environment recognition based on the distance image performed in the above-described embodiment to provide the user with more detailed and highly accurate environmental information. You may.

Further, as a method of three-dimensional measurement of space, in the above-described embodiment, a method of obtaining a distance image by stereo vision from an image captured by a stereo camera (generally called a “passive stereo method”) is employed. In addition, for example, the following various methods can be used.

(1) Active type (method requiring pattern projection) Active stereo method Light cutting method, random dot projection method, spatial coding method, etc. Photometric stereo method Moire method (2) Passive type (under normal illumination Method that does not require light projection) Passive stereo method Two-lens stereo method, three-lens stereo method, multi-lens stereo method (method of the above embodiment) Lens focusing method Monocular method (such as texture and shadow) As the environment notification device, a device that generates various sensory stimuli other than the tactile display device and the stereo headphones described above can be used. For example, a vibrator that emits mechanical vibration, a speaker or earphone that emits a warning sound, and the like can be used. In particular, vibrators are useful for users who have hearing impairments as well as visual impairments.
As a utilization form of the vibrator, for example, a bracelet-shaped vibrator is attached to one arm or desirably both arms, and more desirably, a vibrator having a form suitable for each part is attached to a body part other than the arm, and By controlling the vibration intensity of the vibrator, the vibration waveform, the vibration interval, the selection of the position of the vibrator to be vibrated, and the combination of the vibrations of the vibrators at a plurality of locations, it is possible to determine which obstacles are in which directions in which directions. There is a method of notifying the user whether there is something.

[Brief description of the drawings]

FIG. 1 is a perspective view showing the overall configuration of an embodiment of a visually impaired person environment recognition support apparatus according to the present invention.

FIG. 2 is an explanatory diagram showing a relationship between the direction of a stereo camera 20 and the ground.

FIG. 3 is a diagram showing an example of a method of displaying a position and a height of an object identified by environment recognition on a tactile display device 40;

FIG. 4 is a diagram showing another example of a method of displaying an object identified by environment recognition on a tactile display device 40.

FIG. 5 is a diagram showing still another example of a method of displaying an object identified by environment recognition on the tactile display device 40.

FIG. 6 is a diagram showing still another example of a method of displaying an object identified by environment recognition on the tactile display device 40.

FIG. 7 is a diagram showing another example of a method of displaying an object identified by environment recognition on the tactile display device 40.

FIG. 8 is a diagram showing various examples of how to set a plane when a method of providing an arbitrary plane in the display target space and displaying the presence or absence of an object in the plane on the tactile display device 40 is performed. .

FIG. 9 is a diagram showing a display example of the tactile display device 40 when an arbitrary plane is provided in the display target space and a method of displaying the presence or absence of an object in the plane on the tactile display device 40 is performed.

FIG. 10 is a diagram showing an example of a method of controlling and driving the stereo headphones 50.

FIG. 11 shows a control / drive unit 3 for providing a user with an acoustic three-dimensional image using the principle that dolphins and bats recognize the surrounding environment by ultrasonic waves in addition to the stereophonic sound effect.
9 is a flowchart showing the flow of the processing of FIG.

FIG. 12 is a plan view showing a modification of the shape of the display surface of the tactile display device 40.

FIG. 13 is a perspective view showing a modification of the stereo camera.

[Explanation of symbols]

 Reference Signs List 20 stereo camera 21 video camera (for example, CCD camera) 30 data processing device 31 distance image calculation unit 32 control / drive unit 40 tactile display device 41 pin 50 stereo headphone 100 user 110 stereo camera viewing angle 200 ground

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Hideyuki Wakai 1200 Manda, Hiratsuka-shi, Kanagawa F-term (reference) in Komatsu Ltd. 5B057 AA19 BA02 CA13 CA16 DA06 DA15 DB03 DC01

Claims (7)

[Claims] 1. A space monitoring means for monitoring a three-dimensional space extending in front of a user, and using an output signal of the space monitoring means, three-dimensional coordinates of an object area occupied by an object in the three-dimensional space. Calculating means for calculating; driving means for outputting a driving signal for generating a tactile stimulus or sound corresponding to the three-dimensional coordinates of the object region calculated by the calculating means; and receiving the driving signal from the driving means. And an informing means for generating the tactile stimulus or sound. 2. The haptic display device according to claim 1, wherein the notifying unit has a tactile display device that generates a concavo-convex image that can be sensed by touching the user. A display device is driven.
An environment recognition support device for the visually impaired according to the description. 3. The notifying unit includes a stereo sound generating device that generates a stereo sound that allows a user to sense a distance and a direction of a sound source with left and right ears, and wherein the driving unit determines a distance of the object area from the user. The environment recognition support device for a visually impaired person according to claim 1, wherein the stereo sound generation device is driven so that the direction and the direction are expressed by the stereo sound. 4. The space monitoring means has a multi-eye camera for photographing the three-dimensional space from a plurality of different positions, and the calculation means uses a plurality of images photographed by the multi-eye camera. The environment recognition support apparatus for visually impaired persons according to claim 1, wherein the three-dimensional coordinates of the object area are calculated by stereo vision. 5. An environment recognizing means for identifying the type and the three-dimensional coordinates of each object existing in the three-dimensional space based on the three-dimensional coordinates of the object area calculated by the calculating means, 2. The environment recognition support device for visually impaired persons according to claim 1, wherein the driving means controls the tactile stimulus or the sound in accordance with the type and three-dimensional coordinates of the object identified by the environment recognition means. 6. An environment recognizing means for identifying whether an individual object existing in the three-dimensional space is an obstacle, and the driving means according to three-dimensional coordinates of the obstacle identified by the environment recognizing means. The environment recognition support device for the visually impaired according to claim 5, wherein the tactile stimulus or the sound is controlled by the control. 7. A step of monitoring a three-dimensional space extending in front of the user, and a step of calculating three-dimensional coordinates of an object area occupied by the object in the three-dimensional space based on a result of the space monitoring. Providing a tactile stimulus or a sound to the user in accordance with the three-dimensional coordinates of the object area calculated by the calculation means.