JP5461486B2 - Visually handicapped walking support device - Google Patents

Description

  The present invention is a visually impaired person walking that supports safe walking by notifying that a visually impaired person is approaching a step, a depression, an obstacle, or that the obstacle is approaching. The present invention relates to a support device.

  Various devices for supporting a visually impaired person to act safely alone have been proposed. The invention described in Patent Document 1 to Patent Document 3 is an example, and an outline of these conventional examples will be described.

  The invention described in Patent Document 1 captures a front space with a multi-lens stereo camera, and the data processing device receives the captured image data to calculate the three-dimensional position of the object in the front space, thereby allowing the ground and obstacles to be captured. And so on. Then, the identification result is transmitted to the user by a tactile display device or headphones. In the tactile display device, a large number of pins are arranged on a display surface so as to be able to advance and retreat, and based on the identification result, the ground, obstacles, and the like are displayed by the unevenness of the pins. The user can recognize the presence of the ground or an obstacle by touching the tactile display device to recognize the unevenness of the pins. Further, Patent Document 1 has a description that a hole or the like generated in the front ground can be recognized by a user by vibrating a corresponding pin.

The invention described in Patent Document 2 mainly includes a stereo image capturing device, a mobile computer, and an information transmitter, and character information related to the current position and character information related to a destination from a stereo image captured by the stereo image capturing device. Is extracted, and the content of meaning obtained from the character information is emitted by the information transmitter by voice information and tactile information.
The invention described in Patent Document 3 incorporates a navigation system equipped with a GPS sensor in a mobile terminal, and a destination is set and input by voice. The mobile terminal can be connected to the server by a mobile communication terminal. The server stores information on the current location and the destination to create a travel route, and the mobile terminal that has received the travel route data guides the travel route by voice.

  According to the invention described in Patent Document 1, the user can grasp the state of the front space by continuously touching the tactile display device. Therefore, the user must always touch the tactile display device and always consider what the pin irregularities mean, which is inconvenient. In addition, a visually impaired person who is born has difficulty in imaging the physical shape of an object, and it is difficult to accurately analyze and judge visual information displayed on a tactile display device. Therefore, even if the visual information is reproduced with the unevenness of the pins, it is difficult to grasp the meaning and it is not practical. In addition, when a wall or station platform falls down, or when an obstacle such as a wall appears, it is impossible to give an alarm and call attention.

  In the invention described in Patent Document 2, character information is read from a captured stereo image, and the meaning content is issued to the user by voice information or tactile information. Therefore, character recognition processing in the image, character meaning content It takes time for analysis processing and conversion processing to voice information. For this reason, when a wall or a station platform falls down, or when an obstacle such as a wall appears, it is impossible to give an alarm and call attention. Originally, the invention described in Patent Document 2 is not intended to issue an alarm in the above case. In addition, since there is a difficulty in reading the character information from the visual information and in the reliability of analyzing the semantic content of the read character information, it is necessary to add many technical improvements for practical use. .

  The invention described in Patent Document 3 requires a large-scale device and system such as a navigation system including a GPS sensor, a server, and a portable communication terminal that communicates between the navigation system and the server. In addition, when a wall or station platform falls down, or when an obstacle such as a wall appears, it is impossible to give an alarm and call attention.

JP 2002-65721 A JP 2001-318594 A JP 2003-121193 A

  The present invention eliminates the problems of the prior art described above, i.e., when a wall or a station platform falls down or an obstacle such as a wall appears in the front while having a relatively simple configuration. An object of the present invention is to provide a walking support device for visually impaired people who can give a warning and prompt attention.

The visually impaired person walking support device according to the present invention includes:
A stereo camera capable of obtaining a pair of images by capturing an image of the same subject with a certain baseline length;
An image processing apparatus for dividing the pair of images into a plurality of vertical regions;
From a pair of images of the divided plurality regions, the distance vector calculator for calculating the length of the distance vector is a line connecting the object and the camera that is imaged in each region,
A camera angle calculation unit that calculates a camera angle with respect to a vertical line from the length of the distance vector for each region for each region;
From the distance vector length in each region and the camera angle for each region, a camera height calculation unit that calculates the camera height from the flat ground in each region ,
A camera height average value calculation unit that calculates an average value of the camera height in each of the above areas and sets it as a camera height reference value;
Spatial cross-sectional area that calculates a square cross-sectional area in which the base is delimited by the tip of each distance vector of two adjacent areas, extends vertically from the top of these distance vectors, and the top is delimited by the camera height position. An arithmetic unit;
When a reference space sectional area of the space cross-sectional area in the respective regions obtained by the spatial without walking only One obstacle in the flat surface, the ratio of the spatial cross-sectional area of the production time with respect to the reference space cross-sectional area of each region A spatial cross-sectional area ratio calculation unit that outputs an alarm signal when becomes more than a certain value,
An alarm device that operates in response to the output of the alarm signal and issues an alarm ;
The space cross-sectional area calculation unit calculates the space cross-sectional area at the time of actual operation in each region at a constant time interval,
The most important feature is that the space cross-sectional area ratio calculation unit calculates a ratio of the latest space cross-sectional area during actual operation to the reference space cross-sectional area .

The presence / absence of an obstacle or the like is determined based on a change in the spatial cross-sectional area rather than the presence / absence of an obstacle or the like based on only the change in the distance to the subject on the screen. Since the change in the spatial cross-sectional area appears remarkably with respect to only the change in distance , it is possible to quickly and reliably determine whether there is an obstacle, etc., and to promptly and reliably issue an alarm when an obstacle appears. .

It is a functional block diagram which shows the Example of the visually impaired person walk assistance apparatus which concerns on this invention. It is a perspective view which shows the external appearance of the said Example. It is a visual field figure which shows the example of the imaging screen by the stereo camera with which the said Example is provided. It is a model figure which simplifies and shows the use condition of the said Example. It is a model figure which simplifies and shows the relationship between a user, camera visual field, and a subject at the time of replacing the camera optical axis in the said use state horizontally. It is a model figure for demonstrating the camera angle calculation at the time of the actual use of the said Example. It is a model figure which shows the 1st example of the camera angle calculation as a walk pre-process of the said Example. It is a model figure which shows the 2nd example of the said camera angle calculation. It is a model figure for demonstrating the camera reference height calculation at the time of the actual use of the said Example. It is a model figure which shows the mode of the camera angle fluctuation | variation at the time of the actual use of the said Example. It is a model figure for demonstrating the space cross-sectional area ahead of the camera at the time of the actual use of the said Example. It is a graph which shows the mode of the fluctuation | variation of the camera angle obtained when mounting | wearing with the visually impaired person walking assistance apparatus which concerns on the said Example, and walking on a flat road. It is a graph which shows the mode of the fluctuation | variation of the space cross-sectional area ratio for every image area obtained when the visually impaired person walking assistance apparatus which concerns on the said Example is mounted | worn and it walks on a flat road. It is a graph which shows the mode of the fluctuation | variation of the height data for every image area obtained when the visually impaired person walking assistance apparatus which concerns on the said Example is mounted | worn and it walks on a flat road. It is a graph which shows the average value of the said height data. It is a graph which shows the mode of the fluctuation | variation of the walking distance when mounting | wearing with the visually impaired person walking assistance apparatus which concerns on the said Example, and walking. It is a graph which shows the mode of the fluctuation | variation of the cumulative value of the said walking distance. It is a graph which shows the mode of the fluctuation | variation of the walking angle when mounting | wearing with the visually impaired person walking assistance apparatus which concerns on the said Example, and walking. It is a graph which shows the mode of the fluctuation | variation of the cumulative value of the said walking angle. It is a model figure which shows the example of the camera front space cross-sectional area for every image area when mounting | wearing the visually impaired person walking assistance apparatus which concerns on the said Example, and walking on a flat road. It is a model figure which shows the example of the camera front space cross-sectional area for every image area when mounting | wearing with the visually impaired person's walking assistance apparatus which concerns on the said Example, and walking up and a staircase appears ahead. It is a model figure which shows the example of a camera front space cross-section for every image area when it walks with the visually impaired person walking assistance apparatus which concerns on the said Example, and it descends ahead and a staircase appears. It is a model figure which shows the example of the camera front space cross-sectional area for every image area when mounting | wearing with the visually impaired person walking assistance apparatus which concerns on the said Example, and walking, and the front-end | tip of a platform appears ahead. It is a model figure which shows the example of the camera front space cross-sectional area for every image area when mounting | wearing with the visually impaired person walking assistance device which concerns on the said Example, and walking, and the wall appeared ahead. It is a model figure which shows the example of the camera front space cross-sectional area for every image area when it approaches a wall further from the state shown in FIG. It is a model figure which shows the example of the walk operation | movement determination procedure of the user with which the visually impaired person walking assistance device which concerns on the said Example was mounted | worn. It is a graph which shows the example of the fluctuation | variation of the walk distance which can be obtained with the visually impaired person walk assistance apparatus which concerns on the said Example. It is a graph which shows the fluctuation | variation of the cumulative value of the said walking distance. It is a model figure which shows the mode of the rotation operation | movement of the user with which the visually impaired person walking assistance device which concerns on the said Example was mounted | worn from an upper surface. It is a model figure which shows the mode of the said rotation operation from diagonally upward. It is a visual field figure which shows another example of the picked-up image by the stereo camera with which the visually impaired person walk assistance apparatus which concerns on the said Example is provided, and shows two images obtained by rotation operation | movement. It is a graph which shows a rotation angle fluctuation | variation when the picked-up image of FIG. 31 is obtained by right rotation. It is a graph which shows the fluctuation | variation of the cumulative value of the said rotation angle fluctuation | variation. It is a graph which shows a rotation angle fluctuation | variation when the picked-up image of FIG. 31 is obtained by left rotation. It is a graph which shows the fluctuation | variation of the cumulative value of the said rotation angle fluctuation | variation.

  Embodiments of the visually impaired person walking support apparatus according to the present invention will be described below with reference to the drawings.

  First, an outline of an embodiment of the visually impaired person walking support apparatus will be described with reference to the external view of FIG. The present embodiment includes a right-eye camera 11 and a left-eye camera 12 that constitute a stereo camera, image digitizing devices 15 and 16, a computing device 20, headphones 31 as an alarm device, and a vibration device 32. The right eye camera 11, the left eye camera 12, and the image digitizing devices 15 and 16 are attached to a frame having the same shape as the frame of the glasses. More specifically, the right-eye camera 11 is mounted on the right lens frame 1 and the left-eye camera 12 is mounted on the left lens frame 2 so that their rotational positions can be adjusted around the horizontal and vertical axes, respectively. The converting devices 15 and 16 are attached to the left and right vines 3 of the frame. The frame can be worn by a visually impaired person in the same manner as a normal person wearing glasses, and the left and right cameras 11 and 12 are positioned in front of the left and right eyes of the visually impaired person.

  The right eye camera 11 and the left eye camera 12 constituting the stereo camera are used for measuring the distance to the subject by imaging the same subject with a certain baseline length. The left and right cameras 11 and 12 are provided with CCDs and other image pickup devices, and image signals output from the image pickup devices are converted into digital data by the image digitizing devices 15 and 16, respectively, and are converted into a plurality of image areas. The signal is divided into corresponding signals and input to the arithmetic unit 20. In order to assist the distance measurement by the stereo camera, or instead of the distance measurement by the stereo camera, a distance sensor 13 may be provided between the left and right lens frames 1 and 2.

  The arithmetic unit 20 includes, for example, a CPU or a DSP (digital signal processor) as a main component. In addition to the arithmetic unit 20, a storage unit including a memory (not shown) is provided. Based on the image data signal, the arithmetic unit 20 performs various calculations, which will be described in detail later, so as to output an alarm signal when a step or an obstacle appears in front. The headphone 31 and the vibration device 32 are operated by the alarm signal. The headphone 31 alerts the user with sound, and the vibration device 32 urges the user to be alerted by vibrating itself. The computing device 20 is mounted on the user's waist and the like, and has a built-in power supply battery for driving the visually impaired walking support device including the left and right cameras 11 and 12 and the image digitizing devices 15 and 16.

  Next, an example of a signal processing system of the visually impaired walking support apparatus will be described with reference to FIG. In FIG. 1, the code | symbol 10 has shown the stereo camera provided with the said left and right cameras 11 and 12. In FIG. The left and right image signals output from the stereo camera 10 are converted into digital image data signals by the A / D converter 17 and input to the image processing device 18. The A / D converter 17 corresponds to the image digitizing devices 15 and 16 shown in FIG. The image processing device 18 has a function of dividing a pair of left and right images into a plurality of regions, and inputs left and right image data signals to the arithmetic device 20 for each divided region.

  The computing device 20 includes a distance vector computing unit 21, a camera angle computing unit 22, a camera height computing unit 23, a camera angle average value computing unit 24, a camera height average value computing unit 25, a space cross-sectional area computing unit 26, a space section. An area ratio calculation unit 27, a warning determination unit 28, a relative movement vector calculation unit 32, and a walking distance / walking angle calculation unit 34 are provided. The arithmetic device 20 may include the arithmetic units as a physical configuration, or may realize functions as the arithmetic units by controlling operations of the CPU and DSP by software. . A clock signal oscillator 40, an alarm device 30, and a storage unit 50 are attached to the arithmetic device 20. The storage unit 50 includes a reference space sectional area storage unit 51 and a space sectional area storage unit 52.

  The detailed function of each arithmetic unit will be described later. Here, the flow of signals in each arithmetic unit will be described. The left and right image data signals output from the image processing device 18 are input to the distance vector calculation unit 21 and the length of the distance vector for the subject for each divided image area is calculated. Each distance vector length signal is input to the camera angle calculation unit 22, and the camera angle in use, that is, the angle of the optical axis of the photographing lens with respect to the vertical line is calculated for each image region. Based on the calculated camera angle data, the camera height calculation unit 23 calculates the camera height for each image area. Based on the camera angle data calculated for each image area, the camera angle average value calculation unit 24 calculates the average value of the camera angles, and calculates the camera height average value based on the camera height data calculated for each image area. The average value of the camera height is calculated by the unit 25.

  The distance vector length signal, the average value of the camera angle, and the average value data of the camera height are input to a spatial cross-sectional area calculation unit 26, and a spatial cross-sectional area for each image area is calculated. The spatial cross-sectional area is a cross-sectional area in the vertical plane of the front space for each image area that can be obtained from the length of the distance vector to the subject in each image area and the mounting height of the stereo camera 10. The space cross-sectional area includes a reference space cross-section that is a value obtained in a space where a user is stationary or walking on a flat road and has no obstacles, and a space cross-sectional area that changes every moment during actual operation. The area data is stored in the reference space cross-sectional area storage unit 51, and the space cross-sectional area data that changes every moment is stored in the space cross-sectional area storage unit 52. The spatial cross-sectional area data that changes every moment is calculated by the spatial cross-sectional area calculation unit 26 at regular time intervals based on the clock signal transmitted from the clock signal oscillator 40 and stored in the spatial cross-sectional area storage unit 52. The area data is updated to the latest data.

  The reference space cross-sectional area data stored in the storage units 51 and 52 and the space cross-sectional area data that changes every time during actual operation are input to the space cross-sectional area ratio calculating unit 27 and compared, and the space cross-sectional area ratio calculating unit 27 An alarm signal is output when the ratio of the reference cross-sectional area to the actual cross-sectional area is greater than a certain level. The warning signal is input to the warning device 30, and the warning device 30 issues a warning by sound, a warning that can be recognized by a feeler, etc., and notifies the user that a wall, stairs, the tip of a station platform, etc. have appeared. .

  The present embodiment includes a relative movement vector calculation unit 32 and a walking distance / walking angle calculation unit 34 so that the user's walking behavior can be determined. The relative movement vector calculation unit 32 receives the image data of the specific n frame and the image data of the n−1 frame immediately before the image data from the image processing device 18. From the data, a relative movement vector between the apparatus and the subject is calculated. Based on this relative movement vector, the walking distance / walking angle calculation unit 34 calculates the walking distance and its accumulated value, the walking angle and its accumulated value. These calculation results are input to the warning determination unit 28 and used for the operation of the alarm device 30.

Next, the walking support operation according to the above embodiment will be described more specifically. FIG. 3 is an example of left and right images taken by the stereo camera 10, and is an image of the tip of the platform and the track in front of it. In the left and right images, parallax corresponding to the baseline length of the left and right cameras is generated. An image captured by the left and right cameras is divided into three predetermined image areas. In the example shown in the drawing, the image is divided into an upper image area U, a central image area C, and a lower image area L in the vertical direction of the screen. A phase difference has occurred in the image signals output from the left and right imaging elements in each image area, and the magnitude of this phase difference corresponds to the subject distance reflected in each image area. The subject distance can be obtained from the phase difference. A distance measuring device based on such a principle is well known in an autofocus device of a camera. In the present invention, as shown in FIG. 4, the lines connecting the subject and the camera in the three image areas are referred to as an upper end vector V _U , a center vector V _C , and a lower end vector V _L , respectively. In addition, the distance to the subject reflected in each image area is referred to as the vector length from the camera to the subject, and the vector lengths of the upper end image area U, the central image area C, and the lower end image area L are the upper end vector length. , The center vector length and the bottom vector length.

Assuming that a visually handicapped person uses the walking assistance apparatus according to the present invention while wearing it on the head, as shown in FIG. 4, the optical axis O of the camera 10 faces obliquely downward in front of the user. In FIG. 4, it is assumed that the user is standing on the flat ground 7. The central vector V _C is coincident with the optical axis O, the upper end vector V _U on its upper side around the central vector V _C is, there is a lower vector V _L on the lower side. The length of each vector is obtained by calculation based on the principle of distance measurement by the distance vector calculation unit 21. Referring to FIG. 4, is it possible to always observe the change in the length of each vector and determine that a wall, a step, or an obstacle has appeared in the front when a change of a certain level or more is observed? Seems like. However, it is not possible to make an accurate judgment as it is. First, in the embodiment shown in FIG. 4, when the vertical direction is Y and the horizontal direction is Z, the optical axis O of the camera is inclined with respect to both the Y direction and the Z direction. Therefore, in the image space captured by the camera 10, as shown in FIG. 5, on the assumption that the camera optical axis O is in the horizontal direction, it is assumed that the ground 7 as the subject has risen obliquely. The distance is measured, and the originally flat ground 7 is judged as if it was a slope.

Therefore, it is necessary to return to the coordinate system of the person who uses the walking support device. For this purpose, the angle of the camera optical axis O at that time is obtained, and coordinate rotation is applied by the obtained angle. FIG. 6 shows this state. In general, an angle sensor or an acceleration sensor is attached to the camera 10, and the angle of the optical axis O of the camera 10 in use can be detected based on outputs from these sensors. However, mounting the sensor increases the weight and power consumption, and it is difficult to ensure the synchronism between the image data and the data from the angle sensor or the acceleration sensor. Therefore, in this embodiment, the angle of the camera optical axis O is also calculated from image data obtained every moment. If the angle of the camera optical axis O is known, the coordinate rotation can be performed as described above to return to the human coordinate system. Thus, for the first time, it is possible to determine whether the tips of the upper end vector V _U , the center vector V _C , and the lower end vector V _L are flat road surfaces or uneven surfaces.

In using the visually impaired person walking support apparatus according to the present embodiment, first, processing before walking is performed, and the angle of the camera optical axis O is obtained. This pre-processing mode may be executed by a user by operating a switch as necessary. The user wears the apparatus and stands on a flat road, and operates the apparatus in the pretreatment mode. 7 and 8 show camera angle detection in this operation mode. The lengths of the above three vectors, that is, the upper end vector V _U , the center vector V _C , and the lower end vector V _L are obtained, and the combination of the two vectors is shown. By this, the camera angle is detected. The height of the tip of each vector, that is, the height of the ground 7 is substantially the same. The height H of the camera is also the same. Therefore, the camera height H is used as a common variable, and the angle of the camera optical axis O with respect to the vertical line is determined from a right triangle formed by the three vector lengths and the ground 7.

As shown in FIG. 7, the first camera angle D1 is obtained from two triangles including a triangle including the center vector V _C and a triangle including the upper end vector V _U. If the angle of the upper end vector V _U is D _U , the camera angle D1 is
D1 = tan ⁻¹ ((−V _C / V _U + cos D _U ) / sin D _U )
It can be calculated by the following formula.
As shown in FIG. 8, the second camera angle D2 is obtained from two triangles including a triangle including the center vector V _C and a triangle including the lower end vector V _L. When the angle of the lower end vector V _L is D _L , the camera angle D2 is
D2 = tan ⁻¹ ((V _C / V _L −cosD _L ) / sinD _L )
It can be calculated by the following formula.
These camera angle calculations are performed by the camera angle calculation unit 22 shown in FIG.

If the ground on which the user is standing is completely flat and the height of the camera does not fluctuate, the values of D1 and D2 will match, but in reality there will be a slight difference. The average (C_Deg_stand) is taken as the camera angle at that time.
C_Deg_stand = (D1 + D2) / 2
This calculation is performed by the camera angle average value calculation unit 24 shown in FIG. The camera angle average value is stored in an appropriate memory or the like.

As described above, the calculated camera angle includes an error, and the height position of the ground in each image region may be different. Therefore, as shown in FIG. The camera height in the image area is obtained, and the average value is obtained. In FIG. 9, assuming that the camera height from the ground in the upper end area of the screen is H _U , the camera height from the ground in the central area is H _C , and the camera height from the ground in the lower end area is H _L , Well,
H _U = V _U × cos (C_Deg_stand + D _U )
H _C = V _C × cos (C_Deg_stand)
H _L = V _L × cos (C_Deg_stand−D _L )
It can be calculated by the following formula. This calculation is performed by the camera height calculation unit 23 shown in FIG.
Each height average value is calculated in the camera height average calculator 25 shown in FIG. 1, it is stored in an appropriate memory as height reference value H _S. The equation at a height reference value H _S is as follows.
H _S = (H _U + H _C + H _L ) / 3

After the pre-processing before walking is performed as described above, walking is started. During walking, the angle of the camera optical axis changes as the user's posture changes slightly. FIG. 10 shows this variation. Based on the already calculated height reference value H _S and from the lengths of the vectors V _U , V _C , and V _L , the camera angles D _UW , D _CW , and D _LW during walking in each image region are calculated. Is always calculated at regular time intervals. The calculation formula is as follows.
^{_{D UW = cos -1 (H C}} / V U) -D U
D _CW = cos ⁻¹ (H _C / V _C )
^{_{D LW = cos -1 (H C}} / V L) + D L
Since the camera angle obtained by these equations includes an error, an average value of these camera angles is taken as a camera angle (C_Deg_Walking) as in the following equation.
C_Deg_Walking = (D _UW + D _CW + D _LW ) / 3

Further, camera heights H _UW , H _CW , and H _LW during walking are calculated by applying the following equations, and an average value of these heights _HA is calculated.
H _UW = V _U × cos (C_Deg_Walking + V _U )
H _CW = V _C × cos (C_Deg_Walking)
H _LW = V _L × cos (C_Deg_Walking−V _L )
H _A = (H _UW + H _CW + H _LW ) / 3

  The camera angle calculation and camera height calculation described above are applied only when walking on a flat road, and other types such as ascending stairs, descending stairs, obstacles, etc. appear on the screen. In order to do this, the spatial cross-sectional area calculation described below is performed. Regardless of whether the person is conscious or unconscious, the person captures the expanse of the space in front of him and acts while recognizing whether the captured space can move freely. FIG. 11 shows a model of a state when the visually impaired person walking support apparatus according to the present embodiment is worn and the user walks on a relatively flat ground having no obstacles.

FIG. 11 differs from what has been described so far in that it assumes a cross-sectional area of a space sandwiched between the distance vectors V _U , V _C , and V _L at the upper end, the center, and the lower end. In this embodiment, by calculating these cross-sectional areas, it is possible to quickly and easily identify the ascending stairs, descending stairs, obstacles, depressions, and the like. In FIG. 11, the space in front of the user is regarded as a space in which a person can move freely, and the cross-sectional areas of the three spaces are defined as flat road reference cross-sectional areas. One space bottom is delimited by the tip of the tip and the central vector V _C of the upper vector V _U, extending from these vectors tip vertically in a space of a square in which the top side is delimited by the camera height, the space a reference cross-sectional area and _{S US.} Another space is a rectangular space in which the base is divided by the tip of the center vector V _{C and} the tip of the bottom vector V _L , extends vertically from these vector tips, and the top is divided by the camera height position. a reference cross-sectional area of the space and _{S CS.} The other space is delimited by the intersection of the tip of the lower end vector V _L and the ground of the line perpendicularly descending from the camera, the base extends vertically from the tip of the vector and the intersection, and the upper side is at the camera height position. in the space of a square delimited, the reference cross-sectional area of the space between S _LS.

Each of the reference cross-sectional areas S _US , S _CS , S _LS is calculated by the following equation.
S _LS = (H _S + V _L × cos (C_Deg_Walking−D _L )) / 2
× V _L × sin (C_Deg_Walking-D _L )
S _CS = (V _L × cos (C_Deg_Walking−D _L )
+ V _C × cos (C_Deg_Walking−D _L )) / 2
× (V _C × sin (C_Deg_Walking)
−V _L × sin (C_Deg_Walking−D _L )
S _US = (V _U × cos (C_Deg_Walking + D _U )
+ V _C × cos (C_Deg_Walking)) / 2
× (V _U × sin (C_Deg_Walking + D _U )
−V _C × sin (C_Deg_Walking)
The height reference value H _S , the upper end vector angle D _U , and the lower end vector angle D _L are fixed values, and the camera angle C_Deg_Walking and the three distance vectors V _U , V _C , and V _L are fluctuation values.

Each space cross-sectional area is calculated by a space cross-sectional area calculating unit 26 shown in FIG. The calculated reference sectional areas S _US , S _CS , S _LS are stored in the reference space sectional area storage unit 51. Further, while the user is walking, that is, while the apparatus according to the present embodiment is in operation, the spatial cross-sectional area calculation unit 26 calculates the above three cross-sectional areas at regular time intervals based on the clock signal generated by the clock signal oscillator 40. Calculate. The cross-sectional data during walking, which changes every moment, is temporarily stored in the spatial cross-sectional area calculator 52 and updated to new data.

Assuming that the space cross-sectional areas corresponding to the upper end vector, the center vector, and the lower end vector, which change every moment during walking, are S _U , S _C , and S _L , each of the spatial cross-sectional areas S _U , S _C , and S _L above. The ratio with respect to the reference areas S _US , S _CS , S _LS is calculated by the space cross-sectional area ratio calculation unit 27 shown in FIG. When the spatial cross-sectional area ratios corresponding to the upper image area U, the central image area C, and the lower image area L are R _SU , R _SC , and R _SL , respectively,
R _SU = S _U / S _US
R _SC = _S C _{/ S CS}
R _SL = S _L / S _LS
It can be obtained by the following formula. By performing these calculations, it is possible to know how much the spatial cross-sectional area corresponding to each image region has changed with respect to the reference value. When the amount of change exceeds a certain threshold value, an alarm is issued assuming that a step or obstacle is approaching forward.

  FIG. 12 shows an example of a change in camera angle obtained during actual operation. The horizontal axis is the time axis, and the numbers attached to the horizontal axis indicate the number of captured image frames. Similarly for other graphs, the horizontal axis represents the time axis and the numbers represent the number of captured image frames. The camera angle fluctuates in a range of about 10 degrees as the user of the apparatus walks.

  FIG. 13 shows an example of changes in the respective space area ratios at the upper end, the center, and the lower end of the screen obtained during actual operation. Both FIG. 12 and FIG. 13 are graphs when walking on a flat road. When FIG. 12 and FIG. 13 are compared, it can be seen that the space area ratio also varies as the camera angle varies. Therefore, as described above, the average value of the camera height is used. FIG. 14 shows actual examples of camera height fluctuations at the top, center and bottom of the screen. Even when the user walks on a flat road surface, as shown in FIG. 14, the camera height fluctuates little by little, and the spatial area ratio obtained by the calculation fluctuates as it is. However, when the camera heights at the top, center, and bottom of the screen are averaged, flat height data with little fluctuation is obtained as shown in FIG. Therefore, the average height data does not fluctuate greatly unless the posture changes extremely, such as when the user sits down. Therefore, if the height data has little fluctuation as shown in FIG. It can be determined that the person is standing or walking on a flat road.

  Further, whether or not the user is walking can be detected by motion detection described later. FIG. 16 shows a change in walking distance that can be obtained by the motion detection, in other words, a change in walking speed. Referring to this graph, it seems that walking and stopping are frequently performed, but when the walking distance shown in FIG. 16 is integrated, the walking distance is accumulated in a gentle line shape as shown in FIG. Recognize.

  The same applies to the walking angle of the user, that is, the angle fluctuation in the horizontal plane of the user. FIG. 18 shows an example of the walking angle fluctuation that can be obtained by motion detection. According to FIG. 18, the walking angle is constantly changing in small increments. However, when the angle variation data is integrated, it becomes a relatively gently rising line as shown in FIG. 19, and there is no positive direction change, and it can be determined that the user is walking smoothly on a flat road. In other words, if the graphs fluctuate so as to exceed a predetermined threshold value, it can be determined that the road surface is not flat, such as a step or an obstacle.

Next, the determination operation according to the present embodiment when a flat road approaches a non-flat road will be described. FIG. 20 is a specific example of a state in which a user wearing the visually impaired person walking support apparatus according to the present embodiment stands on an ideal flat road. Each numerical value at this time is set as a model. Height reference value _{H S} for the camera 10 is 1.6 m, the angle C_Deg 50 ° of the optical axis of the camera to the camera angles, or a vertical line, the bottom of the space section of the top of the screen is 1.37 m, the bottom of the space section in the center of the screen 0.75 m, the bottom of the space section at the bottom of the screen is 1.16 m, the length of the distance vector V _{U in} the top area is 3.65 m, the length of the distance vector V _{C in} the center area is 2.49 m, the distance in the bottom area The length of the vector V _L is 1.98 m. Reference space sectional area of the top of the screen area obtained from these values _{S US} is 2.912M ^2, the reference space sectional area _{S CS} is 1.2 m ² of the screen center area, the reference space sectional area _{S LS} bottom of the screen area 1 .856 m ² .

As the user walks, it is assumed that the stairs are approaching as shown in FIG. The specifications of the stairs are 30 degrees, the tread is 0.38 m, and the kick is 0.22 m, based on general building standards. As shown in FIG. 21, in the state where the stairs are approaching only in the upper end area of the screen, H _S , C_Deg, the bottom of the screen center area, the bottom of the lower end area of the screen, V _C , V _L , and the spatial sectional area of the screen center area. The value of S _C , the spatial cross-sectional area S _{L at} the lower end area of the screen does not change. As shown in FIG. 21, when the distance vector V _U of the upper end region is applied to the vertical plane between the first and second steps of the staircase, and the length of the distance vector V _U at that time is 2.98 m, space sectional area _{S U} of the top of the screen area becomes 1.12 m ^2. That is, the change ratio of the distance vector _{V U} of the upper end whereas a 0.816, the rate of change of the spatial cross-sectional area becomes 0.384. Therefore, it can be understood that the staircase and other obstacles can be detected easily and reliably by obtaining the space sectional area and determining the ratio to the reference space sectional area.

  When the rate of change of the spatial sectional area obtained as described above exceeds a certain threshold value, the spatial sectional area ratio calculating unit 27 shown in FIG. 1 outputs an alarm signal, and the alarm device 30 is driven by this alarm signal. The user can be informed that the stairs are approaching. However, if imaging is performed at regular intervals, the spatial cross-sectional area and the spatial cross-sectional area ratio are calculated, and an alarm is issued immediately when the spatial cross-sectional area ratio exceeds a threshold value for the first time, the reliability of the alarm decreases. Therefore, an alarm is issued when the threshold value is exceeded continuously for a certain number of frames. Further, the camera angle C_Deg is held at the immediately preceding value when the counting of the rising staircase detection frame is started.

As mentioned above, the timing when the space cross-sectional area ratio exceeds the threshold and the alarm is output is
Total number of change detection frames / (number of system sampling frames / sec)
It becomes. If the walking speed of a person is 1 m / sec, the user will approach the stairs by xm after x seconds. Therefore, it is set so that a warning is issued before the user approaches the stairs.

FIG. 22 shows a model when the user gets down and approaches the stairs. The specifications of the stairs are the same as in FIG. In a state where the stairs are approaching only the upper end area of the screen, as in the case of FIG. 21, H _S , C_Deg, the bottom of the screen center area, the bottom of the lower end area of the screen, V _C , V _L , the space of the screen center area The values of the cross-sectional area S _C and the spatial cross-sectional area S _{L at} the lower end area of the screen do not change. When comes to the stairs down distance vector V _U of the upper end, the distance vector V _U length becomes suddenly longer, its tip reaches the like landings or walls of the stairs. Therefore, it is possible to easily recognize that the space area ratio at the upper end of the screen suddenly increases 3 to 4 times, and the stairs are approaching. In this case as well, an alarm is issued when the change rate of the spatial cross-sectional area exceeds the threshold value for a certain number of frames continuously.

FIG. 23 shows a model of the situation when the user approaches the tip of the station platform. The height of the home is 1.1 m in accordance with an example of the actual height of the home. In a state where the stairs are approaching only at the upper end area of the screen as shown in FIG. 23, as in the case of FIG. 22, H _S , C_Deg, the bottom of the screen center area, the bottom of the lower end area of the screen, V _C , V _L , The values of the spatial cross-sectional area S _{C in the} center area of the screen and the spatial cross-sectional area S _{L in} the bottom area of the screen do not change. Distance vector V _U of the upper end region Sashikakari the tip of the platform, and when passing the tip of the platform, the length of the distance vector V _U becomes suddenly longer, its tip reaches the laying surface of the line. The length of the distance vector V _U at this time exceeds 6 m. Therefore, it is possible to easily recognize that the space area ratio at the upper end of the screen suddenly increases 3 to 4 times and the front end of the platform approaches. In this case as well, an alarm is issued when the change rate of the spatial cross-sectional area exceeds the threshold value for a certain number of frames continuously.

The specification assumed in FIG. 23, about 3m front from the platform tip distance vector extends is suddenly length of V _U, begin to detect that the previously has fallen from the platform tip. If the rate of change of the spatial cross-sectional area is set to issue an alarm when the 10-frame threshold value is continuously exceeded, assuming that the person's walking speed is 1 m / sec, about 2 m before the platform tip. An alarm can be issued.

FIG. 24 schematically shows a situation when the user approaches the wall 9 that is an obstacle. In this case, the operation is substantially the same as the case where the ascending staircase described with reference to FIG. 21 is detected. In a state where the distance vector V _U of the upper end region looming wall 9 only in the top of the screen area is incoming in a vertical wall surface as shown in FIG. 24, is shortened length of the distance vector V _U of the upper end region, the top of the screen The area of the area is reduced. The values of H _S , C_Deg, the bottom of the screen center region, the bottom of the screen bottom region, V _C , V _L , the spatial cross-sectional area S _C of the screen central region, and the spatial cross-sectional area S _L of the screen bottom region remain unchanged. In the example shown in FIG. 24, the distance length of the vector _{V U} of the upper end region is reduced to 2.98M, the rate of change space sectional area _{S U} of the top of the screen area is a 1.0 m ² and 0.346 Become. Therefore, an alarm can be issued as in the example of FIG.

FIG. 25 shows an operation when the user further approaches 1.5 m to the wall 9 and the wall 9 approaches the center area of the screen. Length shorter 1.67m next distance vector V _U of the upper end region is blocked by the wall 9, also the length distance vector V _C of the central region is shielded by the walls 9 becomes 1.95m Yes. Distance vector _{V U,} the distance vector _{V C} is blocked by the wall 9, the space area _{S U} of the top of the screen area is further reduced and 0.248M ^2, the space sectional area _{S C} of the screen center region and 0.425M ² Get smaller. An alarm is issued continuously from the state shown in FIG.

As mentioned above, although the detection operation | movement of obstacles, such as a stairway and a wall by a present Example was demonstrated, according to a present Example, the user's walking action can be calculated. FIG. 26 explains the principle as a model, and shows how a user wearing a stereo camera acquires a subject image while walking. In FIG. 26, (1) shows the relationship between the camera 10 and the user and the subject 8 when the image of the (n-1) th frame is acquired, and (3) shows an example of the image of the (n-1) th frame. (2) shows the relationship between the camera 10 and the user and the subject 8 when acquiring the next frame, that is, the nth frame image, and (4) shows an example of the nth frame image. In this example, walking user a flat ground 7, they become the example object 8 on the front of the ground 7 is present, reflected statue of n-1 th frame object 8 to the top of the screen area in The image of the subject 8 is reflected at the center of the screen at the nth frame. A substantially similar image is reflected in both the right-eye camera and the left-eye camera. Here, either the left or right image is used.

  In FIG. 26, (5) shows a change in the relative relationship between the camera 10 and the subject 8 from the n−1th frame to the nth frame. The user approaches the subject 8, but when viewed from the camera 10, it appears that the subject 8 moves from the upper end side toward the center on the screen, and the subject 8 approaches the camera 10. Further, since the camera optical axis O at this time faces obliquely forward, as shown in (6), coordinate rotation is applied so that the camera optical axis O coincides with the horizontal line. Therefore, the actual subject 8 viewed from the camera 10 side moves obliquely downward and forward from the top and reaches the center of the screen as indicated by an arrow 64 in (7). In (7), the arrow indicated by reference numeral 64 indicates the relative movement locus of the actual subject 8 as viewed from the camera, and the angle of the movement locus with respect to the vertical line is the camera angle C_Deg shown in (5) and (6). It corresponds to. By recognizing the movement of the subject 8 as a change in the subject distance by the above-described distance vector calculation, it is possible to recognize that the distance from the camera 10 to the subject 8 has become shorter.

  In FIG. 26 (7), reference numeral 62 denotes an image plane of a subject image by the taking lens optical system. An arrow 65 shown on the imaging plane 62 shows how the relative movement of the subject 8 indicated by the arrow 64 with the camera 10 is reflected on the imaging plane 62. The movement of the subject 8 on the image plane 62 and the distance vector calculation can determine how the user of the camera 10 has acted.

  One element of the action of the user of the camera 10 is a walking distance. In this embodiment, the subject distance can be measured and the moving distance per unit time can be calculated. FIG. 27 shows an example of the movement distance per unit time. The walking distance of the camera user can be determined by integrating the moving distance per unit time. The walking distance and its accumulated value are calculated by the walking distance / walking angle calculation unit 34 based on the relative movement vector calculated by the relative movement vector calculation unit 32 shown in FIG. FIG. 28 shows an example in which the walking distance is accumulated.

  FIG. 29 and FIG. 30 show that the walking angle change or the rotation motion recognition in the horizontal plane can be recognized as another element of the camera user's action. As shown in FIGS. 29 and 30, a human being walking does not always walk straight forward, but the walking direction changes to the left or right, or the face direction always changes to the left or right. 29 and 30 show changes in the orientation of the user wearing a stereo camera composed of the right eye camera 11 and the left eye camera 12. The camera user does not always walk straight in front of the subject 8 in front, but the optical axis of the camera 10 swings left and right in accordance with a change in the walking direction or a change in the face direction.

  FIG. 30 shows how the subject 8 is seen by the camera 10 when the optical axis O of the camera swings from side to side. As the camera 10 rotates left and right, the relative positional relationship between the camera 10 and the subject 8 changes. The two subjects 8 shown in FIG. 30 indicate a subject that can be seen from the camera when the camera rotates to the right and a subject that can be seen from the camera when the camera rotates to the left. The rotational movement distance of the subject 8 viewed from the camera is indicated by an arrow 66 in FIG. The relative positional change of the subject 8 changes within a range indicated by an arrow 67 on the imaging plane 62 of the camera. Further, the rotation angle of the camera at this time is represented by θ. f indicates the focal length of the photographic lens, and Z indicates the distance from the camera to the subject 8.

The rotational movement distance seen from the camera indicated by the arrow 66 is:
Rotational movement distance = z / f × (projection plane pixel vector × pixel size)
Can also be obtained at
Walking angle / θ = tan ⁻¹ (rotational distance / Z)
The following equation holds. In this way, it is possible to know the rotation operation of the camera user. FIGS. 29 and 30 are examples in which the subject 8 is an object that does not move. When the subject is an object that moves on the ground (road surface), the movement is added. In this embodiment, the screen is divided into three parts, and the subject in each divided screen is imaged and calculated at a fixed time interval. Therefore, it is possible to identify whether the subject is a stationary subject or a moving subject. In addition, there is no error in recognition of obstacles.

FIG. 31 and subsequent figures show examples of actual images when the camera user rotates as described above, and examples of various signal waveforms obtained thereby. FIG. 31 shows an example of an image reflected on the camera on the station platform. The leading edge of the platform crosses the screen substantially along the diagonal line, and the rail is reflected in front of it. Of course, the rail laying surface is about 1.1 m lower than the platform surface. The left image in FIG. 31 is an image of the (n-1) th frame, the right image is an image of the subsequent n frame, and shows a change in the image when the camera user rotates from left to right. ing. Either the left image or the right image of the stereo camera may be used.

  FIG. 32 shows a rotation angle for each frame in the case of the right rotation, and FIG. 33 shows a case where the rotation angles are accumulated (integrated). Similarly, FIG. 34 shows a rotation angle for each frame in the case of left rotation, and FIG. 35 shows a case where the rotation angles are accumulated (integrated). During rotation, the rotation angle changes in small increments as shown in FIGS. Since the rotation angle is changing while the rotation angle is changing in small increments, it is meaningless to perform measurement and calculation during this time. When a state in which there is no small change in the rotation angle continues for a certain period of time, it can be determined that the user has stopped the rotation. Therefore, the rotation angle is obtained from the accumulated values as shown in FIGS.

  The change of the image as shown in FIG. 31 and the change of the rotation angle shown in FIGS. 32 to 35 accompanying it also occur when the camera user walks. Therefore, the change in the rotation angle shown in FIGS. 32 and 34 can be regarded as a change in the magnitude of the walking vector. Therefore, when the walking vector is extremely small, it is considered that the vehicle is stopped. Further, when the fixed frame number stop state continues, each data is reset once.

  Corresponding to the walking distance change and the rotation angle change described above, the walking distance and walking angle at that time are calculated by the walking distance / walking angle calculator 34, and the walking distance and the rotation angle change are The space area ratio calculation unit 27 calculates the space area ratio. When the change in the space area ratio exceeds a predetermined threshold value, an alarm is issued from the alarm device 30 according to the determination of the alarm determination unit 28.

The visually impaired person walking support device according to the present invention issues a warning when a step, a wall, or other obstacle approaches, and requires a large-scale system, a special warning device, or an output device. Therefore, it can be provided at a low cost and does not require special training, and can be used conveniently.
In addition, for example, if the present invention is attached to an unmanned search robot or the like, it is possible to detect that the robot has approached an obstacle, so that a smooth search can be performed while avoiding the obstacle.
By using a camera that can detect infrared rays, such as mounting an infrared filter on the camera, steps, walls, and other obstacles can be recognized even in the dark. Therefore, the device of the present invention is useful not only for visually impaired people but also for healthy people.

DESCRIPTION OF SYMBOLS 10 Stereo camera 18 Image processing apparatus 20 Calculation apparatus 21 Distance vector calculation part 22 Camera angle calculation part 23 Camera height calculation part 24 Camera angle average value calculation part 25 Camera height average value calculation part 26 Spatial cross-sectional area calculation part 27 Spatial section Area ratio calculation unit 30 Alarm device 50 Storage unit 51 Reference space sectional area storage unit 52 Spatial sectional area storage unit

Claims (6)

A stereo camera capable of obtaining a pair of images by capturing an image of the same subject with a certain baseline length;
An image processing apparatus for dividing the pair of images into a plurality of vertical regions;
From a pair of images of the divided plurality regions, the distance vector calculator for calculating the length of the distance vector is a line connecting the object and the camera that is imaged in each region,
A camera angle calculation unit that calculates a camera angle with respect to a vertical line from the length of the distance vector for each region for each region;
From the distance vector length in each region and the camera angle for each region, a camera height calculation unit that calculates the camera height from the flat ground in each region ,
A camera height average value calculation unit that calculates an average value of the camera height in each of the above areas and sets it as a camera height reference value;
Spatial cross-sectional area that calculates a square cross-sectional area in which the base is delimited by the tip of each distance vector of two adjacent areas, extends vertically from the top of these distance vectors, and the top is delimited by the camera height position. An arithmetic unit;
When a reference space sectional area of the space cross-sectional area in the respective regions obtained by the spatial without walking only One obstacle in the flat surface, the ratio of the spatial cross-sectional area of the production time with respect to the reference space cross-sectional area of each region A spatial cross-sectional area ratio calculation unit that outputs an alarm signal when becomes more than a certain value,
An alarm device that operates in response to the output of the alarm signal and issues an alarm ;
The space cross-sectional area calculation unit calculates the space cross-sectional area at the time of actual operation in each region at a constant time interval,
The spatial cross-sectional area ratio calculation unit is a visually impaired person walking support device that calculates a ratio of the latest actual spatial cross-sectional area and the reference spatial cross-sectional area . The image processing apparatus divides the image into three regions in the vertical direction ,
The distance vector calculation unit is configured to calculate an upper end distance vector, a center distance vector, and a lower end distance vector for each of the three regions.
A camera angle calculation unit that calculates a camera angle with respect to a vertical line from the length of the three distance vectors for each of the regions;
The visually impaired person walking support apparatus according to claim 1 , wherein the camera height calculation unit calculates the camera height for each of the regions based on the camera angle for each of the regions . The visually impaired person walking support apparatus according to claim 1 , further comprising a camera angle average value calculation unit that calculates an average value of camera angles for each image region and sets the average value as a camera angle . The reference spatial cross-sectional area data calculated by the spatial cross-sectional area calculation unit is stored in the storage unit, and is used for calculation of a ratio with the spatial cross-sectional area at the time of actual operation. apparatus. The image processing apparatus can output a multi-frame image including an image of a specific frame and an image immediately before the specific frame,
A relative movement vector calculation unit that calculates the length of the relative movement vector with respect to the subject from the images of the plurality of frames,
Space sectional area ratio calculating section, visually impaired persons support device according to any one of claims 1 to 4 is intended for calculating the cross-sectional open area ratio in each frame corresponding to a change in the relative movement vector. Space sectional area ratio calculation section, according to any one of claims 1 to 5 for outputting an alarm signal when the ratio of the spatial cross-sectional area of the production time for the reference space sectional area continuously certain number of frames exceeds a specific Visually impaired person walking support device.

Images