JP2023073100A - Walking support system - Google Patents

Abstract

To provide a walking support system that enables a signal to be appropriately recognized.SOLUTION: A correction angle which is an angle between a reference transverse direction which is a direction orthogonal to an extending direction of a white line of a crosswalk and an imaging direction of a camera 20 is calculated by a yawing angle calculation unit 85b on the basis of an image obtained by the camera 20, and a movement for making the correction angle small is commanded to a pedestrian by the vibration of a vibration generator 50. Thereby, it is possible to substantially match the imaging direction of the camera 20 with the reference transverse direction, and it is possible to cause a signal to exist in the angle of view of the camera 20 so that the signal can be appropriately recognized even in a situation that the signal does not exist in the angle of view of the camera 20 due to a fact that the correction angle is large.SELECTED DRAWING: Figure 3

Description

The present invention relates to a walking assistance system. In particular, the present invention relates to an improvement for appropriately recognizing a traffic signal in providing walking assistance to a pedestrian such as a visually handicapped person.

2. Description of the Related Art As a system (walking support system) that performs notification for walking support of a pedestrian such as a visually impaired person, the one disclosed in Patent Document 1 is known. This patent document 1 discloses a system for assisting the walking of a person (visually impaired person) who acts without using sight, in which an image captured by a camera carried by the visually impaired person and a pre-stored reference image are used. It is disclosed that walking assistance is provided to a visually impaired person by voice or the like while matching the .

Republished patent WO2018/025531

By the way, in this type of walking support system, when a pedestrian (visually impaired person, etc.) is approaching a crosswalk and the traffic light (for example, a traffic light for pedestrians) is red, the pedestrian cannot When the traffic light turns green while the pedestrian is stopped in front of the crosswalk, the pedestrian is notified to walk (cross). need to do For this reason, in order to appropriately give various notifications to pedestrians (notice to stop walking in front of the crosswalk and notification to start crossing after that), it is necessary to make sure that the traffic light can be photographed by an image acquisition means such as a camera. Therefore, it is essential to accurately recognize the presence or absence of a traffic signal and the state of the traffic signal (whether the signal is green or red).

However, since the photographing direction of the image acquisition means substantially coincides with the walking direction of the pedestrian, even if the pedestrian is approaching a pedestrian crossing with a traffic light, the pedestrian's walking direction (crossing) Depending on the direction to the sidewalk), there is a possibility that a traffic signal does not exist within the angle of view of the image acquisition means, and in this case, the existence of the traffic signal cannot be recognized.

Specifically, FIG. 28 is a plan view showing each route (approaching route) when a pedestrian approaches the crosswalk CW. The approach route indicated by the arrow a in the figure is the direction in which the walking direction toward the pedestrian crossing CW is orthogonal to the longitudinal direction of the white line WL (the direction in which the pedestrian crosses the road by the shortest route; is facing the crosswalk). In this case, when the appropriate transverse direction (the direction orthogonal to the longitudinal direction of the white line WL) is the x-direction and the actual transverse direction is the x'-direction, the x-direction and the x'-direction substantially coincide with each other. Become. That is, the angle θ between the vector x and the vector x' is approximately 0°. FIG. 29 shows an example of an image captured by the image acquiring means in this state. Thus, in the case of the approach route a, the traffic light TL is photographed by the image acquisition means, and the state of the traffic light TL can be accurately recognized.

In general, guidance equipment GE such as Braille blocks is installed on the road (on the sidewalk) before the crosswalk CW, and visually impaired people can cross in the appropriate crossing direction (x' direction) according to the guidance of this guidance equipment GE. The walking direction will be modified to match the direction (x direction).

However, in the case of sidewalks without guidance equipment GE such as Braille blocks, or when the width of the sidewalk is narrow, the crossing direction (x' direction) should substantially match the appropriate crossing direction (x direction). cannot be compensated for, and the walking direction may deviate from the proper direction. The approach route indicated by the arrow b in FIG. 28 is a case where the pedestrian walks from the right end on the front side to the left end on the back side of the crosswalk CW. FIG. 30 shows an example of an image captured by the image acquiring means in this state. Also, the approach route indicated by the arrow c in FIG. 28 is a case where the pedestrian walks from the left end on the front side to the right end on the back side of the pedestrian crossing CW. FIG. 31 shows an example of an image captured by the image acquiring means in this state. As described above, the traffic light TL does not exist within the angle of view of the image acquisition means on the approach route c (see the broken line in FIG. 31), and in this case, the presence of the traffic light TL cannot be recognized. In other words, even though the traffic light TL exists ahead, the presence of the traffic light TL cannot be recognized. As a result, it is not possible to recognize the presence or absence of the traffic light TL and the state of the traffic light TL (whether it is a green light or a red light), and it is not possible to notify pedestrians (a notification instructing to stop or a notification instructing to cross) ) was difficult to perform properly. Thus, whether or not the traffic light TL exists within the angle of view of the image acquisition means depends on the photographing direction of the image acquisition means (when the pedestrian is walking, the direction of It is greatly influenced by the direction the pedestrian is facing (if the pedestrian is facing).

The present invention has been made in view of the above points, and an object of the present invention is to provide a walking support system that enables proper recognition of a traffic light.

The solution means of the present invention for achieving the above object is to recognize a traffic signal located in front of the pedestrian in the walking direction and instruct the pedestrian to stop before the crosswalk according to the state of the traffic signal. and a walking support system that performs at least one of instructions to start crossing the pedestrian crossing. This walking support system includes image acquisition means capable of acquiring an image in front of the pedestrian in the direction in which the pedestrian is facing as a photographing direction, and based on the image acquired by the image acquisition means, a correction angle calculator for calculating a correction angle which is an angle between a reference crossing direction which is a direction orthogonal to the extending direction of the white line of the pedestrian crossing and the photographing direction of the image acquisition means; and an instruction means for instructing the pedestrian to perform an operation for reducing the correction angle calculated by the calculation unit.

The term "pedestrian" as used herein is a concept that includes not only people who are actually walking, but also people who stop walking due to, for example, waiting for a traffic light in front of a pedestrian crossing.

According to the specified matter, in a situation where the pedestrian approaches or reaches the front of the crosswalk, the image acquisition means acquires an image in front of the pedestrian in the walking direction, and based on the acquired image, the correction angle calculation unit A correction angle between the reference transverse direction and the imaging direction of the image acquisition means is calculated. Then, the instruction means instructs the pedestrian to perform an action for reducing the correction angle. For example, the pedestrian is instructed to change the orientation of the body. The pedestrian who receives the instruction from the instruction means changes the direction of the body, for example, so as to reduce the correction angle, and thereby the photographing direction of the image acquisition means substantially coincides with the reference transverse direction. Therefore, even if the traffic light does not exist within the angle of view of the image acquisition means due to the correction angle being large, the traffic light can be made to exist within the angle of view of the image acquisition means. is possible, and it becomes possible to appropriately recognize the traffic light.

Further, the pedestrian is a visually impaired person holding a white cane, the image acquiring means is built in the white cane, and the corrected angle calculating section calculates the A swing angle of the white cane by calculating the inclination when the white line of the pedestrian crossing is inclined on the image due to the white cane being swung to the left and right, and correcting the image based on the inclination. is zero, and a yaw angle calculator for calculating the corrected angle based on the image created by the rolling corrector.

In the image acquired by the image acquisition means, the angle of inclination of the white line of the pedestrian crossing (the angle of inclination with respect to the horizontal direction on the image) indicates the orientation of the pedestrian (visually impaired person) with respect to the pedestrian crossing (the extension direction of the white line of the pedestrian crossing). It is possible to calculate the correction angle that is correlated with a direction that is angled with respect to an appropriate direction that is an orthogonal direction. However, a visually impaired person walks while checking the road surface condition in front by waving a white cane left and right. In other words, by swinging the white cane left and right, the white line of the pedestrian crossing is tilted with respect to the horizontal direction on the image due to the fact that the imaging optical axis of the image acquisition means built in the white cane is swung left and right. I have something to do. In other words, the reason why the white line of the pedestrian crossing is inclined with respect to the horizontal direction on this image is not only the orientation of the visually impaired person to the pedestrian crossing but also the swinging of the white cane left and right by the visually impaired person. For this reason, in the present solution means, an image is created by correcting the image acquired by the image acquisition means by the correction operation of the rolling correction section assuming that the swing angle of the white cane is zero. In other words, an image is created in which the inclination of the white lines of the pedestrian crossing caused by swinging the white cane left and right is eliminated. Then, the yawing angle calculation unit estimates the inclination angle of the white line caused by the orientation of the visually impaired person with respect to the pedestrian crossing, and calculates the difference between the direction perpendicular to the longitudinal direction of the white line and the direction the visually impaired person is facing. A correction angle between the two is calculated, and the visually impaired person is instructed to perform an operation for reducing this correction angle. Then, when the visually impaired person changes the direction according to this instruction, the corrected angle becomes smaller and an image in which the traffic light exists within the angle of view of the image acquisition means is obtained. As a result, it is possible to appropriately recognize the traffic light by giving appropriate instructions to the visually impaired person without being affected by the left and right swinging of the white cane.

Further, the yawing angle calculation unit is configured to calculate the correction angle based on the inclination angle of the white line of the pedestrian crossing with respect to the horizontal direction in the image created by the rolling correction unit.

If the direction perpendicular to the longitudinal direction of the white line of the pedestrian crossing and the direction in which the pedestrian is facing (corresponding to the photographing direction of the image acquisition means) are different, the angle between these directions (the corrected angle ) appears as an angle of inclination of the white line with respect to the horizontal direction on the image. Therefore, the yawing angle calculator calculates the correction angle from this tilt angle. This makes it possible to accurately obtain the correction angle.

Further, the yaw angle calculation unit uses a trained model based on data annotated in advance with respect to the image created by the rolling correction unit, and a first recognition unit that recognizes the white line of the pedestrian crossing; a second recognition unit that recognizes the white line of the pedestrian crossing by computer vision for the image created by the rolling correction unit; a result of the recognition operation by the first recognition unit and a result of the recognition operation by the second recognition unit; a white line identifying unit that identifies the white line of the pedestrian crossing in the image based on each of them, and the correction is performed based on the angle of inclination of the white line identified by the white line identifying unit with respect to the horizontal direction on the image. It is configured to calculate the angle.

According to this, it is possible to obtain sufficient recognition accuracy of the white line of the pedestrian crossing in the image, and it is possible to calculate the corrected angle with high accuracy.

A pedestrian crossing detection unit for detecting the pedestrian crossing based on the image acquired by the image acquisition means, and a coordinate position on the image of the white line located farthest on the image among the white lines of the pedestrian crossing. and a storage unit that stores a plurality of image data specifying the relationship between the correction angle and the correction angle, and the correction angle calculation unit is positioned farthest among the white lines on the pedestrian crossing detected by the pedestrian crossing detection unit. In addition to obtaining the coordinate position of the white line to be crossed, image data matching the coordinate position of the farthest white line detected by the pedestrian crossing detection unit among the white lines in each image data stored in the storage unit is extracted. Then, the corrected angle is calculated based on this image data.

According to this, the correction angle can be obtained by extracting from each image data stored in the storage section image data that matches the coordinate position of the farthest white line in the image acquired by the image acquisition means. Therefore, it is possible to simplify the processing operation for obtaining the corrected angle, and to reduce the load on the control system.

Further, the instruction means is built in a white cane used by the visually impaired person who is a pedestrian, and instructs the visually impaired person by vibration or voice to perform an operation for reducing the correction angle. It is configured.

As a result, it is possible to appropriately instruct a visually impaired person walking while holding a white cane to orient himself/herself so that he/she can recognize the traffic signal.

In the present invention, based on the image acquired by the image acquisition means, the correction is the angle between the reference transverse direction, which is the direction orthogonal to the extending direction of the white lines of the pedestrian crossing, and the photographing direction of the image acquisition means. The angle is calculated, and the pedestrian is instructed to perform an action to reduce the corrected angle. As a result, it is possible to make the photographing direction of the image acquisition means approximately coincide with the reference transverse direction. Even if there is, it is possible to make the traffic signal exist within the angle of view of the image acquisition means, and it is possible to appropriately recognize the traffic signal.

It is a figure which shows the white cane which incorporated the walking assistance system which concerns on embodiment. It is the schematic which shows the inside of the grip part of a white cane. 1 is a block diagram showing a schematic configuration of a control system of a walking support system; FIG. FIG. 3 is a diagram showing an example of an image captured by a camera when a visually impaired person faces a pedestrian crossing; 1 is a front view showing a visually impaired person walking while swinging a white cane; FIG. FIG. 6 is a diagram showing an example of an image captured by a camera when the white cane is swung to the left in the state shown in FIG. 5; 6 is a diagram showing an example of an image captured by a camera when the white cane is swung to the right in the state shown in FIG. 5; FIG. FIG. 3 is a side view of a visually impaired person for explaining a road surface-based global coordinate system; FIG. 4 is a side view showing part of a white cane for explaining a cane coordinate system and a camera coordinate system; FIG. 4 is a front view showing a part of a white cane for explaining a cane coordinate system and a camera coordinate system; FIG. 10 is a diagram showing an example of an image captured by a camera when a white cane is swung to the left; FIG. 12 is a diagram showing an example of an image after the image shown in FIG. 11 is corrected by a rolling correction unit; FIG. 10 is a diagram showing another example of an image after correction by the rolling correction unit; FIG. 10 is a diagram for explaining a setting operation of a Boundary Box by a first recognition unit; FIG. 11 is a diagram for explaining a setting operation of a Boundary Box by a second recognition unit; FIG. 10 is a diagram for explaining an angle formed by a boundary box of an area recognized as a white line by Computer Vision and a horizontal line on an image; FIG. 4 is a diagram showing an example of an image captured by a camera when a visually impaired person is walking toward a pedestrian crossing; FIG. 4 is a diagram showing an example of an image captured by a camera at the timing when a visually impaired person reaches a pedestrian crossing; FIG. 4 is a diagram showing an example of an image captured by a camera when a visually impaired person is crossing a pedestrian crossing; FIG. 10 is a diagram showing an example of an image captured by a camera when a visually impaired person crossing a pedestrian crossing is walking in a direction deviating to the right side of the pedestrian crossing; FIG. 10 is a diagram showing an example of an image captured by a camera when a visually impaired person crossing a pedestrian crossing is walking toward the left side of the pedestrian crossing. FIG. 10 is a diagram showing an image of a recognized pedestrian crossing and a traffic light; FIG. 10 is a diagram for explaining the dimensions of each part in the boundary box of the recognized white line of the pedestrian crossing; It is a flowchart figure which shows the procedure of the walking assistance operation|movement by a walking assistance system. FIG. 10 is a diagram showing an example of an image captured by a camera in modification 1; FIG. 10 is a diagram for explaining the coordinate position and the length dimension of the Boundary Box at the farthest position on the image in Modified Example 2; FIG. 10 is a diagram showing an example of a stored past image; FIG. FIG. 4 is a plan view showing each approach route when a pedestrian approaches a pedestrian crossing; FIG. 10 is a diagram showing an example of an image captured by a camera when approaching route a; FIG. 10 is a diagram showing an example of an image captured by a camera when approaching route b; FIG. 10 is a diagram showing an example of an image captured by a camera when approaching route c.

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings. In this embodiment, a description will be given of a case in which a walking support system according to the present invention is incorporated in a white cane used by a visually impaired person. It should be noted that pedestrians in the present invention are not limited to visually impaired persons.

-Overview of White Cane-
FIG. 1 is a diagram showing a white cane 1 incorporating a walking support system 10 according to this embodiment. As shown in FIG. 1, the white cane 1 includes a shaft portion 2, a grip portion 3, and a tip portion (tip) 4. As shown in FIG.

The shaft portion 2 is a hollow rod having a substantially circular cross section, and is made of aluminum alloy, glass fiber reinforced resin, carbon fiber reinforced resin, or the like.

The grip portion 3 is provided at the base end portion (upper end portion) of the shaft portion 2, and is configured by attaching a cover 31 made of an elastic body such as rubber. In addition, the grip part 3 of the white cane 1 in this embodiment is slightly curved on the tip side (upper side in FIG. 1) in consideration of ease of holding and slip resistance when gripped by a visually impaired person (pedestrian). It has a shape.

The tip portion 4 is a substantially bottomed tubular member made of a hard synthetic resin or the like, and is externally inserted onto the tip portion of the shaft portion 2 and fixed by means of adhesion, screwing, or the like. The tip portion 4 has a hemispherical end face on the tip side.

The white cane 1 according to the present embodiment is a non-foldable straight cane, but it may be made foldable or extendable at one or more points in the middle of the shaft portion 2 .

- Configuration of walking support system -
A feature of this embodiment resides in a walking support system 10 incorporated in the white cane 1 . The walking support system 10 will be described below.

FIG. 2 is a schematic diagram showing the inside of the grip portion 3 of the white cane 1. As shown in FIG. As shown in FIG. 2, the walking support system 10 according to this embodiment is built in the white cane 1. As shown in FIG. FIG. 3 is a block diagram showing a schematic configuration of the control system of the walking support system 10. As shown in FIG.

As shown in these figures, the walking support system 10 includes a camera (image acquisition means) 20, a short-range wireless communication device 40, a vibration generator (instruction means) 50, a battery 60, a charging socket 70, an inertial measurement device (hereinafter referred to as , IMU) 90, a control device 80, and the like.

The camera 20 is embedded in the front surface of the grip portion 3 (the surface facing the traveling direction of the visually impaired person) at the base portion of the grip portion 3, and the front side of the visually impaired person's traveling direction (the pedestrian in the present invention faces). direction). The camera 20 is composed of, for example, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Also, the configuration and arrangement position of the camera 20 are not limited to those described above.

A feature of this camera 20 is that it is an image in front of the walking direction of the visually impaired person, and when the visually impaired person reaches the pedestrian crossing, it is the white line of the pedestrian crossing that is closest to the visually impaired person. It is configured as a wide-angle camera capable of capturing images that include both white lines in position and traffic lights (eg, pedestrian traffic lights) located in front of the visually impaired. In other words, when the visually impaired person reaches the front of the crosswalk, it is installed at the frontmost white line on the pedestrian crossing that exists near the feet of the visually impaired person (a position slightly in front of the feet) and at the crossing point. It has a configuration that allows you to shoot both the signal and the traffic light. The viewing angle required for this camera 20 is appropriately set so that an image including both the white line (the white line of the pedestrian crossing) and the traffic signal at the position closest to the visually impaired person can be obtained (captured) as described above. be done.

The short-range wireless communication device 40 is a wireless communication device for performing short-range wireless communication between the camera 20 and the IMU (Inertial Measurement Unit) 90 and the control device 80 . For example, short-range wireless communication is performed between the camera 20 and the IMU 90 and the control device 80 by means of communication means such as well-known Bluetooth (registered trademark), and information of images captured by the camera 20 and information acquired by the IMU 90 (described later) 3-axis acceleration and 3-axis angular velocity) is wirelessly transmitted to the control device 80 .

The vibration generator 50 is arranged above the camera 20 at the base of the grip portion 3 . This vibration generator 50 vibrates with the operation of the built-in motor, and by transmitting the vibration to the grip portion 3, various notifications ( instructions) can be performed. A specific example of notification to the visually impaired by vibration of the vibration generator 50 will be described later.

The battery 60 is composed of a secondary battery that stores electric power for the camera 20 , the short-range wireless communication device 40 , the vibration generator 50 , the control device 80 and the IMU 90 .

The charging socket 70 is a portion to which a charging cable is connected when power is stored in the battery 60 . For example, the charging cable is connected when a visually impaired person charges the battery 60 from a household power supply while at home.

The IMU 90 is arranged below the camera 20 . This IMU 90 is equipped with a 3-axis gyro and a 3-direction accelerometer, and measures 3-dimensional acceleration (3-axis acceleration) and angular velocity (3-axis angular velocity) at the position where the IMU 90 is arranged on the white cane 1. do. In some cases, the IMU 90 incorporates sensors such as a pressure gauge and GPS for improving reliability.

The control device 80 includes, for example, a processor such as a CPU (Central Processing Unit), a ROM (Read-Only Memory) that stores control programs, a RAM (Random-Access Memory) that temporarily stores data, and an input/output port. etc.

The control device 80 includes an information receiving section 81, a crosswalk detecting section 82, a traffic light recognizing section 83, a switching recognizing section 84, a correction angle calculating section 85, an information transmitting section 86 as functional sections realized by the control program. It has An outline of the functions of these units will be described below.

(Information receiver)
The information receiving section 81 receives the information of the image captured by the camera 20 from the camera 20 via the short-range wireless communication device 40 at predetermined time intervals. The information receiving unit 81 also receives information on the three-axis acceleration and three-axis angular velocity measured by the IMU 90 from the IMU 90 via the short-range wireless communication device 40 at predetermined time intervals. It is preferable that the reception timings of the image information and the information of the triaxial acceleration and triaxial angular velocity are synchronized.

(Pedestrian crossing detector)
The pedestrian crossing detection unit 82 corrects (for example, a rolling correction unit 85a described later) based on the information of the image received by the information reception unit 81 (information of the image captured by the camera 20), and the yaw angle calculation unit described later. Based on the image obtained by the visually impaired person changing the direction by instructing the visually impaired person to change the direction based on the corrected angle calculation result of 85b), the pedestrian crossing can be recognized and the pedestrian crossing Detect the position of each white line in

Specifically, an example of an image taken by the camera 20 when the visually impaired person stops properly in front of the crosswalk CW (faces the crosswalk CW) will be described with reference to FIG. 4 is acquired, a boundary box (see the dashed dotted line in FIG. 4) is set for a plurality of white lines WL1 to WL4 forming the pedestrian crossing CW. The details of this Boundary Box setting operation will be described later.

Then, the pedestrian crossing detection unit 82 detects the lower end position (see LN in FIG. 4) of the Boundary Box closest to the pedestrian among these Boundary Boxes. In this embodiment, a Boundary Box is set for each of the white lines WL1 to WL4, and the lower end position LN of the Boundary Box located at the bottom of the image is detected. , the lower end position of the lowermost white line WL1 among the plurality of white lines WL1 to WL4 determined on the image may be detected.

As will be described later, the Boundary Box is used to specify the stopping position and direction of the visually impaired person, the position of the traffic light TL, the traveling direction when the visually impaired person crosses the pedestrian crossing CW, and the pedestrian crossing CW. It is used to determine the completion of crossing the road, etc. Details of these will be described later.

(traffic signal recognition unit)
The traffic light recognition unit 83 recognizes the position of the traffic light TL and recognizes the state of the traffic light TL (recognizes whether the traffic light is red or green) from the information of the image (FIG. 4) described above. Specifically, the presence area of the traffic light TL on the image is estimated, and the position of the traffic light TL within the estimated area and the state of the traffic light TL are recognized. In estimating the presence area of the traffic light TL on the image, the coordinates in the image of the farthest boundary box among the boundary boxes set for the white lines WL1 to WL4 recognized as described above are specified, As shown in FIG. 4, a rectangle (having a width of w _s and a height of _hs rectangle) is defined, and the crop range is output as the area A of the traffic light TL (existence area of the traffic light TL). At this time, the crop range may be square or rectangular. Although the dimensions w _s and h _s can be set arbitrarily, they are empirically or experimentally set so as to include the arrangement position of the traffic light TL. In addition, a general object detection algorithm or a rule-based algorithm is used for the determination of the state of the traffic light (detection of color) performed by the traffic light recognition unit 83 .

(switch recognition part)
The switching recognition unit 84 recognizes that the state of the traffic light TL recognized by the traffic light recognition unit 83 has changed from red to green. Upon recognizing the switching of the signal, the switching recognizing section 84 transmits a switching signal to the information transmitting section 86 . This switching signal is transmitted from the information transmitting section 86 to the vibration generator 50 . The vibration generator 50 vibrates in a predetermined pattern in conjunction with receiving this switching signal, and instructs the visually impaired person to cross the pedestrian crossing due to the traffic light TL switching from red to green. A notice of permission (notification of start of crossing) is sent.

(Description of the problem and outline of the invention)
Here, the problems to be solved by the present invention will be described. As described above, the visually impaired person crosses the pedestrian crossing along the approach route indicated by the arrow c in FIG. In this case, the image captured by the camera 20 is in the state shown in FIG. 31, for example. Thus, the traffic light TL does not exist within the angle of view of the camera 20 on the approach route c, and in this case, the presence of the traffic light TL cannot be recognized. In other words, even though the traffic light TL exists ahead, the presence of the traffic light TL cannot be recognized. As a result, it is not possible to recognize the presence or absence of the traffic light TL and the state of the traffic light TL (whether it is a green light or a red light). There is a problem that it becomes difficult to properly perform notification).

In this embodiment, in view of this point, in the image acquired by the camera 20, the angle of inclination of the white line WL of the crosswalk CW (the angle of inclination with respect to the horizontal direction on the image) indicates that the visually impaired person Inclination angle with respect to orientation (appropriate direction that is perpendicular to the extending direction of the white line WL of the pedestrian crossing CW (x direction in FIG. 28; direction directly facing the pedestrian crossing CW; reference crossing direction in the present invention); correction angle ), instruct the visually impaired person to face in the proper direction (the direction perpendicular to the longitudinal direction of the white line WL of the pedestrian crossing CW), and follow this instruction to change the direction of the visually impaired person. 4 (an image when a visually impaired person faces the crosswalk CW) as shown in FIG.

However, the visually handicapped person walks while swinging the white cane 1 to the left and right to check the condition of the road ahead. That is, by swinging the white cane 1 left and right, the photographing optical axis of the camera 20 built in the white cane 1 is swung left and right. As a result, the direction of the photographing optical axis of the camera 20 changes greatly with respect to the walking direction of the visually impaired person, and this influence causes the white line WL of the pedestrian crossing CW to be inclined with respect to the horizontal direction on the image. Become. In other words, the reason why the white line WL of the pedestrian crossing CW is inclined with respect to the horizontal direction on this image is not only the direction of the visually impaired person to the pedestrian crossing CW , a visually impaired person swinging the white cane 1 left and right.

For this reason, in the present embodiment, the correction angle calculation unit 85 is provided with a rolling correction unit 85a and a yaw angle calculation unit 85b as functional units realized by the control program, and the image captured by the camera 20 is On the other hand, the correction operation by the rolling correction unit 85a corrects the inclination of the white line WL of the pedestrian crossing CW caused by the visually impaired person swinging the white cane 1 left and right. After creating an image from which the inclination of the white line WL of the pedestrian crossing CW is eliminated, the yaw angle calculator 85b estimates the inclination angle of the white line WL caused by the orientation of the visually impaired person with respect to the pedestrian crossing CW. Change the direction for visually impaired people so that the angle (correction angle) between the x direction, which is the direction orthogonal to the longitudinal direction of the , and the x' direction, which is the actual transverse direction, is approximately 0° It is intended to give instructions to If the visually handicapped person changes the orientation according to this instruction so that the direction is perpendicular to the direction of extension of the white line WL of the pedestrian crossing CW (the direction substantially coincides with the x direction), for example, the image shown in FIG. 4 (camera An image in which the traffic light TL exists within the angle of view of 20) is obtained.

The rolling correction section 85a and the yaw angle calculation section 85b will be described below.

(Rolling correction part)
The rolling correction unit 85a calculates the inclination when the white line WL of the pedestrian crossing CW in the image acquired by the camera 20 is inclined on the image due to the white cane 1 being swung left and right, and calculates the inclination. By correcting the image based on the above, an image is created assuming that the swing angle of the white cane 1 is zero.

When a visually impaired person swings the white cane 1 left and right, the camera 20 is tilted according to the swing angle. That is, the vertical direction of the camera 20 is tilted with respect to the vertical direction. Specifically, when the visually impaired person swings the white cane 1 to the left, the vertical direction of the camera 20 is tilted to the right with respect to the vertical direction (the upper end of the white cane 1 is positioned to the right of the lower end). tilts to do). Conversely, when the visually impaired person swings the white cane 1 to the right, the vertical direction of the camera 20 is tilted to the left with respect to the vertical direction (so that the upper end of the white cane 1 is positioned to the left of the lower end). ). This situation will be described with reference to FIGS. 5 to 7. FIG.

FIG. 5 is a front view showing a state in which the visually impaired person Hu is walking while swinging the white cane 1 left and right. In FIG. 5, the x-axis is the axis along the walking direction of the visually impaired person Hu (the axis in front of the line of sight of the visually impaired person Hu). The y-axis is a horizontal axis orthogonal to the x-axis, and the z-axis is a vertical axis orthogonal to each of the x-axis and the y-axis. Also, FIG. 5 shows a state in which the visually impaired person Hu is walking while holding a white cane 1 in his right hand. In addition, the state in which the white cane 1 is positioned in front of the visually impaired person Hu is indicated by a solid line, and the state in which the visually impaired person Hu swings the white cane 1 to the left (right side in FIG. 5) is indicated by a dashed line. A state in which the visually impaired person Hu swings the white cane 1 to the right (left side in FIG. 5) is indicated by a chain double-dashed line.

In this state where the visually impaired person Hu is swinging the white cane 1 to the left and right while crossing the pedestrian crossing CW, the camera 20 shoots in the situation where the visually impaired person Hu is waving the white cane 1 to the left. Not only is the optical axis swung to the left, but the vertical direction of the camera 20 is tilted to the right with respect to the vertical direction. Even if the vehicle is facing the pedestrian crossing CW, the image captured by the camera 20 will be an image of the left side of the pedestrian crossing CW and will be an image tilted to the left (see FIG. 6, which is an image captured at this time). . On the other hand, in a situation where the visually impaired person Hu swings the white cane 1 to the right, not only is the imaging optical axis of the camera 20 swung to the right, but also the vertical direction of the camera 20 is tilted to the left with respect to the vertical direction. Therefore, even if the visually handicapped person Hu is positioned in the center of the crosswalk CW in the width direction (facing the crosswalk CW), the image captured by the camera 20 will be the image on the right side of the crosswalk CW. As it increases, the image becomes tilted to the right (see FIG. 7, which is the photographed image at this time). The maximum tilt angle of the camera 20 with respect to the y-axis when the visually impaired person Hu swings the white cane 1 to the left with respect to the x-axis is defined as φ _l , When the maximum tilt angle of the camera 20 with respect to the y-axis when the white cane 1 is swung to the right _{is φr} , the maximum angle _θr when the white cane 1 is swung to the right is generally the maximum angle θr when the white cane 1 is swung to the left. Due to the tendency to be larger than the angle _θl , the maximum tilt angle _φr when swinging the white cane 1 to the right is greater than the maximum tilt angle _φl when swinging the white cane 1 to the left. tends to be large (φ _l <φ _r ). 6 and 7 show images captured by the camera 20 in such a situation, where the maximum tilt angle φ _l in the situation where the visually impaired person Hu swings the white cane 1 to the left is The maximum tilt angle _φr is greater when the handicapped person Hu swings the white cane 1 to the right.

In view of this situation, the rolling correction unit 85a obtains the tilt angle of the white line WL caused by the left and right swing of the white cane 1 by the visually impaired person Hu by the following method, and corrects the image accordingly. Therefore, the tilting of the white line WL on the image caused by swinging the white cane 1 to the left and right is eliminated.

<Description of each coordinate system>
First, each coordinate system used for image correction will be described. FIG. 8 is a side view of the visually impaired person Hu for explaining the global coordinate system based on the road surface when the visually impaired person Hu is walking while holding the white cane 1 in his right hand. Also in FIG. 8, the x-axis is the axis along the walking direction of the visually impaired person Hu (the axis in front of the line of sight of the visually impaired person Hu). The y-axis is a horizontal axis orthogonal to the x-axis, and the z-axis is a vertical axis orthogonal to each of the x-axis and the y-axis. A coordinate system having these axes is defined as a global coordinate system x _g =(x, y, z) ^T as a road surface-based orthogonal right-handed coordinate system.

9 (a side view showing a part of the white cane 1 for explaining the cane coordinate system and the camera coordinate system) and FIG. 10 (a part of the white cane 1 for explaining the cane coordinate system and the camera coordinate system). ), the position of the IMU 90 provided in the white cane 1 is set as the origin, the axial direction of the white cane 1 is the _z0 axis (upward is positive), and the z0 axis is perpendicular to the _z0 axis. A cane is an orthogonal right-handed coordinate system in which the forward direction is the x0 _- axis (the forward direction is positive) and the left-right direction orthogonal to the z0 _- axis and the x0 _- axis is the y0 _- axis (the left direction of the visually impaired person Hu is positive). Let the coordinate system x _c = (x ₀ , y ₀ , z ₀ ) ^T . For simplification of explanation, the origin of the global coordinate system _xg and the origin of the cane coordinate system _xc are assumed to be at the same position.

Furthermore, when the white cane 1 is equipped with N (N=1 in this embodiment) cameras 20, the shooting optical axis direction of camera j (j=1, . . . , N) is ξ _j axis (the forward direction is positive), the upward direction orthogonal to this ξj _- axis is the ζj _- axis (upward direction is positive), and the left-right direction orthogonal to the ξj _- axis and ζj _- axis is the ηj _- axis (for the visually impaired Hu). Assume that the camera coordinate system x _p =(ξ _j ,η _j ,ζ _j ) ^T is an orthogonal right-handed coordinate system in which the left direction is positive).

<Description of image rotation processing>
When the rolling correction unit 85a receives information on the three-axis acceleration and the three-axis angular velocity measured by the IMU 90, let the acquired acceleration vector be a=( _ax , _ay , _az ) ^T , and the acquired angular velocity Let the vector be ω=(ω _x , ω _y , ω _z ) ^T .

These three-axis accelerations and three-axis angular velocities are obtained at each time t=t _i (i=1, . . . , n), and the obtained data at time t _i are a=a(t _i ) _and ). In this case, the relationship between the cane coordinate system _xc and the global coordinate system _xg can be expressed by the following equation (1).

Here, _Rc is a coordinate transformation matrix, which is the following equation (2).

On the other hand, regarding the relationship between the cane coordinate system _xc and the camera coordinate system _xp (the camera coordinate system is _xj when one camera 20 is targeted), the position of the camera 20 is unchanged in the white cane 1. Therefore, regardless of the position and posture of the white cane 1, the following equation (3) can be used.

Here, A _j and B _j are affine transformation matrices, which are uniquely determined by the arrangement positions of the camera 20 and the IMU 90, respectively.

Then, from equations (1) and (3) (by substituting equation (1) into equation (3)), the relationship between the camera coordinate system x _j and the global coordinate system x _g is given by the following equation (4) can be obtained as

From this equation (4), as shown in FIG. 11, the z-axis in the global coordinate system _xg (the vertical upward axis with respect to the road surface) and the ζ _j- axis in the camera coordinate system _xj (the upward ₎ can be calculated. That is, it is possible to calculate the tilt of the camera 20 caused by the white cane 1 being swung left and right.

Since the angle φ _j that is the inclination of the camera 20 is calculated in this way, the image is rotated by this angle φ _j (the image in FIG. 11 is rotated clockwise by the angle φ _j ). ), it becomes possible to generate an image when the tilt of the camera 20 is 0, as shown in FIG. That is, it is possible to obtain an image in which each white line WL of the pedestrian crossing CW extends in the horizontal direction. Note that FIG. 12 illustrates an image when the visually impaired person Hu faces the pedestrian crossing CW. When the visually handicapped person Hu is not facing the crosswalk CW (the proper direction (the x direction in FIG. 28; the direction facing the crosswalk CW) that is perpendicular to the extending direction of the white line WL of the crosswalk CW In this case, even if the image is rotated and corrected as described above, the extension direction of the white line WL of the pedestrian crossing CW will be changed due to the existence of this correction angle. It is tilted with respect to the horizontal direction on the image.

11 and 12 exemplify the case where the tilted image is rotated (rotated clockwise) when the white cane 1 is swung to the left, but the white cane 1 is swung to the right. The same processing is performed when performing rotational correction (rotation correction in the counterclockwise direction) on an image that is tilted.

In other words, when an image is generated in which the inclination of the white line WL on the image due to the waving of the white cane 1 to the left and right is eliminated by the correction operation by the rolling correction unit 85a, the white line WL on the image is generated. If WL is slanted, the slant is the orientation of the visually impaired person Hu relative to the crosswalk CW (the actual orientation of the visually impaired person Hu relative to the x direction, which is the direction orthogonal to the longitudinal direction of the white line WL). direction in the x′ direction). The yawing angle calculator 85b calculates a correction angle between the x direction and the x' direction resulting from the orientation of the visually impaired person Hu.

(Yawing angle calculator)
The yawing angle calculation unit 85b refers to the image obtained by the correction operation in the rolling correction unit 85a. It is determined that this is the cause, and a correction angle is calculated based on the inclination angle of the white line WL. A specific description will be given below.

The yawing angle calculation unit 85b includes a first recognition unit 85c, a second recognition unit 85d, and a white line specifying unit 85e as functional units realized by the control program. These functional units will be described below.

Assume that the image shown in FIG. 13 is obtained by the correction operation of the rolling correction section 85a. Let the white line in this figure be Ωi (iεN). In the figure, i=a, b, c, d, that is, white lines Ωa, Ωb, Ωc, Ωd are obtained.

Then, the first recognition unit 85c extracts recognizable white lines from these white lines Ωa, Ωb, Ωc, and Ωd by well-known deep learning. Specifically, a trained model based on pre-annotated data is used to recognize the white line of the pedestrian crossing CW. Here, only the white lines Ωa, Ωb, and Ωc are recognized by the first recognition section 85c, and as shown in FIG. It is in a set state.

Further, the second recognition unit 85d extracts white lines recognizable by computer vision from the white lines Ωa, Ωb, Ωc, and Ωd. Here, the white lines Ωa, Ωb, Ωc, and Ωd are recognized by the second recognition section 85d, and as shown in FIG. reference) is set.

The white line specifying unit 85e specifies the white line of the pedestrian crossing CW in the image based on the result of the recognition operation by the first recognition unit 85c and the result of the recognition operation by the second recognition unit 85d. Specifically, three white lines Ωa, Ωb, and Ωc are determined as the white lines recognized by the respective recognition units 85c and 85d in the above-described case, and these white lines Ωa, Ωb, and Ωc are obtained in a highly reliable state. identified as being

For the white lines Ωa, Ωb, and Ωc thus obtained in a highly reliable state, a boundary box is set in the area recognized as the white line by the computer vision, and this boundary box is formed with the horizontal line on the image. An angle φk (k=a, b, . . . n) is obtained (see FIG. 16), and based on this angle φk (φa, φb), the following equations (5) and (6) are used to determine the direction of the visually impaired person Hu. A correction angle θ, which is the angle between the direction x′ and the x direction, is obtained.

n in Expression (6) is the number of Boundary Boxes in the area recognized as a white line in FIG. α in Equation (5) is a conversion coefficient between the tilt of the image and the walking direction (facing direction) of the visually impaired person Hu, and is a value determined in advance by the angle of view of the lens of the camera 20 .

In this way, the yaw angle calculator 85b obtains the correction angle θ, and this correction angle θ is calculated as the angle between the proper orientation of the visually impaired person Hu (the x direction orthogonal to the extending direction of the white line). will be The yawing angle calculator 85 b then transmits this information to the vibration generator 50 . The vibration generator 50 receives this information (signal) and instructs the visually impaired person Hu to change the orientation. That is, for example, as shown in FIG. 16, when the white lines Ωa, Ωb, and Ωc are slanted to the left on the image, the visually impaired person Hu is assumed to be facing more to the right than the proper orientation. Give an instruction to change the direction to face . In other words, the vibration generator 50 vibrates in a vibration pattern that gives an instruction to change its orientation to face left. Conversely, if the white lines Ωa, Ωb, and Ωc are sloping to the right on the image, the visually impaired person Hu is assumed to be facing leftward rather than the proper orientation, and changes the orientation to face rightward. give instructions to In other words, the vibration generator 50 vibrates in a vibration pattern that gives an instruction to change its orientation to face right. As a result, the correction angle .theta. is reduced, and an image such as that shown in FIG. 4 (an image in which the traffic light TL is present within the angle of view of the camera 20) is obtained.

－Walking support movement－
Next, the walking support operation by the walking support system 10 configured as described above will be described. First, the outline of this walking support operation will be described.

(Outline of walking support operation)
Here, the time during which the visually impaired person is walking is represented by tε[0, T], and the variable (state variable) representing the state of the visually impaired person is represented by ^sεRT . State variables at time t are represented by integers s _t ∈ {0, 1, 2}, and are respectively in the walking state (s _t = 0), the stopped state (s _t = 1), and the crossing state (s _t = 2). The walking state here is assumed to be, for example, a state in which a visually impaired person is walking toward an intersection (an intersection with a traffic light TL and a pedestrian crossing CW). In addition, the stopped state is assumed to be a state in which the visually impaired person has reached the front of the crosswalk CW and is stopped (not walking) by waiting for a traffic light (waiting for the light to switch from red to green). . Also, the crossing state is assumed to be a state in which a visually impaired person is crossing the crosswalk CW.

According to the present embodiment, when an image X _t εR ^w0×h0 (w ₀ and h ₀ represent the vertical and horizontal image sizes of the image, respectively) captured by the camera 20 at time t is input, It proposes an algorithm for obtaining the output ^yεRT for the purpose of assisting human walking. Here, the output for assisting the walking of the visually impaired person is represented by an integer of y _t ε{1, 2, 3, 4}, and is a stop instruction (y _t =1) and a walking instruction (y _t = 2), right departure warning (y _t =3), left departure warning (y _t =4). In the following description, the stop instruction may also be referred to as a stop notification. Also, the walking instruction may be referred to as walking notification or crossing notification. These instructions (notifications) and warnings are given to the visually impaired person by the vibration pattern of the vibration generator 50 . The visually impaired person has previously grasped the relationship between instructions (notifications) and warnings and the vibration pattern of the vibration generator 50, and senses the vibration pattern of the vibration generator 50 from the grip part 3 to give instructions and warnings. Know the type of warning.

In addition, as will be described later, functions for determining the transition of the variable s representing the state of the visually impaired person (hereinafter referred to as state transition functions) f ₀ , f ₁ , and f ₂ , and departure from the pedestrian crossing CW (to the left and right) deviation) exists, and these state transition functions f ₀ to _{f 3 are} stored in the ROM. Specific examples of these state transition functions f ₀ to f ₃ will be described later.

(Summary of Output Variable y and State Transition Function f _i )
The output y _t ε{1, 2, 3, 4} for assisting the walking of the visually impaired person will be described.

As described above, the output y _t includes a stop instruction (y _t =1), a walking instruction (y _t =2), a right deviation warning (y _t =3), a left deviation There are four types of outputs for warnings (y _t =4).

The stop instruction (y _t =1) is to notify the visually impaired person to stop walking when the visually impaired person reaches the crosswalk. For example, if the image captured by the camera 20 is in the state shown in FIG. Since the distance to the crosswalk CW is relatively long, no instruction to stop (y _t =1) is given, and the visually impaired person continues walking (s _t =0), and the image captured by the camera 20 is shown in FIG. 18 (a diagram showing an example of an image captured by the camera 20 at the timing when the visually impaired person reaches the pedestrian crossing CW), the timing at which the visually impaired person reaches the pedestrian crossing CW Therefore, a stop instruction (y _t =1) is output to notify the visually impaired person to stop walking. Determination (determination based on the calculation result of the state transition function) as to whether or not the condition for issuing the stop instruction (y _t =1) is satisfied will be described later.

The walking instruction (y _t =2) notifies the visually impaired person to walk (cross the pedestrian crossing CW) when the traffic light TL has changed from red to green. For example, when the visually impaired person is in a stopped state (s _t =1) in front of the crosswalk CW, and the traffic light TL is switched from red to green based on the image captured by the camera 20, a walking instruction is given. (y _t =2) is output to notify the visually impaired person to start crossing the pedestrian crossing CW. Determination (determination based on the calculation result of the state transition function) as to whether or not the condition for performing this walking instruction (y _t =2) is satisfied will also be described later.

In the present embodiment, the timing at which the walking instruction (y _t =2) is issued is the timing at which the state of the traffic light TL changes from red to green. In other words, even if the visually handicapped person reaches the crosswalk CW and the traffic light TL is already green, the walking instruction (y _t =2) is not given. A walking instruction (y _t =2) is given at the timing of switching to . As a result, when a visually impaired person crosses the pedestrian crossing CW, the time when the traffic light TL is green is sufficiently ensured, This makes it difficult to invite a situation in which the TL switches from a green light to a red light.

The right departure warning (y _t = 3) is issued to the visually impaired person crossing the pedestrian crossing CW when the visually impaired person is walking in the direction of deviating from the pedestrian crossing CW to the right. This warns that there is a risk of deviating to the right from the sidewalk CW. For example, if the image captured by the camera 20 is in the state shown in FIG. The image captured by the camera 20 in a state where a person is crossing the pedestrian crossing CW (s _t = 2) is shown in FIG. A diagram showing an example of an image taken by the camera 20) when the visually impaired person is walking in the direction of deviating from the pedestrian crossing CW to the right. Therefore, a right deviation warning (y _t =3) is output to warn the visually impaired person.

The left departure warning (y _t = 4) is issued to the visually impaired person crossing the pedestrian crossing CW when the visually impaired person is walking in the direction of deviating left from the pedestrian crossing CW. This warns that there is a risk of deviating to the left from the sidewalk CW. For example, when the image captured by the camera 20 is in the state shown in _FIG . 21 (a diagram showing an example of an image captured by the camera 20 when a visually impaired person crossing the pedestrian crossing CW is walking toward the left side of the pedestrian crossing CW); When it becomes , the visually impaired person is walking in the direction of deviating to the left from the pedestrian crossing CW, so a left deviation warning (y _t = 4) is output to warn the visually impaired person. will do.

Determination (determination based on the calculation result of the state transition function) as to whether or not the conditions for issuing the right departure warning (y _t =3) and the left departure warning (y _t =4) are satisfied will also be described later.

(Feature value used for walking support)
Next, the feature quantity used for walking assistance for the visually impaired will be described. In order to appropriately notify the visually impaired to stop walking in front of the crosswalk CW and to start crossing after that, the position of the crosswalk CW (the position closest to the crosswalk CW It is essential to accurately recognize the position of the white line WL1 on the side) and the state of the traffic light TL (whether it is a green light or a red light) from the information from the camera 20. In other words, it is necessary to construct a model formula that reflects the position of the white line WL1 and the state of the traffic light TL, and to comprehend the current situation of the visually impaired person according to this model formula.

FIGS. 22 and 23 outline feature quantities {w ₃ , w ₄ , w ₅ , h ₃ , r, b} ^T εR ⁶ used in walking assistance for the visually impaired. r and b each represent the detection result (0: not detected, 1: detected) of the state (red signal and green signal) of the traffic light TL. When detecting the state of the traffic light TL, the state of the traffic light TL is recognized by extracting the area A surrounded by the dashed line in FIG. 4 as described above. Also, w ₃ , w ₄ , w ₅ , and h ₃ are plotted using the boundary box for the white line WL1 positioned closest to the front among the white lines WL1 to WL4 of the crosswalk CW recognized by the crosswalk detection unit 82. 23 is defined. That is, _w3 is the distance from the left edge of the image to the left edge of the Boundary Box (equivalent to the left edge of the white line WL1), _w4 is the width dimension of the Boundary Box (equivalent to the width dimension of the white line WL1), and _w5 is The distance from the right edge of the image to the right edge of the Boundary Box (equivalent to the right edge of the white line WL1), and _h3 is the distance from the bottom edge of the image to the bottom edge of the Boundary Box (equivalent to the front edge of the white line WL1). .

Let g be a function for detecting crosswalks CW and traffic lights TL using deep learning, predicted crosswalks CW and traffic lights TL using an image X _t ^εR If the Boundary Box of is expressed as g(X _t ), the feature quantity necessary for assisting the walking of the visually impaired can be expressed as Equation (7) below.

here,

is an operator for extracting the feature quantity j(t) and for post-processing the g(X _t ), and p1 is the maximum number of Boundary Boxes per frame.

(state transition function)
Next, the state transition function will be explained. As described above, this state transition function is for each of the stop instruction (y _t =1), the walking instruction (y _t =2), the right departure warning (y _t =3), and the left departure warning (y _t =4). It is used to determine whether or not the conditions for notification are satisfied.

The state quantity (state variable) s _{t+1 at time t+1} is the time history information J={j(0), j(1), . Using the state variable s _t and the image X _t+1 taken at time t+1, the following equation (9) can be used.

The state transition function f in Equation (9) can be defined as in Equation (10) below according to the state quantity at the current time.

In other words, the transition of the walking of the visually impaired person is as follows: walking (for example, walking towards the crosswalk CW) → stopping (for example, stopping before the crosswalk CW) → crossing (for example, crossing the crosswalk CW) → walking (for example, crossing (walking after crossing the sidewalk CW) is repeated, and it is determined whether or not the condition for giving a stop instruction (y _t = 1) to the visually impaired person in the walking state (s _t = 0) is established. The state transition function for is f ₀ (J, X _t+1 ), and the establishment of the condition for instructing the visually impaired person in the stopped state (s _t = 1) to cross (walk) (y _t = 2) The state transition function for determining presence/absence is f ₁ (J, X _t+1 ), and the conditions for notifying the visually impaired person in the crossing state (s _t = 2) of walking (crossing completion) are established. A state transition function for determining the presence or absence of is f ₂ (J, X _t+1 ). Also, the state transition function for determining whether or not the condition for warning the visually impaired person in the crossing state (s _t =2) to deviate from the pedestrian crossing CW is satisfied is f ₃ (J, X _t+1 ) is.

The state transition function corresponding to each state quantity (state variable) will be specifically described below.

(State transition function applied in walking state)
The state transition function f ₀ (j, X _t+1 ) used when the state quantity at the current time is the walking state (s _t =0) is expressed by the following formula (11) using the feature quantity of the formula (7). ) to (13).

where H is the Heaviside function and δ is the Delta function. Also, α ₁ and α ₂ are parameters used as criteria, and t0 is a parameter that designates the past state to be used. Also, I ₂ ={0,1,0,0,0,0} ^T and I ₄ ={0,0,0,1,0,0} ^T .

Using this formula (11), "1" can be obtained only when the conditions of α ₁ >h ₃ and w ₄ >α ₂ have not been satisfied in the past time t0 and have been satisfied for the first time at time t+1. , otherwise "0" will be obtained. That is, since α ₁ >h ₃ is established, the white line WL1 (the lower end of the boundary box of the white line) positioned closest to the pedestrian crossing CW is positioned at the feet of the visually impaired person, and w ₄ >α ₂ is established. As a result, when it is determined that the white line WL1 extends in a direction orthogonal to the traveling direction of the visually impaired person (the width dimension of the boundary box of the white line exceeds a predetermined dimension), "1 ” is obtained.

In this way, when "1" is obtained in the equation (11), it is assumed that the condition for issuing the stop instruction (y _t =1) is established, and the visually impaired person in walking state is instructed to stop (for example, at a pedestrian crossing). An instruction to stop walking in front of the CW; a stop notification) is given.

In addition, in this embodiment, not only the condition that the pedestrian crossing CW is under the feet of the visually impaired person (α ₁ >h ₃ ), but also the width constraint of the detected pedestrian crossing CW (w ₄ >α ₂ ) is added. This prevents an error when a crosswalk other than the crosswalk CW in the traveling direction of the visually impaired person (such as a pedestrian crossing in a direction perpendicular to the traveling direction of the visually impaired person at an intersection) is included in the image X _t+1. Prevent detection. In other words, even if there are a plurality of crosswalks with different crossing directions at an intersection of roads, etc., the pedestrian crossing CW that the visually impaired person should cross (the white line WL1 is the direction that the visually impaired person should cross) ) and other pedestrian crossings (crossing where the width of the white line WL1 is recognized to be relatively narrow) It is possible to clearly discriminate between the sidewalk and the sidewalk, and it is possible to accurately notify the visually impaired person of the start of crossing with a high degree of accuracy.

(State transition function applied in stop state)
The state transition function f ₁ (j, X _t+1 ) used when the state quantity at the previous time is the stopped state (s _t =1) can be the following equations (14) to (16).

Here, X' _t+1 is obtained by performing image trimming and enlargement processing from X _t+1 . In other words, the image X′ _t+1 has a sufficiently high recognition accuracy for the traffic light TL. Also, I ₅ ={0,0,0,0,1,0} ^T and I ₆ ={0,0,0,0,0,1} ^T .

In expression (14), "1" is obtained only when a green light is detected for the first time at time t+1 after detecting a red light in the past time t0, and "0" is obtained in other cases. Become.

In this way, when "1" is obtained in equation (14), it is assumed that the condition for instructing walking (crossing) (y _t =2) is satisfied, and the visually impaired person in the stopped state is instructed to cross ( For example, a crosswalk crossing instruction; crossing notification) is performed.

In addition, there are cases in which state transitions cannot be made according to the logic described above at crosswalks at intersections without traffic lights. In order to solve this problem, a new parameter t1>t0 may be introduced, and transition to the walking state may be made when it is determined that there is no state transition from the stop state during time t1.

(state transition function applied in traverse state)
The state transition function f ₂ (j, X _t+1 ) used when the state quantity at the previous time is the crossing state (s _t =2) can be given by the following equation (17).

In this formula (17), "1" is obtained only when the traffic light TL and the pedestrian crossing CW at the foot cannot be detected even once during the current time t+1 from the past t−t0. "0" will be obtained. In other words, "1" is obtained only when the traffic light TL and the crosswalk CW at the foot cannot be detected because the crosswalk CW has been completely crossed.

In this way, when "1" is obtained in the expression (17), it is assumed that the condition for notifying the completion of the crossing is met, and the visually impaired person in walking state has completed the crossing (completion of crossing the pedestrian crossing). will be notified.

(State transition function for judging departure from crosswalk)
The state transition function f ₃ (j, X _t+1 ) for determining deviation from the pedestrian crossing CW while the visually impaired person is crossing the pedestrian crossing CW can be represented by the following equations (18) to (20).

where _α3 is a parameter used as a criterion. Also, I ₁ ={1,0,0,0,0,0} ^T and I ₃ ={0,0,1,0,0,0} ^T .

In this equation (18), "1" is obtained if the amount of deviation from the center of the frame at the detected position of the pedestrian crossing CW is greater than or equal to the allowable amount, and "0" is obtained otherwise. In other words, "1" is obtained when the value of _w3 is greater than a predetermined value (left deviation) or when the value of _w5 is greater than a predetermined value (right deviation). become.

If "1" is thus obtained in equation (18), a right departure warning (y _t =3) or a left departure warning (y _t =4) is issued.

(Walking support motion)
Next, the flow of the walking support operation by the walking support system 10 will be described. Here, a situation in which a visually impaired person approaches a pedestrian crossing CW with a traffic light TL will be described as an example.

FIG. 24 is a flow chart showing a series of flows of the walking support operation described above. This flowchart is repeatedly performed at predetermined time intervals such that one routine is performed from predetermined time t to predetermined time t+1 in a situation where a visually impaired person is walking. Description of variables (J, X _t+1 ) in each state transition function is omitted in the following description.

First, in step ST1, the visually impaired person is in a walking state, and in step ST2, the position of the white line WL1 of the crosswalk CW in the image area including the crosswalk CW, which is recognized by the crosswalk detection unit 82 (more Specifically, the state transition function f ₀ (the position of the boundary box of the frontmost white line WL1) for determining whether or not the condition for performing the stop instruction (y _t = 1) is satisfied (the above It is determined whether or not "1" is obtained in expression 11).

When "0" is obtained in this state transition function f ₀ , the condition for issuing the stop instruction (y _t =1) is not met, that is, the visually impaired person has not yet reached the crosswalk CW. A NO determination is made as it is not, and the process returns to step ST1. Since the determination in step ST2 is NO until the visually impaired person reaches the crosswalk CW, the operations in steps ST1 and ST2 are repeated.

If the visually handicapped person has reached the front of the pedestrian crossing CW and the state transition function f0 is " ₁ ", a YES determination is made in step ST2, and the process proceeds to step ST3. In this step ST3, a stop instruction (y _t =1) is given to the visually impaired person. Specifically, the vibration generator 50 of the white cane 1 held by the visually impaired person vibrates in a pattern indicating a stop instruction (stop notification). As a result, the visually impaired person holding the grip part 3 of the white cane 1 perceives the vibration pattern of the vibration generator 50, recognizes the stop instruction, and stops walking.

In a situation where the visually impaired person is in a stopped state in step ST4, it is determined in step ST5 whether or not the traffic light TL is recognized within the image (within the angle of view) taken by the camera 20 or not. That is, the traffic light recognition unit 83 determines whether or not the position of the traffic light TL and the state of the traffic light TL can be recognized.

If the traffic signal TL is recognized in the image and YES is determined in step ST5, the process proceeds to step ST7. On the other hand, if the traffic light TL is not recognized in the image and the determination in step ST5 is NO, the process proceeds to step ST6, and the image is corrected by the rolling correction section 85a described above, and then the yaw angle calculation section Calculation of the corrected angle by 85b, and instruction to change the direction of the visually impaired person by the vibration generator 50 (instruction to change the direction according to the corrected angle) are performed.

After instructing the visually impaired person (instructing to change direction) in this manner, the process returns to step ST5, and it is determined whether or not the traffic light TL is recognized in the image. That is, when the visually impaired person changes direction according to the instruction and the traffic light TL is recognized in the image, a YES determination is made in step ST5, and the process proceeds to step ST7. Therefore, the operations of steps ST5 and ST6 are repeated until the traffic light TL is recognized in the image, and the process proceeds to step ST7 when the traffic light TL is recognized.

At step ST7, it is determined whether or not "1" is obtained in the state transition function f ₁ (equation 14) for determining whether or not the condition for performing the walking instruction (y _t =2) is satisfied. During the determination operation by the state transition function _f1 , as shown in FIG. 4 described above, the area A surrounded by the broken line is extracted and, for example, the area A is enlarged to facilitate the state of the traffic light TL. It becomes possible to judge

When "0" is obtained in this state transition function _f1 , the condition for giving a walking instruction (y _t =2) is not satisfied, that is, the traffic light TL is not yet switched to green, and a NO determination is made. and returns to step ST4. Since the determination in step ST7 is NO until the traffic light TL is switched to green, the operations of steps ST4 to ST7 are repeated.

When the traffic light TL is switched to green and "1" is obtained in the state transition function _f1 , a YES determination is made in step ST7, and the process proceeds to step ST8. This operation corresponds to the operation of the switching recognition unit (the switching recognition unit that recognizes that the state of the traffic light has changed from the stop instruction state to the crossing permission state) 84 .

In step ST8, a walking (crossing) instruction (y _t =2) is given to the visually impaired person. Specifically, the vibration generator 50 of the white cane 1 held by the visually impaired person vibrates in a pattern indicating a walking instruction (a notification to start crossing). As a result, the visually impaired person holding the grip portion 3 of the white cane 1 recognizes that the walking instruction has been given, and starts crossing the pedestrian crossing CW.

In step ST9, the visually impaired person is crossing the pedestrian crossing CW, and in step ST10, the state transition function f ₃ (the above formula 18) determines whether or not "1" is obtained.

When "1" is obtained in the state transition function _f3 and a YES determination is made in step ST10, it is determined in step ST11 whether or not the direction of departure from the crosswalk CW is the right direction (right departure). . If the direction of departure from the crosswalk CW is the right direction and the determination in step ST11 is YES, the process proceeds to step ST12, and a right departure warning (y _t =3) is issued to the visually impaired person. Specifically, the vibration generator 50 of the white cane 1 held by the visually impaired person vibrates in a pattern indicating a right deviation warning. As a result, the visually impaired person holding the grip portion 3 of the white cane 1 recognizes that the right deviation warning has been issued, and changes the walking direction to the left.

On the other hand, if the direction of departure from the pedestrian crossing CW is the left direction and the determination in step ST11 is NO, the process proceeds to step ST13, and a left departure warning (y _t =4) is issued to the visually impaired person. Specifically, the vibration generator 50 of the white cane 1 held by the visually impaired person vibrates in a pattern indicating a left deviation warning. As a result, the visually impaired person holding the grip portion 3 of the white cane 1 recognizes that the left deviation warning has been issued, and changes the walking direction to the right. After the departure warning is given in this manner, the process proceeds to step ST16.

If there is no deviation from the crosswalk CW and "0" is obtained in the state transition function _f3 , a NO determination is made in step ST10, and the process proceeds to step ST14. In this step ST14, it is determined whether or not the deviation warning in step ST12 or step ST13 is currently occurring. If the departure warning is not occurring and the determination in step ST14 is NO, the process proceeds to step ST16. On the other hand, if the departure warning is being issued and the determination in step ST14 is YES, the process proceeds to step ST15, cancels the departure warning, and proceeds to step ST16.

At step ST16, it is determined whether or not "1" is obtained in the state transition function f ₂ (equation 17) for determining whether or not the conditions for notifying the completion of crossing are satisfied.

When "0" is obtained in this state transition function _f2 , the condition for notifying the completion of crossing is not satisfied, that is, the visually impaired person is crossing the pedestrian crossing CW, and NO determination is made. , the process returns to step ST9. Since a NO determination is made in step ST16 until the crossing of the pedestrian crossing CW is completed, the operations of steps ST9 to ST16 are repeated.

In other words, when the visually impaired person deviates from the pedestrian crossing CW while crossing, the aforementioned deviation warning is issued, and when the deviation is resolved, the deviation warning is cancelled. This is done until the sidewalk CW is completely crossed.

When the visually impaired person has completed crossing the pedestrian crossing CW and "1" is obtained in the state transition function _f2 , a YES determination is made in step ST16, the process proceeds to step ST17, and the visually impaired person completes crossing. notification. Specifically, the vibration generator 50 of the white cane 1 held by the visually impaired person vibrates in a pattern indicating the completion of crossing. As a result, the visually impaired person holding the grip portion 3 of the white cane 1 recognizes that the completion of the crossing has been notified, and returns to normal walking.

In this way, the above operation is repeated each time the visually impaired person crosses the crosswalk CW.

- Effects of the embodiment -
As described above, in this embodiment, based on the image acquired by the camera 20, the reference crossing direction (the is calculated as the angle between the x-direction and the imaging direction of the camera 20, and the visually handicapped person is instructed to reduce the correction angle .theta. As a result, it is possible to make the photographing direction of the camera 20 substantially coincide with the reference transverse direction. However, the traffic light TL can be present within the angle of view of the camera 20, and the traffic light TL can be appropriately recognized.

Further, the correction operation by the rolling correction unit 85a is performed on the image acquired by the camera 20 to create an image assuming that the swing angle of the white cane 1 is zero. In other words, an image is created in which the inclination of the white line WL of the pedestrian crossing CW caused by swinging the white cane 1 left and right is eliminated. Then, the yawing angle calculation unit 85b estimates the inclination angle of the white line WL caused by the orientation of the visually impaired person Hu with respect to the pedestrian crossing CW. A correction angle between the facing direction is calculated, and an operation for reducing this correction angle is instructed to the visually handicapped person Hu. This makes it possible to appropriately recognize the traffic light TL without being affected by the white cane 1 being swung left and right.

Further, the white line WL of the crosswalk CW in the image created by the rolling correction unit 85a is corrected based on the result of the recognition operation by the first recognition unit 85c and the result of the recognition operation by the second recognition unit 85d. A white line identification unit 85e for identification is provided. As a result, it is possible to sufficiently obtain recognition accuracy of the white line WL of the pedestrian crossing CW in the image, and it is possible to calculate the corrected angle with high accuracy.

Further, in this embodiment, the visually impaired person is notified of the start of crossing on the condition that the state of the traffic light TL has changed from red to green. Therefore, when a visually impaired person crosses the pedestrian crossing CW, it is possible to secure a sufficient amount of time during which the traffic light TL is green.

In addition, in the present embodiment, by incorporating the components of the walking support system 10 into the white cane 1, the walking support system 10 is realized only with the white cane 1, so that the walking support system 10 with high practicality is provided. can do.

-Modification 1-
Next, modification 1 will be described. This modified example differs from the above-described embodiment in the processing operation after the correction angle is calculated as described above. Since the correction angle calculation operation is the same as that of the above-described embodiment, only the processing operation after calculation of the correction angle will be described here.

FIG. 25 is a diagram showing an example of an image captured by the camera 20 in this modified example. As shown in FIG. 25, by calculating the correction angle θ, it is possible to grasp the position of the traffic light TL with respect to the image captured by the camera 20 (see the traffic light TL indicated by the dashed line in the figure). In other words, it can be grasped that the traffic light TL is currently present within the angle of view Ω _s which is swung to the left by the angle θ from the angle of view of the image captured by the camera 20 . Therefore, the visually impaired person is provided with information suggesting the direction of the traffic light TL, and the visually impaired person changes the direction of the camera 20 according to this information (changes the orientation of the camera 20 to the direction in which the traffic light TL exists). do. That is, the visually handicapped person changes the way of holding the white cane 1 to change the photographing direction of the camera 20 .

In this modification, as in the case of the above-described embodiment, even if the traffic light TL does not exist within the angle of view of the camera 20 due to the large correction angle, the camera 20 The traffic light TL can be present within the angle of view of , and the traffic light TL can be appropriately recognized.

-Modification 2-
Next, modification 2 will be described. This modification differs from the above-described embodiment in the method for obtaining the correction angle. Therefore, only the method for obtaining the correction angle will be described here.

FIG. 26 is for explaining the coordinate position and length dimension of the Boundary Box in a state where the entire Boundary Box at the farthest position on the image captured by the camera 20 exists within the angle of view of the camera 20. is a diagram. Here, the coordinate point of the upper left corner of the Boundary Box at the farthest position on the image is (x _u , y _u ), and the width dimension of this Boundary Box is w _u .

As shown in FIG. 4, when the entire Boundary Box at the farthest position on the image captured by the camera 20 exists within the angle of view of the camera 20, the traffic light TL is present in the image. It can be determined that it exists (the traffic light TL is in a recognizable state). For example, if the condition of x _u > 0 and x _u + w _u < w ₀ (w ₀ is the number of pixels in the width direction of the image; corresponds to the image size), the farthest Boundary It is determined that the entire Box exists within the angle of view of the camera 20, and that the traffic light TL exists within the image.

In this modification, a plurality of images (see FIG. 27) representing the relationship between the coordinate points (x _u , y _u ) and the width dimension w _u described above and the correction angle are stored in a RAM or a ROM (storage unit referred to in the present invention). ), and by extracting an image that substantially matches the coordinate point (x _u , _yu ) and the width dimension w _u of the Boundary Box on the image captured by the camera 20, the extracted image has By reading the assigned correction angle, the correction angle in the image photographed by the camera 20 is obtained. The plurality of stored images may be created in advance (created and stored when the white cane 1 is manufactured), or the correction angle is calculated by the processing operations of the above-described embodiment. Each piece of information (the coordinate point of the white line WL, the correction angle, etc.) may be associated with each other and stored.

According to this modified example, it is possible to reduce the number of calculation processes compared to the case of the above-described embodiment, and it is possible to reduce the load on the control system.

-Other embodiments-
It should be noted that the present invention is not limited to the above-described embodiments and modifications, and all modifications and applications within the scope of the claims and their equivalents are possible.

For example, in the above-described embodiment and each of the modified examples, the case where the walking support system 10 is incorporated in the white cane 1 used by the visually impaired has been described. The present invention is not limited to this, and may be a cane, a wheelbarrow, or the like when the pedestrian is an elderly person.

Further, the correction angle calculation method is not limited to the above-described embodiment and each of the modifications, and the correction angle may be calculated in consideration of the angular velocity obtained by the gyro provided in the IMU 90. In this case, even if a part of the pedestrian crossing CW cannot be recognized due to an obstacle such as a vehicle, the correction angle can be calculated with high accuracy. Also, when the absolute direction of the crossing direction of the visually impaired person is clear (for example, when the direction is clear based on GPS information and map information), the compass information of the IMU 90 is used to calculate the corrected angle. can be

Further, in the above-described embodiment and each of the above-described modified examples, an instruction to stop in front of the pedestrian crossing CW and an instruction to start crossing the pedestrian crossing CW are given to the visually impaired person according to the state of the traffic light TL. The present invention may be applied to only one of them.

Further, in the above embodiment and each of the modifications, the white cane 1 is provided with the charging socket 70 to charge the battery (secondary battery) 60 from a household power source. The present invention is not limited to this, and a photovoltaic sheet may be attached to the surface of the white cane 1 and the battery 60 may be charged with power generated by the photovoltaic sheet. Also, a primary battery may be used instead of the secondary battery. Alternatively, the white cane 1 may incorporate a pendulum generator, and the battery 60 may be charged using the pendulum generator.

Further, in the above-described embodiment and each of the modified examples, the type of notification is classified according to the vibration pattern of the vibration generator 50 . The present invention is not limited to this, and notification may be made by voice.

INDUSTRIAL APPLICABILITY The present invention is applicable to a walking support system that provides walking support to a visually impaired person who walks.

1 white cane 10 walking support system 20 camera (image acquisition means)
50 vibration generator (instruction means)
80 Control device 82 Pedestrian crossing detection unit 85 Correction angle calculation unit 85a Rolling correction unit 85b Yawing angle calculation unit 85c First recognition unit 85d Second recognition unit 85e White line identification unit TL Traffic light CW Pedestrian crossing WL White line Hu Visually impaired person (pedestrian )
x reference transverse direction θ correction angle

Claims (6)

A traffic signal located in front of the pedestrian's walking direction is recognized, and at least one of instructing the pedestrian to stop before the pedestrian crossing and instructing the pedestrian to start crossing the pedestrian crossing according to the state of the traffic signal. A walking support system,
an image acquisition means capable of acquiring an image in front of the pedestrian in the walking direction, with the direction in which the pedestrian is facing as the photographing direction;
Based on the image acquired by the image acquisition means, a correction angle that is an angle between a reference transverse direction that is a direction perpendicular to the extending direction of the white line of the pedestrian crossing and the photographing direction of the image acquisition means. a correction angle calculation unit that calculates
instruction means for instructing the pedestrian to perform an action for reducing the correction angle calculated by the correction angle calculation unit;
A walking support system comprising: In the walking support system according to claim 1,
The pedestrian is a visually impaired person holding a white cane, and the image acquisition means is built in the white cane,
The correction angle calculation unit
When the white line of the pedestrian crossing in the image acquired by the image acquisition means is tilted in the image due to the swinging of the white cane to the left and right, the tilt is calculated, and the image is obtained based on the tilt. a rolling correction unit that creates an image assuming that the swing angle of the white cane is zero by correcting the
and a yaw angle calculator that calculates the corrected angle based on the image created by the rolling corrector. In the walking support system according to claim 2,
The walking support, wherein the yaw angle calculation unit is configured to calculate the correction angle based on an inclination angle of the white line of the pedestrian crossing with respect to the horizontal direction in the image created by the rolling correction unit. system. The walking support system according to claim 2 or 3,
The yaw angle calculator,
a first recognition unit that recognizes the white line of the pedestrian crossing using a trained model based on pre-annotated data for the image created by the rolling correction unit;
a second recognition unit that recognizes the white lines of the pedestrian crossing by computer vision of the image created by the rolling correction unit;
a white line identifying unit that identifies the white line of the pedestrian crossing in the image based on the result of the recognition operation by the first recognition unit and the result of the recognition operation by the second recognition unit,
A walking support system, wherein the correction angle is calculated based on an inclination angle of the white line specified by the white line specifying unit with respect to a horizontal direction on the image. In the walking support system according to claim 1,
a pedestrian crossing detection unit that detects the pedestrian crossing based on the image acquired by the image acquisition means;
a storage unit that stores a plurality of image data specifying the relationship between the coordinate position on the image of the white line positioned farthest on the image among the white lines on the pedestrian crossing and the correction angle,
The correction angle calculation unit obtains the coordinate position of the farthest white line among the white lines on the pedestrian crossing detected by the pedestrian crossing detection unit, and calculates the coordinate position of the white line in each image data stored in the storage unit. Image data matching the coordinate position of the furthest white line detected by the pedestrian crossing detection unit is extracted, and the correction angle is calculated based on this image data. walking support system. In the walking support system according to any one of claims 1 to 5,
The instruction means is built in a white cane used by the visually impaired person who is a pedestrian, and is configured to instruct the visually impaired person to perform an operation for reducing the correction angle by vibration or sound. A walking support system characterized by:

Images