WO2023243338A1

WO2023243338A1 - Information processing device, information processing method, program, and information processing system

Info

Publication number: WO2023243338A1
Application number: PCT/JP2023/019250
Authority: WO
Inventors: 正幸横山; 孝悌清水
Original assignee: ソニーグループ株式会社
Priority date: 2022-06-15
Filing date: 2023-05-24
Publication date: 2023-12-21

Abstract

An information processing device according to one aspect of the present technology comprises a peripheral information acquiring unit, a sound information generating unit, and a notification control unit. The peripheral information acquiring unit acquires first peripheral information and second peripheral information relating to a peripheral environment on the basis of a detection result from one or more object detecting sensors. The sound information generating unit generates first sound information using first musical sound data for notifying the first peripheral information, on the basis of the first peripheral information, and generates second sound information using second musical sound data for notifying the second peripheral information, on the basis of the second peripheral information. The notification control unit outputs both the first sound information and the second sound information.

Description

Information processing device, information processing method, program, and information processing system

The present technology relates to an information processing device, an information processing method, a program, and an information processing system that can be applied to broadcasting information about the surrounding environment.

In the scanning distance measuring device described in Patent Document 1, the reliability of the optical distance detector is determined based on the difference between the distance information obtained from the optical distance detector and the distance information obtained from the ultrasonic distance detector. degree is calculated. If the difference between the two pieces of distance information is large, it is determined that the reliability is low, and the distance information obtained from the optical distance detector is corrected using the distance information obtained from the ultrasonic distance detector.

Japanese Patent Application Publication No. 2018-189494

There is a need for technology that can notify users of information about the surrounding environment with high accuracy.

In view of the above circumstances, the purpose of the present technology is to provide an information processing device, an information processing method, a program, and an information processing system that are capable of reporting information about the surrounding environment with high accuracy.

In order to achieve the above object, an information processing device according to an embodiment of the present technology includes a peripheral information acquisition section, an audio information generation section, and a notification control section.
The surrounding information acquisition unit obtains first surrounding information and second surrounding information regarding the surrounding environment based on detection results of one or more object detection sensors.
The audio information generation unit generates first audio information using first musical tone data for notifying the first peripheral information based on the first peripheral information, and generates first audio information based on the second peripheral information. second audio information is generated using second musical tone data for notifying the second peripheral information based on the second peripheral information.
The notification control unit outputs both the first audio information and the second audio information.

In this information processing device, first peripheral information and second peripheral information are acquired based on detection results by one or more object detection sensors. Then, first audio information is generated using the first musical tone data based on the first peripheral information. Also, second musical tone data is used to generate second audio information based on the second peripheral result. Both the first audio information and the second audio information are output.
Thereby, it becomes possible to notify both the first surrounding information and the second surrounding information via audio, and it becomes possible to notify the user of information about the surrounding environment with high accuracy.

The one or more object detection sensors may include a first object detection sensor and a second object detection sensor. In this case, the peripheral information acquisition unit acquires the first peripheral information based on the detection result of the first object detection sensor, and the peripheral information acquisition unit acquires the first peripheral information based on the detection result of the second object detection sensor. Surrounding information may also be acquired.

The first object detection sensor may be a first ranging sensor that operates according to a first method. In this case, the second object detection sensor may be a second ranging sensor that operates according to a second method different from the first method.

The first object detection sensor may be a first distance measurement sensor arranged with the first direction as the detection direction. In this case, the second object detection sensor may be a second distance measurement sensor arranged with a detection direction in a second direction different from the first direction.

The one or more object detection sensors may be sensors that generate image information. In this case, the peripheral information acquisition unit acquires the first peripheral information based on information on some pixel areas of the image information, and acquires the first peripheral information based on information on other pixel areas of the image information. The second surrounding information may be acquired using the second peripheral information.

The one or more object detection sensors may be sensors that generate image information. In this case, the peripheral information acquisition unit acquires information regarding the first type of object detected based on the image information as the first peripheral information, and Information regarding a second type of object different from the type may be acquired as the second peripheral information.

The surrounding information acquisition unit may generate integrated surrounding information based on the first surrounding information and the second surrounding information.

The first ranging sensor may operate using an optical laser method. In this case, the second ranging sensor may operate using an ultrasonic method. Further, the surrounding information acquisition unit may generate the integrated surrounding information based on the stability of detection by the first distance measurement sensor and the stability of detection by the second distance measurement sensor.

When the detection stability of the first distance measurement sensor is low and the detection stability of the second distance measurement sensor is high, the peripheral information acquisition unit includes a light-transmitting member or a light-absorbing member in the periphery. The integrated surrounding information indicating that the member exists may be generated.

The peripheral information acquisition unit further includes hardness information acquired as the second peripheral information based on the detection result of the second distance measurement sensor, and the peripheral information acquisition unit is configured to determine whether the light transmitting member or the light absorbing member Information regarding at least one of the material and type of object may be generated as the integrated surrounding information.

The audio information generation unit determines whether or not to output the first audio information based on the first peripheral information, and if it is determined that the first audio information is not output, the notification control The output of the first audio information by the unit may be restricted.

The first audio information may be first musical tone information constituting a predetermined song. In this case, the second audio information may be second musical tone information constituting the predetermined music piece.

The audio information generation unit may generate the first audio information by controlling musical tone parameters of the first musical tone data based on the first peripheral information.

The musical sound parameters may include at least one of volume, frequency, pitch, speed, BPM, or tempo.

The first surrounding information may include distance information. In this case, the audio information generating section may generate the first audio information by controlling the musical tone parameters based on the distance information.

The audio information generation unit may control localization of the first audio information based on a detection direction of the first ranging sensor.

An information processing method according to an embodiment of the present technology is an information processing method executed by a computer system, and includes first peripheral information and second peripheral information regarding the surrounding environment based on detection results of one or more object detection sensors. including obtaining.
First musical tone data for notifying the first peripheral information based on the first peripheral information is used to generate first audio information, and the second audio information is generated based on the second peripheral information. Second audio information is generated using second musical tone data for reporting surrounding information.
Both the first audio information and the second audio information are output.

A program according to one embodiment of the present technology causes a computer system to execute the following steps.
A step of acquiring first peripheral information and second peripheral information regarding the surrounding environment based on detection results of one or more object detection sensors.
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. A step of generating second audio information using second musical tone data for notifying surrounding information.
A step of outputting both the first audio information and the second audio information.

An information processing system according to an embodiment of the present technology includes one or more object detection sensors, the surrounding information acquisition section, the audio information generation section, and the notification control section.

The information processing system may further include an audio output unit that outputs the first audio information and the second audio information, and an information output unit that outputs information to the user.

FIG. 1 is a schematic diagram for explaining an overview of a surrounding information notification system according to an embodiment of the present technology. FIG. 1 is a schematic diagram showing an example of a functional configuration of a surrounding information notification system. 2 is a flowchart showing an example of basic operation of the surrounding information notification system. FIG. 2 is a schematic diagram for explaining a configuration example of a sensor section. FIG. 7 is a schematic diagram for explaining another configuration example of the sensor section. FIG. 7 is a schematic diagram for explaining another configuration example of the sensor section. FIG. 2 is a schematic diagram showing an example of the configuration of one or more object detection sensors. FIG. 3 is a schematic diagram showing another example of the configuration of one or more object detection sensors. FIG. 2 is a block diagram for explaining notification of surrounding information according to the first embodiment. It is a schematic diagram which shows an example of 1st audio|voice information and 2nd audio|voice information. It is a table showing differences between a laser ranging sensor and an ultrasonic ranging sensor. FIG. 3 is a schematic diagram for explaining an example in which an image sensor is arranged as an object detection sensor. FIG. 7 is a block diagram showing a configuration example for realizing notification of surrounding information according to a second embodiment. 2 is a flowchart showing an example of notification of surrounding information according to the present embodiment. 3 is a table showing an example of determination of the situation of the surrounding environment. FIG. 3 is a schematic diagram showing a case where an obstacle exists in a position in the front direction of the user. FIG. 3 is a schematic diagram showing a case where an obstacle exists on the ground in front of the user. Schematic diagram showing a case where a fall danger point exists in the front direction of the user 3 is a table showing an example of a process for notifying obstacles and falling danger points. FIG. 7 is a schematic diagram for explaining another example of the notification process of obstacles and falling danger points. It is a schematic diagram which shows the detection example of "ground obstacle (large)" and "ground obstacle (small)." FIG. 7 is a block diagram showing an example of a configuration for realizing notification of surrounding information according to a third embodiment. 2 is a flowchart showing an example of notification of surrounding information according to the present embodiment. It is a schematic diagram which shows an example of an obstacle space map. FIG. 3 is a schematic diagram showing another configuration example of the surrounding information notification system according to the present embodiment. FIG. 7 is a schematic diagram showing a configuration example of a surrounding information notification system according to a fourth embodiment. 2 is a flowchart showing an example of notification of surrounding information according to the present embodiment. FIG. 7 is a schematic diagram showing another example of a method of outputting audio according to distance. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer (information processing device) that can be used to construct a peripheral information notification system according to the present technology. FIG. 1 is a block diagram showing an example of a schematic configuration of a vehicle control system. FIG. 2 is an explanatory diagram showing an example of installation positions of an outside-vehicle information detection section and an imaging section.

Hereinafter, embodiments according to the present technology will be described with reference to the drawings.

[Summary of surrounding information notification system]
FIG. 1 is a schematic diagram for explaining an overview of a surrounding information notification system according to an embodiment of the present technology.
In this embodiment, the surrounding information notification system 1 is constructed as a system that can be used by visually impaired people such as blindness and amblyopia. That is, the user 2 of the surrounding information notification system 1 is a visually impaired person.
The surrounding information notification system 1 corresponds to an embodiment of an information processing system according to the present technology.

As shown in FIG. 1, the user 2 uses a white cane 4 to grasp the situation of the surrounding environment when moving on the ground 3. For example, the user 2 can grasp the situation on the ground 3 based on the sensation (tactile sensation) obtained through the white cane 4.

For example, it is possible to grasp objects 5 such as cars (vehicles), utility poles, signboards, etc. that are present in the direction in which the user 2 is traveling. Further, it is also possible to grasp stairs going upward (up stairs), escalators going upward (up escalator), and the like.
In addition, there are various objects 5 such as stairs going downwards (down stairs), escalators going downwards (down escalators), the edge of the station platform (boundary between the platform and the railroad crossing), Braille blocks installed on the ground 3, etc. It is possible to grasp the situation.

In FIG. 1, a car is illustrated as the object 5. In addition, any object 5 on the ground 3, such as a telephone pole, a signboard, a wall, an upward staircase, an upward escalator, a pedestrian, a bicycle, a motorbike, etc., is included in the concept of "object" according to the present technology.
Furthermore, any shape or area where there is a risk of the user 2 falling, such as stairs going downwards (downstairs), escalators going downwards (downward escalators), edges of station platforms, holes, etc. Included in the concept of "fall hazard point". In the present disclosure, the "fall danger point" is included in the "downwardly concave area."

In the environment in which the user 2 walks, there are various dangerous objects that may collide or fall. Examples of accidents include collisions with broken signs, collisions with truck beds, and falls from platforms where the floor suddenly becomes empty. For visually impaired people, it is extremely important to grasp information about the surrounding environment and avoid danger.

The surrounding information notification system 1 is capable of reporting surrounding information regarding the surrounding environment to the user 2 with high precision. For example, the user 2 can avoid various dangers based on the notified surrounding information.
For example, it is possible to avoid collision with the object 5 that becomes an obstacle during movement (walking, etc.). It is also possible to avoid falling at dangerous points, such as falling down stairs or falling from a station platform onto the tracks.
Note that the surrounding information notification system 1 can also be called a danger avoidance system. Further, the notification of surrounding information can also be called notification of surrounding information.

As schematically shown in FIG. 1, the surrounding information notification system 1 includes a sensor section 6, an information output section 7, and a controller 8.
The sensor unit 6 performs sensing regarding the surrounding environment.
The information output unit 7 outputs information to the user 2.
The controller 8 controls the operation of the sensor section 6 and the information output section 7. The controller 8 acquires surrounding information regarding the surrounding environment and notifies the user 2 of the surrounding information.

[Configuration example of surrounding information notification system]
FIG. 2 is a schematic diagram showing an example of the functional configuration of the surrounding information notification system 1. As shown in FIG.
As shown in FIG. 2, in this embodiment, the sensor section 6 includes one or more object detection sensors 10.

In the present disclosure, the object detection sensor 10 includes any sensor capable of outputting information capable of detecting the object 5, a signal including information capable of detecting the object 5, or the like. For example, any sensor that can output information (signal) that can determine whether or not the object 5 is detected (ON/OFF) is included.
In addition to detecting the object 5 (ON/OFF), the object detection sensor 10 also detects various information regarding the object 5, such as the distance to the object 5, the shape of the object 5, the size of the object 5, the material of the object 5, etc. Any sensor capable of detecting information may be used. Note that "detection" can also be referred to as "detection." Further, a sensor that acquires biological information such as pulse, heartbeat, body temperature, and brain waves may be used as necessary.

As the object detection sensor 10, for example, a distance measurement sensor, an image sensor (digital camera), an infrared sensor, etc. can be used.
Examples of the distance measurement sensor include an optical laser distance measurement sensor (hereinafter referred to as a laser distance measurement sensor), an ultrasonic distance measurement sensor (hereinafter referred to as an ultrasonic distance measurement sensor), a stereo camera, It is possible to use various types of distance measurement sensors, such as a ToF (Time of Flight) sensor, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), and a structured light type distance measurement sensor.

As the image sensor, for example, a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor is used. A sensor having both the functions of an image sensor and a ranging sensor may be used. For example, a ToF sensor or the like that can detect distance information for each pixel may be used. An image sensor corresponds to one embodiment of a sensor that generates image information.

The information output unit 7 is configured by an arbitrary device for outputting information to the user 2. As shown in FIG. 2, in this embodiment, a speaker 11 and a vibration device 12 are used as an example of the information output section 7.
The speaker 11 outputs audio. By driving the speaker 11, it becomes possible to notify the user 2 of information via audio. In the example shown in FIG. 1, a headset 13 including a speaker 11 is used as the information output section 7 and is attached to the user's 2 head. Note that the headset 13 is not limited to an overhead type, but may be an in-ear type, a canal type, an open-ear type, or a head-mounted device. Further, it may be a device having hearing aid processing such as a hearing aid or a sound collector.

In this embodiment, the speaker 11 functions as an audio output section. That is, the information output section 7 is configured to include an audio output section.

The vibration device 12 outputs vibration. For example, the vibration device 12 is placed at any position that contacts the user's 2 body. For example, any vibration motor or the like that can generate notification vibrations or the like can be used as the vibration device 12. By driving the vibration device 12, it becomes possible to present a tactile sensation to the user 2, and it becomes possible to notify information.

In the example shown in FIG. 2, the surrounding information notification system 1 further includes a communication section 14 and a storage section 15. The communication unit 14 and the storage unit 15 are connected to the controller 8 via a bus or the like.
The communication unit 14 is a module for performing network communication, short-range wireless communication, etc. with other devices. For example, a wireless LAN module such as WiFi or a communication module such as Bluetooth (registered trademark) is provided.

Note that the sensor unit 6 shown in FIG. 2 and the controller 8 may be communicably connected via wireless communication or the like. In this case, a communication section is also configured in the sensor section 6 (not shown). For example, an object detection sensor 10 including a communication section is used.
Similarly, the information output unit 7 shown in FIG. 2 and the controller 8 may be communicably connected via wireless communication or the like. In this case, the information output section 7 is also configured with a communication section (not shown). For example, the headset 13 shown in FIG. 1 includes a communication section, and is connected to the controller 8 via wireless communication.

The storage unit 15 is a storage device such as a nonvolatile memory, and for example, an HDD (Hard Disk Drive) or SSD (Solid State Drive) is used. In addition, any computer-readable non-transitory storage medium may be used.

For example, the storage unit 15 stores a control program for controlling the overall operation of the surrounding information notification system 1. The storage unit 15 also contains a history of detection results (sensing results) by the sensor unit 6, a history of acquired surrounding information, user information regarding the user 2, and information such as methods and characteristics regarding the sensor unit 6 and the information output unit 7. , and other various information necessary for operating the surrounding information notification system 1 are stored. Note that the method of installing the control program etc. is not limited.

The controller 8 controls the operation of each block included in the surrounding information notification system 1. The controller 8 includes hardware necessary for a computer, such as a processor such as a CPU, GPU, or DSP, memory such as a ROM or RAM, and a storage device such as an HDD. The information processing method according to the present technology is executed by the CPU loading the program according to the present technology stored in the storage unit 15 or the memory into the RAM and executing it.
As the controller 8, a device such as a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array), or another ASIC (Application Specific Integrated Circuit) may be used.

In this embodiment, the processor of the controller 8 executes a program (for example, an application program) according to the present technology, so that the peripheral information acquisition section 17, the notification information generation section 18, and the notification control section 19 are configured as functional blocks. Realized.
The information processing method according to this embodiment is executed by these functional blocks. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

The surrounding information acquisition unit 17 obtains surrounding information regarding the surrounding environment based on the detection results of one or more object detection sensors 10. For example, the peripheral information acquisition unit 17 can acquire the presence or absence of a peripheral object 5, the distance to the object 5, the shape of the object 5, the size of the object 5, the material of the object 5, etc. as peripheral information.
Further, in cases where a distance measurement sensor is used as the object detection sensor 10, it is possible to obtain information regarding fall danger points such as descending stairs or the edge of a station platform as surrounding information. For example, the peripheral information acquisition unit 17 can also acquire information such as the presence or absence of a fall danger point, the distance to the fall danger point, the shape of the fall danger point, the size of the fall danger point, and the like.
In addition, various peripheral information regarding the surrounding environment may be acquired.

When a distance measurement sensor is used as one or more object detection sensors 10, the distance to the object 5 (measured distance value) is detected by the distance measurement sensor. That is, surrounding information is detected by the ranging sensor.
In this way, peripheral information may be detected by the object detection sensor 10 in some cases. That is, the surrounding information may be generated by the sensor unit 6 in some cases. In this case, the surrounding information acquisition unit 17 obtains surrounding information by receiving the surrounding information from the sensor unit 6.

On the other hand, peripheral information may be generated by performing recognition processing, analysis processing, etc. based on information, signals, etc. detected by one or more object detection sensors 10. In this case, the peripheral information acquisition unit 17 acquires peripheral information by generating peripheral information based on detection results by one or more object detection sensors.

That is, in the present disclosure, acquiring peripheral information based on detection results by one or more object detection sensors 10 means receiving peripheral information detected by one or more object detection sensors 10 and detecting one or more objects. This includes both generating surrounding information based on the detection results by the sensor 10.

In this surrounding information notification system 1, it is also possible to acquire the following detection results as surrounding information.
When the distance measurement value (distance information) of a distance sensor whose detection direction is set to the front direction (progressing direction), upward direction, downward direction, left side, right side, etc. of user 2 reaches a threshold value or less, each detection direction detects that an obstacle is approaching.
By performing object recognition on image information acquired by an image sensor, specific objects such as people and cars are detected.
When the distance measurement value of the distance measurement sensor directed downward and forward reaches a threshold value or more, it is detected that a step or the like is approaching.
When the surrounding information notification system 1 is configured as a vehicle-mounted sensor, it detects pedestrians in front and obstacles and people behind when entering the garage.
When the present surrounding information notification system 1 is mounted on a drone, buildings, trees, cliffs, etc. are detected using an image sensor or a ranging sensor.
Here, the threshold value may be automatically or dynamically set by the surrounding information notification system 1 side, or may be set by the user 2 as appropriate.

As object recognition technology for image information acquired by an image sensor, any image recognition technology such as image size conversion, character recognition, shape recognition, matching processing using a model image of the object, edge detection, projective transformation, etc. can be used. May be used.
Furthermore, even if any machine learning algorithm using, for example, DNN (Deep Neural Network), RNN (Recurrent Neural Network), CNN (Convolutional Neural Network), etc. is used, good. For example, by using AI (artificial intelligence) that performs deep learning, it becomes possible to generate and output peripheral information with high accuracy.
For example, by performing semantic segmentation on image information, it is also possible to determine the type of object for each pixel in the image.
Note that the application of the machine learning algorithm may be performed to any processing within the present disclosure.

As material information, it is possible to acquire information related to hardness, such as amplitude information of ultrasonic reflected waves. Of course, information regarding other materials may also be acquired.

The notification information generation unit 18 generates notification information for notifying the user 2 of surrounding information. The notification information includes any information for realizing output of peripheral information by the speaker 11 and the vibration device 12 arranged as the information output section 7.

For example, the notification information includes audio information to be output from the speaker 11 and output control information for specifying how to output the audio information.
Audio information can be in various forms, such as a message such as "There is an obstacle ahead", musical sound information (melody, accompaniment, etc.) that makes up a certain song, or a notification sound such as "beep beep". may be output.
As the output control information, arbitrary information defining volume, pitch, playback speed, BPM (Beats Per Minute), sound localization (localization direction), etc. may be generated. For example, by controlling the localization of sound, it is also possible to provide information using stereophonic sound.

Additionally, the notification information generation unit 18 generates vibration information for vibrating the vibration device 12 as notification information. For example, vibration information for realizing various vibration patterns in which vibration strength (amplitude), frequency, tempo, etc. are specified is generated as notification information.

The notification control section 19 controls the information output section 7 based on the notification information. The speaker 11 is driven by the notification control unit 19, and audio information generated as notification information is output. Further, the vibration device 12 is driven, and a vibration pattern corresponding to the vibration information generated as notification information is output.

In this embodiment, a device including the controller 8 corresponds to an embodiment of an information processing device according to the present technology. For example, when the sensor section 6 and the controller 8 are integrally configured, one embodiment of the information processing apparatus according to the present technology is realized in a form that includes one or more object detection sensors 10.
The information output section 7 and the controller 8 may be integrally configured. For example, the controller 8 may be configured in the headset 13 worn on the user's 2 head. In this case, an embodiment of the information processing apparatus according to the present technology is implemented in a form that includes a device for notification such as the speaker 11.

The sensor section 6, the information output section 7, and the controller 8 may be integrally configured. In this case, one embodiment of the information processing apparatus according to the present technology is realized in a form including one or more object detection sensors 10 and a notification device such as a speaker 11.
In this way, it is possible to adopt various forms as the peripheral information notification system 1.

[Example of basic operation of surrounding information notification system]
FIG. 3 is a flowchart showing an example of the basic operation of the surrounding information notification system 1.
The surrounding information acquisition section 17 obtains surrounding information based on the detection result by the sensor section 6 (step 101).
The broadcast information generation unit 18 generates broadcast information for broadcasting surrounding information (step 102). In step 102, notification information corresponding to the surrounding information to be notified to the user 2 is generated.

For example, assume that in step 101, surrounding information indicating that the edge of the station platform, which is a falling danger point, exists in the immediate vicinity of the user 2 is acquired. In this case, for example, assuming that the danger level (degree of danger) is very high, audio information corresponding to a high-pitched alarm sound (warning sound) and output control for outputting the alarm sound from the headset 13 at a relatively high volume. information is generated as broadcast information.
Further, vibration information such that a powerful vibration pattern with a large amplitude and a suppressed frequency is output from the vibration device 12 is generated as notification information. In addition, it is possible to generate various types of notification information depending on surrounding information.

The information output unit 7 is controlled by the notification control unit 19 based on the notification information, and surrounding information is notified to the user 2 (step 103). In this embodiment, the speaker 11 and the vibration device 12 are controlled by the notification control section 19.
The user 2 can grasp the situation of the surrounding environment through sound and vibration (tactile sense), and can move while avoiding danger.

[Configuration example of sensor section 6]
4 to 6 are schematic diagrams for explaining configuration examples of the sensor section 6. FIG.
In the example shown in FIG. 4A, the sensor section 6 is configured at the user's 2 waist position (belt position), and at a position on the front side of the user 2.
In the example shown in FIG. 4B, the sensor unit 6 is located on the head of the user 2, on the front side of the user 2.

For example, the sensor main body 21 in which one or more object detection sensors 10 are arranged is configured as a wearable device that can be worn by the user 2. The sensor section 6 is realized by the wearable device. In this way, one or more object detection sensors 10 may be placed in a wearable device worn by the user 2.

As shown in FIGS. 4A and 4B, the user 2 can realize the sensor unit 6 at various positions by wearing the sensor main body 21 configured as a wearable device. For example, a wristband type worn on the wrist, a bracelet type worn on the upper arm, a headband type worn on the head (head mounted type), a neckband type worn around the neck, a torso type worn on the chest, and a type worn on the waist. Various forms may be adopted, such as a belt type worn on the ankle or an anklet type worn on the ankle.

In addition, wearable devices in the form of glasses, rings, necklaces, earrings, or piercings, a form that can be attached to the toe of a shoe, a form that can be attached to any position with a clip, etc. may be adopted.

In the example shown in FIG. 5, the sensor section 6 is realized in a form that can be held by the user 2. For example, a sensor main body 21 in which one or more object detection sensors 10 are arranged is configured as a device that can be held by the user 2.
In the example shown in FIG. 5A, the sensor main body 21 is held by the right hand holding the white cane 4. In the example shown in FIG. 5B, the sensor main body 21 is held by the left hand on the opposite side from the right hand holding the white cane 4. In this way, one or more object detection sensors 10 may be placed on a device held by the user 2.

In the example shown in FIG. 6, the sensor main body 21 (sensor section 6) is mounted on another device held by the user 2. In the example shown in FIG. 6A, a sensor body 21 on which one or more object detection sensors 10 are arranged is mounted on a carrier 22 that is pulled and moved by the user 2.
In the example shown in FIG. 6B, the sensor body 21 is mounted on a handcart 23 that the user 2 pushes to move. In this way, the sensor unit 6 may be realized by mounting the sensor main body 21 on another device held by the user 2.

Of course, the sensor body 21 (sensor section 6) may be mounted on the white cane 4 held by the user 2. A configuration in which the sensor main body 21 (sensor unit 6) is mounted on another device held by the user 2 is included in a configuration in which one or more object detection sensors 10 are placed in a device held by the user 2.

[Example of configuration of one or more object detection sensors]
Various variations can also be considered for the configuration of the one or more object detection sensors 10.
For example, the number of object detection sensors 10, the type (method, etc.) of object detection sensors 10, the attitude (detection direction, etc.) of object detection sensors 10, the sensing parameters of object detection sensors 10 (frame rate, gain, laser intensity, etc.), etc. By arbitrarily selecting and setting , it is possible to realize various configurations.

FIG. 7 is a schematic diagram showing an example of the configuration of one or more object detection sensors 10.
In the example shown in FIG. 7, one or more object detection sensors 10 are arranged in a sensor main body 21 that can be held by the user 2. By changing the orientation of the sensor body 21, the user 2 can scan and sense the surrounding environment.

In the example shown in FIG. 7, two ranging sensors with different methods are used as the one or more object detection sensors 10. Specifically, an optical laser distance measurement sensor (laser distance measurement sensor) 25 and an ultrasonic distance measurement sensor (ultrasonic distance measurement sensor) 26 are used. The two distance measuring sensors 25 and 26 are arranged on the sensor main body 21 so that their detection directions are the same.

Therefore, sensing is performed by the laser ranging sensor 25 and the ultrasonic ranging sensor 26, with the direction in which the sensor body 21 is directed by the user 2 as the detection direction. The detection results by the two types of distance measuring sensors 25 and 26 make it possible to acquire highly accurate surrounding information and to notify the user 2 of the same. Of course, the method of the distance measuring sensor employed is not limited and may be set arbitrarily.

FIG. 8 is a schematic diagram showing another example of the configuration of one or more object detection sensors 10.
In the example shown in FIG. 8, one or more object detection sensors 10 are arranged, for example, in a wearable device (not shown) that can be worn on the user's 2 hand. Alternatively, one or more object detection sensors 10 are arranged near the portion of the white cane 4 that the user 2 holds in his/her hand.

In the example shown in FIG. 8, two ranging

sensors

27 and 28 having different detection directions are used (the detection directions are denoted by the symbols of the ranging sensors 27 and 28).
The example shown in FIG. 8 includes a first ranging sensor arranged with a first direction as a detection direction, and a second ranging sensor arranged with a second direction different from the first direction as a detection direction. This is an embodiment. Either of the

distance measurement sensors

27 and 28 may be used as the first distance measurement sensor.

In this embodiment, the distance measurement sensor 27 is the first distance measurement sensor. The distance measurement sensor 27 is arranged so that the front direction of the user 2 is the detection direction. Therefore, the first direction is the front direction of the user 2. Hereinafter, the distance measuring sensor 27 will be referred to as the front side distance measuring sensor 27 using the same reference numerals. In the example shown in FIG. 8, the front distance measuring sensor 27 is arranged at a height H from the ground 3 so that the direction parallel to the ground 3 is the detection direction. Note that the front direction of the user 2 can also be said to be the direction of movement of the user 2.

The distance measurement sensor 28 serves as a second distance measurement sensor, and is arranged so that the direction toward the measurement point P set on the ground 3 is the detection direction. Therefore, the second direction is the direction from the position of the user's 2 hand (the position at the height H) toward the measurement point P. Hereinafter, the distance measuring sensor 28 will be referred to as the ground side distance measuring sensor 28 using the same reference numerals.

The measurement point P is set at a position on the ground 3 that is a predetermined distance D away from the user 2 along the front direction. The size of the distance D is determined, for example, depending on how far the object 5 on the ground 3 or the falling dangerous point is desired to be detected.
For example, if it is desired to quickly detect an object 5 on the ground 3 or a fall danger point at a relatively far position, the distance D is set to be relatively long. If it is desired to detect the object 5 on the ground 3 or the fall danger point when it is in a relatively close position, the distance D is set to be relatively short.
For example, the distance D to the measurement point P may be set taking into consideration the moving speed of the user 2 and the like. Of course, the distance D to the measurement point P may be arbitrarily set by the user 2, for example, based on other viewpoints.

For example, the intersection angle θ between the detection direction of the ground-side ranging sensor 28 and the ground 3 can be calculated based on the desired distance D using the following trigonometric positioning formula.
Intersection angle θ=arctan(H/D)...(1)

For example, it is assumed that the height H from the ground 3 of the front-side distance measurement sensor 27 and the ground-side distance measurement sensor 28 is 0.75 m. In this case, if it is desired to set the measurement point P so that the distance D=1.5 m, the intersection angle θ=26.565° according to equation (1).
That is, by setting the detection direction to be tilted downward by 26.565 degrees with respect to the front direction, it is possible to arrange the ground-side distance measuring sensor 28 toward the measurement point P at the distance D = 1.5 m. It is possible.

By arranging the two ranging

sensors

27 and 28 whose detection directions are the front direction and the ground direction (direction toward the measurement point P on the ground 3), two channels of detection can be performed in the front direction and the ground direction. It becomes possible to obtain the results. Thereby, it becomes possible to acquire highly accurate surrounding information, and it becomes possible to notify the user 2.

The detection directions of the two

distance measuring sensors

27 and 28, that is, the first direction and the second direction, are not limited and may be set arbitrarily.
For example, any combination of directions may be employed, such as (front direction, back direction), (front direction, upper direction), (front direction, left direction), (front direction, right direction). For example, the distance measuring sensor may be arranged with the direction in which the situation is desired to be known as the detection direction.
For example, if you want to know the situation between the front and the left wall, two distance measuring sensors are arranged so that the detection direction is (front direction, left direction).The measurement point P is, for example, It may be set at a position a predetermined distance D away from the user 2 along the front direction on the left wall.

Further, an arbitrary number of three or more distance measuring sensors may be arranged so that their detection directions are different from each other. For example, three ranging sensors may be arranged so as to provide three channels in the front direction and in the left and right directions perpendicular to the front direction (directions of both left and right walls).
Any variation may be adopted as the number of ranging sensors and the detection direction of each ranging sensor.
Furthermore, for example, the user 2 may be able to set the detection directions of the

distance measuring sensors

27 and 28 on the application. On the other hand, it is generally assumed that it is difficult for visually impaired people to configure various settings on a GUI (Graphical User Interface). It may be something that can be set and changed as appropriate.

[Variations of notifications to users]
There are also various variations regarding the notification of surrounding information to the user 2 in the surrounding information notification system 1. Hereinafter, variations in notification to the user 2 will be described as first to fourth embodiments.

(First embodiment)
FIG. 9 is a block diagram for explaining notification of surrounding information according to the first embodiment.
In the example shown in FIG. 9, the peripheral information acquisition unit 17 acquires first peripheral information 30 and second peripheral information 31. That is, at least two different types of peripheral information are acquired.

For example, assume that the sensor section 6 having the configuration illustrated in FIG. 7 is adopted. That is, it is assumed that a laser ranging sensor 25 and an ultrasonic ranging sensor 26 are arranged. In this case, the surrounding information obtained based on the detection result of the laser ranging sensor 25 is obtained as the first surrounding information 30. Additionally, surrounding information acquired based on the detection result of the ultrasonic ranging sensor 26 is acquired as second surrounding information 31.

For example, the presence or absence (ON/OFF) of detection of the object 5 output from the laser ranging sensor 25, the distance to the object 5, the material (hardness) of the object 5, etc. are acquired as the first peripheral information 30. Ru. In addition, information such as whether or not the object 5 is detected (ON/OFF), the distance to the object 5, and the material (hardness) of the object 5 outputted from the ultrasonic ranging sensor 26 is acquired as second peripheral information 31. Ru.

Note that the laser distance measurement sensor 25 can detect the distance to the object 5, etc. without being affected by the hardness of the object 5. Conversely, it is often difficult to detect the material (hardness) of the object 5 using the laser ranging sensor 25. In this case, for example, the laser distance measurement sensor 25 may output information indicating that the object 5 is undetectable as information about the material (hardness) of the object 5.

In the example shown in FIG. 9, the broadcast information generation section 18 includes audio

signal processing sections

32 and 33 and a speech synthesis processing section 34. The storage unit 15 also stores first musical tone data for notifying the first peripheral information 30 and second musical tone data for notifying the second peripheral information 31.

The musical tone data includes any data that constitutes a musical tone. For example, the data includes data in which a predetermined scale or melody is defined, audio data of a specific musical instrument, and the like.
For example, the musical tone data of the main melody, the musical tone data of the sub melody, etc. of a predetermined song may be used individually. Or, specific instruments playing a given song (melody instruments such as piano, violin, vocals, bass instruments such as bass guitar, contrabass, bass drum, percussion instruments such as glockenspiel, drums, bells, chimes, etc.) ) audio data may be used.
Of course, data such as an electronic sound that is discontinuously and periodically reproduced as "pippipippi" may also be used.

Based on the first peripheral information 30, the audio signal processing unit 32 generates first audio information using the first musical tone data. The audio signal processing unit 33 generates second audio information using the second musical tone data based on the second peripheral information 31. The first audio information and the second audio information are generated as broadcast information.

FIG. 10 is a schematic diagram showing an example of first audio information and second audio information.
Typically, the first musical sound data and the second musical sound data are musical sound data that, when output together to the user 2, form a combination that can be listened to without any musical discomfort. .

For example, musical tone data of a certain part of a predetermined musical piece is set as the first musical tone data. As the second musical sound data, musical sound data of other parts constituting the same music piece is set.
For example, first musical sound data and second musical sound data, such as (main melody, sub melody), (melody, accompaniment), (melody of high instrument, melody of low instrument), (melody of melody instrument, sound of percussion instrument), etc. Any combination may be adopted as the combination.

The audio signal processing unit 32 generates first audio information by controlling musical tone parameters based on the first peripheral information 30. Examples of musical sound parameters include volume, frequency, pitch, speed of song reproduction, BPM, and tempo.
For example, when the first peripheral information 30 is distance information, as the distance approaches, musical tone parameter control such as increasing the volume, pitch, and tempo is executed to generate the first audio information. In this way, the first audio information may be generated by controlling the musical tone parameters based on the distance information.
Alternatively, audio data of a specific musical instrument may be generated as the first audio information in response to the detection of the object 5.

Similarly, the audio signal processing unit 33 generates second audio information by controlling musical tone parameters based on the second peripheral information 31.
In this embodiment, first musical tone information constituting a predetermined music piece is generated as the first audio information. As the second audio information, second musical tone information constituting the same song is generated.

The speech synthesis processing unit 34 synthesizes the first speech information and the second speech information to generate one speech information (synthesized speech information). That is, the synthesized speech information is generated by the speech synthesis processing unit 34 so that both the first speech information and the second speech information illustrated in FIG. 10 are output. Any mixing technique or the like for synthesizing audio information (audio data) may be used.

In the example shown in FIG. 9, an audio output section 35 is configured within the notification control section 19. The audio output unit 35 controls the speaker 11 to output synthesized audio information in which the first audio information and the second audio information are synthesized. As a result, both the first audio information and the second audio information are output from the speaker 11.

The user 2 is able to grasp the first peripheral information 30 through the first audio information and the second peripheral information 31 through the second audio information. That is, the user 2 can grasp the first peripheral information 30 and the second peripheral information 31 at the same time via voice. As a result, it becomes possible to notify the user 2 of information about the surrounding environment with high accuracy.

For example, a melody of a certain musical instrument is output as the first audio information. The melody of another musical instrument is output as the second audio information. When the first audio information and the second audio information are combined and output together, they are played back to the user 2 as one melody.
Further, as the first audio information, a sound of a certain scale is output. Sounds of different scales are output as second audio information. When the first audio information and the second audio information are synthesized and output together, they form a chord and are played back to the user 2.
As a method for outputting both the first audio information and the second audio information, various variations as described above can be considered, and can be adopted as the notification method according to the present embodiment.

For example, it is assumed that the user 2 moves while scanning the surrounding area using the sensor section 6 having the configuration shown in FIG. 7 in which the laser distance measurement sensor 25 and the ultrasonic distance measurement sensor 26 are arranged. Of course, the sensor section 6 may be mounted on the white cane 4.

The first peripheral information 30 is acquired by the peripheral information acquisition unit 17 based on the detection result of the laser ranging sensor 25. Further, second surrounding information 31 is acquired based on the detection result of the ultrasonic ranging sensor 26. It is assumed that the detection direction of the laser distance measurement sensor 25 and the detection direction of the ultrasonic distance measurement sensor 26 are set in the same direction.

FIG. 11 is a table showing differences between the laser distance measurement sensor 25 and the ultrasonic distance measurement sensor 26.
As schematically shown in FIG. 7, since ultrasonic waves have a spread, the detection range of the ultrasonic ranging sensor 26 becomes wide. On the other hand, since the laser has high directivity, the detection range of the laser ranging sensor 25 is narrow. Both the ultrasonic distance measurement sensor 26 and the laser distance measurement sensor 25 return the distance measurement value when an object 5 exists within the detection range, so the laser distance measurement sensor 25 is easier to aim at, It becomes possible to know with high accuracy whether or not there is an object 5 at the point where the cane 4 is pointed.

On the contrary, the ultrasonic ranging sensor 26 has a wide detection range, so while only the approximate direction of the detected object 5 can be known, it enables detection over a wide range. If we compare it with vision, it can be said that the laser distance measurement sensor 25 is close to the fovea, which has a narrow field of view, and the ultrasonic distance measurement sensor 26 is close to the peripheral field, which has a wide field of view.

For example, if the laser of the laser ranging sensor 25 straddles the boundary of a thin object 5 by slightly moving the hand, it is possible to know the width of the object 5 in conjunction with the hand movement. On the other hand, with the ultrasonic ranging sensor 26 having a wide detection range, the object 5 does not exceed the detection range unless the user moves his or her hand widely, so it is difficult for the user 2 to know information such as the width and height of the object 5 in detail.

From these points, for example, in response to the detection results of the laser distance measurement sensor 25, musical sounds (for example, piano, violin, vocals, etc.) made by instruments suitable for the main melody with many scales can be used to convey detailed changes in the detection results. musical sound data).
Since the ultrasonic ranging sensor 26 has a wide detection range, it will notify the detection of some object 5 relatively frequently compared to the laser ranging sensor 25, which has a narrow detection range. For this reason, for example, musical tones of accompaniment or bass instruments (eg, bass guitar, contrabass, bass drum, etc.) suitable for continuous notification are assigned.
If the detection result is not a continuous value such as a distance value, but a binary value such as whether or not the object 5 is detected (ON/OFF), for example, musical sounds of percussion instruments such as glockenspiel, drums, bells, and chimes are assigned. It's okay.

As shown in FIG. 11, another difference between the laser distance measurement sensor 25 and the ultrasonic distance measurement sensor 26 is that the laser distance measurement sensor 25 has difficulty detecting objects with low light reflectance such as glass or black materials; The difference is that the ultrasonic ranging sensor 26 is not suitable for detecting soft materials.

As described above, both types of distance measuring sensors have advantages and disadvantages. In the surrounding information notification system 1 according to the present embodiment, it is possible to compensate for the shortcomings of both distance measuring sensors.
For example, even if there is a wall or door made of glass that cannot be detected by a laser, the ultrasonic wave returns a measured distance value, so it is possible to notify the user 2 that there is some object 5 in the vicinity. In this way, it is possible to increase the types of objects 5 that can be detected. It is also possible to improve environmental resistance.

For example, the laser distance measurement sensor 25 is used as the main distance measurement sensor, and musical tone data of a high-pitched musical instrument is assigned. The secondary ultrasonic ranging sensor 26 receives musical tone data from a bass musical instrument. In addition, musical sound data is assigned so that the user 2 can clearly hear the voices corresponding to both distance measuring sensors. This makes it possible to notify the user 2 of information about the surrounding environment with very high accuracy.

For example, if there is a glass wall, if the main melody is output according to the distance measurement value of the laser distance measurement sensor 25, the output may be interrupted or become inaudible. It is difficult for the user 2 to judge whether this is due to the influence of small objects 5 such as trees or snow, or whether it is due to the light transmitting/absorbing material.
In this embodiment, it is possible to listen to accompaniment sounds according to the distance measurement value of the ultrasonic distance measurement sensor 26. As a result, even if the main melody becomes choppy or inaudible, if the accompaniment sound is stable, it can be easily determined that this is due to the material. In this way, it is possible to judge whether the material is difficult for the laser distance measurement sensor 25 to detect by simultaneously listening to the auxiliary sound (accompaniment) that reflects the distance measurement value of the ultrasonic distance measurement sensor 26. Become.

In addition, since the distance measurement value output (main melody) of the laser distance measurement sensor 25 is interrupted and the distance measurement value output (accompaniment sound) of the ultrasonic distance measurement sensor 26 is stable, the object 5 is If it can be determined that the material is a transparent material or a light-absorbing material, it is also possible to further utilize information on hardness obtained from the ultrasonic ranging sensor 26. For example, it is assumed that objects 5 that are light-transmissive or light-absorbing and have high hardness are often highly dangerous obstacles such as glass walls or black walls. Based on such an assumption, it is possible to estimate that an object having light transmittance or light absorption and low hardness is not glass or the like but a soft light-absorbing material such as black clothing. Furthermore, if it is moving, it can be estimated that it is likely to be a person wearing black clothing. For example, by performing such estimation using machine learning or the like, it becomes possible to realize more advanced material estimation by combining the laser distance measurement sensor 25 and the ultrasonic distance measurement sensor 26. It is also possible to estimate the type of object 5, such as whether it is a wall such as glass or a person wearing clothes made of light-absorbing material.

In this way, the peripheral information acquisition unit 17 can further generate peripheral information based on the first peripheral information 30 and the second peripheral information 31. That is, it is possible to integrate the first surrounding information and the second surrounding information to generate surrounding information regarding the surrounding environment (hereinafter referred to as integrated surrounding information).

It is also possible for the surrounding information acquisition unit 17 to generate integrated surrounding information based on the stability of detection by the laser ranging sensor 25 and the stability of detection by the ultrasonic ranging sensor 26. Note that in the present disclosure, the stability of sensor detection is included in the sensor detection result.

As explained above, when the detection stability of the laser distance measurement sensor 25 is low and the detection stability of the ultrasonic distance measurement sensor 26 is high, there is a light-transmitting member or a light-absorbing member in the vicinity. It is possible to generate integrated peripheral information indicating that the peripheral information exists.

Further, based on the hardness information acquired as the second peripheral information 31 based on the detection result of the ultrasonic ranging sensor 26, information regarding at least one of the material and the type of object for the light-transmitting member or the light-absorbing member is determined. It is also possible to generate detailed information as integrated peripheral information.

For example, the shape of an object can be grasped in detail by converting the detection results from the laser distance measurement sensor 25 into minute changes in tone that are linked to the movements of the body or movable parts. At the same time, by converting the wide range detection results by the ultrasonic ranging sensor 26 into audio such as accompaniment, it is possible to detect in advance the presence of obstacles that are likely to block the route. It is also possible to realize the division of roles between the fovea and the peripheral visual field in vision.
It is also possible to share roles that compensate for the shortcomings of each distance measurement sensor. For example, although the laser distance measurement sensor 25 is normally used, it is also possible to rely on information from the ultrasonic distance measurement sensor 26 in places where there is glass or black material with low light reflectivity.

The specific types of the first peripheral information 30 and the second peripheral information 31 shown in FIG. 9 are not limited. For example, as illustrated in FIG. 8, it is assumed that two distance measuring sensors are set so that their detection directions are different from each other.
In this case, surrounding information obtained based on the detection result of a ranging sensor arranged with the first direction as the detection direction may be obtained as the first surrounding information 30. Further, surrounding information obtained based on the detection result of a ranging sensor arranged with the second direction as the detection direction may be obtained as the second surrounding information 31.

In the example shown in FIG. 8, the first surrounding information 30 is acquired based on the detection result of the front distance measuring sensor 27 arranged with the front direction as the detection direction, for example. Further, the second surrounding information 31 may be acquired based on the detection result of the ground-side ranging sensor 28 arranged with the direction toward the measurement point P as the detection direction. This allows the user 2 to simultaneously grasp information about the environment on the front side and information on the environment on the ground side via audio. Note that the first surrounding information 30 may be acquired based on the detection result of the ground-side distance measurement sensor 28, and the second surrounding information 31 may be obtained based on the detection result of the front-side distance measurement sensor 27.

For example, the distance measurement value of the front side distance measurement sensor 27 is acquired as the first peripheral information 30. Then, musical tone parameters are controlled according to the measured distance value, and a predetermined melody is output as first audio information.
On the other hand, as second surrounding information, the presence or absence of a fall danger point is acquired based on the distance measurement value of the ground-side distance measurement sensor 28. For example, when the distance measurement value of the ground-side distance measurement sensor 28 becomes large, it is determined that there is a fall danger point. In response to detection of a fall danger point, second audio information is generated and output from audio data of a percussion instrument or the like.
As a result, the melody, which is the first audio information, and the audio of a percussion instrument, which is the second audio information, are simultaneously played back to the user 2. The user 2 can simultaneously grasp the proximity of the object 5 in the front direction and the presence or absence of a fall danger point on the ground 3.

Distance measuring sensors of the same type are arranged so that their detection directions are different from each other. For example, a plurality of laser ranging sensors are arranged with different detection directions such as front, back, left, right, top and bottom.
In this case, for example, the main melody is assigned to the laser distance measurement sensor in the front direction. Accompaniment etc. are assigned to the laser ranging sensor in a direction other than the front direction. It is also possible to assign different localizations to each laser distance measurement sensor using stereophonic sound. That is, the localization of the output audio information may be controlled based on the detection direction of each laser ranging sensor.
For example, the localization of the first audio information is controlled based on the detection direction of the first ranging sensor among the plurality of ranging sensors. The localization of the second audio information is controlled based on the detection direction of the second ranging sensor among the plurality of ranging sensors. Such processing is also possible.

FIG. 12 is a schematic diagram for explaining an example in which an image sensor is arranged as the object detection sensor 10.
Image information 38 is generated by the image sensor and output to the surrounding information acquisition unit 17 as a detection result. The surrounding information acquisition unit 17 performs object recognition processing on the image information 38. As a result, for the area included in the angle of view, the presence or absence of detection of the object 5 (ON/OFF), the type of the object 5, the distance to the object 5, the material (hardness) of the object 5, etc. are used as peripheral information. It is possible to obtain.

As shown in FIG. 12A, the first peripheral information 30 may be acquired based on the information of the upper half pixel region 38a of the image information 38. Further, the second peripheral information 31 may be acquired based on the information of the lower half pixel region 38b of the image information 38.
In this way, the first peripheral information 30 may be acquired based on information on a part of the pixel area of the image information 38. Further, the second peripheral information may be acquired based on information on other pixel areas in the image information 38. This allows the user 2 to simultaneously grasp information on the upper environment and information on the lower environment via audio.

In the example shown in FIG. 12B, information regarding a car 39 detected based on image information 38 is acquired as first surrounding information 30. Further, information regarding the person 40 detected based on the image information 38 is acquired as the second peripheral information 31.
In this way, the first peripheral information 30 and the second peripheral information 31 may be acquired for each type of object detected based on the image information 38. That is, information regarding a first type of object detected based on the image information 38 is acquired as first peripheral information, and information regarding a second type different from the first type is acquired as second peripheral information. may be done.

The first type and second type of object can be set arbitrarily. For example, it is possible to set any combination of (person, vehicle), (motorcycle, automobile), (adult, child) (pedestrian, bicycle) as the first type and second type. be.
The user 2 can simultaneously grasp information regarding two different types of objects via voice.

The notification information generation unit 18 may determine whether to output the first audio information based on the first peripheral information 30. For example, it is determined whether or not to output the first audio information based on whether or not the object 5 is detected, the distance to the object 5, and the like. For example, if the object 5 is not detected, or if the distance to the object 5 is greater than a predetermined threshold (eg, 5 m, etc.), it is determined that the first audio information is not output.

If it is determined that the first audio information is not to be output, the output of the first audio information by the audio output unit 35 is restricted. That is, when the first peripheral information 30 satisfies a predetermined condition, the first audio information is output, and when the predetermined condition is not satisfied, the first audio information is not output. Such processing is also possible. The conditions that serve as the criteria for determining whether or not to output the first audio information may be set arbitrarily.

Similarly, regarding the output of the second audio information, it is determined whether or not to output the second audio information based on predetermined conditions, and the output of the second audio information is controlled based on the determination result. Good too.
The same determination condition may be set as the determination condition regarding the output of the first audio information and the determination condition regarding the output of the second audio information, or different determination conditions may be set separately. .

Three or more types of peripheral information may be acquired, and musical tone data for reporting each peripheral information may be used to generate and output a plurality of pieces of audio information.
For example, first musical tone data, second musical tone data, and third musical tone data are prepared for each of first peripheral information, second peripheral information, and third peripheral information.
First musical tone data is used to generate first audio information based on the first peripheral information. Second audio information is generated using the second musical tone data based on the second peripheral information. Third audio information is generated using third musical tone data based on the third peripheral information.
By outputting the first to third audio information together, the user 2 can grasp the first to third peripheral information at the same time. Note that the type and number of peripheral information to be simultaneously notified via voice and the type and number of musical tone data linked to the peripheral information are not limited.

Even if there is a large amount of peripheral information that the user 2 would like to be notified of, it is possible to notify the user 2 of each peripheral information by appropriately converting each peripheral information into audio information. For the user 2, it becomes possible to instantly perceive a plurality of peripheral information without being deprived of vision.

As the object detection sensor 10 disposed in the sensor unit 6, a sensor capable of outputting multidimensional information may be disposed. For example, an image sensor capable of outputting image information 38 shown in FIG. 12 and the like can also be said to be a sensor capable of outputting multidimensional information, with information on each pixel being one-dimensional information. As the object detection sensor 10 that can output multidimensional information, it is also possible to use a ToF (Time of Flight) sensor that can acquire distance information for each pixel.

For example, in the case of VGA resolution, it is 640×480=307200 pixels. In the case of Full HD, the number is 1920×1080=2073600 pixels. For example, audio information may be generated using musical tone data for each pixel information and output together.

On the other hand, when there are such a large number of dimensions, it is difficult to assign different audio information to each dimension unless the number of dimensions is thinned out or grouped.
For example, if you want to reduce the amount of processing when placing stereophonic sound source localization for each pixel, you can increase the speed by reducing the image size. In addition, if a problem arises in which it is relatively easy to distinguish between the left and right localization differences, but it is difficult to distinguish between the top and bottom, it is possible for user 2 to use settings such that the higher the pixel is in the upper direction, the higher the sound is, and the lower the pixel is, the lower the sound is. Announcements that are easy to hear are realized.

As described above, in the peripheral information notification system 1 according to the present embodiment, the first peripheral information 30 and the second peripheral information 31 are acquired by the controller 8 based on the detection results by the one or more object detection sensors 10. . Then, first audio information is generated using the first musical tone data based on the first peripheral information 30. Also, second musical tone data is used to generate second audio information based on the second peripheral result. Both the first audio information and the second audio information are output.
Thereby, it becomes possible to notify both the first surrounding information and the second surrounding information via voice, and it becomes possible to notify the user 2 of information on the surrounding environment with high accuracy.

Conventionally, some vehicles, drones, etc. are equipped with sensors to grasp information about the surrounding space and avoid collisions with surrounding objects. In such cases, in order to avoid collisions with obstacles in the path, a technology is needed to measure the distance to the obstacles and notify the driver in advance.

Distance sensors include, for example, optical laser systems, ultrasonic systems, and stereo cameras, but optical laser systems have problems such as being unable to measure distances from objects with low light reflectance and being affected by environmental light. there were. Another problem with the ultrasonic method was that it was difficult to narrow down the distance measurement range because the sound waves spread out. In order to solve such problems, a configuration in which a plurality of distance measuring sensors of different methods are combined can be cited.

For example, there is a method in which one of a plurality of different distance measuring sensors is selectively switched and used. Alternatively, there is a method of merging the values of a plurality of distance measuring sensors according to the reliability and outputting the result as one detection result.

When these methods are implemented, it is difficult to obtain stable detection results because the characteristics of distance information obtained depending on the environment (e.g. detection range, detection accuracy, fluctuation error range, etc.) differ depending on the distance measurement sensor method. Output becomes difficult. Furthermore, since the detection results are not output consistently, there is a problem in that it is difficult for the user 2 to intuitively understand the characteristics of the measured distance values, making it difficult to use.

For example, it is assumed that a laser distance measurement sensor and an ultrasonic distance measurement sensor are selectively switched and used as appropriate. In this case, the highly directional laser distance measurement sensor outputs detection results for obstacles in the front direction, but the ultrasonic distance measurement sensor with a wide detection range suddenly outputs detection results for objects that are widely present in the surrounding area. It is also possible that the output is switched and output.
It may be difficult for the user 2 to understand whether the detection result currently being reported via audio or the like is information for the front direction or information for a wide area around the user, making it difficult to avoid danger. could be.

In this embodiment, the first peripheral information 30 and the second peripheral information 31 are converted into audio information, and it becomes possible to notify the user 2 at the same time. This makes it possible to intuitively grasp multiple pieces of peripheral information through hearing without selectively switching distance sensors or merging outputs.
For example, it is possible to simultaneously grasp surrounding information based on the detection result by the laser distance measurement sensor 25 and surrounding information based on the detection result by the ultrasonic distance measurement sensor 26 via audio. As a result, it becomes possible to grasp, for example, the situation in the front direction and the situation in a wide range of surroundings at the same time, and it becomes possible to sufficiently avoid danger.

In the first embodiment described above, the peripheral information acquisition unit 17 corresponds to an embodiment of the peripheral information acquisition unit according to the present technology.
The audio

signal processing units

32 and 33 and the audio synthesis processing unit 34 configured in the notification information generation unit 18 correspond to an embodiment of the audio information generation unit according to the present technology.
The audio output unit 35 configured in the notification control unit 19 corresponds to an embodiment of the notification control unit according to the present technology, which outputs both first audio information and second audio information. Note that the speech synthesis processing section 34 can also be regarded as a block that also functions as a notification control section according to the present technology.

Moreover, in the example shown in FIG. 7, the laser ranging sensor 25 corresponds to an embodiment of the first object detection sensor according to the present technology. Further, the laser distance measurement sensor 25 also corresponds to an embodiment of a first distance measurement sensor that operates according to the first method (optical laser method).
The ultrasonic ranging sensor 26 corresponds to an embodiment of the second object detection sensor according to the present technology. Further, the ultrasonic ranging sensor 26 also corresponds to an embodiment of a second ranging sensor that operates according to a second method (ultrasonic method) different from the first method.

In the example shown in FIG. 8, the front distance measuring sensor 27 corresponds to an embodiment of the first object detection sensor according to the present technology. Further, the front side distance measuring sensor 27 also corresponds to one implementation value of the first distance measuring sensor arranged with the first direction (front direction) as the detection direction.
The ground-side ranging sensor 28 corresponds to an embodiment of the second object detection sensor according to the present technology. Further, the ground-side ranging sensor 28 also corresponds to an embodiment of a second ranging sensor arranged with a detection direction in a second direction (ground direction) different from the first direction.

(Second embodiment)
FIG. 13 is a block diagram showing a configuration example for realizing notification of surrounding information according to the second embodiment.
In the surrounding information notification system 41 according to this embodiment, the configuration of the sensor unit 6 shown in FIG. 8 is adopted. That is, a front-side distance measurement sensor 27 whose detection direction is in the front direction, and a ground-side distance measurement sensor 28 whose detection direction is in a direction toward the measurement point P on the ground 3 (ground direction) are used.

Further, in this embodiment, the surrounding information acquisition section 17 includes a distance information acquisition section 42 and a situation determination section 43.
The distance information acquisition unit 42 receives first distance information detected by a first ranging sensor arranged with a first direction as a detection direction, and a second direction different from the first direction as a detection direction. and second distance information detected by the second distance measuring sensor arranged.

In this embodiment, the distance information detected by the front distance measuring sensor 27 (hereinafter referred to as front distance information) is acquired as the first distance information. Further, distance information detected by the ground-side distance measuring sensor 28 (hereinafter referred to as ground-side distance information) is acquired as second distance information.

The situation determination unit 43 determines the surrounding environment based on at least one of first detection information including fluctuations and dispersion of the first distance information and second detection information including fluctuations and dispersion of the second distance information. Determine the situation.
In this embodiment, both the first detection information including fluctuations and variations in the front side distance information and the second detection information including fluctuations and variations in the ground side distance information are used. That is, the situation of the surrounding environment is determined using four pieces of information: variation in front distance information, variation in front distance information, variation in ground side distance information, and variation in ground side distance information.

In this disclosure, "variation in distance information" includes any information related to variation in distance information, such as the magnitude of variation in distance information, the direction of variation (increase/decrease), time of variation, and time of no variation. It will be done.

Further, "distance information variations" includes any information regarding variations in a plurality of distance information detected in time series at a predetermined frame rate. For example, information regarding dispersion of a plurality of pieces of distance information detected in the most recent predetermined period or information regarding dispersion of a predetermined number of distance information detected most recently may be used.
For example, values such as variance and deviation (standard deviation) indicating variations are calculated. The variation time of the deviation, the magnitude of the variation in the deviation, the direction of the variation in the deviation (increase/decrease), the time during which the deviation does not vary, etc. are acquired as information regarding the "distance variation". Note that the values of variance and deviation (standard deviation) can be determined using known arithmetic expressions.

As the situation of the surrounding environment, for example, the presence or absence of a surrounding object 5, the distance to the object 5, the shape of the object 5, the type of the object 5, the size of the object 5, the material of the object 5, etc. may be output as a determination result. is possible.
Further, for example, the presence or absence of a fall danger point, the distance to the fall danger point, the type of fall danger point, the shape of the fall danger point, the size of the fall danger point, etc. can be output as the determination results. Of course, other situations may also be determined.

Note that the front side distance information and the ground side distance information acquired by the distance information acquisition unit 42 are information included in the surrounding information according to the present technology. Further, the determination result of the situation of the surrounding environment outputted by the situation judgment unit 43 is also information included in the surrounding information according to the present technology.

FIG. 14 is a flowchart showing an example of notification of surrounding information according to this embodiment.
When the user 2 turns on the power of the main body, the surrounding information notification system 41 starts up (step 201). For example, a power button or the like is installed on the sensor body 21, and the user 2 presses the power button or the like. Alternatively, if a voice input device such as a microphone is installed, the power may be turned on by voice input by the user 2.

Automatic calibration of the ranging sensor is started (step 202). In the present embodiment, automatic calibration is performed on the front-side ranging sensor 27 and the ground-side ranging sensor 28.

It is determined whether the automatic calibration results are normal (step 203). Specifically, the distance information (distance measurement value) of each distance measurement sensor is acquired, and it is determined whether the value is appropriate. For example, if the sensor body 21 is attached incorrectly or if the user's 2 hand or the like is covering each distance measurement sensor, the distance measurement value will not be appropriate and it will be determined that there is an abnormality. Further, even if each distance measuring sensor is out of order, it is determined that there is an abnormality.

If the result of automatic calibration is determined to be abnormal (No in step 203), a voice guide indicating an error is output from the speaker 11 (step 204). For example, a voice guide such as "Please check whether the device is worn properly" is output.
The user 2 corrects the orientation of each distance measuring sensor as a countermeasure for the error (step 205).

If it is determined that there is no abnormality as a result of the automatic calibration (Yes in step 203), the user 2 moves while searching for obstacles (objects 5) and falling danger points. In the surrounding information notification system 41, the distance information acquisition unit 42 acquires front side distance information and ground side distance information detected at a predetermined frame rate (step 206).

The situation determination unit 43 determines the situation of the surrounding environment based on fluctuations and dispersion of front side distance information and ground side distance information, which are two channels of distance information on the front side and the ground side. In this embodiment, determination results regarding obstacles and fall danger points are output. Specifically, it is determined whether an obstacle and a falling point are present (step 207).

FIG. 15 is a table showing an example of determination of the situation of the surrounding environment.
16 to 18 are schematic diagrams for explaining the determination example shown in FIG. 15.
FIG. 16 is a schematic diagram showing a case where an obstacle exists in a position in the front direction of the user 2.
FIG. 17 is a schematic diagram showing a case where an obstacle exists on the ground 3 in front of the user 2.
FIG. 18 is a schematic diagram showing a case where a fall danger point exists at a position in the front direction of the user 2.

As shown in FIGS. 16A and 16B, when an obstacle 44 (object 5) with a height of H or more exists at a position in the front direction, the front distance becomes smaller as the user 2 moves in the front direction. Furthermore, the deviation of the front distance also continues to fluctuate. Therefore, the situation determining unit 43 can determine that the obstacle 44 is present at the position in the front direction when the front distance becomes small and the variation time of the deviation of the front distance is longer than a predetermined time. .
More specifically, it is possible to determine that an obstacle 44 with a height of at least H or more exists in a position in the front direction.

Note that when determining the presence of the obstacle 44, a threshold regarding front distance information may be set. For example, when the front distance information becomes smaller than a predetermined threshold value, it may be determined that the obstacle 44 exists.
For example, when the front distance becomes smaller than a threshold value such as 5 m or 10 m, it is determined that the obstacle 44 exists at a position in the front direction. Thereby, it becomes possible to detect with high precision the obstacle 44 that exists within the distance where there is a possibility of collision.

Note that the "predetermined time" that serves as the criterion for determination may be appropriately set when constructing the surrounding information notification system 41. For example, by arranging the obstacle 44 at a position in the front direction and performing calibration or the like, an appropriate time during which the obstacle 44 can be detected is calculated. A threshold value is set as a "predetermined time" based on the calculated time. For example, the calculated time may be directly used as the threshold, or a time close to the calculated time may be used as the threshold. When the variation time of the deviation of the front side distance is longer than the threshold value, it is possible to determine that the variation time of the deviation of the front side distance is "longer than a predetermined time". Of course, the settings are not limited to this.

In FIG. 16A, a car 45 exists as an obstacle 44. In FIG. 16B, an upward staircase 46 is present as the obstacle 44.

As shown in FIG. 16A, when the user 2 moves toward the car 45, the car 45 comes relatively close to the measurement point P. When a portion of the vehicle 45 (for example, tires, bumper, etc.) comes into the detection range of the ground-side distance measuring sensor 28, the ground-side distance information becomes smaller. In other words, the ground side distance information does not substantially change until the vehicle 45 approaches the measurement point P.
As shown in FIG. 16A, when the car 45 approaches near the measurement point P, the front distance information becomes approximately equal to the distance D from the user 2 to the measurement point P.

As shown in FIG. 16B, when the user 2 moves toward the up stairs 46, the up stairs 46 relatively approaches the vicinity of the measurement point P. When the lowest step (first step) of the ascending stairs 46 comes into the detection range of the ground-side distance measuring sensor 28, the ground-side distance information becomes smaller. In other words, the ground side distance information does not substantially change until the up stairs 46 approaches the measurement point P.

The step portion on the upper side of the ascending staircase 46 is located further back in the front direction than the lowest step, and is located away from the user 2. Therefore, when the upward stairs 46 approaches the measurement point P, the front side distance information has a value larger than the distance D from the user 2 to the measurement point P.
After that, when the user 2 moves toward the upward stairs, the ground side distance information becomes further smaller, and the deviation of the ground side distance information continues to fluctuate. That is, until the front distance information becomes approximately equal to the distance D to the measurement point P, the ground side distance information becomes smaller and the deviation of the ground side distance information continues to fluctuate.

By focusing on these points, it is possible to determine whether the obstacle 44 present in the front direction is, for example, a car 45 or an upward staircase 46. Specifically, it is possible to determine whether the obstacle present in the front direction is a car 45 or an upward staircase 46 by focusing on the front side distance information at the timing when the ground side distance information becomes small. It is possible.

In this embodiment, the front side distance information becomes smaller and the fluctuation time of the deviation of the front side distance information is longer than a predetermined time, and the ground side distance information is changed until the front side distance information becomes smaller than a predetermined threshold. If there is no change, it is determined that there is an object other than the object constructed obliquely upward and away from the user at a position in the front direction.

In addition, if the front distance information becomes small and the fluctuation time of the deviation of the front distance information is longer than a predetermined time, and before the front distance information becomes smaller than the predetermined threshold, the ground side distance information becomes small and the ground side If the variation time of the deviation of the distance information is longer than a predetermined time, it is determined that there is an object constructed diagonally upward and away from the user at a position in the front direction.

The "predetermined threshold" regarding the front distance information is set based on the distance D on the ground 3 from the user 2 to the measurement point P. For example, the distance D may be used as it is as the threshold value. Alternatively, a value close to the distance D may be used as the threshold. For example, the "predetermined threshold" may be calculated by calibration or the like, or may be set arbitrarily by the user 2.

"A state in which the distance information does not change" includes not only a state in which the distance information does not change at all, but also a state in which there is almost no change. For example, a state in which a range with a relatively small width is set and the distance information falls within the range can be defined as a "state in which the distance information does not change." For example, it is possible to set a range of ±10% with respect to the reference distance information as the range for determining whether "the distance information is in a state where there is no change." Of course, it is not limited to this range. Alternatively, it may be set arbitrarily on the system side or the user side.

In the present embodiment, the presence of an object in the shape of an upward staircase is determined as "an object constructed diagonally upward and away from the user." Therefore, "an object other than an object obliquely constructed upward and away from the user" is an object other than an upward staircase-shaped object.

In FIG. 16A, a car 45 is an embodiment of an object other than an upwardly directed staircase-shaped object. Of course, the object is not limited to the car 45, and includes any object other than an object shaped like an upward staircase.
In FIG. 16B, an ascending staircase 46 is one embodiment of an upwardly directed staircase-shaped object. The object is not limited to the upward staircase 46, and for example, an upward escalator is also included in the upward staircase-shaped object. Other arbitrary step-shaped objects are also included.

In the table shown in FIG. 15, a "frontal obstacle" corresponds to an object other than an upward staircase-shaped object. The "up stairs/up escalator" corresponds to a staircase-shaped object heading upward.
In the table shown in FIG. 15, the condition for the ground side distance information for determining a "frontal obstacle" and "up stairs/up escalator" is that the front side distance information becomes smaller than a predetermined threshold. is listed.

In FIG. 17, an obstacle 48 (object 5) whose height is lower than H exists on the ground 3. Hereinafter, an obstacle 48 whose height is lower than H will be referred to as a ground obstacle 48 using the same reference numeral. The ground obstacle 48 shown in FIG. 17A is larger in size than the ground obstacle 48 shown in FIG. 17B.

As shown in FIG. 15, in this embodiment, when the ground side distance information becomes small and the variation time of the deviation of the ground side distance information is longer than a predetermined time in a state where the front side distance information does not change, the ground side distance information is changed. It is determined that a ground obstacle 48 larger than a predetermined size exists on the ground.
In addition, in a state where the front side distance information does not change, if the ground side distance information becomes small and the variation time of the deviation of the ground side distance information is shorter than a predetermined time, a ground smaller than the predetermined size is placed on the ground 3. It is determined that an obstacle 48 exists.

For example, when the ground obstacle 48 shown in FIG. 17A exists on the ground 3, the deviation of the ground side distance information is higher than when the ground obstacle 48 shown in FIG. 17B exists on the ground 3. The fluctuation time becomes longer. Based on the variation time of the deviation of the ground distance information, it is possible to determine the relative size of the ground obstacle 48 existing on the ground 3. That is, it is possible to determine whether the ground obstacle 48 is larger than a "predetermined size".

For example, a threshold value ("predetermined time") is set for the variation time of the deviation of the ground side distance information. Then, when the variation time of the deviation of the ground side distance information is longer than the threshold value, it is determined that a ground obstacle 48 having a relatively large size exists on the ground 3. If the variation time of the deviation of the ground side distance information is shorter than the threshold value, it is determined that a ground obstacle 48 with a relatively small size exists on the ground 3.

By setting the threshold value appropriately, what size of the ground obstacle 48 is considered to be a relatively large ground obstacle 48, and what size of the ground obstacle 48 is considered to be a relatively small ground obstacle 48. It becomes possible to set as appropriate. That is, by appropriately setting the threshold value, it is possible to appropriately set a "predetermined size" that is a criterion for determining the size of the ground obstacle 48.
For example, a "predetermined size" is set as a criterion for determining the size of the ground obstacle 48, and the ground side distance information is set so that determination can be made based on the set "predetermined size". A threshold value (“predetermined time”) may be set for the variation time of the deviation. The threshold value (“predetermined time” or “predetermined size”) serving as a criterion for determination may be arbitrarily set by the system or the user.

The moving speed (walking speed) of the user 2 may be acquired and the threshold value may be set using the information.
The size of the ground obstacle 48 is also defined based on the height of the ground obstacle 48, the area seen from the information on the ground obstacle 48, or both of these parameters. It may be set arbitrarily.

"Ground obstacle (large)" listed in the table of FIG. 15 corresponds to a ground obstacle 48 that is larger than a predetermined size (relatively large in size) that exists on the ground 3. “Ground obstacle (small)” corresponds to a ground obstacle 48 existing on the ground 3 that is smaller than a predetermined size (relatively small in size).

In FIG. 18A, a downward staircase 51 exists as a fall danger point 50 at a position in the front direction of the user 2. In FIG. 18B, an edge 52 of the platform exists as a fall danger point 50 in a position in the front direction.

As shown in FIG. 15, in this embodiment, when the ground side distance information becomes large, it is determined that a fall danger point 50, which is an area concave downward, exists in a position in the front direction. In the table of FIG. 15, the front side distance information is described as being unchanged or decreasing. The present invention is not limited to this, and regardless of the conditions of the front side distance information, it may be determined that the fall danger point 50 exists when the ground side distance information becomes large.

In FIGS. 18A and 18B, a descending staircase 51 and an edge 52 of the platform are illustrated as falling danger points 50. The present invention is not limited to this, and it is also possible to determine the presence of a downward escalator or any fall danger point 50 where there is a risk of the user 2 falling based on the ground side distance information. Note that the falling point in FIG. 15 corresponds to the falling danger point 50.

The notification information generation unit 18 and the notification control unit 19 execute notification processing of the obstacles 44 (48) and the falling danger points 50 (step 208).

FIG. 19 is a table showing an example of a process for notifying obstacles and falling danger points.
In this embodiment, the notification information generation unit 18 determines the danger level for movement of the user 2 based on the determination result by the situation determination unit 43. Then, notification information is generated and output so as to correspond to the danger level.

In this embodiment, it is possible to determine each of the presence of the obstacle 44 (48) and the presence of the fall danger point 50. As shown in FIG. 19, the danger level is determined to be higher when the fall danger point 50 exists than when the obstacle 44 (48) exists.
Specifically, "down stairs, down escalators, and other falling points" pose a high degree of danger to the user 2, and the danger level is set to "high." The danger level of the obstacle 44 (48) is set to "medium" or "low," which is lower than "high." This makes it possible to more strongly notify the user 2 of the danger of falling.

Furthermore, in this embodiment, the danger level is determined depending on the type of obstacle 44 (48). Specifically, the danger level of the "frontal obstacle" is set to "medium", assuming that the degree of injury to the user 2 at the time of a collision is moderate. The risk level of the "ground obstacle (large)" is set to "medium" because the possibility of tripping or the degree of injury to the user 2 in the event of a collision is moderate.

The danger level of the "ground obstacle (small)" is set to "small" because the possibility of tripping or the degree of injury to the user 2 in the event of a collision is low. The danger level of the "up stairs/up escalator" is set to "low" because the degree of injury to the user 2 is low. By setting the danger level in this way, highly accurate notification can be achieved. Note that the setting of the danger level is not limited to this example. In addition, it may be dynamically set by the system based on data on the user's status at the time (gender, age, health condition, whether or not hearing aids are worn, etc.), or it may be set arbitrarily by the user. It's okay. For example, even if a visually impaired person is the same, it is thought that the degree of risk of falling may differ between a young person and an elderly person. Therefore, if User 2 is a young person, the risk level of "Ground Obstacle (Large)" is set to "Medium", but if User 2 is an elderly person, it is changed to "High". may be performed.
Note that the distance D and musical tone data may be set as appropriate based on data on user status (gender, age, health condition, presence or absence of hearing aids, etc.). For example, it is generally said that people with hearing loss have difficulty hearing sounds in high frequency ranges compared to healthy people, so if user 2 is hard of hearing, he or she may use a low-frequency instrument instead of a high-frequency instrument as the main source of musical sound data. Processing such as allocation may also be performed.

Furthermore, as shown in FIG. 19, in this embodiment, notification by voice and notification by vibration are used. Specifically, audio notifications are performed regarding "frontal obstacles" and "up stairs/up escalators." Vibration notification is performed for "ground obstacles (large)", "ground-side obstacles (small)", "down stairs, down escalators, and other falling points".

That is, in this embodiment, audio information is generated as notification information in order to notify the situation of the surrounding environment corresponding to the front direction. On the other hand, vibration information is generated as notification information in order to notify the situation of the surrounding environment corresponding to the ground direction.
As a result, the user 2 can grasp the situation in the front direction through the sound, and can also grasp the situation in the ground direction through the vibration. That is, it becomes possible to broadcast information about the surrounding environment with high accuracy.
Of course, simultaneous notification to the user 2 via voice as described in the first embodiment may be adopted. Further, vibration information may be generated as notification information to notify the situation of the surrounding environment corresponding to the front direction, and audio information may be generated as the notification information to notify the situation of the surrounding environment corresponding to the ground direction.

As shown in FIG. 19, in this embodiment, a "safety distance range" and a "notification distance range" are set.
The "safe distance range" is a distance range that is far from the obstacle 44 (48) or the fall danger point 50 and is determined to be safe. If the distance from the obstacle 44 (48) or fall danger point 50 is included in the "safe distance range", no notification is necessary and reproduction of the notification information is stopped. That is, the output of audio information from the speaker 11 and the output of the vibration pattern according to the vibration information of the vibration device 12 are stopped.

The "notification distance range" is a distance range in which it is determined that the obstacle 44 (48) or fall danger point 50 is approaching and that notification is necessary. When the distance from the obstacle 44 (48) or the falling danger point 50 is included in the "reporting distance range", the notification according to the danger level is executed as follows, for example.
"Front obstacle" (danger level "medium")...Outputs discontinuous mid-range sound "Up stairs/up escalator" (danger level "low")...Outputs discontinuous low-range sound "Ground obstacle (Large)" (Danger level "Medium")...Outputs discontinuous mid-range vibrations. "Ground obstruction (Small)" (Danger level "Low")...Outputs discontinuous high-range vibrations. "Downward escalators and other falling points"...outputs discontinuous low-frequency vibrations.In this way, by executing notifications according to the danger level, highly accurate notifications can be achieved. This allows the user 2 to intuitively grasp the danger level.

The "safety distance range" and the "notification distance range" can be set using, for example, a threshold related to distance. If the obstacle 44 (48) or the fall danger point 50 is further away than the threshold value, it is determined that the distance is within the "safe distance range". If the obstacle 44 (48) or the falling danger point 50 is closer than the threshold value, it is determined that the obstacle 44 (48) or the fall danger point 50 is included in the "reported distance range".

The threshold value is set, for example, based on the distance D on the ground 3 from the user 2 to the measurement point P. For example, the distance D may be used as it is as the threshold value. Alternatively, a value close to the distance D may be used as the threshold.
For example, when the distance D from the user 2 to the measurement point P is relatively small, the distance D is used as it is as the threshold value. When the distance D from the user 2 to the measurement point P is relatively large, a value shorter than the distance D is used as the threshold value. In addition, any other setting method may be adopted.

Note that the method of determining the danger level, the method of outputting notification information, the method of setting the "safe distance range" and the "notification distance range", etc. are not limited and may be set arbitrarily. For example, personalization and customization can be freely performed based on the environment in which User 2 moves (such as what objects are present on the route he walks every day), information about User 2's walking (such as walking speed), etc. It's fine.

FIG. 20 is a schematic diagram for explaining another example of the process for notifying obstacles and falling danger points.
In the example shown in FIG. 20, the "notification distance range" is further divided into a "soft notification distance range" and a "danger notification distance range."

The "soft notification distance range" is a distance range for notifying that the obstacle 44 (48) or fall danger point 50 is approaching. In other words, it is a distance range that notifies you of the need for vigilance.
The "danger warning distance range" is a distance range in which the obstacles 44 (48) and falling danger points 50 are approaching and the danger level is high. In other words, this is a distance range that notifies you that the vehicle is about to collide with the obstacle 44 (48) or fall into the fall danger point 50.

In the example shown in FIG. 20, 4 m is set as the distance indicating the boundary between the "safety distance range" and the "notification distance range". Further, 2 m is set as the distance indicating the boundary between the "soft alert distance range" and the "danger alert distance range."
In other words, the range from 0m to 2m from an obstacle or a fall danger point is the "dangerous distance range." The "soft distance range" is a range of 2m to 4m from an obstacle or fall hazard point. The "safe distance range" is a range of 4 meters or more from an obstacle or fall hazard point. Here, the distance indicating the boundary between the "safety distance range" and the "notification distance range" may be set arbitrarily.

In the "danger warning distance range", in order to notify the user 2 that there is more danger than in the "soft warning distance range", a notification method that can sufficiently call attention is adopted. .

In the example shown in FIG. 20, the tempo is switched for the discontinuously output sounds and vibrations. Specifically, in the "soft notification distance range", discontinuous sounds and vibrations are output at a relatively low tempo. In the "danger warning distance range", discontinuous sounds and vibrations are output at a relatively high tempo. Of course, the present invention is not limited to this, and the intensity of the sound, the playback speed of the song, the BPM, the intensity of vibration, the frequency, etc. may be controlled.
By changing the notification method between the "soft notification distance range" and the "danger notification distance range," the user 2 can intuitively grasp how close the obstacle 44 (48) or falling danger point 50 is. This makes it possible to grasp the surrounding situation with high precision.

The notification of surrounding information continues until the user 2 turns off the main body power. When the user 2 turns off the main body power, the operation of the peripheral information notification system 41 ends (step 209).

FIG. 21 is a schematic diagram showing an example of detection of "ground obstacle (large)" and "ground obstacle (small)."
As shown in FIG. 21, until the detection range of the ground-side distance measuring sensor 28 reaches the ground obstacle 48, the ground-side distance information remains unchanged based on the distance to the ground 3, and the ground-side distance information The deviation of is stable. That is, as shown in FIG. 21, there is a "deviation stabilization period".

When the ground obstacle 48 approaches the position of the measurement point P, detection of the ground obstacle 48 is started, and the ground side distance information becomes smaller. While the ground obstacle 48 is being detected, the deviation of the ground side distance information fluctuates, resulting in a "deviation fluctuation period" as shown in FIG. 21.

When the ground side distance information returns to the value before the obstacle detection, the detection of the ground obstacle 48 ends. Thereafter, the ground side distance information remains unchanged with the distance to the ground 3 as a reference, and the "deviation stable period" again occurs as shown in FIG. 21.

Here, as shown in FIG. 21, when detecting the ground obstacle 48, the deviation of the ground side distance information may become an unspecified value at the detection start timing and the detection end timing. Therefore, the deviation of the ground side distance information at the detection start timing of the ground obstacle 48 and the deviation of the ground side distance information at the obstacle detection end timing may be processed so as not to be included in the obstacle determination.
In this case, as shown in FIG. 21, deviation data excluding the deviation of the ground distance information at the detection start timing and the detection end timing is used as data that can be used to determine an obstacle. This makes it possible to detect the ground obstacle 48 with high accuracy.

As described above, in the surrounding information notification system 41 according to the present embodiment, the controller 8 controls the fluctuation of the front side distance information (first distance information) detected by the front side distance measurement sensor 27 (first distance measurement sensor) and First detection information including variations, and second detection information including variations and variations in ground-side distance information (second distance information) detected by the ground-side distance measurement sensor 28 (second distance measurement sensor). Broadcast information is generated based on this. This makes it possible to detect information about the surrounding environment with high accuracy and notify the user 2 of the information.

Dangerous issues that can occur while walking for people with visual impairments such as amblyopia or total blindness include ``collisions with obstacles'' and ``falls due to irregularities on the floor.'' It is important to realize an accessibility device that can detect both of these factors and help visually impaired people avoid danger using an inexpensive device and an inexpensive system configuration.

In the surrounding information notification system 41 according to the present embodiment, danger avoidance does not use camera recognition technology that is unsuitable for low cost, weight reduction, and miniaturization, but uses a lightweight and inexpensive distance measurement sensor to implement the above-mentioned accessibility device. It is possible to achieve this.
As shown in Figure 15, by focusing on the fluctuations and dispersion of distance information in two channels, ``front direction'' and ``ground direction,'' it is possible to grasp the surrounding environment and situation while walking in detail and instantly. Become. In other words, it is possible to simultaneously detect "collision with an obstacle" and "fall/fall due to an abnormality on the floor". This has the effect of improving the safety and sense of security of the visually impaired user 2.

Additionally, in this embodiment, it is possible to estimate the obstacle characteristics (danger level). In addition, the hurdles for introducing it as a safe and inexpensive accessibility device will be lowered, and it can contribute to society.

As mentioned above, the detection directions of the plurality of ranging sensors may be set arbitrarily. Based on the fluctuations and variations in distance information of a plurality of distance measuring sensors whose detection directions are set in various directions, it is possible to notify the user 2 of various environmental information with high accuracy.

The type of fall danger point may be able to be determined by appropriately setting the number of ranging sensors and the detection direction of each ranging sensor. For example, it may be possible to determine whether the point is a fall danger point in the shape of a staircase going downward or a fall danger point in another shape. Furthermore, it may be possible to determine that it is at the edge of a station platform. Then, the danger level may be determined according to the type of fall danger point.

The threshold value for determining whether or not an obstacle is detected in the front direction and the distance D from the user 2 to the measurement point P (crossing angle θ in the direction toward the ground 3) are adjusted based on the route and path that the user 2 takes every day. may be done.
For example, route data frequently used by user 2 is acquired using GPS or the like. Based on the route data, settings such as the maximum distance for obstacle detection in the front direction and the irradiation angle in the ground direction may be automatically adjusted (personalized).

For example, in the case of a route with many upward stairs, settings suitable for detecting upward stairs are adopted. If the route has many dangerous falling points, settings suitable for detecting the dangerous falling points are adopted. The settings may be automatically adjusted during the route.

If a voice input device such as a microphone is installed, settings such as the maximum distance for obstacle detection in the front direction and the irradiation angle in the ground direction may be adjusted by voice input by the user 2.
Note that a motor mechanism, an actuator mechanism, or the like can be appropriately configured as a mechanism for automatically adjusting settings such as the maximum distance for obstacle detection in the front direction and the irradiation angle in the ground direction.

The configuration that can control the detection direction of each ranging sensor can also be said to be a direction control unit that changes at least one of the first direction and the second direction.

Furthermore, it may be possible to selectively switch between a mode in which only the front-side distance measurement sensor 27 is driven, a mode in which only the ground-side distance measurement sensor 28 is driven, and a mode in which both the two

distance measurement sensors

27 and 28 are driven. . For example, a mode in which only important channels are driven may be automatically set depending on the remaining battery level, or the user 2 may be able to set the mode as appropriate.

If an image sensor or the like is installed and a person can be detected based on image information, the detected person may be excluded from obstacle detection.
Further, as the configuration of the sensor section 6, an integral configuration with a smartphone may be adopted by utilizing a distance measuring sensor etc. mounted on the smartphone.

In the second embodiment described above, the distance information acquisition section 42 corresponds to one embodiment of the distance information acquisition section according to the present technology.
The situation determining unit 43 corresponds to an embodiment of the situation determining unit according to the present technology.
The notification information generation section 18 and the notification control section 19 function as an embodiment of a notification section that generates and outputs notification information for notifying the determination result by the situation determination section.

(Third embodiment)
FIG. 22 is a block diagram showing a configuration example for realizing notification of surrounding information according to the third embodiment.

In the surrounding information notification system 54 according to this embodiment, the sensor unit 6 is further equipped with a 9-axis sensor 55 and a GPS 56.
The 9-axis sensor 55 includes a 3-axis acceleration sensor, a 3-axis gyro sensor, and a 3-axis compass sensor. The nine-axis sensor 55 can detect acceleration, angular velocity, and orientation in three axes of the sensor section 6 (sensor main body 21). In addition, an IMU (Inertial Measurement Unit) sensor having any configuration may be used. The GPS 56 acquires information on the current position of the sensor section 6 (sensor main body 21). Further, a sensor that acquires biological information such as pulse, heartbeat, body temperature, and brain waves may be used as necessary.

In this embodiment, the controller 8 further includes a self-position estimating section 57 and a map information generating section 58. These blocks are realized by the processor of the controller 8 executing a program according to the present technology. In order to realize each functional block, dedicated hardware such as an IC (integrated circuit) may be used as appropriate.

The self-position estimating section 57 estimates the self-position of the sensor section 6 (sensor main body 21). In the present disclosure, the self-position includes the position and orientation of the sensor body 21. The self-position estimating unit 57 can calculate position information indicating where the sensor body 21 is located and posture information such as which direction the sensor body 21 is facing.
Furthermore, based on the posture information of the sensor body 21, it is possible to detect which direction the user 2 is currently facing. In other words, it is possible to detect which direction the front distance measuring sensor 27 is facing.

The attitude, position, and movement (movement) of the sensor body 21 can also be regarded as the attitude, position, and movement (movement) of the user 2.

The self-position of the sensor body 21 is calculated based on the detection results from the sensor section 6. In order to estimate the self-position, an image sensor or the like for acquiring surrounding image information may be installed.

For example, in order to calculate the self-position of the sensor body 21, a three-dimensional coordinate system is set for the surrounding space. For example, coordinate values (for example, XYZ coordinate values) defined by an absolute coordinate system (world coordinate system) may be used. Alternatively, coordinate values (for example, xyz coordinate values or uvd coordinate values) defined by a relative coordinate system with a predetermined point as a reference (origin) may be used. When a relative coordinate system is used, the origin serving as a reference may be set arbitrarily.

The self-position estimating unit 57 calculates position coordinates in the set three-dimensional coordinate system. In addition, when the X axis is the pitch axis, the Y axis is the roll axis, and the Z axis is the yaw axis, the pitch angle, roll angle, and yaw angle are calculated based on the front direction of user 2 (sensor body 21). Ru. Of course, the specific format of the position information and posture information of the user 2 (sensor body 21) is not limited.

The algorithm for estimating the self-position of the sensor body 21 is not limited, and any algorithm such as SLAM (Simultaneous Localization and Mapping) may be used. In addition, any machine learning algorithm or the like may be used.

The map information generation unit 58 generates an obstacle space map corresponding to the surrounding environment based on the history of determination results by the situation determination unit 43. The obstacle space map corresponds to one embodiment of surrounding map information according to the present technology.

As described in the second embodiment, the situation determination unit 43 can detect the obstacles 44 (48) and the fall danger points 50. In this embodiment, it is possible to calculate the position information in the three-dimensional coordinate system of the detected obstacles 44 (48) and fall danger points 50 based on the front direction information detected by the self-position estimating unit 57. It is.
That is, based on the history of the judgment results of the obstacles 44 (48) and fall danger points 50 detected up to now, the position information of the obstacles 44 (48) and fall danger points 50) detected up to now is determined. An obstacle space map containing the obstacles is generated.

FIG. 23 is a flowchart showing an example of notification of surrounding information according to this embodiment.
Steps 301-307 in FIG. 23 are similar to steps 201-207 shown in FIG.
In this embodiment, in step 308, the map information generation unit 58 generates an obstacle space map.

FIG. 24 is a schematic diagram showing an example of an obstacle space map. In FIG. 24, a three-dimensional coordinate system is set in which the XY direction is the horizontal direction and the Z direction is the height direction.
In step 308, based on the self-position of the user 2 and the yaw angle (rotation angle with respect to the Z axis) of the distance measurement sensor (front distance measurement sensor 27), detected obstacles 44 (48) and falling danger are detected. The XY coordinate values of point 50 are calculated as position information. Then, an obstacle space map 60 containing positional information of the detected obstacles 44 (48) and fall danger points 50 is generated.

In the example shown in FIG. 24, the obstacles 44 (48) and fall danger points 50 are all schematically illustrated in the same manner. Of course, it is possible to generate the obstacle space map 60 including information such as the types of obstacles 44 (48) and the types of fall danger points 50.
In other words, it is possible to generate an obstacle space map 60 that includes spatial position information of the positions, types, and attributes of the obstacles 44 (48) and fall danger points 50.
Hereinafter, the obstacles 44 (48) and the fall danger points 50 may be collectively referred to as objects to be avoided. The object to be avoided can also be called a dangerous object.

In step 309, the notification information generation unit 18 and notification control unit 19 execute notification via audio. In this embodiment, notification is performed using stereophonic sound regarding the object to be avoided that is closest to the user 2 on the obstacle space map 60. That is, the localization of the audio information is set so that the audio can be heard from the position of the object to be avoided at the closest distance with respect to the current direction of the user 2 (front direction). Additionally, audio information is output such that the volume is attenuated depending on the distance to the object to be avoided. In addition, in this embodiment, as an example, it is assumed that notification using stereophonic sound is performed for the object to be avoided at the closest distance from the user 2, but which object to avoid is targeted for notification using stereophonic sound? may be set arbitrarily. For example, notification using stereophonic sound may be performed regarding the object to be avoided that is closest to the user 2 and the object to be avoided that is the second closest to the user 2 . Furthermore, if a plurality of objects to be avoided are located at the same distance from the user 2, notification using stereophonic sound may be performed for all of the plurality of objects.

This allows the user 2 to intuitively grasp in which direction the closest object to be avoided exists with respect to the current front direction. As a result, it becomes possible to avoid danger with high precision.

For example, assume that instead of notification using stereophonic sound, notification is performed in response to the detection of an object to be avoided that exists in a position in the front direction. In this case, for example, in the state shown in the obstacle space map 60 of FIG. 24A, it is possible to detect and notify an object to be avoided that exists in a position in front of the user 2.

As shown in FIG. 24B, it is assumed that the user 2 changes the direction of travel to the right in order to avoid the notified object to be avoided. In this case, when the previously detected object to be avoided deviates from the front direction of the user 2, the detection and notification end. After that, the user 2 may walk right next to the avoidable object without being detected.
As a result, the vehicle may collide with the end of the obstacle 44 (48) or fall from the end of the fall danger point 50.

For example, when the sensor unit 6 (sensor main body 21) is attached to the head or arm of the user 2 to search for an object to be avoided, from a state in which an object to be avoided is detected in the front direction and a notification is performed, For example, let us assume that a person shakes his or her head or swings an arm. In this case, the detection of the object to be avoided may fail and the notification may end. In this state, the vehicle may move toward the object to be avoided that was detected immediately before, causing a collision with or falling from the object to be avoided. That is, it may become impossible to avoid an object to be avoided once it has been detected.

In this embodiment, an obstacle space map 60 is generated that includes position information of objects to be avoided that have been detected in the past. Then, an object to be avoided that is close to the user 2 is notified by stereophonic sound so that the direction toward the user 2 is reflected.
Thereby, even if the object to be avoided deviates from the front direction of the user 2, it is possible to notify the user 2 of the object to be avoided. This makes it possible to solve the above-mentioned problems and improve the success rate of avoiding the detected object to be avoided.

The notification method described in the second embodiment and the notification method described in the third embodiment may be used together. That is, both real-time notification for the front direction and ground direction and notification using stereophonic sound using the obstacle space map 60 may be performed. Of course, the notification method described in the first embodiment may be used in combination.

Note that if the user 2 is also a hearing-impaired person (for example, is hard of hearing or uses a hearing aid/sound collector, etc.), it is also effective to provide notification using vibration instead of stereophonic sound. That is, vibrations are presented to the body part corresponding to the position of the object to be avoided at the closest distance with respect to the current direction of the user 2 (front direction). Further, the strength of the vibration may be attenuated depending on the distance to the object to be avoided. The user 2 may be able to appropriately set whether to notify using stereophonic sound or vibration. Further, if the user 2 is using a hearing aid, a sound collector, etc., processing such as hearing aid processing may be performed on the stereophonic sound that is notified.

FIG. 25 is a schematic diagram showing another configuration example of the surrounding information notification system according to the present embodiment.
In the example shown in FIG. 25, the peripheral information notification system 64 is realized by the controller 8 shown in FIG. 1 and the server device 63 arranged on the network 62 working together.

The network 62 is constructed by, for example, the Internet or a wide area communication network. In addition, any WAN (Wide Area Network), LAN (Local Area Network), etc. may be used, and the protocol for constructing the network 62 is not limited.

The server device 63 includes hardware necessary for configuring a computer, such as a CPU, ROM, RAM, and HDD. For example, the server device 63 can be realized by any computer such as a PC (Personal Computer).
As shown in FIG. 25, in this embodiment, the map information generation unit 58 is implemented by the server device 63.

From the controller 8, the determination result by the situation determination unit 43 ("obstacle/fall danger point information" in the figure) is transmitted to the server device 63 via the network 62. Although not shown, information on the self-position estimated by the self-position estimating unit 57 is also transmitted to the server device 63 via the network 63.

In the server device 63, a history of determination results by the situation determining section 43 is stored in an obstacle information DB. Then, the map information generation unit 58 generates an obstacle space map 60 as illustrated in FIG. 24 . The obstacle space map 60 generated by the server device 63 is transmitted to the controller 8 via the network 62.
Note that the DB may be constructed in a storage device within the server device 63, or may be constructed in an external storage device that the server device 63 can access.

The notification information generation unit 18 and notification control unit 19 of the controller 8 execute notification using stereophonic sound based on the received obstacle space map 60.

In this way, the surrounding information notification system 64 can be realized by a cloud system using the cloud (cloud computing). By using a cloud system, even if the edge terminal configured on the user 2 side is not equipped with high processing capacity, it is possible to realize notification using a wide range obstacle space map 60.
As a result, it is possible to realize an accessibility device that is lightweight and inexpensive, yet can report information about the surrounding environment with high accuracy.

The server device 63 on the network 62 may integrate "obstacle/fall danger point information" sent from multiple users 2 using the surrounding information notification system 64. Then, the obstacle space map 60 for each user 2 may be generated based on the integrated "obstacle/fall danger point information".

For example, the sensor main body 21 worn by the user 2 walking in a certain area performs a search in the front direction and the ground direction. “Obstacle/fall danger point information” related to the detection of the detected object to be avoided is transmitted from the controller 8 to the server device 63 via the network 62 .

Here, it is assumed that there is another user 2 walking in the same area. The sensor main body 21 attached to the other user 2 performs a search in the front direction and the ground direction. “Obstacle/fall danger point information” related to the detection of the detected object to be avoided is transmitted from the controller 8 to the server device 63 via the network 62 .

The server device 63 integrates the information on the avoidable object detected by the user 2's search and the information on the avoidable object detected by the other user 2's search, and stores it in the obstacle information DB. Then, an obstacle space map 60 is generated that includes both the positional information of the avoidable object detected by the user 2's search and the positional information of the avoidable object detected by the search of another user 2. The generated obstacle space map 60 is transmitted to both the user 2 and other users 2 via the network 62.

As a result, it is possible to send to user 2 an obstacle space map that includes position information of objects to be avoided that were not detected by user 2's search but were detected by other user 2's searches. becomes. As a result, it becomes possible to notify the user 2 of information on objects to be avoided existing in the vicinity with higher accuracy.
Of course, it is also possible to notify other users 2 of information on objects to be avoided in the vicinity with higher precision. For example, by sharing information on objects to be avoided that have been detected by User 2 and his or her family, it is possible to notify information on objects to be avoided in the surrounding area with higher accuracy. is assumed.

(Fourth embodiment)
FIG. 26 is a schematic diagram showing a configuration example of a surrounding information notification system according to the fourth embodiment.
The surrounding information notification system 66 according to this embodiment is configured by a cloud system. That is, the peripheral information notification system 66 is realized by the controller 8 shown in FIG. 1 and the server device 63 arranged on the network 62 working together.

Additionally, the surrounding information notification system 66 includes a guide device 67 that is communicably connected to the controller 8 and the server device 63 via the network 62. The guidance device 67 has a display and is used as a remote terminal used by an operator (route guidance etc. sender) 68 who provides route guidance etc. to the user 2. The route guidance can also be called a walking guidance notification.

As shown in FIG. 26, the server device 63 stores the information on the object to be avoided ("obstacle/fall danger point information") sent from the user 2 in the obstacle information DB. Further, the server device 63 has built therein a real world map information DB in which map information of the real world is stored. For example, map information of various regions is acquired from a map server that provides map services on the network 62, and stored in the real world map information DB.

FIG. 27 is a flowchart showing an example of notification of surrounding information according to this embodiment.
Steps 401-408 in FIG. 27 are similar to steps 301-308 shown in FIG. 23.

In step 408 , the controller 8 transmits the determination result by the situation determining unit 43 (“obstacle/fall danger point information”) and the self-position information estimated by the self-position estimating unit 57 to the server device via the network 63. 63. Then, the map information generation unit 58 in the server device 63 generates an obstacle space map 60.

In this embodiment, the map information generation unit 58 generates a real space dangerous object map 69 in which surrounding real world information is added to the obstacle space map 60 based on the user 2's position information in the real world. . As shown in FIG. 27, in this embodiment, the obstacle space map 60 and real world map information are linked to generate a real space dangerous object map 69 to which landmark information and the like are added (step 409).

The real space dangerous object map 69 includes the user 2's real world position information, such as the real world position information and attribute information of the obstacles 44 (48) that are objects to be avoided (dangerous objects) and the falling danger points 50, and Contains real world information such as landmark information. Note that the real world information includes arbitrary geographic information such as place names and topography.
The real space dangerous object map 69 is an embodiment of real surrounding map information according to the present technology.

The real space dangerous object map 69 is transmitted to the guide device 67 via the network 62 and displayed on the display of the guide device 67. The display mode of the real space dangerous object map 69 is not limited and may be set arbitrarily.
For example, an icon of an object to be avoided may be superimposed on map information of the real world, and when the icon is selected, detailed information of the object to be avoided may be displayed. Alternatively, the display area of the display may be divided and information regarding the avoidable object may be displayed in a list.

In this embodiment, the operator 68 executes route guidance using the real space dangerous object map 69. For example, ``There are stairs leading down to the entrance of 〇〇 station 5 meters ahead. Please proceed with caution.'' ``A car is parked at the entrance of 〇〇 parking lot. Please stop temporarily.'' ``There is a large obstacle on the ground. It becomes possible to provide route guidance that combines information from the real world with information about dangerous objects to avoid, such as "There is an object falling. Please slow down." As a result, the user 2 can obtain information normally obtained by a healthy person, and it is possible to further improve safety in avoiding danger.

On the user 2 side, the notification information generation unit 18 in the controller 8 receives guidance information including the contents of the route guidance of the operator 68 (step 410). Based on the received guidance information, the notification control unit 19 outputs the contents of the route guidance from the speaker 11.

The configuration and method for transmitting the guidance information to the controller 8 on the user 2 side via the network 62 are not limited. For example, the guidance information can be transmitted using a well-known technique using a voice input device such as a microphone of the guidance device 67, a communication device, or the like.

The route guidance provided by the operator 68 can also be said to be notification using real world information based on the real space dangerous object map 69. Therefore, it can be said that the mechanism for transmitting guidance information including the contents of route guidance for the operator 68 provided in the guidance device 67 functions as a "notification section" in the surrounding information notification system 66.

Note that instead of the route guidance by the operator 68, automatic voice route guidance based on the real space dangerous object map 69 may be performed. Automatic voice route guidance based on the real space dangerous object map 69 is also included in the notification using real world information based on the real space dangerous object map 69. Further, the mechanism that executes route guidance using automatic voice functions as a "notification unit" in the surrounding information notification system 66.

For example, the real space dangerous object map 69 may be transmitted to the controller 8 on the user 2 side via the network 62. Then, the notification information generation section 18 and the notification control section 19 may perform route guidance or the like based on the real space dangerous object map 69.

By applying the surrounding information notification system according to each of the above embodiments, the image of a company that creates smart accessibility products will be enhanced, and the company's brand image and existence value can be expected to be enhanced as a company that aims to contribute to society. .

<Other embodiments>
The present technology is not limited to the embodiments described above, and various other embodiments can be realized. In the following, techniques applicable to the surrounding information notification system of each embodiment described above will be listed as appropriate.

FIG. 28 is a schematic diagram showing another example of a method of outputting sound according to distance.
In the example shown in FIG. 28, musical tone information such as a predetermined song is played in the "safe distance range." For example, songs that the user 2 likes are played.
When entering the "soft notification distance range", a detection notification sound that notifies that an object to be avoided has been detected is faded in so as to be superimposed on the musical sound information. As the detection notification sound, for example, discontinuous mid-range sound is output. In the example shown in FIG. 28, the mixing amount for the detection notification sound is increased linearly, but the present invention is not limited to this, and various other fade-in controls may be adopted.
When the distance to the object to be avoided becomes shorter in the "soft notification distance range", both the musical tone information and the detection notification sound are output at the maximum standard level.

When the distance to the object to be avoided approaches 2 m, which indicates the boundary between the "soft alert distance range" and the "danger alert distance range," both the musical tone information and the detection notification sound are faded out. In the example shown in FIG. 28, the mixing amount for the musical tone information is reduced in a curve (rapidly), and the mixing amount for the detection notification sound is reduced linearly (reduced at a constant rate). Various such fade-out controls may be employed.

At the same time as the musical sound information and the detection notification sound fade out, the danger notification sound indicating that danger is approaching is faded in. As the danger notification sound, for example, discontinuous high-frequency sound is output.
When the distance to the object to be avoided falls within the "danger warning distance range", the danger notification sound becomes louder to the maximum standard level to strongly alert the user 2.

For example, when a buzzer sound is output when an object to be avoided is detected, the buzzer sound may continue to sound in a crowded station premises, inside an elevator, or when using an escalator. It may also be accompanied by discomfort.

In the example shown in FIG. 28, music information of "favorite music" is used as the main sound source, and detection notification sound such as sonar sound is mixed in a fading manner according to the approach distance to the object to be avoided. To go. When the object to be avoided is extremely close, the main music, detection notification sound, and danger notification sound are cross-faded and mixed. Such notification becomes possible, and it becomes possible to eliminate notifications that are unpleasant for the user 2.

By adding a determination that the object to be avoided is a person (not the object to be avoided), it is also possible to filter out unnecessary notifications in the case of extremely close proximity to an object that is not a person. For example, if it is determined that the object that approaches is a person based on the amplitude of ultrasonic waves detected by an ultrasonic ranging sensor or person recognition based on image information from an image sensor, surrounding information will not be reported. A setting that does not occur may also be adopted.

Information on environmental sounds (sounds of trains running, crowds, etc.) acquired as surrounding information in cases where multiple ranging sensors with different methods or detection directions are arranged, etc. Alternatively, the necessary distance measurement channels may be automatically switched based on map information of the real world or the like. For example, in the configuration shown in FIG. 8, even if sensing in the front direction by the front-side distance measurement sensor 27 and sensing in the ground direction by the ground-side distance measurement sensor 28 are automatically switched based on surrounding information. good.
For example, when the user 2 is moving on a station platform, it is possible to prioritize sensing toward the ground.

As a result of self-position estimation using SLAM or the like, the walking speed of the user 2 is estimated from the change in the front side distance information, and is used to determine the size of the ground obstacle 48 along with the variation time of the deviation of the ground side distance information. good.

By applying the peripheral information notification system according to the present technology, it is also possible to realize a device compatible with a speech-type UI for visually impaired people.
On the other hand, it is also possible to construct a surrounding information notification system according to the present technology for healthy people. For example, a front-side distance measuring sensor whose detection direction is in the front direction and a back-side distance measuring sensor whose detection direction is in the back direction (direction toward the rear side) are arranged. Then, detection of a suspicious person sneaking up from behind may be performed based on the back side distance information detected by the back side distance measuring sensor.

For example, if three distance measurement sensors are arranged so that there are three channels in the front direction and in the left and right directions, it is possible to determine whether the user is walking straight based on the distance information of the left and right distance measurement channels. It is also possible to determine whether
Alternatively, based on the distance information of each distance measurement channel, it may be determined whether or not the route has deviated from a predetermined route, whether it has deviated from a Braille block, etc.
Of course, machine learning algorithms may be used for these determinations.

For example, assume that an object is detected at a constant distance on the front side based on the front side distance information with no variation in the deviation of the ground side distance information. In this case, it is assumed that a stationary object has been detected, and the output of the alarm sound etc. is stopped. It is also possible to adopt such a setting that prevents the alarm sound from continuously sounding.

By collecting sounds from the surrounding environment while the user is walking, it is also possible to guide the user 2 in the direction of a person who will support him (for example, the user's family, caregiver, etc.) using stereophonic sound or the like.
Furthermore, when approaching an obstacle, the effects of NR (noise reduction) and NC (noise canceling) may be suppressed to make it easier to hear surrounding environmental sounds. For example, when a train approaches, it is possible to turn off the NR and NC and let the user 2 hear the sound of the train to alert the user.

The surrounding information notification system according to the present technology may be provided to blindfolded healthy people at a theme park or the like where they can experience the experience of blind people.
Furthermore, when a healthy person uses a smartphone while walking, a surrounding information notification system according to the present technology may be constructed by attaching a distance measurement sensor or the like in the same direction as the outward-facing camera.
Further, a surrounding information system according to the present technology may be constructed for a vehicle, a drone, or the like, and the surrounding information may be notified to a pilot or the like.

FIG. 29 is a block diagram showing an example of a hardware configuration of a computer (information processing device) 70 that can be used to construct a peripheral information notification system according to the present technology.
The computer 70 includes a CPU 71, a ROM 72, a RAM 73, an input/output interface 75, and a bus 74 that connects these to each other. A display section 76, an input section 77, a storage section 78, a communication section 79, a drive section 80, and the like are connected to the input/output interface 75.
The display section 76 is a display device using, for example, liquid crystal, EL, or the like. The input unit 77 is, for example, a keyboard, pointing device, touch panel, or other operating device. If the input section 77 includes a touch panel, the touch panel can be integrated with the display section 76.
The storage unit 78 is a nonvolatile storage device, such as an HDD, flash memory, or other solid-state memory. The drive section 80 is a device capable of driving a removable recording medium 81 such as an optical recording medium or a magnetic recording tape.
The communication unit 79 is a modem, router, or other communication equipment that can be connected to a LAN, WAN, etc., and is used to communicate with other devices. The communication unit 79 may communicate using either wired or wireless communication. The communication unit 79 is often used separately from the computer 70.
Information processing by the computer 70 having the above hardware configuration is realized by cooperation between software stored in the storage unit 78 or the ROM 72, and hardware resources of the computer 70. Specifically, the information processing method according to the present technology is realized by loading a program constituting software stored in the ROM 72 or the like into the RAM 73 and executing it.
The program is installed on the computer 70 via the recording medium 61, for example. Alternatively, the program may be installed on the computer 70 via a global network or the like. In addition, any computer-readable non-transitory storage medium may be used.

The information processing method (peripheral information notification method) and program according to the present technology are executed by multiple computers communicably connected via a network etc., and the information processing device according to the present technology is constructed. It's okay.
That is, the information processing method and program according to the present technology can be executed not only in a computer system configured by a single computer but also in a computer system in which multiple computers operate in conjunction with each other.
Note that in the present disclosure, a system means a collection of multiple components (devices, modules (components), etc.), and it does not matter whether all the components are in the same housing or not. Therefore, a plurality of devices housed in separate casings and connected via a network and a single device in which a plurality of modules are housed in one casing are both systems.
Execution of the information processing method and program according to the present technology by a computer system includes, for example, acquisition of surrounding information (obtaining distance information, situation determination), generation of notification information (generation of audio information, generation of vibration information), and notification control. This includes both the case where each process is executed by a single computer and the case where each process is executed by different computers. Furthermore, execution of each process by a predetermined computer includes causing another computer to execute part or all of the process and acquiring the results.
That is, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

Each configuration of the surrounding information notification system, sensor section, information output section, controller, etc., described with reference to each drawing, acquisition of surrounding information (obtainment of distance information, situation determination), generation of notification information (generation of audio information, (generation of vibration information), the processing flow of notification control, etc. are just one embodiment, and can be arbitrarily modified without departing from the spirit of the present technology. That is, any other configuration, algorithm, etc. may be adopted for implementing the present technology.

In this disclosure, words such as "approximately,""approximately," and "approximately" are used as appropriate to facilitate understanding of the explanation. On the other hand, there is no clear difference between when words such as "abbreviation,""approximately," and "approximately" are used and when they are not.
That is, in the present disclosure, "center", "center", "uniform", "equal", "same", "orthogonal", "parallel", "symmetrical", "extending", "axial direction", "cylindrical shape", "cylindrical shape", "ring shape" Concepts that define the shape, size, positional relationship, state, etc., such as "circular shape", include "substantially centered,""substantiallycentral,""substantiallyuniform,""substantiallyequal," and "substantially "Substantially perpendicular""Substantiallyparallel""Substantiallysymmetrical""Substantiallyextending""Substantiallyaxial""Substantiallycylindrical""Substantiallycylindrical" The concept includes "substantially ring-shaped", "substantially annular-shaped", etc.
For example, "perfectly centered", "perfectly centered", "perfectly uniform", "perfectly equal", "perfectly identical", "perfectly orthogonal", "perfectly parallel", "perfectly symmetrical", "perfectly extended", "perfectly It also includes states that fall within a predetermined range (e.g. ±10% range) based on the following criteria: axial direction, completely cylindrical, completely cylindrical, completely ring-shaped, completely annular, etc. It will be done.
Therefore, even when words such as "approximately,""approximately," and "approximately" are not added, concepts that can be expressed by adding so-called "approximately,""approximately," and "approximately" may be included. On the other hand, when a state is expressed by adding words such as "approximately", "approximately", "approximately", etc., a complete state is not always excluded.

In this disclosure, expressions using "more" such as "greater than A" and "less than A" are inclusive of both concepts that include the case of being equivalent to A and concepts that do not include the case of being equivalent to A. This is an expression included in For example, "greater than A" is not limited to not including "equivalent to A", but also includes "more than A". Moreover, "less than A" is not limited to "less than A", but also includes "less than A".
When implementing the present technology, specific settings etc. may be appropriately adopted from the concepts included in "greater than A" and "less than A" so that the effects described above are exhibited.

It is also possible to combine at least two of the feature parts according to the present technology described above. That is, the various characteristic portions described in each embodiment may be arbitrarily combined without distinction between each embodiment. Further, the various effects described above are merely examples and are not limited, and other effects may also be exhibited.

<Application example>>
The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can be applied to any type of transportation such as a car, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility vehicle, an airplane, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc. It may also be realized as a device mounted on the body.

FIG. 30 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied. Vehicle control system 7000 includes multiple electronic control units connected via communication network 7010. In the example shown in FIG. 30, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an outside vehicle information detection unit 7400, an inside vehicle information detection unit 7500, and an integrated control unit 7600. . The communication network 7010 connecting these plurality of control units is, for example, a communication network based on any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark). It may be an in-vehicle communication network.

Each control unit includes a microcomputer that performs calculation processing according to various programs, a storage unit that stores programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various devices to be controlled. Equipped with Each control unit is equipped with a network I/F for communicating with other control units via the communication network 7010, and also communicates with devices or sensors inside and outside the vehicle through wired or wireless communication. A communication I/F is provided for communication. In FIG. 30, the functional configuration of the integrated control unit 7600 includes a microcomputer 7610, a general-purpose communication I/F 7620, a dedicated communication I/F 7630, a positioning section 7640, a beacon receiving section 7650, an in-vehicle device I/F 7660, an audio image output section 7670, An in-vehicle network I/F 7680 and a storage unit 7690 are illustrated. The other control units similarly include a microcomputer, a communication I/F, a storage section, and the like.

The drive system control unit 7100 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 7100 includes a drive force generation device such as an internal combustion engine or a drive motor that generates drive force for the vehicle, a drive force transmission mechanism that transmits the drive force to wheels, and a drive force transmission mechanism that controls the steering angle of the vehicle. It functions as a control device for a steering mechanism to adjust and a braking device to generate braking force for the vehicle. The drive system control unit 7100 may have a function as a control device such as ABS (Antilock Brake System) or ESC (Electronic Stability Control).

A vehicle state detection section 7110 is connected to the drive system control unit 7100. The vehicle state detection unit 7110 includes, for example, a gyro sensor that detects the angular velocity of the axial rotation movement of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, or an operation amount of an accelerator pedal, an operation amount of a brake pedal, or a steering wheel. At least one sensor for detecting angle, engine rotational speed, wheel rotational speed, etc. is included. The drive system control unit 7100 performs arithmetic processing using signals input from the vehicle state detection section 7110, and controls the internal combustion engine, the drive motor, the electric power steering device, the brake device, and the like.

The body system control unit 7200 controls the operations of various devices installed in the vehicle body according to various programs. For example, the body system control unit 7200 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as a headlamp, a back lamp, a brake lamp, a turn signal, or a fog lamp. In this case, radio waves transmitted from a portable device that replaces a key or signals from various switches may be input to the body control unit 7200. The body system control unit 7200 receives input of these radio waves or signals, and controls the door lock device, power window device, lamp, etc. of the vehicle.

The battery control unit 7300 controls the secondary battery 7310, which is a power supply source for the drive motor, according to various programs. For example, information such as battery temperature, battery output voltage, or remaining battery capacity is input to the battery control unit 7300 from a battery device including a secondary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals, and controls the temperature adjustment of the secondary battery 7310 or the cooling device provided in the battery device.

The external information detection unit 7400 detects information external to the vehicle in which the vehicle control system 7000 is mounted. For example, at least one of an imaging section 7410 and an external information detection section 7420 is connected to the vehicle exterior information detection unit 7400. The imaging unit 7410 includes at least one of a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The vehicle external information detection unit 7420 includes, for example, an environmental sensor for detecting the current weather or weather, or a sensor for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle equipped with the vehicle control system 7000. At least one of the surrounding information detection sensors is included.

The environmental sensor may be, for example, at least one of a raindrop sensor that detects rainy weather, a fog sensor that detects fog, a sunlight sensor that detects the degree of sunlight, and a snow sensor that detects snowfall. The surrounding information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. The imaging section 7410 and the vehicle external information detection section 7420 may be provided as independent sensors or devices, or may be provided as a device in which a plurality of sensors or devices are integrated.

Here, FIG. 31 shows an example of the installation positions of the imaging section 7410 and the vehicle external information detection section 7420. The

imaging units

7910, 7912, 7914, 7916, and 7918 are provided, for example, at at least one of the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle 7900. An imaging unit 7910 provided in the front nose and an imaging unit 7918 provided above the windshield inside the vehicle mainly acquire images in front of the vehicle 7900.

Imaging units

7912 and 7914 provided in the side mirrors mainly capture images of the sides of the vehicle 7900. An imaging unit 7916 provided in the rear bumper or back door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 provided above the windshield inside the vehicle is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, and the like.

Note that FIG. 31 shows an example of the imaging range of each of the

imaging units

7910, 7912, 7914, and 7916. Imaging range a indicates the imaging range of imaging unit 7910 provided on the front nose, imaging ranges b and c indicate imaging ranges of

imaging units

7912 and 7914 provided on the side mirrors, respectively, and imaging range d is The imaging range of an imaging unit 7916 provided in the rear bumper or back door is shown. For example, by superimposing image data captured by

imaging units

7910, 7912, 7914, and 7916, an overhead image of vehicle 7900 viewed from above can be obtained.

The external

information detection units

7920, 7922, 7924, 7926, 7928, and 7930 provided at the front, rear, sides, corners, and the upper part of the windshield inside the vehicle 7900 may be, for example, ultrasonic sensors or radar devices. External

information detection units

7920, 7926, and 7930 provided on the front nose, rear bumper, back door, and upper part of the windshield inside the vehicle 7900 may be, for example, LIDAR devices. These external information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, and the like.

Returning to FIG. 30, the explanation continues. The vehicle exterior information detection unit 7400 causes the imaging unit 7410 to capture an image of the exterior of the vehicle, and receives the captured image data. Further, the vehicle exterior information detection unit 7400 receives detection information from the vehicle exterior information detection section 7420 to which it is connected. When the external information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the external information detection unit 7400 transmits ultrasonic waves, electromagnetic waves, etc., and receives information on the received reflected waves. The external information detection unit 7400 may perform object detection processing such as a person, car, obstacle, sign, or text on the road surface or distance detection processing based on the received information. The external information detection unit 7400 may perform environment recognition processing to recognize rain, fog, road surface conditions, etc. based on the received information. The vehicle exterior information detection unit 7400 may calculate the distance to the object outside the vehicle based on the received information.

Additionally, the outside-vehicle information detection unit 7400 may perform image recognition processing or distance detection processing for recognizing people, cars, obstacles, signs, characters on the road, etc., based on the received image data. The outside-vehicle information detection unit 7400 performs processing such as distortion correction or alignment on the received image data, and also synthesizes image data captured by different imaging units 7410 to generate an overhead image or a panoramic image. Good too. The outside-vehicle information detection unit 7400 may perform viewpoint conversion processing using image data captured by different imaging units 7410.

The in-vehicle information detection unit 7500 detects in-vehicle information. For example, a driver condition detection section 7510 that detects the condition of the driver is connected to the in-vehicle information detection unit 7500. The driver state detection unit 7510 may include a camera that images the driver, a biosensor that detects biometric information of the driver, a microphone that collects audio inside the vehicle, or the like. The biosensor is provided, for example, on a seat surface or a steering wheel, and detects biometric information of a passenger sitting on a seat or a driver holding a steering wheel. The in-vehicle information detection unit 7500 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 7510, or determine whether the driver is dozing off. You may. The in-vehicle information detection unit 7500 may perform processing such as noise canceling processing on the collected audio signal.

The integrated control unit 7600 controls overall operations within the vehicle control system 7000 according to various programs. An input section 7800 is connected to the integrated control unit 7600. The input unit 7800 is realized by a device that can be inputted by the passenger, such as a touch panel, a button, a microphone, a switch, or a lever. The integrated control unit 7600 may be input with data obtained by voice recognition of voice input through a microphone. The input unit 7800 may be, for example, a remote control device using infrared rays or other radio waves, or an externally connected device such as a mobile phone or a PDA (Personal Digital Assistant) that is compatible with the operation of the vehicle control system 7000. It's okay. The input unit 7800 may be, for example, a camera, in which case the passenger can input information using gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by a passenger may be input. Furthermore, the input section 7800 may include, for example, an input control circuit that generates an input signal based on information input by a passenger or the like using the input section 7800 described above and outputs it to the integrated control unit 7600. By operating this input unit 7800, a passenger or the like inputs various data to the vehicle control system 7000 and instructs processing operations.

The storage unit 7690 may include a ROM (Read Only Memory) that stores various programs executed by the microcomputer, and a RAM (Random Access Memory) that stores various parameters, calculation results, sensor values, etc. Further, the storage unit 7690 may be realized by a magnetic storage device such as a HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

The general-purpose communication I/F 7620 is a general-purpose communication I/F that mediates communication with various devices existing in the external environment 7750. The general-purpose communication I/F7620 supports cellular communication protocols such as GSM (registered trademark) (Global System of Mobile communications), WiMAX (registered trademark), LTE (registered trademark) (Long Term Evolution), or LTE-A (LTE-Advanced). , or other wireless communication protocols such as wireless LAN (also referred to as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). The general-purpose communication I/F 7620 connects to a device (for example, an application server or a control server) existing on an external network (for example, the Internet, a cloud network, or an operator-specific network) via a base station or an access point, for example. You may. In addition, the general-purpose communication I/F 7620 uses, for example, P2P (Peer To Peer) technology to communicate with a terminal located near the vehicle (for example, a driver, a pedestrian, a store terminal, or an MTC (Machine Type Communication) terminal). You can also connect it with

The dedicated communication I/F 7630 is a communication I/F that supports communication protocols developed for use in vehicles. The dedicated communication I/F 7630 uses standard protocols such as WAVE (Wireless Access in Vehicle Environment), which is a combination of lower layer IEEE802.11p and upper layer IEEE1609, DSRC (Dedicated Short Range Communications), or cellular communication protocol. May be implemented. The dedicated communication I/F 7630 typically supports vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication. ) communications, a concept that includes one or more of the following:

The positioning unit 7640 performs positioning by receiving, for example, a GNSS signal from a GNSS (Global Navigation Satellite System) satellite (for example, a GPS signal from a GPS (Global Positioning System) satellite), and determines the latitude, longitude, and altitude of the vehicle. Generate location information including. Note that the positioning unit 7640 may specify the current location by exchanging signals with a wireless access point, or may acquire location information from a terminal such as a mobile phone, PHS, or smartphone that has a positioning function.

The beacon receiving unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from a wireless station installed on the road, and obtains information such as the current location, traffic jams, road closures, or required travel time. Note that the function of the beacon receiving unit 7650 may be included in the dedicated communication I/F 7630 described above.

The in-vehicle device I/F 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I/F 7660 may establish a wireless connection using a wireless communication protocol such as wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). In addition, the in-vehicle device I/F 7660 connects to USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), or MHL (Mobile High The in-vehicle device 7760 may include, for example, at least one of a mobile device or wearable device owned by a passenger, or an information device carried into or attached to the vehicle. In addition, the in-vehicle device 7760 may include a navigation device that searches for a route to an arbitrary destination. or exchange data signals.

The in-vehicle network I/F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I/F 7680 transmits and receives signals and the like in accordance with a predetermined protocol supported by the communication network 7010.

The microcomputer 7610 of the integrated control unit 7600 communicates via at least one of a general-purpose communication I/F 7620, a dedicated communication I/F 7630, a positioning section 7640, a beacon reception section 7650, an in-vehicle device I/F 7660, and an in-vehicle network I/F 7680. The vehicle control system 7000 is controlled according to various programs based on the information obtained. For example, the microcomputer 7610 calculates a control target value for a driving force generating device, a steering mechanism, or a braking device based on acquired information inside and outside the vehicle, and outputs a control command to the drive system control unit 7100. Good too. For example, the microcomputer 7610 realizes ADAS (Advanced Driver Assistance System) functions, including vehicle collision avoidance or impact mitigation, following distance based on vehicle distance, vehicle speed maintenance, vehicle collision warning, vehicle lane departure warning, etc. Coordination control may be performed for the purpose of In addition, the microcomputer 7610 controls the driving force generating device, steering mechanism, braking device, etc. based on the acquired information about the surroundings of the vehicle, so that the microcomputer 7610 can drive the vehicle autonomously without depending on the driver's operation. Cooperative control for the purpose of driving etc. may also be performed.

The microcomputer 7610 acquires information through at least one of a general-purpose communication I/F 7620, a dedicated communication I/F 7630, a positioning section 7640, a beacon reception section 7650, an in-vehicle device I/F 7660, and an in-vehicle network I/F 7680. Based on this, three-dimensional distance information between the vehicle and surrounding objects such as structures and people may be generated, and local map information including surrounding information of the current position of the vehicle may be generated. Furthermore, the microcomputer 7610 may predict dangers such as a vehicle collision, a pedestrian approaching, or entering a closed road, based on the acquired information, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or lighting a warning lamp.

The audio and image output unit 7670 transmits an output signal of at least one of audio and images to an output device that can visually or audibly notify information to the occupants of the vehicle or to the outside of the vehicle. In the example of FIG. 30, an audio speaker 7710, a display section 7720, and an instrument panel 7730 are illustrated as output devices. Display unit 7720 may include, for example, at least one of an on-board display and a head-up display. The display section 7720 may have an AR (Augmented Reality) display function. The output device may be other devices other than these devices, such as headphones, a wearable device such as a glasses-type display worn by the passenger, a projector, or a lamp. When the output device is a display device, the display device displays results obtained from various processes performed by the microcomputer 7610 or information received from other control units in various formats such as text, images, tables, graphs, etc. Show it visually. Further, when the output device is an audio output device, the audio output device converts an audio signal consisting of reproduced audio data or acoustic data into an analog signal and audibly outputs the analog signal.

Note that in the example shown in FIG. 30, at least two control units connected via the communication network 7010 may be integrated as one control unit. Alternatively, each control unit may be composed of a plurality of control units. Furthermore, vehicle control system 7000 may include another control unit not shown. Further, in the above description, some or all of the functions performed by one of the control units may be provided to another control unit. In other words, as long as information is transmitted and received via the communication network 7010, predetermined arithmetic processing may be performed by any one of the control units. Similarly, sensors or devices connected to any control unit may be connected to other control units, and multiple control units may send and receive detection information to and from each other via communication network 7010. .

Note that a computer program for realizing each function of the information processing apparatus according to the present embodiment described using FIG. 2 and the like can be implemented in any control unit or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Furthermore, the above computer program may be distributed, for example, via a network, without using a recording medium.

In the vehicle control system 7000 described above, the information processing device according to the present embodiment described using FIG. 2 and the like can be applied to the integrated control unit 7600 of the application example shown in FIG. 30.

Furthermore, at least some of the components of the information processing device described using FIG. 2 etc. are included in the module for the integrated control unit 7600 shown in FIG. May be realized. Alternatively, the information processing device described using FIG. 2 and the like may be realized by a plurality of control units of vehicle control system 7000 shown in FIG. 30.

Note that the present technology can also adopt the following configuration.
(1)
a surrounding information acquisition unit that obtains first surrounding information and second surrounding information regarding the surrounding environment based on detection results of one or more object detection sensors;
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. an audio information generation unit that generates second audio information using second musical tone data for notifying surrounding information;
An information processing device comprising: a notification control unit that outputs both the first audio information and the second audio information.
(2) The information processing device according to (1),
The one or more object detection sensors include a first object detection sensor and a second object detection sensor,
The peripheral information acquisition unit acquires the first peripheral information based on the detection result of the first object detection sensor, and acquires the second peripheral information based on the detection result of the second object detection sensor. Information processing device to obtain.
(3) The information processing device according to (2),
The first object detection sensor is a first ranging sensor that operates according to a first method,
The second object detection sensor is a second distance measuring sensor that operates according to a second method different from the first method.
(4) The information processing device according to (2) or (3),
The first object detection sensor is a first ranging sensor arranged with a first direction as a detection direction,
The second object detection sensor is a second ranging sensor arranged with a second direction different from the first direction as a detection direction. Information processing device.
(5) The information processing device according to (1),
The one or more object detection sensors are sensors that generate image information,
The peripheral information acquisition unit acquires the first peripheral information based on information on some pixel regions of the image information, and acquires the first peripheral information based on information on other pixel regions of the image information. An information processing device that acquires peripheral information of No. 2.
(6) The information processing device according to (1),
The one or more object detection sensors are sensors that generate image information,
The peripheral information acquisition unit acquires information regarding a first type of object detected based on the image information as the first peripheral information, and acquires information regarding the first type of object detected based on the image information. An information processing device that acquires information regarding a different second type of object as the second peripheral information.
(7) The information processing device according to (2),
The surrounding information acquisition unit generates integrated surrounding information based on the first surrounding information and the second surrounding information. Information processing device.
(8) The information processing device according to (7),
The first ranging sensor operates by an optical laser method,
The second ranging sensor operates using an ultrasonic method,
The surrounding information acquisition unit generates the integrated surrounding information based on the stability of detection by the first ranging sensor and the stability of detection by the second ranging sensor. Information processing device.
(9) The information processing device according to (8),
When the detection stability of the first distance measurement sensor is low and the detection stability of the second distance measurement sensor is high, the peripheral information acquisition unit includes a light-transmitting member or a light-absorbing member in the periphery. An information processing device that generates the integrated peripheral information indicating that a member exists.
(10) The information processing device according to (9),
The peripheral information acquisition unit further includes hardness information acquired as the second peripheral information based on the detection result of the second distance measurement sensor, and the peripheral information acquisition unit is configured to determine whether the light transmitting member or the light absorbing member An information processing device that generates information regarding at least one of a material and a type of an object as the integrated peripheral information.
(11) The information processing device according to any one of (1) to (10),
The audio information generation unit determines whether or not to output the first audio information based on the first peripheral information, and if it is determined that the first audio information is not output, the notification control An information processing device that regulates output of the first audio information by the unit.
(12) The information processing device according to any one of (1) to (11),
The first audio information is first musical tone information constituting a predetermined song,
The second audio information is second musical tone information constituting the predetermined music piece. Information processing device.
(13) The information processing device according to any one of (1) to (12),
The audio information generation unit generates the first audio information by controlling musical tone parameters of the first musical tone data based on the first peripheral information.
(14) The information processing device according to (13),
The musical tone parameter includes at least one of volume, frequency, pitch, speed, BPM, and tempo. Information processing device.
(15) The information processing device according to (13) or (14),
The first peripheral information includes distance information,
The audio information generation unit generates the first audio information by controlling the musical tone parameters based on the distance information.
(16) The information processing device according to (13) or (14),
The audio information generation unit controls the localization of the first audio information based on the detection direction of the first ranging sensor.
(17)
acquiring first peripheral information and second peripheral information regarding the surrounding environment based on detection results of one or more object detection sensors;
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. generating second audio information using second musical tone data for notifying surrounding information;
An information processing method in which a computer system executes the steps of: outputting both the first audio information and the second audio information.
(18)
acquiring first peripheral information and second peripheral information regarding the surrounding environment based on detection results of one or more object detection sensors;
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. generating second audio information using second musical tone data for notifying surrounding information;
A program that causes a computer system to execute the steps of: outputting both the first audio information and the second audio information.
(19)
one or more object detection sensors;
a surrounding information acquisition unit that obtains first surrounding information and second surrounding information regarding the surrounding environment based on detection results of the one or more object detection sensors;
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. an audio information generation unit that generates second audio information using second musical tone data for notifying surrounding information;
An information processing system comprising: a notification control unit that outputs both the first audio information and the second audio information.
(20) The information processing system according to (19), further comprising:
An information processing system, comprising: an audio output section that outputs the first audio information and the second audio information; and an information output section that outputs information to a user.
(21) The information processing device according to any one of (1) to (16),
The one or more object detection sensors are arranged in a device worn by a user or a device held by the user. Information processing apparatus.
(22) The information processing device according to any one of (1) to (16) or (21), further comprising:
An information processing device comprising the one or more object detection sensors.

D... Distance from user to measurement point P...

Measurement point

1, 41, 54, 64, 66... Surrounding information notification system 2... User 3... Ground 5... Object 8... Controller 10... Object detection sensor 17... Surrounding information acquisition unit 18... Notification information generation section 19... Notification control section 25... Laser ranging sensor 26... Ultrasonic ranging sensor 27... Front side ranging sensor 28... Ground side ranging sensor 30... First peripheral information 31... Second peripheral

information Surrounding information

32, 33...Audio signal processing unit 34...Speech synthesis processing unit 35...

Audio output unit

38a, 38b...Pixel area 42...Distance information acquisition unit 43...Situation determination unit 44...Obstacle 48...Ground obstacle 50...Falling Danger point 58...Map information generation unit 60...Obstacle space map 63...Server device 68...Operator 69...Real space dangerous object map 70...Computer

Claims

a surrounding information acquisition unit that obtains first surrounding information and second surrounding information regarding the surrounding environment based on detection results of one or more object detection sensors;
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. an audio information generation unit that generates second audio information using second musical tone data for notifying surrounding information;
An information processing device comprising: a notification control unit that outputs both the first audio information and the second audio information.
The information processing device according to claim 1,
The one or more object detection sensors include a first object detection sensor and a second object detection sensor,
The peripheral information acquisition unit acquires the first peripheral information based on the detection result of the first object detection sensor, and acquires the second peripheral information based on the detection result of the second object detection sensor. Information processing device to obtain.
The information processing device according to claim 2,
The first object detection sensor is a first ranging sensor that operates according to a first method,
The second object detection sensor is a second distance measuring sensor that operates according to a second method different from the first method.
The information processing device according to claim 2,
The first object detection sensor is a first ranging sensor arranged with a first direction as a detection direction,
The second object detection sensor is a second ranging sensor arranged with a second direction different from the first direction as a detection direction. Information processing device.
The information processing device according to claim 1,
The one or more object detection sensors are sensors that generate image information,
The peripheral information acquisition unit acquires the first peripheral information based on information on some pixel regions of the image information, and acquires the first peripheral information based on information on other pixel regions of the image information. An information processing device that acquires peripheral information of No. 2.
The information processing device according to claim 1,
The one or more object detection sensors are sensors that generate image information,
The peripheral information acquisition unit acquires information regarding a first type of object detected based on the image information as the first peripheral information, and acquires information regarding the first type of object detected based on the image information. An information processing device that acquires information regarding a different second type of object as the second peripheral information.
The information processing device according to claim 2,
The surrounding information acquisition unit generates integrated surrounding information based on the first surrounding information and the second surrounding information. Information processing device.
The information processing device according to claim 7,
The first ranging sensor operates by an optical laser method,
The second ranging sensor operates using an ultrasonic method,
The surrounding information acquisition unit generates the integrated surrounding information based on the stability of detection by the first ranging sensor and the stability of detection by the second ranging sensor. Information processing device.
The information processing device according to claim 8,
When the detection stability of the first distance measurement sensor is low and the detection stability of the second distance measurement sensor is high, the peripheral information acquisition unit includes a light-transmitting member or a light-absorbing member in the periphery. An information processing device that generates the integrated peripheral information indicating that a member exists.
The information processing device according to claim 9,
The peripheral information acquisition unit further includes hardness information acquired as the second peripheral information based on the detection result of the second distance measurement sensor, and the peripheral information acquisition unit is configured to determine whether the light transmitting member or the light absorbing member An information processing device that generates information regarding at least one of a material and a type of an object as the integrated peripheral information.
The information processing device according to claim 1,
The audio information generation unit determines whether or not to output the first audio information based on the first peripheral information, and if it is determined that the first audio information is not output, the notification control An information processing device that regulates output of the first audio information by the unit.
The information processing device according to claim 1,
The first audio information is first musical tone information constituting a predetermined song,
The second audio information is second musical tone information constituting the predetermined music piece. Information processing device.
The information processing device according to claim 1,
The audio information generation unit generates the first audio information by controlling musical tone parameters of the first musical tone data based on the first peripheral information.
The information processing device according to claim 13,
The musical tone parameter includes at least one of volume, frequency, pitch, speed, BPM, and tempo. Information processing device.
The information processing device according to claim 13,
The first peripheral information includes distance information,
The audio information generation unit generates the first audio information by controlling the musical tone parameters based on the distance information.
The information processing device according to claim 13,
The audio information generation unit controls the localization of the first audio information based on the detection direction of the first ranging sensor.
acquiring first peripheral information and second peripheral information regarding the surrounding environment based on detection results of one or more object detection sensors;
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. generating second audio information using second musical tone data for notifying surrounding information;
An information processing method in which a computer system executes the steps of: outputting both the first audio information and the second audio information.
acquiring first peripheral information and second peripheral information regarding the surrounding environment based on detection results of one or more object detection sensors;
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. generating second audio information using second musical tone data for notifying surrounding information;
A program that causes a computer system to execute the steps of: outputting both the first audio information and the second audio information.
one or more object detection sensors;
a surrounding information acquisition unit that obtains first surrounding information and second surrounding information regarding the surrounding environment based on detection results of the one or more object detection sensors;
First audio information is generated using first musical tone data for notifying the first peripheral information based on the first peripheral information, and first audio information is generated based on the second peripheral information. an audio information generation unit that generates second audio information using second musical tone data for notifying surrounding information;
An information processing system comprising: a notification control unit that outputs both the first audio information and the second audio information.
The information processing system according to claim 19, further comprising:
An information processing system, comprising: an audio output section that outputs the first audio information and the second audio information; and an information output section that outputs information to a user.