JP6500139B1 - Visual support device - Google Patents

Abstract

[PROBLEMS] To provide support information notification in consideration of the circumstances of individual users. SOLUTION: A visual support device (100) has conditions specific to a user regarding notification of support information for supporting user's visual recognition. Based on the storage unit (130) stored, the imaging unit (110) that acquires image data representing the scene that the user is viewing, the image data acquired by the imaging unit, and the conditions, 3 includes a generation unit (120) that generates support information and a notification unit (140) that notifies the generated support information.

Description

  The present invention relates to an apparatus for assisting vision.

  There is known a visual assistance device that assists a visually impaired person in real time by voice or the like when the presence of an obstacle is detected by recognizing a scene with a camera (Patent Document 1). ).

JP 2000-325389 A

For example, when a plurality of dangerous objects are detected using the visual assistance device as described above, the user may be confused if all of them are notified at once. Further, depending on the situation and circumstances of the user, such as the extent of the user's failure and whether the user is familiar with the current location, notification may not be necessary. If notifications are made one by one in such a case, the user may feel annoying. As described above, in the conventional technique, the notification is uniform, and the circumstances of individual users are not taken into consideration when the support information is notified.
An object of this invention is to notify the support information based on the situation of each user.

In one aspect, the present invention relates to a storage unit that stores a condition specific to a user regarding notification of support information for supporting the user's visual recognition, and image data representing a scene that the user is viewing. An imaging unit to be acquired, a generation unit that generates support information based on the image data acquired by the imaging unit and the condition, a notification unit that notifies the user of the generated support information , Means for acquiring reaction information representing a reaction of the user made in response to the notification of support information, and the generation unit further includes a visual support device that generates the support information based on the acquired reaction information. provide.

  According to the present invention, support information suitable for each user's situation is notified.

1 is an external view of a visual assistance device 100. FIG. The figure which shows the hardware constitutions of the vision assistance apparatus 100. FIG. The functional block diagram of the vision assistance apparatus 100. FIG. The figure which shows the example of the information memorize | stored in condition table T1. The figure which shows the example of the information memorize | stored in the correction table T2. The figure which shows an example of 100 operation | movement. The figure which shows the example of the notification produced | generated based on an image. The figure which shows the example of the notification produced | generated based on an image. The functional block diagram which concerns on 2nd Example. The figure which shows the example of the information memorize | stored in shared information table T4. The figure which shows the operation example which concerns on 2nd Example.

<Example 1>
FIG. 1 is an example of the appearance of the visual assistance device 100.
The visual assistance device 100 is worn by the user like normal glasses, and includes a frame 20, a projection device 21, a microphone 24, a speaker 25, a camera 22, a projection surface 23, and a switch 26. including. Here, the shape and function of the frame 20 and the projection surface 23 are arbitrary, and may have the function of normal glasses. For example, the projection surface 23 may or may not function as a normal eyeglass lens. The projection device 21, the camera 22, the projection surface 23, and the speaker 25 are operated by electric power supplied from a power source (not shown) built in the frame 20. The switch 26 is operated by the user. In addition to the control unit, the frame 20 incorporates a unit (not shown) for realizing various functions described below. Note that the vision support apparatus 100 may not be provided with a power source, and may be supplied with power from a battery unit that is provided separately and carried by the user.

FIG. 2A is a diagram illustrating an example of a hardware configuration of the vision support apparatus 100.
The visual assistance device 100 includes a processor 201, a memory 202, a camera 22, a microphone 204, an operation switch 205, a sensor 206, a timing device 207, a projection device 208, a speaker 25, and a vibration motor 210. The processor 201 is one or more general-purpose or dedicated processors, and includes, for example, a CPU, a sound processor, and an image processor. The memory 202 is a storage device such as a semiconductor memory. The camera 22 is an image pickup mechanism including an optical element such as a lens and a light receiving element, a photographing control circuit such as a focus and exposure. A plurality of cameras 22 may be provided. For example, the visual assistance device 100 includes normal glasses and a belt unit for winding and fixing the glasses around the head, and in addition to a camera for imaging the front as shown in FIG. 1, a camera for imaging the rear At a position corresponding to the back of the head on the belt. In this way, even when silent electric vehicles or bicycles approach from behind or from the side by acquiring a scene in a wider range (ideally 360 degrees around) than the normal field of view. , Can notify the user. The microphone 204 is a sound collection mechanism including a capacitor microphone and a signal processing circuit. The operation switch 205 is a switch operated by the user.

  The sensor 206 is one or more sensors for detecting the state of the visual assistance device 100 and the environment around the visual assistance device 100, such as an acceleration sensor, an orientation sensor, and an illuminance sensor. The time measuring device 207 is a time measuring means such as a timer. The projection device 208 includes a semiconductor laser, a drive circuit, and the like, and is a mechanism that projects characters and images onto the projection surface 23. The speaker 25 is a speaker and notifies the user of information by voice. The vibration motor 210 is a vibrator such as a small motor, and functions as support information notification means. The communication unit 211 is a wireless communication module such as WiFi, and transmits and receives data to and from an external device. The position acquisition unit 212 is a mechanism for measuring the position of the visual assistance device 100 such as a GPS reception module.

FIG. 2B is a functional block diagram of the visual assistance device 100.
The visual assistance device 100 includes a notification unit 140, a reaction acquisition unit 150, a voice input unit 160, a generation unit 120, a mode switching unit 170, a storage unit 130, a generation unit 120, and an imaging unit 110. .

  The imaging unit 110 is realized by the camera 22 and intermittently acquires image data representing a scene viewed by the user at a predetermined frame rate. Strictly speaking, the acquired image is not the same as the world that the user is viewing, depending on the camera characteristics, the human eye mechanism, and the visual ability of each user. It can be said that it reflects the scene that humans who have

The mode switching unit 170 is realized by the operation switch 205 and accepts switching of conditions related to support information. For example, a plurality of operation modes are registered in advance in the generation unit 120, and the user uses the mode switching unit 170 to specify or change the operation mode. The operation mode designates a mode of control related to notification of support information, and is defined by, for example, the frequency of notification of support information and the amount of notification content. Therefore, the control content regarding the support information generated by the generation unit 120 differs depending on the operation mode. For example, it is conceivable that three operation modes of “high”, “medium”, and “small” are set in advance and the user designates one mode to be applied. Alternatively, the “indoor mode” in which the notification control parameter is optimized when the user is indoors, and the “night mode” in which the notification control parameter is optimized when the user moves at night, the current position and time It may be possible to switch the operation mode according to the situation where other users are placed. The condition can be switched at any time during use.

  The voice input unit 160 is realized by the microphone 204 and acquires a user's voice and surrounding sounds.

  The reaction acquisition unit 150 is realized by any one or more of the microphone 204, the operation switch 205, and the sensor 206, and acquires reaction information indicating a user's reaction to the notification of support information. In addition, the reaction acquisition unit 150 acquires the current user operation state. The motion state includes, for example, information on whether or not the user is performing a motion such as walking or the posture (sitting, lying down, etc.).

  The position specifying unit 190 is realized by the position acquisition unit 212, and specifies the position of the visual assistance device 100. Note that map information and facility information are stored in the storage unit 130, and the position specifying unit 190 is based on the current position in a situation where the user is currently located (whether the user is outdoors or indoors, outdoor In the case of, it may be specified whether the user is in a registered facility or on a road, and if indoors, the user is in a registered facility such as an office or home.

  The timer unit 180 is realized by the timer device 207 and acquires the current time.

  Based on the information acquired from the timing unit 180 and the position specifying unit 190, the generation unit 120 calculates the stay time in a predetermined area (for example, in a pre-registered facility) when an image is acquired by the imaging unit 110. It may be calculated. The generation unit 120 may further determine the user's action based on the information acquired from the reaction acquisition unit 150. The user's action is, for example, information on whether lying down, resting, walking, or exercising. Thus, information such as position, place, and behavior is accumulated in the user specific information T3.

The notification unit 140 includes an audio output unit 141, a display unit 142, and a vibration generation unit 143, and notifies the user of support information generated by the generation unit 120 under the control of the generation unit 120.
The voice output unit 141 is realized by the speaker 25 and notifies the support information by voice. Specifically, support information is generated as text data, and this text data is read out by speech synthesis processing.
The display unit 142 is realized by the projection device 208 and notifies the support information by a video. The vibration generating unit 143 is realized by the vibration motor 210 and notifies support information by vibration.
Note that one or two of the audio output unit 141, the display unit 142, and the vibration generation unit 143 may be omitted. For example, if a blind user is assumed, the function of the display unit 142 may be omitted.

The storage unit 130 is realized by the memory 202, stores a program for controlling the generation unit 120 such as an OS, and stores conditions specific to the user regarding notification of support information for supporting the user's visual recognition. The Specifically, a condition table T1, a correction table T2, and user-specific information T3 are stored, and information corresponding to each user is stored therein.

  Information unique to each user is stored in the user-specific information T3. The user-specific information includes, for example, information (such as a face image or facial feature amount) for specifying a person (eg, a relative or friend) having a predetermined relationship with the user. Further, information related to the user's own state such as information related to the user's physical ability may be stored. Specifically, it is a user's physical condition and physical ability (sight, myopia, hyperopia, presbyopia, content / degree of disability, hearing, taste, touch, smell, disability, motor ability, etc.). Further, the user specific information T3 may include a user's behavior history (walking route, staying place, staying time, etc.). The user's action history may include information indicating the reaction of the user who has received the notification.

  FIG. 3 shows a specific example of information stored in the condition table T1. In this example, the content of notification control of support information is described in association with the notification condition. Notification conditions are roughly classified into detection contents and status. The detected content is the type of an object (artificial object such as a car, a building, a sign, or a natural object such as a person or a tree) extracted as a result of image analysis. It may also indicate an event other than the presence of an object, such as the fact that there is a puddle or the fact that a tree branch is falling down in front of you. Hereinafter, for convenience of explanation, an object to be extracted from an image and to be notified to a user is referred to as “object” regardless of whether it is a tangible object or an objectless object. The details in the figure indicate detailed information of the object. For example, it is information such as an aspect (time change) and an attribute of the object. The situation includes information indicating the situation when the image is acquired, such as the time when the image was acquired, the location, and the brightness of the shooting location.

  The elements constituting the notification control contents include necessity, priority, timing, and notification method in the example of FIG. Necessity defines whether or not to notify each detected object. The priority defines the priority at the time of notification. Generally, a high priority is set for an object that is highly likely to be life threatening. Timing defines the timing of execution and / or termination of notification. The notification method defines which function of the notification unit 140 is used.

In the example of the figure, for example, when a vehicle is detected, the vehicle is running but not facing the user, and when the detection occurs outdoors, other situations Notification is required regardless of (time zone or ambient brightness), and the priority classification of notification is “2” (second highest), and the target of the target is determined by means of voice, text, and vibration. The control content is defined such that notifications continue to be issued while the vehicle is detected.
When a plurality of objects requiring notification are detected, an object with a higher priority is notified with higher priority. “Preferential notification” means, for example, that the notification is given earlier in time, the notification is given at the same time, but it is displayed in a mode that draws more attention than other objects (such as the character size or volume is high). is there.

  One or more items of the notification control content are generated based on the content of the user specific information T3, or can be rewritten by the user. As a result, the contents of notification control reflect user preferences, user circumstances, and the like. Moreover, after the content of the user specific information T3 is updated or the user's reaction to the notification is obtained, the generation unit 120 may automatically rewrite the content as needed or periodically. In this case, the content changes according to the period during which the user uses the visual assistance device 100.

  The storage unit 130 may store information other than user-specific information. For example, typical obstacle types (vehicles, traffic signs, steps, etc.) and their danger levels (importance of notification) are registered in the storage unit 130.

  FIG. 4 shows an example of information stored in the correction table T2. In the correction table T2, the user's reaction (reaction) and the correction content of the priority are stored in association with each other. The generation unit 120 refers to the correction table T2 and rewrites the condition table T1 when detecting a predetermined reaction. For example, in the example shown in the figure, when a response indicating that the notification is turned off by operating a switch or the like is stored as a response to the notification of the notified support information, the priority of the support information is lowered by one step. . The reason for changing the priority in this way is that it can be assumed that the user's reaction reflects the necessity of the notification and the user's judgment regarding the importance. Even when the user changes the frequency of notification in response to notification of certain support information, the priority of the support information is corrected.

  When the reaction acquisition unit 150 detects a user's reaction (stops, changes the direction of the face, etc.), it is presumed that the user is responding to the object of the support information and that the user needs to be notified. Therefore, the priority is corrected so as to increase.

  In this way, by updating the contents of the condition table T1 according to the user's response to the support information (including a response of no response), notification control when the same or similar situation is encountered after the next time The possibility that the contents are more appropriate according to the user's intention is increased.

The generation unit 120 repeatedly performs image analysis at predetermined intervals on the image data obtained by the processor 201 and acquired by the imaging unit 110, and whether or not a preset target object or target object exists. Determine whether. Then, the generation unit 120 generates support information based on the image data acquired by the imaging unit 110 and the conditions unique to the user stored in the storage unit 130.
Specifically, when a plurality of one or more specific objects or objects are extracted as a result of the image analysis, the generation unit 120 refers to the user specific information T3 and needs to notify each object or object. Determine no or priority. Then, one or more objects are selected from the one or more objects or objects, and support information related to the selected one or more objects is generated.

  Here, the objects to be extracted are typically objects such as objects such as vehicles, pedestrians, utility poles, streets, curbs, gutters, road surface irregularities, and other surrounding topography. It is. In addition, a target that contributes to the safe movement of the user (when walking or driving a vehicle) such as a traffic sign, a mark drawn on a road, a pedestrian crossing, a railroad crossing, and a floor plan of an outdoor room may be included.

  The generation unit 120 may generate support information by further using the reaction information acquired by the reaction acquisition unit 150. In addition, the generation unit 120 may generate support information when an obstacle with a risk level equal to or higher than a predetermined value is extracted from the image data. Further, when a person is detected in the image data, the generation unit 120 may determine support information based on the person information.

  The condition may be defined using information related to the position of the user. In addition, the condition may be defined using information related to stay time. The condition may be defined using information related to the user's behavior.

  Further, when the operation mode is designated, the support information is generated in consideration of the conditions specified by the operation mode. For example, if “notification frequency“ low ”” is designated as the operation mode and this operation mode prescribes a rule of “not notify for an object with a priority of 2 or less”, the priority “1”. Only objects for which is set are subject to notification.

As shown in FIG. 3, the control contents need not be prepared in a table format in which the control contents corresponding to the conditions are defined. For example, the control algorithm may be determined using a machine learning technique.
Specifically, the situation at the time of imaging (in other words, the situation encountered by the user) that can be represented by the extracted object, the image feature amount, etc., and the user's reaction obtained for the executed notification are shown. , By inputting it into the learning device as training data, a classifier that classifies the objects that need to be notified and the objects that do not need to be notified, or a regression model to determine the importance or necessity of the notification (learned Model).

Here, the user's reaction is whether or not he / she has turned around for the notification supplied from the reaction acquisition unit 150, whether or not he / she has lost the balance of his / her body, such as whispering immediately after the notification. The validity of the notification, such as whether the user has stopped or changed the course of walking, whether the person has spoken or raised a hand after notifying the presence of a person, and the physical response of the user (including the case of no response) This is information that can be used to estimate the degree of influence on the user's behavior.
In addition to or in addition to the user's reaction, information (for example, a walking route or an activity time zone) indicating user behavior included in the user information is accumulated, and the accumulated user information is stored in the learning device. You may enter.

  As a machine learning method for generating a learning model, for example, Deep Neural Network (DNN, deep learning) can be used. Specifically, CNN (Conversion Neural Network), RNN (Recurrent Natural Network), etc. are applicable.

The learning timing may be executed before using the visual support device 100, or whenever the user's reaction to the situation / object or notification that the user has encountered after use is obtained. May be.
In the former case, for example, image data simulating the field of view captured by the user is prepared, and the user is asked questions regarding the conditions regarding notification such as necessity of notification, priority of notification, timing, etc. And learning based on answers obtained from the user. After the learned model is generated in this way, the visual assistance device 100 is used by the user.
In the latter case, an unlearned algorithm is used at the start of use of the visual support device 100, but learning progresses as information about user responses and behavior patterns is accumulated.

  In short, the visual support device 100 is acquired with a learning device that learns conditions regarding notification of support information based on a plurality of acquired image data and a plurality of reaction information obtained corresponding to each image data. It is only necessary to have a function of generating support information corresponding to the image data using the learning device.

  In addition, the hardware configuration illustrated in FIG. 2A or a part of the functional configuration illustrated in FIG. 2B may be realized by an apparatus provided separately from the vision support apparatus 100. For example, the notification function of the portable information processing terminal such as a smartphone carried by the user may be used without providing the notification unit in the visual support device 100. Specifically, wireless communication is performed between the visual assistance device 100 and the portable information processing terminal, and a notification execution request and notification content are transmitted from the visual assistance device 100 to the portable information processing terminal.

Further, other functions may be added to the hardware configuration or functional configuration shown in FIGS. 2A and 2B. For example, a function for guiding the movement to the visual assistance device 100 is provided. Specifically, the map information is stored in the storage unit 130, the information on the current position and the orientation of the user acquired by the position acquisition unit 212 and the sensor 206, respectively, and the voice input unit 160 acquired. Based on the information on the position and facility name of the movement target and the map information, information that supports the movement of the user such as the route and direction the user should take and the distance to the destination may be generated and notified to the user. .
As another example, a function of authenticating a user may be provided. Specifically, the visual support device 100 is provided with a means for acquiring user-specific biometric information such as a voiceprint or a fingerprint, and determines whether or not the information matches with previously registered information. The timing for executing the authentication is, for example, when the visual assistance device 100 is activated.

FIG. 5 shows an example of the operation of the visual assistance device 100. The visual assistance device 100 performs acquisition of the current time and location, sensing of the surrounding situation and the user's state, reception of designation of the operation mode, and the like, and determines information defined as notification conditions (S500). ).
When the captured images are accumulated for a predetermined frame, the generation unit 120 analyzes the accumulated images (S510).
As a result of the analysis, the generation unit 120 determines whether or not notification is necessary when an object is detected (S521). If the notification is necessary (S521; YES), the generation unit 120 generates the notification content (S522). Here, when there are a plurality of notification targets, the order and mode of each notification are further determined. The notification unit 140 executes notification according to the timing information (S530). If there is no timing information, a notification is immediately executed.
When the generation unit 120 obtains a reaction from the user (S540: YES), the generation unit 120 writes information indicating the reaction in the storage unit 130, and updates the notification condition based on the reaction information as necessary (S550). ). If notification is unnecessary (S521; NO) or no reaction is acquired (S540; NO), the process returns to S500.

FIG. 6 shows a specific example of the notification generated based on the acquired image.
Assume that an image indicated by D1 is captured. As a result of analyzing this image, the presence of the object OB1 that is a car, the object OB2 that is a tree, the object OB3 as a step that is a kind of obstacle, and the object OB4 as a signboard that is a kind of obstacle is detected. To do. In addition, it is assumed that the positional relationship between the objects and the positional relationship with the user are specified, and that the moving direction of the object OB1 is detected.
Suppose now that user A and user B, who are both users of the visual assistance device 100, encounter the situation shown in this image outdoors. In this example, the user A is first notified of a message that calls attention to the vehicle coming as support information corresponding to the object OB1, and subsequently, as support information corresponding to the object OB3, A message urging attention to the front signboard is notified, and then a message urging attention to the step is notified.

  Such notification control is performed because, for example, information indicating that the user A has considerably poor eyesight is written in the user specific information T3, and based on this information, a high notification is given to the moving object in the condition table T1. This is a case where priority is given and the control content is that a stationary obstacle has a low priority but notification is required. In this example, the reason why the notification is not executed even though it is determined that the object OB4 is required to be notified is that, for example, the upper limit of the number that can be notified at one time is set to 3 so that the user is not confused. This is a case where the priority is determined to be lower than the other three objects because the user is far from the position of the tree at present.

On the other hand, only the support information corresponding to the object OB3 is notified to the user B. In such a situation, for example, information indicating that the user B has no problem with visual acuity but has a problem with walking ability is stored in the user-specific information T3. As a result, only the object OB3 related to walking is notified. On the other hand, it can be considered that the object OB1, the object OB4, and the object OB2 may occur when the generation unit 120 determines that the notification is unnecessary.

  FIG. 7 shows another example of the generated notification. D2 is an example of an image acquired when the user is captured in an indoor corridor such as an office. Now, it is assumed that user C and user D, who are both users of the visual assistance device 100, have encountered this situation. The generation unit 120 detects the object OB21 and the object OB22 as the objects. In addition, by the face authentication process, it is determined that the object OB21 is a person X registered as a friend of C, and the object OB22 is a person who is neither a friend of the user C nor a friend of the user D.

  The user C is informed of the appearance of an acquaintance person X. Thereby, the user C can respond to X with greetings. In such a situation, for example, the user C has a complete understanding of the current location and does not want the guidance information. In accordance with this situation, the support information related to the location is not notified or is notified. Appears when the priority is set to be low.

  On the other hand, the user D is first explained about the situation that he / she is in the hallway, and then alerted that the person is heading from the front. Such a situation occurs, for example, when the current position is the highest priority indoors for the user D, and the setting according to the request of the user D is made in the storage unit 130.

  Alternatively, a setting for detecting only a person from the imaging result is registered in the storage unit 130, and the user wants to search for a waiting person (registered in advance in the storage unit 130) from a large number of people at a meeting place or the like. In addition, a method of using the visual assistance device 100 such as operating the mode switching unit 170 to validate this setting is also conceivable.

  As described above, even if the same situation is encountered, the support information selection method varies depending on the user, and as a result, the support information provided may differ depending on the user.

According to the present embodiment, in consideration of user-specific information such as user circumstances, convenience, needs, preferences, current situation, at least one of notification necessity, contents, timing, and order is It is determined. As a result, useless notification is performed for the user, and as a result, the situation where the user feels bothersome is expected to be reduced. On the other hand, too much pursuit of annoyance is less likely to cause a situation where a necessary notification is not performed.
Moreover, since the notification control content can be changed based on the user's reaction to the notification and / or the user's action history, the support provided to the user is optimized as the visual assistance device 100 is used.

The above embodiment is mainly intended to supplement the visual ability of a user who has reduced the visual ability originally possessed by a normal person, and to support daily life. You may apply to the scene where the display of ability is required.
For example, when an operation mode for recognizing a distant object is provided, and a scene that requires recognition of a distant object is encountered, the user manually switches to that mode, or the object is not detected for perception by image analysis. As a result of recognizing that it does not exist, a method of using the visual assistance device 100 is conceivable in which the generation unit 120 automatically switches to notify an object located far away as assistance information. Alternatively, a “navigation mode” in which the destination is specified in advance and a “free mode” in which the user walks freely without a specific destination point are provided, and the contents and timing of the support information in both modes, The frequency or the like may be varied. In the “navigation mode”, in addition to providing the support information described above, the route to the destination is guided by voice based on the map information and the current position, while in the “free mode”, the above-described support is provided. In addition to providing information, information such as nearby restaurant information and park presence information, which is not directly related to walking, can be set so that information according to the user's hobbies and preferences is notified. For example, information about restaurants and other commercial facilities and public facilities is notified when the user approaches within 200 meters.
In addition, the visual assistance device 100 may be used to provide visual assistance information corresponding to an activity scene, such as when performing special work or exercise.
That is, the present invention can be used for all humans other than so-called healthy persons. In other words, the “support” in the present invention is not limited to the meaning of compensating for the reduced or lacking visual ability.

  In addition to the glasses type, the vision support device 100 may be an earphone type worn on the ear, a broach type hanging from the neck, a type worn on the waist, and the like. That is, there is no need to acquire an image that accurately reflects the scene that a so-called healthy person would see. For example, in the broach type, an image of a viewpoint viewed from the neck or chest is acquired, but there is no particular problem for the purpose of detecting an object. Further, the visual assistance device 100 may intentionally acquire a video from a different viewpoint from the normal viewpoint. For example, the visual support device 100 is designed in the shape of a portable information terminal such as a smartphone, and an instrument (a so-called self-shooting stick) for pointing the visual support device 100 at a position away from the hand is attached. Alternatively, the visual assistance device 100 may be attached to a flying object (such as a drone) and moved in the air to shoot. By doing so, it may be easy to detect and determine a person or an obstacle.

  As a notification method, voice notification and image display on the projection surface may be combined. For example, when an obstacle is detected, a warning sound is emitted and the content of the warning is displayed on the projection plane.

The above embodiment is merely an example of a method for identifying a situation from a photographed image and a recognition target. For example, the generation unit 120 may recognize only that the object is an obstruction without specifying the type of object or other details.
Further, it is not necessary to set the entire range shown in the acquired image as the object extraction range. For example, an object is extracted from a predetermined area (for example, the center of the screen (the center of the field of view), the lower half of the screen (the field of view around the ground), or the area corresponding to the predetermined field of view)) of the image. It is conceivable to set the range. Specifically, the position where the object is detected in the visual field may be added to the notification condition of the condition table T1.

By narrowing down the object extraction range in this way, the processing speed of image processing is improved, and as a result, the real-time property of notification is improved.
In addition, the user may be allowed to select an extraction range of the object. For example, it may be possible to define different object extraction ranges depending on the operation mode. For example, if you are in a city with many people, you may get confused if you notify the information about all detected people, so if you go out in the city, extract the area near the center of the screen (the front of the user) Switch to range and set operation mode. By narrowing down the information in this way, the possibility of an accident of a frontal collision with a person coming from at least the front is reduced, and the possibility of being confused by a large number of notifications is reduced.

  As a method of acquiring information indicating a scene, an image reflecting the scene is acquired based on data detected or imaged using sound waves, millimeter waves, or other electromagnetic waves instead of or in addition to the visible light image described above. May be. For example, the visual support device 100 is provided with an infrared camera or millimeter wave radar, and at night, the visible light camera is turned off and the infrared camera or millimeter wave radar is activated, and data measured by the infrared camera or millimeter wave radar is obtained. Make an image.

<Example 2>
Information related to the scene that the user of the visual assistance device 100 actually encountered at a certain position, which is information that assists other users with visual assistance at that position (for example, notification facts that have been notified to each user and its information The notification content itself may be shared by a plurality of users. Specifically, as shown in FIG. 8, a plurality of visual assistance devices 100 (100-1, 100-2, 100-3) and a server 300 are connected via 900.

  The server 300 is a general-purpose server and includes a storage unit 310, a communication unit 320, and a control unit 330. The storage unit 310 is a storage device such as a hard disk, and stores shared information collected from the vision support device 100, which will be described later, in addition to the control program of the server 300. The communication unit 320 is a network interface for exchanging information with each vision support apparatus 100 via the Internet 900. The control unit 330 is a processor and controls the server 300.

  The generation unit 120 supplies the generated support information to the notification unit 140 and determines whether the support information should be shared with other users. For the support information determined to be shared, the generation unit 120 generates shared information based on the generated support information.

  The shared information is not generated based on the support information, but may be determined by the user and input to the visual support device 100. Specifically, recording is started by the voice input unit 160 when notification is given or when a predetermined operation is performed by the user, and the user's utterance made within a predetermined period is acquired. The generation unit 120 performs processing such as converting the acquired speech data into text data, and transmits the text data to the server 300. The generation unit 120 sends a voice message such as “Do you want to post the current information?” Via the notification unit 140 when the notification is made or when it is detected that the user has performed a predetermined action or utterance. Notification may be performed to prompt the user to post information or ask for confirmation.

In addition, regarding the posting timing of shared information, regardless of whether or not support information notification is received and the position and timing at which the notification is received, information that is considered to be shared by the user is accepted at any timing or location. May be. In this case, the user inputs position information to be associated with the shared information. Here, when it is desired to input shared information related to a position different from the current position of the visual assistance device 100, the user executes an application for displaying or searching for map information in the visual assistance device 100, for example, and sets the position on the map. By designating, the coordinates corresponding to the designated position are acquired from the application.
The generated shared information is transmitted to the server 300 using the function of the communication unit 211 together with the current position specified by the position specifying unit 190.

  FIG. 9 is an example of the shared information table T4 registered by each vision support device 100 and stored in the storage unit 310. The shared information table T4 stores shared information in association with the position information. The position information may be described using a coordinate format such as latitude and longitude, an address format determined by the government, and building landmark information. In short, the position information may be in a format that allows the visual support device 100 to determine whether or not the position of the terminal itself is a position corresponding to the position stored in the position information.

The shared information describes the presence of an obstacle, the degree of danger of the obstacle (may be subjective information of the registered user), and other detailed information, and the coping action for the obstacle. It is preferable that the information is more detailed than information that can be detected by the visual assistance device 100 using the condition table T1 or the like. For example, even if it can be detected that there is a step by image analysis by the imaging unit 110, how dangerous the step is, or how to act (be careful about how to detour) It is often difficult to determine whether it should be overcome), but more detailed support is possible by notifying information based on the actual experience of others.

The visual assistance device 100 makes an inquiry to the server 300 using the current position of the visual assistance device 100 as a key periodically or when the change in the position of the visual assistance device 100 satisfies a predetermined criterion, and shared information corresponding to the current position is obtained. It is confirmed whether it is stored, and if such shared information is stored, it is obtained from the server 300. The visual assistance device 100 notifies the user of information based on the acquired shared information by at least one of voice, text, and image. Here, when the support information includes information on the degree of danger, the display mode may be varied depending on the degree of danger.
Note that the notification timing of the shared information to the user may be performed separately from the above-described support information, or may be notified together with the support information at all times. When notification of shared information is based on notification of support information, the content, timing, and mode of notification may be determined according to the support information and the shared information. For example, when there is shared information related to the support information at that position, the priority of the support information may be raised and notified in a manner that stands out from the other support information to be notified. Also good.

Here, the information to be shared may be all the notifications that have been executed or a part of them. The determination as to whether or not to share may be performed by each visual assistance device 100 or may be performed by the server 300. When the visual support device 100 determines whether or not to share, for example, only information whose notification priority is equal to or higher than a predetermined value is set as a sharing target. Alternatively, the user who is notified of the information may determine whether to share the information.
When determining whether or not to be shared by the server 300, from the viewpoint of ensuring reliability, for example, when there is registration from a predetermined number or more of visual support devices 100 for the same or related information, the shared information table Register at T4. In addition, when a plurality of pieces of shared information associated with the same or substantially the same position are acquired, the server 300 edits the acquired shared information or uses the acquired timing to perform information overwriting or other processing editing processing. May be. Further, the content notified to the user may be changed based on the content of the processing edit, the number of shared information received corresponding to a certain position, timing, frequency, and the like. As described above, the content of the shared information notified to the user can dynamically change based on the posted information and the attribute of the poster.
In addition, the information to be shared does not necessarily have to be related to safety or danger when the user moves, and may include subjective information such as the user's impression and impression of the scene. In short, it may be scene information regarding the scene at a certain position. In other words, the shared information is active footprint information posted by other users.

  In short, each visual assistance device 100 encounters a user of the visual assistance device at a certain position, and transmits information for assisting the visual recognition of other users at the position to the server 300, and the visual information from the server 300. The server 300 includes a receiving unit that receives the information corresponding to the position of the support device, and the server 300 encounters a user of the first visual support device from a first visual support device at another position, and receives another information at the position. If it has a receiving unit that collects information for assisting the user's visual recognition and a transmission unit that transmits the information corresponding to the position of the second visual assistance device to the second visual assistance device Good.

  FIG. 10 shows an operation example according to the present embodiment. In this example, the visual support device 100-1 executes registration of shared information upon receiving the support information notification, and the visual support device 100-2 is independent from the support information notification. Execute sharing information notification.

When the notification is executed by the notification unit 140 (S530), the generation unit 120 determines whether information related to the notified information should be registered as shared information (S601). If it is determined that it should not be registered, no special processing is performed. When it is determined that registration is to be performed, the notified content and position information are set as a set, and a registration request is transmitted to the server 300 (S602).
When receiving the registration request (S603), the server 300 adds a record to the shared information table T4. When it is determined that information equivalent to the shared information related to the registration request has already been registered, only information that the information has been newly posted may be stored.

  The visual assistance device 100-2 periodically notifies the server 300 of the position information of the terminal itself, and inquires whether there is shared information corresponding to the current position (S604). Corresponding to the current position is, for example, a case where the position information associated with the shared information indicates a place within a predetermined distance from the current position of the visual assistance device 100-2. If there is shared information that has been inquired, the server 300 transmits the shared information to the vision support apparatus 100-2 (S605). The visual assistance device 100-2 that has received the shared information determines whether the visual assistance device 100-2 is located at the shared information. If the shared information is located, the visual assistance device 100-2 receives the information based on the received shared information. It produces | generates and notifies to the user of the visual assistance apparatus 100-2 (S606).

  The information based on the shared information may be the shared information itself, or if the support information has been notified (or is about to be notified), a new information generated based on the support information and the shared information Information. Alternatively, information may be generated based on user attributes and shared information stored in the storage unit 130. For example, when the shared information includes information that there is a step and the risk level for a user with limited mobility is “large”, the user attribute of the visual assistance device 100-2 includes If the information indicating that it is inconvenient is included, the priority of notification, the degree of warning, etc. are determined so that it is transmitted as a dangerous obstacle.

  In the navigation mode described above, when shared information exists, the shared information may be included in the guidance information. For example, when the user sets the navigation mode and the user inputs a destination voice before departure, the visual assistance device 100 is the information about the entire route to the destination and is on the route. Alternatively, shared information associated with a position close to the route is acquired from the server 300 and presented to the user by voice. For example, “Leave the house, go straight about 100 meters to the left, turn left at the traffic light, and go up the station stairs after 500 meters. This stairs has a footprint (shared information). The content is "Because the edges are slippery". As described above, by using the shared information to inflate the image about the route to the destination and the scene, it is possible to give a sense of security to visually impaired users.

  In short, in the information processing apparatus according to the present invention, conditions specific to the user regarding the notification of the support information for supporting the user's visual recognition are stored, and image data representing the scene being visually recognized by the user is stored. The step of acquiring, the step of generating support information based on the acquired image data and the condition, and the step of notifying the generated support information may be executed. A program for executing the above steps may be stored in advance in the information processing apparatus, or may be downloaded via a network.

100, 100-1, 100-2, 100-3 ... Vision support device, 20 ... Frame, 21 ... Projection device, 22 ... Camera, 23 ... Projection plane, 24 ... Microphone, 25 ... speaker, 26 ... switch, 140 ... notification unit, 150 ... reaction acquisition unit, 160 ... voice input unit, 120 ... generation unit, 170 ... mode switching , 130 ... storage unit, 110 ... imaging unit, 180 ... timing unit, 190 ... position specifying unit, 201 ... processor, 202 ... memory, 204 ... microphone, 205 ... Operation switch, 206 ... Sensor, 207 ... Time measuring device, 208 ... Projection device, 210 ...
Vibration motor, 211 ... communication unit, 212 ... position acquisition unit, 141 ... sound output unit, 142 ... display unit, 143 ... vibration generation unit, 300 ... server, 310 ... Storage unit 320 ... Communication unit 330 ... Control unit T1 ... Condition table T2 ... Correction table T3 ... User-specific information OB ... Object, T4 -Shared information table

Claims (13)

A storage unit for storing conditions unique to the user regarding notification of support information for supporting the user's visual recognition;
An imaging unit that acquires image data representing a scene that the user is viewing;
A generating unit that generates support information based on the image data acquired by the imaging unit and the condition;
A notification unit for notifying the user of the generated support information ;
Means for obtaining reaction information representing a reaction of the user made to the notification of the support information;
With
The generation unit is a visual support device that generates the support information based on the acquired reaction information . A storage unit for storing conditions unique to the user regarding notification of support information for supporting the user's visual recognition;
An imaging unit that acquires image data representing a scene that the user is viewing;
A generating unit that generates support information based on the image data acquired by the imaging unit and the condition;
A notification unit for notifying the user of the generated support information ;
Means to get the current time
With
The condition is defined using information about the timing of notification,
Visual support device. A storage unit for storing conditions unique to the user regarding notification of support information for supporting the user's visual recognition;
An imaging unit that acquires image data representing a scene that the user is viewing;
A generating unit that generates support information based on the image data acquired by the imaging unit and the condition;
A notification unit for notifying the user of the generated support information ;
Means for acquiring the position of the own device;
With
The condition is defined using information about the user's location,
Visual support device. A storage unit for storing conditions unique to the user regarding notification of support information for supporting the user's visual recognition;
An imaging unit that acquires image data representing a scene that the user is viewing;
A generating unit that generates support information based on the image data acquired by the imaging unit and the condition;
A notification unit for notifying the user of the generated support information ;
Means for obtaining a stay time of the user;
With
The condition is defined using information on stay time,
Visual support device. A storage unit for storing conditions unique to the user regarding notification of support information for supporting the user's visual recognition;
An imaging unit that acquires image data representing a scene that the user is viewing;
A generating unit that generates support information based on the image data acquired by the imaging unit and the condition;
A notification unit for notifying the user of the generated support information ;
Means for identifying the user's behavior;
With
The condition is defined using information about the user's behavior,
Visual support device. A storage unit for storing conditions unique to the user regarding notification of support information for supporting the user's visual recognition;
An imaging unit that acquires image data representing a scene that the user is viewing;
A generating unit that generates support information based on the image data acquired by the imaging unit and the condition;
A notification unit for notifying the user of the generated support information;
With
The condition is defined using information on the physical ability of the user.
Visual support device. A storage unit for storing conditions unique to the user regarding notification of support information for supporting the user's visual recognition;
An imaging unit that acquires image data representing a scene that the user is viewing;
A generating unit that generates support information based on the image data acquired by the imaging unit and the condition;
A notification unit for notifying the user of the generated support information;
A switching unit for accepting switching of the condition by the user;
A visual support device comprising: The generating unit selects one or more objects based on the condition from a plurality of objects detected from the image data, and generates information on the selected one or more objects.
The visual assistance device according to claim 1. The generation unit generates support information representing an object of a type corresponding to the condition among objects included in the image data acquired by the imaging unit.
The visual assistance device according to claim 8 . Person information about a person is registered in the storage unit,
The notification unit determines the support information based on the person information when a person is detected in the image data;
Vision assistance device according to any one of claims 1 to 9. A transmission unit that transmits to the server the scene information about the position and the scene of the position;
A receiving unit that receives scene information according to the position of the visual assistance device from the server;
Further comprising
The notification unit vision assistance device according to any one of claims 1 to 10 for notifying the user based on the received Jokei information. A computer storing conditions specific to the user regarding notification of support information for supporting the user's visual recognition,
Obtaining image data representing a scene viewed by the user;
Generating support information based on the acquired image data and the condition; notifying the user of the generated support information ;
A step of acquiring reaction information representing a reaction of the user made in response to the notification of the support information ,
In the step of generating the support information, the acquired reaction information is further used.
program. The first visual assistance device and the second visual assistance device according to any one of claims 1 to 11, each used by a different user;
A receiving unit that collects information from the first visual assistance device that the user of the first visual assistance device encounters at a certain position and assists the visual recognition of another user at the position;
Vision assistance system comprising the information corresponding to the position of the second vision assistance device, a server and a transmission unit for transmitting to the second vision assistance device.

Images