JP7360775B2 - Smart glasses, program and display control method - Google Patents

Description

The present invention relates to smart glasses technology. In particular, it relates to see-through smart glasses that can be used as augmented reality wearable computers.

Smart glasses are worn like glasses in front of the wearer's field of vision, and are equipped with a camera and a retinal projection display. From the wearer's perspective, information is displayed superimposed on the field of view of the real space, unlike virtual reality, which displays an unreal space. Some smart glasses are equipped with natural language voice recognition functionality, allowing users to operate them using their own utterances.

In recent years, deterioration in the quality of nursing care services has become a social issue at elderly care facilities and medical institutions due to an increase in the number of people being cared for and a shortage of caregivers. In particular, caregivers need to respond differently to each person being cared for, depending on the location within the facility and the time of day. However, in nursing care facilities where a large number of care recipients reside, caregivers are often unable to remember the characteristics of each care recipient.

On the other hand, there is a technique for monitoring the behavior of a person to be monitored (a person being cared for) in a facility (for example, see Patent Document 1). According to this technology, multiple cameras are installed in a facility, the person being monitored is identified in the images through facial recognition, and information related to the person's care is notified to the supervisor (caregiver). be able to. The system also analyzes images taken by the camera, recognizes typical movements of the cared person, and sends notification information based on the movements to the supervisor. Here, in consideration of the privacy of the person to be monitored, the person being monitored is not notified of the captured monitoring image itself, but only of the analysis results obtained by analyzing the monitoring image. Furthermore, the supervisor can be notified of information related to the degree of care required according to the level of care that differs for each person to be monitored.

Patent No. 6579411

However, according to the technique described in Patent Document 1, it is necessary to arrange a monitoring camera in a living place such as a nursing care facility, which may make the person being cared for feel uncomfortable. Furthermore, wearable devices such as electronic tags and sensors can be affixed to the cared person, but the cared person may similarly feel a sense of resistance and aversion. On the other hand, in the case of a large-scale nursing care facility, for example, it is difficult for caregivers to know personal information such as names and symptoms of all care recipients.

In response, the inventors of the present application developed smart glasses that allow caregivers to quickly identify the care recipient in front of them and immediately recognize the care recipient's personal information. I thought it might be possible to use it. At this time, we thought that the information that a caregiver would need about the care recipient would differ depending on the location within the facility and the time of day.

SUMMARY OF THE INVENTION Therefore, the present invention aims to provide smart glasses, a program, and a display control method that allow a wearer to quickly identify a user in front of the user's view and immediately recognize personal information of the user in front of the user's view. .

According to the present invention, see-through smart glasses have a camera that captures an image of the line of sight, and a display that projects display information in the field of view of the user wearing the glasses.
Display information storage means that stores in advance display information according to a user ID (identifier) and display information according to a position ;
a face recognition means for recognizing a human face area from a video, detecting the size of the face area, and identifying a user ID from the face area;
Sensing means for detecting the current position;
If the size of the face area is equal to or larger than the first predetermined range, display information corresponding to the specified user ID is displayed on the display; otherwise, display information corresponding to the detected current position is displayed on the display. It is characterized by having a display control means.

According to another embodiment of the smart glasses of the present invention,
The displayed information may be text, images, and/or videos.

According to another embodiment of the smart glasses of the present invention,
The display information storage means is
For each user ID, display information according to each set of location and time zone,
For each location, display information according to the time zone is stored.
The display control means displays display information corresponding to the user ID, current location, and time on the display when the size of the recognized face area is equal to or larger than a first predetermined range; otherwise , display information corresponding to the current location and time is displayed on the display. It is also preferable to display display information according to the band on the display.

According to another embodiment of the smart glasses of the present invention,
It further includes a speaker that outputs audio to the hearing of the user who is wearing it,
If the size of the recognized face area is smaller than the second predetermined range, and if the display information to be displayed on the display is text, the display control means converts it into speech by speech synthesis and outputs the speech from the speaker. It is also preferable to do so.

According to another embodiment of the smart glasses of the present invention,
The speaker may be a bone conduction speaker.

According to another embodiment of the smart glasses of the present invention,
The larger the size of the face area of the person recognized by the face recognition means, the shorter the distance between the person and the user who wears it, and the smaller the size, the farther the distance between the person and the user who wears it.
It is also preferable that the first predetermined range of the size of the face area is based on the distance between the person and the wearer user.

According to another embodiment of the smart glasses of the present invention,
Wireless tags are placed in the surrounding environment where the user wearing them can move, depending on their location.
The display control means is
It has a position table storing the MAC address and/or tag ID of the wireless tag for each position,
When multiple wireless tags are detected, the location is identified using a position table based on the MAC address and/or tag ID of the wireless tag that receives the wireless signal with the strongest radio field strength, or
Alternatively, it is also preferable to specify the position by triangulation using a position table from the MAC addresses and/or tag IDs of three wireless tags that receive wireless signals in descending order of radio field strength.

According to another embodiment of the smart glasses of the present invention,
further comprising a wireless communication interface connecting to the server via a wireless network;
It is also preferable that the display information storage means receives the display information corresponding to the user ID in advance from the server via the wireless communication network.

According to the invention,
In a program that operates a computer installed in see-through smart glasses that have a camera that captures images of the line of sight and a display that displays information in the field of view of the user wearing the glasses,
Display information storage means that stores in advance display information according to a user ID (identifier) and display information according to a position ;
a face recognition means for recognizing a human face area from a video, detecting the size of the face area, and identifying a user ID from the face area;
Sensing means for detecting the current position;
If the size of the face area is equal to or larger than the first predetermined range, display information corresponding to the specified user ID is displayed on the display; otherwise, display information corresponding to the detected current position is displayed on the display. It is characterized in that a computer functions as a display control means.

According to the invention,
In a display control method for see-through smart glasses that have a camera that captures an image of the line of sight and a display that projects display information in the field of view of a user who wears the glasses,
Smart glasses are
It has a display information storage section that stores in advance display information according to a user ID (identifier) and display information according to a position ,
A first step of recognizing a person's face area from the video, detecting the size of the face area, identifying a user ID from the face area , and detecting the current position ;
If the size of the face area is equal to or larger than the first predetermined range, display information corresponding to the specified user ID is displayed on the display; otherwise, display information corresponding to the detected current position is displayed on the display. The second step is performed.

According to the smart glasses, program, and display control method of the present invention, the user who wears the glasses can quickly identify the user who is in the field of view, and immediately recognize the personal information of the user in the field of view.

FIG. 2 is an external view of smart glasses according to the present invention. It is an explanatory diagram showing a display of a display of smart glasses. FIG. 2 is a functional configuration diagram of smart glasses according to the present invention. FIG. 3 is an explanatory diagram showing processing functions of a face recognition unit. FIG. 3 is an explanatory diagram illustrating a display display according to the distance from the cared person facing the person. FIG. 3 is an explanatory diagram of a position table in the display control unit. FIG. 3 is an explanatory diagram showing display items in a display control unit. This is an explanation showing the switching of display items according to the field of view of the wearer user.

Hereinafter, embodiments of the present invention will be described in detail using the drawings.

FIG. 1 is an external view of smart glasses according to the present invention.

The smart glasses 1 are of a see-through type that can display images in the user's line of sight.
According to FIG. 1, the smart glasses 1 include a display 101, a camera 102, a speaker 103, a sensing unit 104, and a wireless communication interface 105 as hardware.
The display 101 projects specific display information in the field of view of the wearer user. The display information is generally text, but may of course be images or videos.
The camera 102 captures an image of the field of view of the wearer user. In some cases, the user in the visual field is reflected in the video, and in other cases, only facility equipment is reflected in the video without the user in the visual field.
The speaker 103 emits an audio signal to the auditory sense of the wearer user. When the information displayed on the display 101 is text, the audio signal is obtained by converting the text by speech synthesis. Moreover, the speaker 103 may be a bone electric speaker. By vibrating in contact with the temple of the wearing user, the wearing user can audibly recognize the characters. This allows the wearer user to recognize the characters without being heard by the user in sight.
The sensing unit 104 is, for example, Bluetooth (registered trademark), and receives a wireless signal transmitted from a BLE (Bluetooth Low Energy) tag. It is assumed that the BLE tag is placed in advance at each location within the facility where the user who wears it moves. Thereby, the smart glasses 1 can detect the current position.
The wireless communication interface 105 communicates with the user management server 2 connected to the Internet via a wireless LAN access point or a mobile phone network base station. The user management server 2 manages information displayed on the display 101 of the smart glasses 1 for each user.

FIG. 2 is an explanatory diagram showing the display of the smart glasses.

According to FIG. 2, it is assumed that a caregiver (wearer user) is wearing the smart glasses 1, for example, in a nursing care facility. Further, it is assumed that Mr. A, who is cared for, is within the field of view of the caregiver.
At this time, from the caregiver's perspective, the cared person A is visible through the smart glasses 1, and at the same time, the cared person A's "personal information" is displayed so as to be reflected therein. The personal information is, for example, information such as symptoms necessary for care recipient A's care.
As a result, the caregiver can make an appropriate call to the cared person A as soon as the cared person A comes into view.

FIG. 3 is a functional configuration diagram of smart glasses according to the present invention.

According to FIG. 3, the smart glasses 1 include a face recognition section 11, a user feature accumulation section 110, a display control section 12, and a display information storage section 120. These functional components are realized by executing a program that causes a computer installed in the smart glasses to function. Furthermore, the processing flow of these functional components can also be understood as a display control method for smart glasses.

[Face recognition unit 11]
The face recognition unit 11 recognizes a human face area from a video, detects the size of the face area, and identifies a user ID (IDentifier) from the face area. The identified user ID is output to the display control unit 12.

FIG. 4 is an explanatory diagram showing the processing functions of the face recognition section.
According to FIG. 4, the processing functions of the face recognition unit 11 include a face area detection unit 111, a facial feature detection unit 112, a user feature accumulation unit 110, and a high-speed classifier 113.

(Face area detection unit 111)
The face area detection unit 111 cuts out a face area (for example, a bounding box) from the image of the video frame captured by the camera 102. Specifically, R-CNN (Regions with Convolutional Neural Networks) and SSD (Single Shot Multibox Detector) are used.
R-CNN combines rectangular face regions with convolutional neural network features to detect a subset of face regions (region proposal). Next, CNN features are extracted from the region proposal. Then, the bounding box of the region proposal is adjusted using a support vector machine trained in advance using CNN features.
SSD is a general object detection algorithm using machine learning, and determines rectangular frames (bounding boxes) called default boxes. A large number of default boxes of different sizes are superimposed on one image, and a predicted value is calculated for each box. For each default box, it is possible to predict the position of the box, such as how far it is from the object and how different its size will be.

(Facial feature detection unit 112)
(S2) The facial feature detection unit 112 converts the extracted facial region into a 128/256/512-dimensional feature amount (Euclidean distance) using a face recognition model. The closer the distance, the more likely it is that the person can be recognized as the same person. Alternatively, a triplet of Anchor, Positive, and Negative may be created as facial images of the same person, and training may be performed to minimize Triplet Loss. .
Specifically, Google's (registered trademark) Facenet (registered trademark) algorithm can also be used as a face recognition model. With this, it is possible to convert the facial region into a feature amount of a multidimensional vector.

(User feature accumulation unit 110)
The user feature amount storage unit 110 stores in advance feature amounts of facial images of actual users to be identified. For example, it may be a feature amount of a facial image of a cared person living in a facility. Only the users stored in the user feature storage unit 110 are to be identified.

(High-speed discriminator 113)
The high-speed discriminator 113 uses the user feature storage unit 110 to check against a set of features of the user's face, and identifies the user ID that has the shortest distance or a feature that is less than or equal to a predetermined threshold.

The face recognition unit 11 has a lightweight learning model so that it can be implemented in a small IoT device. The amount of calculation has been significantly reduced by selecting the middle layer features of the trained model (deleting layers) and replacing it with a lightweight architecture. Specifically, using the C++ Native library, even a small device like smart glasses can identify 10,000 people in one second.

[Display information storage unit 120]
The display information storage unit 120 stores display information for each user ID in advance.
The "display information" is information that can be displayed on the display of the smart glasses 1, and is preferably "characters." Of course, the information is not limited thereto, and may be an image or video.
Further, the display information storage unit 120 downloads and stores display information for each user ID in advance from the user management server 2 via the wireless communication interface 105. By downloading it in advance, the smart glasses 1 can function as a standalone. Of course, the display information storage unit 120 may be always connected to the user management server 2 via the wireless communication interface 105 so as to be able to communicate therewith.

The display information storage unit 120 may further store personal information for each user as display information according to time zone and/or location.
(When the user ID can be identified by the face recognition unit 11)
・Personal information for each user according to the time zone ・Personal information for each user according to the time zone and location (if the user ID cannot be identified by the face recognition unit 11)
・Information according to location ・Information according to time and location

[Display control unit 12]
When the size of the face area is equal to or larger than the first predetermined range, the display control unit 12 displays display information corresponding to the specified user ID on the display 101.
The size of the face area of the user in view recognized by the face recognition unit 11 is determined as follows.
(1) The larger the size of the face area, the shorter the distance between the user at the end of the field of view and the user at the wearer.
(2) The smaller the size of the face area, the greater the distance between the user at the end of the field of view and the user from whom the wearer is wearing.
For this purpose, the first predetermined range of the size of the face area, which is the threshold value, is based on the distance between the user in view and the wearer user.

The display control unit 12 controls the display as follows.
(1) When the size of the recognized face area is equal to or larger than the first predetermined range, display information corresponding to the user ID, current location, and time zone is displayed on the display 101.
(2) If the size of the recognized face area is smaller than the first predetermined range, display information corresponding to the detected current position is displayed on the display 101. At this time, it is also preferable to display display information according to the time zone in addition to the current location.

FIG. 5 is an explanatory diagram showing the display according to the distance to the cared person facing each other.

According to FIG. 5(a), two care recipients, Mr. A and Mr. B, can be seen in the field of view of the wearing source user through the smart glasses 1. At this time, if the size of Mr. A's face area exceeds the first predetermined range, Mr. A's personal information (display information) is displayed on the display 101. Mr. A's personal information, such as the following, is displayed in the field of view of the wearer.
[Basic information]: Name, age, level of care required, date of birth, room number [Behavior information]: Wake-up time, presence of meals, presence of hydration, presence of medication [Notification]: Health condition, body temperature, blood pressure, pulse, symptoms , medication information, rehabilitation information

According to FIG. 5(b), similarly to FIG. 5(a), the wearer user can see two care recipients, Mr. A and Mr. B. At this time, unlike in FIG. 5A, the size of the face area of both Mr. A and Mr. B is not larger than the first predetermined range. At this time, only the names of Mr. A and Mr. B are displayed in the field of view of the wearer user.

FIG. 6 is an explanatory diagram of a position table in the display control section.

For example, a BLE tag is placed as a wireless tag at each position in the surrounding environment where the wearer user can move.
On the other hand, the display control unit 12 of the smart glasses 1 has a position table.
The "location table" stores the MAC address and/or tag ID of a wireless tag for each location.

According to the position table of FIG. 6, the following information is associated with each tag ID and MAC address.
[Tag ID] [MAC address]: Location classification, room number, indoor location classification The current location of the user wearing the smart glasses 1 is specified by one of the following methods.
(1) When a plurality of wireless tags are detected, the position is specified using a position table based on the MAC address and/or tag ID of the wireless tag that receives the wireless signal with the strongest radio field strength.
(2) The position is determined by triangulation using a position table based on the MAC (Media Access Control) addresses and/or tag IDs of the three wireless tags that receive wireless signals in order of radio field strength (RSSI (Received Signal Strength Indication)). Identify.

FIG. 7 is an explanatory diagram showing display items in the display control section.

According to FIG. 7, display items of display information shown to the wearer user are shown according to the time zone and position. As a result, display information is customized from the display information storage unit 120 according to the time of day (morning, afternoon, evening) and location (inside the facility, entrance, toilet/bathroom, bedroom), and is shown to the wearer. be able to.

There are the following three display items (basic information)
Name, age, level of care, date of birth, room number, care plan (behavior information)
Wake-up time, meals (completed/not yet), hydration (completed/not), medication (completed/not),
Excretion (present/absent), bathing (completed/unfinished),
(Transfer information)
Health status, body temperature, blood pressure (upper/lower), pulse, treatment information, symptoms, treatment information,
Number of days without defecation, time of defecation, abnormal excretion (presence/absence), time of morning/lunch/night meals, amount of breakfast/noon/night meals,
Hydration time, water amount, medication time, bathing time, abnormal bathing (presence/absence)

According to FIG. 7, display items are defined as follows.
Assume that there is a user in view from the wearer's perspective regardless of the time of day. At this time, if the size of the face area exceeds the first predetermined range and the user ID is specified, display item A in FIG. 7 becomes the display target. On the other hand, if the size of the face area is smaller than the first predetermined range, display item B (name only) in FIG. 7 becomes the display target.
Assume that it is morning time and that there is no visible user from the wearer's perspective. Here, for the wearer user, if the current location is "toilet/bath", display item C becomes the display target, and if the current location is "bedroom", display item D becomes the display target.
Assume that it is daytime and there is no visible user from the wearer's perspective. Here, for the wearer user, if the current location is "toilet/bath", display item E becomes the display target, and if the current location is "bedroom", display item F becomes the display target.
Assume that it is night time and there is no visible user from the wearer's perspective. Here, for the wearer user, if the current location is "toilet/bath", display item G becomes the display target, and if the current location is "bedroom", display item H becomes the display target.

FIG. 8 is an explanation showing switching of display items according to the field of view of the wearer user.

According to FIG. 8A, since the size of the face area of the sighted user is greater than or equal to the first predetermined range when viewed from the wearing source user, display information corresponding to the specified user ID is displayed on the display 101. has been done.
According to FIG. 8(b), there is no visible user from the wearer's perspective. Further, it is assumed that the wearer user is currently located in the toilet/bathroom of room 201. At this time, display items according to the time zone and location are displayed on the display 101 using the display items shown in FIG.
According to FIG. 8C, there is no visible user from the wearer's perspective. Further, it is assumed that the wearer user is currently located in the bedroom of room 201. At this time, display items according to the time zone and location are displayed on the display 101 using the display items shown in FIG.

According to the embodiments described above, the display information of, for example, characters was described as being displayed on the display 101. On the other hand, as another embodiment, the display information of characters may be recognized aurally through the ear of the wearer user.

If the size of the recognized face area is smaller than the second predetermined range, the display control unit 12 converts the display information to be displayed on the display into audio by voice synthesis, and outputs the audio from the speaker 103.
Regarding the size of the face area, the second predetermined range may be the same as the first predetermined range, or may be larger or smaller than the first predetermined range.
In addition, for speech synthesis, we use a lightweight Japanese voice reading function that can be installed on a standalone microcomputer board for IoT/embedded devices. However, homographs such as personal names are controlled so that they can be read out appropriately by adding pronunciations. For example, in the case of ``Ohara-san'', various readings are assumed, such as ``Kohara'', ``Ohara'', and ``Obara''.

The farther the distance between the wearing source user and the sighted user is (the smaller the size of the face area of the sighted user), the more the voice may be output from the speaker 103 to read out the characters. On the other hand, the closer the distance between the wearing source user and the sighted user is (the larger the size of the face area of the sighted user), the more display information is displayed on the display 101.
In this way, the method of presenting information to the wearer user can be dynamically changed depending on the distance between the wearer user and the sighted user.
For example, in the case of a nursing facility, when a caregiver (wearer user) and a cared person (viewer user) meet face-to-face, they often have a direct conversation at a close distance. This is because if the speaker 103 makes a sound, the caregiver's attention may be distracted.

As described above in detail, according to the smart glasses, program, and display control method of the present invention, the user who wears the glasses can quickly identify the user in front of the field of view and immediately recognize the personal information of the user in the front of the field of view. Can be done.

For example, by wearing the smart glasses of the present invention, a caregiver at a nursing care facility can immediately identify the person being cared for within their field of view, and can also read personal information (name, age, symptoms, etc.) about that person hands-free. It can be recognized by It is possible to quickly and accurately present information to the caregiver while suppressing the psychological resistance of the care recipient.
Further, even if the cared person is not within the field of view, information according to the current position can be immediately recognized using the BLE tag placed within the facility.
The smart glasses of the present invention can also be applied to nursing care work support AR systems and suspicious person detection systems (schools, nursery schools, etc.).

Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The invention is limited only by the claims and their equivalents.

1 Smart glasses 101 Display 102 Camera 103 Speaker, bone electric speaker 104 Sensing unit 105 Wireless communication interface 11 Face recognition unit 110 User feature storage unit 111 Face area detection unit 112 Face feature detection unit 113 High-speed classifier 12 Display control unit 120 Display Information storage unit 2 User management server

Claims (10)

See-through smart glasses have a camera that captures images of the line of sight, and a display that projects information into the user's field of vision.
Display information storage means that stores in advance display information according to the user ID (identifier) and display information according to the position ;
a face recognition means for recognizing a human face area from a video, detecting the size of the face area, and identifying a user ID from the face area;
Sensing means for detecting the current position;
If the size of the face area is equal to or larger than the first predetermined range, display information corresponding to the specified user ID is displayed on the display; otherwise, display information corresponding to the detected current position is displayed on the display. Smart glasses characterized by having display control means. The smart glasses according to claim 1, wherein the displayed information is characters, images, and/or videos. The display information storage means is
For each user ID, display information according to each set of location and time zone,
For each location, display information according to the time zone is stored.
The display control means displays display information corresponding to the user ID, current location, and time on the display when the size of the recognized face area is equal to or larger than a first predetermined range; otherwise , display information corresponding to the current location and time is displayed on the display. The smart glasses according to claim 1 or 2, wherein display information corresponding to the band is displayed on the display. It further includes a speaker that outputs audio to the hearing of the user who is wearing it,
If the size of the recognized face area is smaller than the second predetermined range, and if the display information to be displayed on the display is text, the display control means converts it into speech by speech synthesis and outputs the speech from the speaker. The smart glasses according to any one of claims 1 to 3 , characterized in that: The smart glasses according to claim 4 , wherein the speaker is a bone conduction speaker. The larger the size of the face area of the person recognized by the face recognition means, the shorter the distance between the person and the user who wears it, and the smaller the size, the farther the distance between the person and the user who wears it.
6. The smart glasses according to claim 1, wherein the first predetermined range of the size of the face area is based on the distance between the person and the user wearing the smart glasses. Wireless tags are placed in the surrounding environment where the user wearing them can move, depending on their location.
The display control means is
It has a position table storing the MAC address and/or tag ID of the wireless tag for each position,
When multiple wireless tags are detected, the location is determined using a position table based on the MAC address and/or tag ID of the wireless tag that receives the wireless signal with the strongest radio field strength, or
Alternatively, the position is specified by triangulation using a position table from the MAC addresses and/or tag IDs of three wireless tags that receive wireless signals in order of strength of radio waves. The smart glasses according to item 1. further comprising a wireless communication interface connecting to the server via a wireless network;
8. The smart glasses according to claim 1, wherein the display information storage means receives in advance display information corresponding to a user ID from a server via a wireless communication network. In a program that operates a computer installed in see-through smart glasses that have a camera that captures images of the line of sight and a display that displays information in the field of view of the user wearing the glasses,
Display information storage means that stores in advance display information according to a user ID (identifier) and display information according to a position ;
a face recognition means for recognizing a human face area from a video, detecting the size of the face area, and identifying a user ID from the face area;
Sensing means for detecting the current position;
If the size of the face area is equal to or larger than the first predetermined range, display information corresponding to the specified user ID is displayed on the display; otherwise, display information corresponding to the detected current position is displayed on the display. A program for smart glasses characterized by causing a computer to function as a display control means. In a display control method for see-through smart glasses that have a camera that captures an image of the line of sight and a display that projects display information in the field of view of a user who wears the glasses,
Smart glasses are
It has a display information storage section that stores in advance display information according to a user ID (identifier) and display information according to a position ,
A first step of recognizing a person's face area from the video, detecting the size of the face area, identifying a user ID from the face area , and detecting the current position ;
If the size of the face area is equal to or larger than the first predetermined range, display information corresponding to the specified user ID is displayed on the display; otherwise, display information corresponding to the detected current position is displayed on the display. A display control method for smart glasses, the method comprising: performing the second step.

Images