JP2023047410A - Portable device, program and method for presenting information based on object recognition in accordance with user's dynamic state - Google Patents

Abstract

To provide a portable device, etc., for presenting information based on object recognition in accordance with a user's dynamic state.SOLUTION: There is provided, a portable device having a camera for capturing videos, the portable device comprising: presentation information storage means in which presentation information is preliminarily stored in association for object ID (identifier); object recognition means for recognizing an image region of an object from a video, detecting the size of the image region, and identifying an object ID from the image region; movement detection means for detecting whether or not the portable device is in progress of movement; object ID selection means for selecting an object ID on the basis of whether or not being in progress of movement and the selected object ID; and presentation means for presenting a user with presentation information that corresponds to the selected object ID. The object ID is an individual ID, the presentation information is individual information, and the image region in the object recognition means can be a face region.SELECTED DRAWING: Figure 2

Description

The present invention relates to technology for presenting information to a user based on object recognition. Especially suitable for smart glasses and smartphones.

Smart glasses are see-through type glasses as an augmented reality wearable computer. It is worn in front of the user's (wearer's) field of vision and carries a projection display to the lens. From the wearer's point of view, information is superimposed on the field of view of the real space, unlike the virtual reality that displays the unreal space.

In recent years, at nursing care facilities and medical institutions for the elderly, deterioration in the quality of nursing care services has become a social issue due to an increase in the number of care recipients and a shortage of caregivers. In particular, the caregiver needs to treat each care recipient differently. In particular, in a facility where a large number of care recipients live, it is natural that caregivers cannot remember the personal information of each care recipient (for example, information such as symptoms required for care).

On the other hand, there is a technology in which a caregiver wears smart glasses and hands-free display of the personal information of the cared person on the lens (see, for example, Non-Patent Document 1). According to this technology, the cared person can be identified from the face image of the cared person captured by the camera of the smart glasses, and the cared person's personal information can be immediately projected onto the lens. Therefore, the cared person does not need to wear a wireless tag or marker for personal recognition.

FIG. 1 is an explanatory diagram showing the field of view through smart glasses.

For example, assume that a caregiver wears the smart glasses 1 at a care site. At this time, the caregiver (wearer) can see the cared person in the field of view through the lens of the smart glasses 1 and at the same time see the cared person's "personal information" projected on the lens. As a result, the caregiver can read the personal information of the cared person hands-free while carrying out the care work.

KDDI Research Institute, Zenkokai News Release "KDDI Research Institute and Zenkokai Develop Hands-free Nursing Work Support System Using AR Glasses" (February 2, 2021), [online], [2021] Searched on August 30], Internet <URL: https://www.kddi-research.jp/newsrelease/2021/020201.html> "The story of the unit of internal parameter (focal length) ~ conversion between pixel and mm ~", [online], [searched on September 17, 2021], Internet <URL: https://mem-archive.com/ 2018/02/25/post-201/> ``Investigation of technology to restore 3D data from images Principle of calculation of 3D coordinates-Epipolar geometry & estimation of camera pose-'', [online], [searched on September 17, 2021], Internet <URL: https://qiita.com/akaiteto/items/f5857c7774794a6e5f5e＞

According to the technique described in Non-Patent Document 1 mentioned above, when a plurality of interpersonal persons or objects are captured in an image captured by a camera, an object image having the largest size of the image area or a front image is displayed. It tries to recognize the object given the object image.
On the other hand, the inventors of the present application have considered that, among a plurality of objects reflected in an image, different objects should be focused on by the user depending on the dynamic state of a mobile device equipped with a camera. rice field. In other words, it was thought that the user should select an object of interest in conjunction with the dynamic state of the mobile device, and then display information related to the object to the user.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a mobile device, a program, and a method for presenting information based on object recognition according to the user's dynamic state.

According to the present invention, in a portable device having a camera for capturing images,
presentation information storage means for pre-stored in association with presentation information for each object ID (IDentifier);
an object recognition means for recognizing an image area of an object from an image, detecting the size of the image area, and specifying an object ID from the image area;
movement detection means for detecting whether the mobile device is moving;
an object ID selection means for selecting an object ID based on the presence or absence of movement and the size of the image area;
and presenting means for presenting presentation information corresponding to the selected object ID to the user.

According to another embodiment of the portable device of the present invention,
The object ID is a personal ID,
Presented information is personal information,
It is also preferable that the image area in the object recognition means is a face area.

According to another embodiment of the portable device of the present invention,
The object ID selection means is
When stopped, select an object ID whose image area size is greater than or equal to the first predetermined range,
When moving, it is also preferable to select an object ID whose image area size is equal to or less than the second predetermined range.

According to another embodiment of the portable device of the present invention,
The first predetermined range and the second predetermined range are based on the distance to the subject. The larger the distance to the subject, the smaller the size of the image area, and the shorter the distance to the subject. It is also preferable for the size of the image area to be as large as possible.

According to another embodiment of the portable device of the present invention,
The movement detection means is capable of detecting movement speed,
It is also preferable that the object ID selection means selects the object ID based on the moving speed and the size of the image area.

According to another embodiment of the portable device of the present invention,
It is also preferable that the object ID selection means selects an object ID having a smaller image area size as the movement speed increases.

According to another embodiment of the portable device of the present invention,
It is also preferable that the movement detection means is an acceleration sensor and detects whether or not the person is walking.

According to another embodiment of the portable device of the present invention,
further comprising a visually projecting display;
It is also preferable that the presentation means displays the presentation information corresponding to the object ID on the display.

According to another embodiment of the portable device of the present invention,
A camera is integrally mounted to capture an image of the user's field of view,
A display is a lens that projects presentation information into the user's field of vision.
It is also preferable that the portable device is see-through type smart glasses.

According to another embodiment of the portable device of the present invention,
the camera and/or display are integrally constructed,
It is also preferred that the mobile device is a smart phone.

According to another embodiment of the portable device of the present invention,
further having a speaker for audio output,
It is also preferable that the presenting means reads out the presentation information in a sentence and outputs it by voice from a speaker.

According to another embodiment of the portable device of the present invention,
It is also preferred that the speaker is a bone conduction speaker.

According to the present invention, in a program for functioning a computer mounted on a portable device having a camera for capturing images,
presentation information storage means for pre-stored in association with presentation information for each object ID (identifier);
an object recognition means for recognizing an image area of an object from an image, detecting the size of the image area, and specifying an object ID from the image area;
movement detection means for detecting whether the mobile device is moving;
an object ID selection means for selecting an object ID based on the presence or absence of movement and the size of the image area;
The present invention is characterized by causing the computer to function as presentation means for presenting presentation information corresponding to the selected object ID to the user.

According to the present invention, in an information presentation method for a mobile device having a camera for capturing images,
The mobile device
Presented information is associated with each object ID (identifier) and stored in advance,
a first step of recognizing an image area of an object from a video, detecting the size of the image area, and identifying an object ID from the image area;
a second step of detecting whether the mobile device is in motion;
a third step of selecting an object ID based on whether it is moving and the size of the image area;
and a fourth step of presenting presentation information corresponding to the selected object ID to the user.

According to the portable device, program, and method of the present invention, information based on object recognition can be presented according to the user's dynamic state.

It is an explanatory view showing a view through smart glasses. 1 is a functional configuration diagram of smart glasses in the present invention. FIG. It is a flow chart of an object ID selection part. FIG. 5 is an explanatory diagram showing information presented on smart glasses while the vehicle is stopped; FIG. 4 is an explanatory diagram showing information presented on smart glasses while moving; FIG. 4 is an explanatory diagram showing information displayed on a smartphone while moving;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 2 is a functional configuration diagram of smart glasses in the present invention.

The smart glasses 1 are of a see-through type that can be worn in the field of view of the user.
According to FIG. 2, the smart glasses 1 are equipped with, as hardware, a display 101 for projecting information on the lens, a camera 102 for photographing a person or an object, and a speaker 103 for outputting sound to the wearer.
The smart glasses 1 also have a presentation information storage unit 100, an object recognition unit 11, a movement detection unit 12, an object ID selection unit 13, and a presentation unit 14 as software. These functional components are implemented by executing a program that causes a computer mounted on smart glasses to function.

[Display 101]
The display 101 can project presentation information on lenses through which the wearer's field of view of the smart glasses 1 passes. This allows the wearer to visually recognize the presented information. The presentation information is generally text, but may of course be an image or video.

[Camera 102]
The camera 102 may be configured integrally with the smart glasses 1 or may be connected to the outside of the smart glasses 1 . Camera 102 captures an image of the wearer's field of view. An image may include an interpersonal person (for example, a cared person) or an object (for example, facility equipment). The camera 102 always outputs the captured image to the object recognition unit 11 .

[Speaker 103]
The speaker 103 outputs an audio signal to the hearing of the wearer of the smart glasses 1 . The audio signal may be obtained by synthesizing and converting the characters to be displayed on the display 101 .
Also, the speaker 103 may be, for example, a bone conduction speaker. By vibrating the wearer's temple in contact with the wearer's temple, the wearer can be made to audibly recognize the voice without being heard by the other person.

[Presentation information storage unit 100]
The presentation information storage unit 100 stores in advance presentation information associated with each object ID (identifier).
The object ID may be a personal ID, and the presentation information may be personal information. For example, when assuming a nursing care site, the personal ID may be an ID assigned to each care recipient, and the presentation information may be the care information of the care recipient.

The smart glasses 1 may store personal information for each personal ID in advance (stand-alone type), or access a server via a network and download it (server-client type). good.
Examples of the personal information of the care receiver include the following.
(basic information)
Name, age, care level, date of birth, room number, care plan, etc.
(action information)
Time to wake up, meals (done/not done), hydration (done/not done), medication (done/not done), excretion (yes/no), bathing (done/not done), etc.
(Transfer information)
Health condition, body temperature, blood pressure (upper/lower), pulse, treatment information, symptoms, treatment information, number of days without excretion, excretion time, excretion abnormality (presence/absence), morning/lunch/night meal time, morning/lunch/night meal amount , rehydration time, amount of water, medication time, bathing time, bathing abnormality (yes/no), etc.

[Object recognition unit 11]
The object recognition unit 11 recognizes the target object from the image captured by the camera 102 .
The object recognition unit 11 recognizes image areas of one or more objects from a video, detects the size of each image area, and specifies an object ID from each image area. The object recognition unit 11 is a general machine learning engine applied to image classification and face recognition based on deep learning. It recognizes specific classes (categories like people and things) in the video.
The image area may be a face image of a person, and the recognition target may be identification of the person.

The object recognition unit 11 has an <image area detection function> for detecting an image area of an object and an <object detection function> for detecting an object in the image area.

<Image area detection function>
The image area detection function detects the image area of the object from the video. It cuts out the object region (eg, bounding box) from the image of the video frame. Specifically, R-CNN (Regions with Convolutional Neural Networks) and SSD (Single Shot Multibox Detector) are used.
R-CNN combines rectangular object regions with convolutional neural network features to detect subsets of object regions (region proposal). Next, CNN features are extracted from the region proposal. Then, the bounding box of the region proposal is adjusted by a support vector machine pre-trained using the CNN features.
SSD is a general object detection algorithm using machine learning, and determines a rectangular frame (bounding box) called a default box. A large number of default boxes with different sizes are superimposed on one image, and a predicted value is calculated for each box. For each default box, we can make a prediction of its position, how far it is from the object and how different it is in size.

<Object detection function>
The object detection function detects objects from the image area. The object may be a human body, a face, or an object such as equipment.
Since the embodiment of the present invention assumes a nursing care site, the object detection function recognizes a person's face area from an image and identifies an individual ID from the face area.
The object detection function is preliminarily storing feature amounts of actual individual face images to be identified. For example, the feature amount of the care-receiver's face image is accumulated.
After that, the object detection function converts the cut-out face area, which is an image area, into a 128/256/512-dimensional feature quantity (Euclidean distance) using a face recognition model. Specifically, the Facenet (registered trademark) algorithm of Google (registered trademark) can also be used as the face recognition model. As a result, the facial region can be converted into a multidimensional vector feature quantity.
Then, the object recognition unit 11 collates with a set of facial feature amounts of the individual, and specifies an individual ID that has the shortest distance or a feature amount that is equal to or less than a predetermined threshold.

The learning model of the object recognition unit 11 is lightened so that it can be mounted on a small IoT device. The amount of computation is greatly reduced by selecting the intermediate layer features of the trained model (deleting layers) and replacing the lightweight architecture. Specifically, using a C++ native library, even a small device such as smart glasses can identify 10,000 people in 1 second.

Here, the relationship between the size of the image area in which the object is captured and the distance from the object will be described.
If the object is a human face, the width of the face is about 15 cm. Therefore, the distance between the person and the person can be estimated from the number of pixels in the width of the image area of the recognized face. See Patent Document 2). Depending on the camera, for example, 10 pixels may be assumed to be 1 cm. Of course, the size of a specific object is measured, the object is photographed from a specific distance, and the size-to-distance ratio is fixedly calculated from the size of the image area (the number of pixels). There may be.
Note that the number of pixels in the image area is directly proportional to the resolution of the camera. Therefore, the number of pixels in the image area corresponding to the distance can be determined in accordance with the resolution of the camera.
Further, as another embodiment, there is also a technique of estimating the distance using a plurality of images of video from a camera without measuring the size of a specific object (see Non-Patent Document 3, for example). In this case, three-dimensional coordinates can be calculated from a plurality of monocular camera images, and the Z-axis can also be estimated as the distance from the camera to the object.

[Movement detection unit 12]
The movement detection unit 12 detects whether the smart glasses 1 themselves are "moving". The device may be an acceleration sensor. In particular, whether or not the person is "walking" may be detected as whether or not the person is moving, based on the cycle detected by the acceleration sensor.
Further, the movement detection unit 12 may be capable of detecting "moving speed". For example, the smart glasses 1 themselves may be positioned, and the movement speed may be derived from the movement distance per unit time.

[Object ID selection unit 13]
The object ID selection unit 13 selects an object ID based on whether the object is moving and the size of the image area.

<Selection of Object ID Based on Moving/Standing and Size of Image Area>
FIG. 3 is a flowchart of the object ID selection section.

(S1) The object ID selection unit 13 selects an object ID whose image area size is equal to or larger than a first predetermined range when the movement detection unit 12 detects that it is "stopping".
The predetermined range is based on the distance to the subject. The longer the distance to the subject, the smaller the size of the image area, and the closer the distance to the subject, the larger the size of the image area.

FIG. 4 is an explanatory diagram showing information presented to the smart glasses while stopped.

According to FIG. 4 , for example, at a nursing care site, a caregiver (wearer) can see two care recipients “Mr. A” and “Mr. B” in the field of vision wearing the smart glasses 1 . Further, the object recognition unit 11 has already identified the individual IDs of both care recipients "Mr. A" and "Mr. B" from the face area by the object detection function.
Here, when the caregiver himself/herself is stopped, it can be assumed that the caregiver is paying attention to an interpersonal person or an object at a short distance. For example, it is assumed that a caregiver has a face-to-face conversation with a care-receiver who is close to him/her.

Assume that the caregiver is stopped and the object recognition unit 11 recognizes the care recipient "Mr. A" as an object ID whose image area size is greater than or equal to the first predetermined range. That is, it is determined that the distance to the cared person "Mr. A" is short.
At this time, on the display 101 of the lens of the smart glasses 1, the "personal information" of the cared person "Mr. A" is displayed. The personal information is, for example, information such as symptoms necessary for caring for Mr. A, who is cared for.
As a result, the caregiver can appropriately speak to the care-receiver A, who is relatively nearby, while the car is stopped.

(S2) When the movement detection unit 12 detects that the movement detection unit 12 is moving, the object ID selection unit 13 selects an object ID whose image area size is equal to or smaller than the second predetermined range.

FIG. 5 is an explanatory diagram showing information presented on smart glasses while moving.

According to FIG. 5 , as in FIG. 4 , the caregiver (wearer) can see two care recipients “Mr. A” and “Mr. B” in the field of vision wearing the smart glasses 1 . Further, the object recognition unit 11 has already identified the individual IDs of both care recipients "Mr. A" and "Mr. B" from the face area by the object detection function.
Here, it can be assumed that the caregiver himself/herself is moving means that the caregiver is paying attention to a distant person or object. For example, it is assumed that a cared person who is close to the cared person will quickly pass by.

Assume that the caregiver is moving, and the object recognition unit 11 recognizes the care recipient "Mr. B" as an object ID whose image area size is equal to or smaller than the second predetermined range. That is, it is determined that the distance from the cared person "Mr. B" is long.
At this time, on the display 101 of the lens of the smart glasses 1, the "personal information" of the cared person "Mr. B" is displayed so as to be reflected. The personal information is, for example, information such as symptoms necessary for care recipient B's care.
As a result, the caregiver can know appropriate information in advance before approaching the care-receiver B, who is located relatively far away, while moving.

Specifically, the distance according to the second predetermined range of the size of the image area may be, for example, about 4 m. Of course, the stride length of a person is considered to be approximately height×0.45, although it differs from person to person. Then, the stride length of a 170 cm tall person is about 77 cm. In other words, a distance of about 4m means about 5 steps ahead. In this case, the wearer is approaching a person who is more than five steps ahead while moving, so the personal information of the person should be notified in advance.

<Selection of object ID based on moving speed and size of image area>
As another embodiment, the object ID selection unit 13 may select the object ID based on the moving speed and the size of the image area.
The object ID selection unit 13 performs control so that an object ID having a smaller image area size is selected as the movement speed increases. Specifically, it is preferable to make the second predetermined range variable so that the faster the moving speed, the smaller the second predetermined range. This is because the faster the movement speed, the higher the possibility that the wearer is paying attention to a distant person or object.

[Presentation unit 14]
The presentation unit 14 presents presentation information corresponding to the selected object ID to the user.
The presentation unit 14 may display the presentation information corresponding to the object ID on the display 101 or may output the presentation information from the speaker 103 by voice synthesis.

The presentation information is stored in advance in the presentation information storage unit 100, and may be characters, images, or videos. When the object ID is a personal ID that identifies a care recipient, the presentation information is the personal information of the care recipient. For example, the following personal information is displayed as presentation information.
[Basic information]: Name, age, level of nursing care required, date of birth, room number [Activity information]: Wake-up time, whether to eat, whether to drink water, whether to take medication , medication information, rehabilitation information

When the presentation information is text, it can be output from the speaker 103 as voice by voice synthesis. For voice synthesis, we use a lightweight Japanese text-to-speech function that is installed on a single microcomputer board for IoT and embedded.
That is, the speech based on speech synthesis reads out different information depending on the distance between the wearer and the person or the object. That is, when the mobile phone is stopped, the information about interpersonal information at a short distance is read aloud, and when the mobile phone is moving, the information such as interpersonal information at a long distance is read aloud.

FIG. 6 is an explanatory diagram showing information displayed on the smartphone while moving.

According to the above-described embodiments, the mobile device of the present invention is described as a see-through type smart glasses, but it is not limited to this, and may be a mobile terminal such as a smart phone. In the case of smartphones, the camera and/or the display are integrated.

As described in detail above, according to the portable device, program, and method of the present invention, it is possible to present information based on object recognition according to the user's dynamic state.
According to the present invention, it is possible to dynamically change the information to be presented to the user according to the dynamic state of the mobile device and the distance between the person or object captured by the camera. For example, in the case of a nursing care site, when multiple care recipients are reflected in the caregiver's field of vision, the caregiver's need for caregiver support is determined according to the caregiver's dynamic state and the distance between the caregiver and the caregiver. Personal information of the caregiver can be presented.

It should be noted that, as a result, for example, ``a caregiver at a nursing care site can recognize appropriate personal information of the care recipient in a hands-free manner according to the dynamic state by wearing smart glasses or smartphones.'' From this, it will be possible to contribute to Goal 3 of the Sustainable Development Goals (SDGs) led by the United Nations, "Ensure healthy lives and promote well-being for all at all ages."

For the various embodiments of the present invention described above, various changes, modifications and omissions within the spirit and scope of the present invention can be easily made by those skilled in the art. The foregoing description is exemplary only and is not intended to be limiting. The invention is to be limited only as limited by the claims and the equivalents thereof.

1 smart glasses 100 presentation information storage unit 101 display 102 camera 103 speaker 11 object recognition unit 12 movement detection unit 13 object ID selection unit 14 presentation unit

Claims (14)

In a portable device having a camera that captures images,
presentation information storage means for pre-stored in association with presentation information for each object ID (identifier);
an object recognition means for recognizing an image area of an object from an image, detecting the size of the image area, and specifying an object ID from the image area;
movement detection means for detecting whether the mobile device is moving;
an object ID selection means for selecting an object ID based on the presence or absence of movement and the size of the image area;
and presenting means for presenting presentation information corresponding to the selected object ID to a user. The object ID is a personal ID,
Presented information is personal information,
2. The mobile device according to claim 1, wherein the image area in the object recognition means is a face area. The object ID selection means is
When stopped, select an object ID whose image area size is greater than or equal to the first predetermined range,
3. The mobile device according to claim 1, wherein, when the mobile device is moving, an object ID whose image area size is equal to or smaller than a second predetermined range is selected. The first predetermined range and the second predetermined range are based on the distance to the subject. The larger the distance to the subject, the smaller the size of the image area, and the shorter the distance to the subject. 4. A portable device according to claim 3, wherein the size of the image area increases as the number increases. The movement detection means is capable of detecting movement speed,
5. The portable device according to any one of claims 1 to 4, wherein the object ID selection means selects the object ID based on the moving speed and the size of the image area. 6. The portable device according to claim 5, wherein the object ID selection means selects an object ID having a smaller image area size as the moving speed is faster. 7. The portable device according to any one of claims 1 to 6, wherein the movement detection means is an acceleration sensor and detects whether or not the user is walking. further comprising a visually projecting display;
8. The portable device according to any one of claims 1 to 7, wherein the presentation means displays the presentation information corresponding to the object ID on the display. A camera is integrally mounted to capture an image of the user's field of view,
A display is a lens that projects presentation information into the user's field of vision.
9. The mobile device according to claim 8, wherein the mobile device is see-through type smart glasses. the camera and/or display are integrally constructed,
9. The mobile device of claim 8, wherein the mobile device is a smart phone. further having a speaker for audio output,
11. The portable device according to any one of claims 1 to 10, wherein the presenting means reads out the presentation information in a sentence and outputs it by voice from a speaker. 12. The portable device of claim 11, wherein the speaker is a bone conduction speaker. In a program that operates a computer mounted on a portable device having a camera that captures images,
presentation information storage means for pre-stored in association with presentation information for each object ID (identifier);
an object recognition means for recognizing an image area of an object from an image, detecting the size of the image area, and specifying an object ID from the image area;
movement detection means for detecting whether the mobile device is moving;
an object ID selection means for selecting an object ID based on the presence or absence of movement and the size of the image area;
A program for causing a computer to function as presentation means for presenting presentation information corresponding to a selected object ID to a user. In an information presentation method for a mobile device having a camera for capturing images,
The mobile device
Presented information is associated with each object ID (identifier) and stored in advance,
a first step of recognizing an image area of an object from a video, detecting the size of the image area, and identifying an object ID from the image area;
a second step of detecting whether the mobile device is in motion;
a third step of selecting an object ID based on whether it is moving and the size of the image area;
and a fourth step of presenting presentation information corresponding to the selected object ID to the user.

Images