JP7453193B2 - Mobile device, program, and method for controlling speech based on speech synthesis in conjunction with user's surrounding situation - Google Patents

Description

The present invention relates to a technique for presenting information based on object recognition to a user. Particularly suitable for smart glasses and smartphones.

Smart glasses are see-through glasses that serve as augmented reality wearable computers. This is worn in front of the user's (wearer's) field of vision and is equipped with a projection type display for the lens. From the wearer's perspective, information is displayed superimposed on the field of view of the real space, unlike virtual reality, which displays an unreal space.

In recent years, deterioration in the quality of nursing care services has become a social issue at elderly care facilities and medical institutions due to an increase in the number of people being cared for and a shortage of caregivers. In particular, caregivers need to respond differently to each care recipient. Particularly in facilities where a large number of care recipients reside, it is natural for caregivers to be unable to remember the personal information of each care recipient (for example, information on symptoms necessary for care).

On the other hand, there is a technology in which a caregiver wears smart glasses and displays the personal information of the cared person on the lenses in a hands-free manner (for example, see Non-Patent Document 1). According to this technology, the cared person can be identified from the cared person's face image taken by the camera of the smart glasses, and the cared person's personal information can be immediately displayed on the lens. It is also possible to synthesize the cared person's personal information into voice and reproduce the voice from a bone-electric speaker to the caregiver's hearing. Therefore, the cared person does not need to wear a wireless tag or marker for personal recognition.

FIG. 1 is an explanatory diagram showing the field of view through smart glasses.

For example, assume that a caregiver is wearing smart glasses 1 at a nursing care site. At this time, the caregiver (wearer) can see the cared person in the field of view through the lenses of the smart glasses 1, and at the same time see the cared person's "personal information" reflected on the lenses. As a result, the caregiver can proceed with care work while visually reading or audibly hearing the cared person's personal information hands-free.

KDDI Research Institute/Zenkokai News Release “KDDI Research Institute and Zenkokai develop hands-free nursing care work support system using AR glasses” (February 2, 2021), [online], [Reiwa 3 Searched on September 17], Internet <URL: https://www.kddi-research.jp/newsrelease/2021/020201.html> “A story about the unit of internal parameters (focal length) ~ Conversion between pixel and mm ~”, [online], [Retrieved September 17, 2021], Internet <URL: https://mem-archive.com/ 2018/02/25/post-201/＞ “Survey on technology for restoring 3D data from images: Principles of calculating 3D coordinates ~Estimation of epipolar geometry & camera pose~”, [online], [Retrieved September 17, 2021], Internet <URL: https://qiita.com/akaiteto/items/f5857c7774794a6e5f5e＞

According to the technology described in Non-Patent Document 1 mentioned above, when the personal information of the cared person reflected on the smart glasses is reproduced by voice, the characters of the personal information are directly synthesized into speech. That is, the longer the number of characters in the personal information, the longer the audio reproduction time will be, and the shorter the number of characters in the personal information, the shorter the audio reproduction time.

In contrast, the inventors of the present application have proposed that, depending on the surroundings of a mobile device equipped with a camera, there may be cases where, among multiple objects reflected in the video, you want to hear only the outline of the object in the shortest possible time. I thought that there might be. In other words, we thought that there may be situations in which the wearer wants to hear information about objects in their field of vision clearly, and there may be situations in which they want to hear the information in the shortest possible time.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a portable device, a program, and a method for controlling speech based on speech synthesis in conjunction with the user's surrounding situation.

According to the present invention, in a mobile device having a camera that captures images and a speaker that outputs audio,
Presentation information storage means that associates and stores sentence presentation information for each object ID (identifier);
Object recognition means for recognizing an image area of the object from the video, detecting the size of the image area, and identifying an object ID from the image area;
When the size of the image area is larger than the predetermined range, the amount of reading information that affects the length of audio playback for the sentence of the presentation information corresponding to the object ID is set to be lower than when the size of the image area is less than the predetermined range. A reading information amount control means for controlling the amount of information to be read out so as to reduce the amount of information to be read out;
The present invention is characterized by having a voice synthesis means for voice-synthesizing the presentation information corresponding to the object ID so as to read it out based on the amount of read-out information, and outputting it from a speaker.

According to another embodiment of the mobile device of the present invention,
The object ID is a personal ID,
The information presented is personal information,
It is also preferable that the image area in the object recognition means is a face area.

According to another embodiment of the mobile device of the present invention,
It is also preferable that the reading information amount control means sets the reading information amount to the speaking speed.

According to another embodiment of the mobile device of the present invention,
The reading information amount control means is configured to control the amount of reading information as follows:
If the size of the image area is below the predetermined range, select the normal speaking speed,
When the size of the image area is larger than a predetermined range, it is also preferable to select a speaking speed faster than the normal speaking speed.

According to another embodiment of the mobile device of the present invention,
The presentation information storage means stores in advance a normal sentence and a short sentence shorter than the normal sentence in association with each other as presentation information,
The reading information amount control means is configured to control the amount of reading information as follows:
If the size of the image area is below the specified range, select the normal sentence,
When the size of the image area is larger than a predetermined range, it is also preferable to select a short sentence.

According to another embodiment of the mobile device of the present invention,
The presentation information storage means stores in advance normal sentences in association with each other as presentation information,
The reading information amount control means is configured to control the amount of reading information as follows:
If the size of the image area is below the specified range, select the normal sentence,
When the size of the image area is larger than a predetermined range, it is also preferable to select only the usages and proverbs in the normal sentence.

According to another embodiment of the mobile device of the present invention,
The predetermined range is based on the distance to the subject; the farther the distance to the subject, the smaller the size of the image area, and the closer the distance to the subject, the larger the size of the image area. It is also preferable.

According to another embodiment of the mobile device of the present invention,
The camera is integrally mounted to capture images of the user's field of view,
It is also preferable that the mobile device is see-through smart glasses or a smartphone.

According to another embodiment of the mobile device of the present invention,
The speaker may be a bone conduction speaker.

According to the present invention, in a program that causes a computer installed in a mobile device having a camera that captures images and a speaker that outputs audio to function,
Presentation information storage means that associates and stores sentence presentation information for each object ID (identifier);
Object recognition means for recognizing an image area of the object from the video, detecting the size of the image area, and identifying an object ID from the image area;
When the size of the image area is larger than the predetermined range, the amount of reading information that affects the length of audio playback for the sentence of the presentation information corresponding to the object ID is set to be lower than when the size of the image area is less than the predetermined range. A reading information amount control means for controlling the amount of information to be read out so as to reduce the amount of information to be read out;
The present invention is characterized in that the computer functions as a voice synthesizing means that synthesizes the presentation information corresponding to the object ID to be read aloud based on the amount of readout information and outputs it from a speaker.

According to the present invention, in an information presentation method for a mobile device having a camera that captures images and a speaker that outputs audio,
The mobile device is
Sentence presentation information is associated and stored in advance for each object ID (identifier),
a first step of recognizing an image area of the object from the video, detecting the size of the image area, and identifying an object ID from the image area;
When the size of the image area is larger than the predetermined range, the amount of reading information that affects the length of audio playback for the sentence of the presentation information corresponding to the object ID is set to be lower than when the size of the image area is less than the predetermined range. a second step of controlling to reduce
The present invention is characterized by performing a third step of synthesizing the presentation information corresponding to the object ID to be read aloud based on the amount of information to be read out and outputting it from a speaker.

According to the portable device, program, and method of the present invention, it is possible to control speech based on speech synthesis in conjunction with the user's surrounding situation.

It is an explanatory diagram showing visibility through smart glasses. FIG. 2 is a functional configuration diagram of smart glasses according to the present invention. FIG. 3 is an explanatory diagram showing normal sentences/short sentences in the presentation information storage unit of the present invention. It is a flowchart in the reading information amount control unit of the present invention. It is a functional block diagram in a speech synthesis part. FIG. 3 is an explanatory diagram showing speech synthesis when the distance between the wearer and the other person is long. FIG. 3 is an explanatory diagram showing speech synthesis when the distance between the wearer and the other person is short. FIG. 2 is an explanatory diagram showing a usage pattern of a smartphone according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail using the drawings.

FIG. 2 is a functional configuration diagram of smart glasses according to the present invention.

The smart glasses 1 are of a see-through type that can be worn in the user's field of vision.
According to FIG. 2, the smart glasses 1 are equipped with, as hardware, a display 101 that projects information on lenses, a camera 102 that photographs a person or an object, and a speaker 103 that outputs audio to the wearer.
The smart glasses 1 also include a presentation information storage section 100, an object recognition section 11, a reading information amount control section 13, and a speech synthesis section 14 as software. These functional components are realized by executing a program that causes a computer installed in the smart glasses to function.

[Display 101]
The display 101 can project the presented information on the lens through which the wearer of the smart glasses 1 sees. This allows the wearer to visually recognize the presented information. The presentation information is generally text, but may of course be images or videos.

[Camera 102]
The camera 102 may be configured integrally with the smart glasses 1 or may be connected to the outside of the smart glasses 1. The camera 102 captures an image of the wearer's field of view. The video may include a person (for example, a care recipient) or an object (for example, facility equipment). The camera 102 always outputs captured images to the object recognition unit 11.

[Speaker 103]
The speaker 103 outputs an audio signal to the hearing of the person wearing the smart glasses 1 . The audio signal may be obtained by converting the characters to be displayed on the display 101 by performing speech synthesis.
Further, the speaker 103 may be a bone conduction speaker, for example. By vibrating in contact with the wearer's temple, the wearer can aurally recognize the voice without being audible to others.

[Presentation information storage unit 100]
The presentation information storage unit 100 stores presentation information in advance in association with each object ID (identifier).
The object ID may be a personal ID, and the presentation information may be personal information. For example, when assuming a nursing care site, the personal ID may be an ID assigned to each cared person, and the presentation information may be nursing care information of the cared person.

Further, the presentation information storage unit 100 may store personal information for each individual ID in advance in a memory device (stand-alone type), or may download personal information by accessing a server via a network (server-client type).

FIG. 3 is an explanatory diagram showing normal sentences/short sentences in the presentation information storage section of the present invention.

According to FIG. 3, in the presentation information storage unit 100, a plurality of pieces of personal information are associated with each individual ID. Furthermore, "regular sentences" are associated with presentation information for combinations of multiple pieces of personal information.
According to FIG. 3, personal ID 0001 is associated with the following "regular sentences" for personal information.
<Personal ID=0001>
Injury check: Performed Check for pain: Performed Check walking condition: Performed Results: No particular problems Complaints by the person: None <Normal sentence> ``We performed an injury check and confirmed the presence of pain and confirmed the walking condition.Results There were no particular problems and no complaints were made against him."
Furthermore, in addition to regular sentences, "short sentences" such as the following may be associated.
<Short sentence>"There were no injuries, pain, or problems with walking condition. There were no complaints by the patient."

In addition, as another embodiment, the personal information of the cared person may include, for example, the following. A text serving as presentation information may be created from this personal information.
(basic information)
Name, age, level of care, date of birth, room number, care plan, etc.
(behavior information)
Wake up time, meals (done/not), hydration (done/not), medication (done/not), excretion (done/not), bathing (done/not), etc.
(Transfer information)
Health status, body temperature, blood pressure (upper/lower), pulse rate, treatment information, symptoms, treatment information, number of days without defecation, time of defecation, abnormal excretion (presence/absence), breakfast/noon/nighttime meal times, breakfast/noon/nighttime meal amount , time of hydration, amount of water, time of taking medication, time of bathing, abnormal bathing (presence/absence), etc.

[Object recognition unit 11]
The object recognition unit 11 recognizes a target object from an image captured by the camera 102.
The object recognition unit 11 recognizes image areas of one or more objects from a video, detects the size of each image area, and identifies an object ID from each image area. The object recognition unit 11 is a general machine learning engine applied to image classification and face recognition based on deep learning. This recognizes specific classes (categories such as people and objects) that appear in the video.
Note that the image region may be a face image of a person, and the recognition target may be identification of the person.

The object recognition unit 11 has an <image area detection function> that detects an image area of an object, and an <object detection function> that detects an object in the image area.

<Image area detection function>
The image area detection function detects an image area of a target object from a video. This cuts out an object region (for example, a bounding box) from an image of a video frame. Specifically, R-CNN (Regions with Convolutional Neural Networks) and SSD (Single Shot Multibox Detector) are used.
R-CNN combines rectangular object regions with convolutional neural network features to detect a subset of object regions (region proposal). Next, CNN features are extracted from the region proposal. Then, the bounding box of the region proposal is adjusted using a support vector machine trained in advance using CNN features.
SSD is a general object detection algorithm using machine learning, and determines rectangular frames (bounding boxes) called default boxes. A large number of default boxes of different sizes are superimposed on one image, and a predicted value is calculated for each box. For each default box, it is possible to predict the position of the box, such as how far it is from the object and how different its size will be.

<Object detection function>
The object detection function detects a target object from an image area. The target object may be a human body or a face, or may be an object such as equipment.
Since the embodiment of the present invention assumes a nursing care site, the object detection function recognizes a person's face area from the video and identifies the individual ID from the face area.
The object detection function stores in advance feature amounts of facial images of actual individuals to be identified. For example, the feature amount of the face image of the cared person is accumulated.
Then, the object detection function converts the face region, which is the cut out image region, into a 128/256/512-dimensional feature amount (Euclidean distance) using a face recognition model. Specifically, Google (registered trademark)'s Facenet (registered trademark) algorithm can also be used as a face recognition model. With this, it is possible to convert the face region into a feature amount of a multidimensional vector.
Then, the object recognition unit 11 collates the feature amount with the set of facial features of the individual, and identifies the individual ID that has the shortest distance or the feature amount that is less than or equal to a predetermined threshold.

The object recognition unit 11 has a lightweight learning model so that it can be implemented in a small IoT device. The amount of calculation has been significantly reduced by selecting the middle layer features of the trained model (deleting layers) and replacing it with a lightweight architecture. Specifically, using the C++ Native library, even a small device like smart glasses can identify 10,000 people in one second.

Here, the relationship between the size of the image area in which the target object appears and the distance between the target object and the target object will be explained.
When the target object is a human face, the width of the face is approximately 15 cm, so the distance between the person and the person can be estimated from the number of pixels in the width of the image area of the recognized face (for example, (See Patent Document 2). Depending on the camera, it may be assumed that 10 pixels equals 1 cm, for example. Of course, this method involves measuring the size of a specific object, photographing the object from a specific distance, and then calculating the ratio of size and distance in a fixed manner from the size of the image area (number of pixels). There may be.
Note that the number of pixels in the image area is directly proportional to the resolution of the camera. For this purpose, the number of pixels in the image area can be determined according to the distance according to the resolution of the camera.
Furthermore, as another embodiment, there is a technique for estimating the distance using a plurality of images of video from a camera without measuring the size of a specific object (for example, see Non-Patent Document 3). In this case, three-dimensional coordinates can be calculated from a plurality of monocular camera images, and the Z axis can also be estimated as the distance from the camera to the object.

[Reading information amount control unit 13]
The reading information amount control unit 13 controls the “reading information amount” in the presentation information corresponding to the object ID based on the size of the image area.
The amount of information to be read out affects the "speech playback time" of a sentence or sentence of presentation information by speech synthesis. That is, the audio playback time is controlled by controlling the amount of reading information.
The amount of reading information can be controlled using one of the following three patterns.
<Set the amount of reading information to normal><Reduce the amount of reading information>
(Pattern 1) Play normal sentences at normal speaking speed Play normal sentences at fast speaking speed (Pattern 2) Play normal sentences Play short sentences (Pattern 3) Play normal sentences Play only the idioms and nominals of normal sentences

FIG. 4 is a flowchart of the reading information amount control section of the present invention.

The reading information amount control unit 13 determines whether or not the size of the image area of the object ID recognized by the object recognition unit 11 is less than or equal to a “predetermined range”.
The predetermined range is based on the distance to the subject. The farther the distance to the subject, the smaller the size of the image area, and the shorter the distance to the subject, the larger the size of the image area.

FIG. 5 is an explanatory diagram illustrating speech synthesis when the distance between the wearer and the other person is long.
FIG. 6 is an explanatory diagram illustrating speech synthesis when the distance between the wearer and the other person is short.

According to FIGS. 5 and 6, for example, at a nursing care site, the caregiver (wearer) sees the cared person "Mr. A" in the field of view wearing the smart glasses 1. Furthermore, the object recognition unit 11 has already identified the individual ID of the care recipient "Mr. A" from the face area using the object detection function.

<S1: When the size of the image area is below the predetermined range>
According to FIG. 5, the read-aloud information amount control unit 13 determines that the distance between the cared person "Mr. A" and the cared person is long. In such a case, playback is performed with the reading information amount set to "normal". Specifically, "normal sentences" are played back at "normal speech speed."
As a caregiver, you have plenty of time until you can meet someone or an object that is relatively far away. To this end, the wearer should be able to clearly hear information about other people, etc. that is reflected in the field of view.

<S2: When the size of the image area is larger than the predetermined range>
According to FIG. 6, the readout information amount control unit 13 determines that the distance between the cared person "Mr. A" and the cared person is short. In such a case, playback is performed to "reduce" the amount of information to be read aloud. Specifically, playback is performed in any of S21 to S23 below.

(S21) Select a speaking speed faster than the normal speaking speed Even if it is a normal sentence, by playing the voice at a faster speaking speed, the wearer can hear it in a short time. However, the fastest speed is about 1.5 times, and if it is faster than that, it becomes difficult to hear.

(S22) Reproducing short sentences relative to normal sentences The presentation information storage unit 100 stores in advance a normal sentence and a short sentence shorter than the normal sentence in association with each other as presentation information. Even if the speaking speed is normal, the wearer can listen to the short sentences in a short time by playing back the short sentences.

(S23) Reproducing only the idioms and nominals of normal sentences "Uses" are verbs, adjectives, and adjectival verbs, and are words that can be used as predicates. In addition, ``noun'' refers to words that are nouns and can become subjects by adding ``ga'', ``wa'', etc. A word string consisting only of idioms and nominal pronunciations is extracted from normal sentences recorded as presentation information in the presentation information storage unit 100. Even if the speaking speed is normal, the wearer can hear the word string in a short time by playing back the word string.

As a caregiver, it is necessary to immediately meet face-to-face and start a conversation with a person or object that is relatively close, and there is no time to spare. To this end, the wearer is made to listen to information about other people, etc. reflected in the field of vision in as short a time as possible.

Specifically, the distance depending on the predetermined range of the size of the image area may be, for example, about 4 m. Of course, it differs from person to person, but a person's stride length is generally considered to be 0.45 x height. Therefore, the stride length of a person who is 170 cm tall is approximately 77 cm. In other words, a distance of about 4 meters means about 5 steps ahead. In this case, since the person who is closer than 5 steps away will be facing the person immediately, the wearer is made to listen to the person's personal information in the shortest possible time.

[Speech synthesis unit 14]
The speech synthesis unit 14 synthesizes speech to read the presentation information corresponding to the object ID based on the amount of information to be read, and outputs the result from the speaker 103.

FIG. 7 is a functional configuration diagram of the speech synthesis section.
According to FIG. 7, it is a general speech synthesis configuration. In the case of text presentation information, it can be output as voice from the speaker 103 by voice synthesis. For voice synthesis, we use a lightweight Japanese voice reading function that is installed on a single microcomputer board for IoT/embedded devices.

FIG. 8 is an explanatory diagram showing how the smartphone is used in the present invention.

According to the embodiments described above, the portable device of the present invention has been described as a see-through type smart glass, but is not limited thereto, and may be a portable terminal such as a smartphone. In the case of smartphones, the camera is integrated and is also connected to external speakers such as earphones.

As described above in detail, according to the portable device, program, and method of the present invention, it is possible to control speech based on speech synthesis in conjunction with the user's surrounding situation.
According to the present invention, it is possible to dynamically change the amount of information read out based on voice synthesis, depending on the surrounding situation of the mobile device and the distance between the person and the target object shown in the camera.

For example, in the case of a nursing care site, the personal information of the cared person can be reproduced as audio for a length of time that the carer can afford, depending on the distance between the carer and the cared person. Specifically, as a caregiver, it is necessary to immediately meet face-to-face and start a conversation with a person or object that is relatively close, and there is no time to spare. To this end, the wearer is made to listen to information about other people, etc. reflected in the field of vision in as short a time as possible.

Furthermore, this will allow, for example, ``caregivers at nursing care sites to play back personal information of cared for an appropriate length of time by wearing smart glasses or smartphones.'' It will be possible to contribute to Goal 3 of the Sustainable Development Goals (SDGs), ``Ensure healthy lives and promote well-being for all people of all ages.''

Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The invention is limited only by the claims and their equivalents.

100 Presentation information storage section 101 Display 102 Camera 103 Speaker 11 Object recognition section 13 Reading information amount control section 14 Speech synthesis section

Claims (11)

In a mobile device that has a camera that captures video and a speaker that outputs audio,
a presentation information storage means that associates and stores sentence presentation information for each object ID (identifier);
Object recognition means for recognizing an image area of the object from the video, detecting the size of the image area, and identifying an object ID from the image area;
When the size of the image area is larger than a predetermined range, the amount of reading information that affects the length of audio playback for sentences of presentation information corresponding to the object ID is set to be lower than when the size of the image area is smaller than the predetermined range. A reading information amount control means for controlling the amount of information to be read out so as to reduce the amount of information to be read out;
1. A portable device comprising a voice synthesizing means for voice-synthesizing presentation information corresponding to an object ID so as to read it out based on an amount of read-out information, and outputting the synthesized sound from a speaker. The object ID is a personal ID,
The information presented is personal information,
2. The portable device according to claim 1, wherein the image area in the object recognition means is a face area. 3. The portable device according to claim 1, wherein the reading information amount control means sets the reading information amount to a speaking speed. The reading information amount control means is configured to control the amount of reading information as follows:
If the size of the image area is below the predetermined range, select the normal speaking speed,
4. The portable device according to claim 3, wherein when the size of the image area is larger than a predetermined range, a speech speed faster than a normal speech speed is selected. The presentation information storage means stores in advance a normal sentence and a short sentence shorter than the normal sentence in association with each other as presentation information,
The reading information amount control means is configured to control the amount of reading information as follows:
If the size of the image area is below the specified range, select the normal sentence,
3. The portable device according to claim 1, wherein a short sentence is selected when the size of the image area is larger than a predetermined range. The presentation information storage means stores in advance normal sentences in association with each other as presentation information,
The reading information amount control means is configured to control the amount of reading information as follows:
If the size of the image area is below the specified range, select the normal sentence,
3. The portable device according to claim 1, wherein when the size of the image area is larger than a predetermined range, only the idioms and proverbs in the normal sentence are selected. The predetermined range is based on the distance to the subject; the farther the distance to the subject, the smaller the size of the image area, and the closer the distance to the subject, the larger the size of the image area. The portable device according to any one of claims 4 to 6. The camera is integrally mounted to capture images of the user's field of view,
The portable device according to any one of claims 1 to 7, wherein the portable device is a see-through smart glass or a smartphone. 9. The portable device according to claim 8, wherein the speaker is a bone conduction speaker. In a program that operates a computer installed in a mobile device that has a camera that captures images and a speaker that outputs audio,
Presentation information storage means that associates and stores sentence presentation information for each object ID (identifier);
Object recognition means for recognizing an image area of the object from the video, detecting the size of the image area, and identifying an object ID from the image area;
When the size of the image area is larger than the predetermined range, the amount of reading information that affects the length of audio playback for the sentence of the presentation information corresponding to the object ID is set to be lower than when the size of the image area is less than the predetermined range. A reading information amount control means for controlling the amount of information to be read out so as to reduce the amount of information to be read out;
A program that causes a computer to function as a voice synthesis means that synthesizes presentation information corresponding to an object ID to read out loud based on the amount of information to be read out, and outputs it from a speaker. In an information presentation method for a mobile device having a camera that captures video and a speaker that outputs audio,
The mobile device is
Sentence presentation information is associated and stored in advance for each object ID (identifier),
a first step of recognizing an image area of the object from the video, detecting the size of the image area, and identifying an object ID from the image area;
When the size of the image area is larger than the predetermined range, the amount of reading information that affects the length of audio playback for the sentence of the presentation information corresponding to the object ID is set to be lower than when the size of the image area is less than the predetermined range. a second step of controlling to reduce
An information presentation method characterized by performing a third step of synthesizing presentation information corresponding to an object ID to read out aloud based on an amount of readout information and outputting the result from a speaker.

Images