JP7390891B2 - Client device, server, program, and information processing method - Google Patents

Description

The present invention relates to a client device, a server, and a program.

Generally speaking, audio guidance is known as a method for assisting the actions of visually impaired people.
For example, Japanese Patent Application Laid-Open No. 2004-016578 discloses that by measuring the distance between a transmitter installed at a destination and a receiver carried by a visually impaired person, the distance from the current location to the destination can be determined by voice. A technology has been disclosed to notify the user.

However, in Japanese Patent Laid-Open No. 2004-016578, visually impaired people cannot receive voice guidance in places where transmitters are not installed. Therefore, the range in which visually impaired people can receive audio guidance is limited. As a result, the range in which visually impaired people can move with peace of mind becomes narrower.

An object of the present invention is to eliminate restrictions on the range in which visually impaired people can receive audio guidance.

One aspect of the present invention is
A client device connected to a server capable of generating audio output data regarding an object included in an image, the client device comprising:
comprising means for acquiring image data of an image including at least one attachment worn on a user's finger and at least one object;
comprising means for transmitting the image data to the server,
comprising means for receiving audio output data from the server for outputting audio related to the object included in the image;
comprising means for outputting audio based on the audio output data;
It is a client device.

According to the present invention, it is possible to eliminate restrictions on the range in which a visually impaired person can receive audio guidance.

FIG. 1 is a schematic diagram of an information processing system according to the present embodiment. 2 is a block diagram showing the configuration of the information processing system in FIG. 1. FIG. 2 is a diagram showing the configuration of a camera unit 50 in FIG. 1. FIG. FIG. 2 is a diagram showing the configuration of the nail cap shown in FIG. 1. FIG. It is an explanatory diagram of an outline of this embodiment. It is a sequence diagram of information processing of this embodiment. 7 is an explanatory diagram of S500 in FIG. 6. FIG. 7 is an explanatory diagram of S100 in FIG. 6. FIG. 7 is an explanatory diagram of S502 in FIG. 6. FIG. It is a figure which shows the data structure of the gesture database of a modification. FIG. 7 is a sequence diagram of information processing in a modified example. It is a figure which shows an example of the gesture of a modification.

Hereinafter, one embodiment of the present invention will be described in detail based on the drawings. In addition, in the drawings for explaining the embodiments, the same components are generally designated by the same reference numerals, and repeated explanations thereof will be omitted.

(1) Configuration of information processing system The configuration of the information processing system will be explained. FIG. 1 is a schematic diagram of an information processing system according to this embodiment. FIG. 2 is a block diagram showing the configuration of the information processing system in FIG. 1.

As shown in FIG. 1, the information processing system 1 includes a client device 10, a server 30, and a camera unit 50.
The client device 10 and the server 30 are connected via a network (eg, the Internet or an intranet) NW.
The client device 10 and camera unit 50 are connected via wireless communication.

The client device 10 is an example of an information processing device that sends a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. Server 30 is, for example, a web server.

The camera unit 50 is configured to capture an image and generate image data of the captured image.

A user (for example, a visually impaired person) accesses the server 30 by wearing a nail cap NC (an example of a "wearing item") on his or her finger and carrying the client device 10 with him or her.

(1-1) Configuration of Client Device The configuration of the client device 10 will be explained.

As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input/output interface 13, and a communication interface 14.

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of ROM (Read Only Memory), RAM (Random Access Memory), and storage (for example, flash memory or hard disk).

The programs include, for example, the following programs.
・OS (Operating System) program ・Application program that executes information processing (e.g. web browser)

The data includes, for example, the following data.
・Databases referenced in information processing ・Data obtained by executing information processing (that is, execution results of information processing)

The processor 12 is configured to implement the functions of the client device 10 by activating a program stored in the storage device 11. Processor 12 is an example of a computer.

The input/output interface 13 is configured to obtain user instructions from an input device connected to the client device 10 and output information to an output device connected to the client device 10 .
The input device is, for example, a keyboard, pointing device, touch panel, microphone, or a combination thereof.
The output device is, for example, a display, a speaker, or a combination thereof.

Communication interface 14 is configured to control communication between client device 10 and server 30.

(1-2) Server Configuration The configuration of the server 30 will be explained.

As shown in FIG. 2, the server 30 includes a storage device 31, a processor 32, an input/output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of ROM, RAM, and storage (eg, flash memory or hard disk).

The programs include, for example, the following programs.
・OS program ・Application program that executes information processing ・Learning data set regarding the relationship between image object features and language (e.g. object name)

The data includes, for example, the following data.
・Databases referenced in information processing ・Execution results of information processing

The processor 32 is configured to implement the functions of the server 30 by activating a program stored in the storage device 31. Processor 32 is an example of a computer.

The input/output interface 33 is configured to obtain user instructions from an input device connected to the server 30 and output information to an output device connected to the server 30.
The input device is, for example, a keyboard, pointing device, touch panel, or a combination thereof.
The output device is, for example, a display.

Communication interface 34 is configured to control communication between server 30 and client device 10 .

(1-3) Configuration of camera unit The configuration of the camera unit 50 will be explained. FIG. 3 is a diagram showing the configuration of the camera unit 50 of FIG. 1.

FIG. 3A is a front view of the camera unit 50. FIG. 3B is a top view of the camera unit 50. FIG. 3C is a side view of the camera unit 50.

As shown in FIG. 3, the camera unit 50 includes a lens 50a, a speaker 50b, a clip 50c, an image sensor 50d, and a camera controller 50e.

As shown in FIGS. 3A to 3C, the lens 50a is arranged on the front surface (Z-side surface) of the camera unit 50.

As shown in FIGS. 3B to 3C, the speaker 50b is arranged on the top surface (Y+ side) of the camera unit 50.

The clip 50c is arranged on the back surface (Z+ side surface) of the camera unit 50. In other words, the clip 50c is placed on the opposite surface from the lens 50a.
By hooking the clip 50c onto his or her clothing, the user can wear the camera unit 50 so that the lens 50a faces in front of the user (that is, the direction of the user's line of sight).

As shown in FIGS. 3A and 3C, the image sensor 50d is arranged inside the camera unit 50. The light that has passed through the lens 50a forms an image on the image sensor 50d. The image sensor 50d is configured to generate image data based on the light that has passed through the lens 50a by converting the imaged light into an electrical signal.

The camera controller 50e is arranged inside the camera unit 50. The camera controller 50e is a processor that controls the entire camera unit 50.

(1-4) Structure of Nail Cap The structure of the nail cap NC will be explained. FIG. 4 is a diagram showing the structure of the nail cap shown in FIG. 1.

As shown in FIG. 4A, the nail cap NC includes five right-hand nail caps NCR and five left-hand nail caps (that is, 10 nail caps) NCL. Different patterns (for example, texts "L1" to "L5" and "R1" to "R5") are formed on the ten nail caps NCL and NCR. Each nail cap NC is distinguished by the pattern formed on the nail cap NC.

As shown in FIG. 4B, each nail cap NC can be attached to a user's finger.

(2) Overview of this embodiment An overview of this embodiment will be explained. FIG. 5 is an explanatory diagram outlining the present embodiment.

As shown in FIG. 5, when the nail cap NC attached to the nail of a user (for example, a visually impaired person) enters the imaging range, the client device 10 detects an object OBJ1 (apple) and an object OBJ2 around the nail cap NC. Image data of an image IMG containing (banana) is acquired.
The client device 10 transmits the acquired image data to the server 30.

The server 30 determines the position of the nail cap NC and the positions of the objects OBJ1 to OBJ2 in the image IMG by performing image analysis on the image data transmitted from the client device 10.
Based on the specified position, the server 30 specifies the object OBJ1 closest to the nail cap NC among the objects OBJ1 to OBJ2.
The server 30 refers to the learning data set stored in the storage device 31 and estimates the object name (that is, apple) based on the feature amount of the identified object OBJ1.
The server 30 generates audio output data for outputting audio of the estimated object name.
The server 30 transmits the generated audio output data to the client device 10.

The client device 10 outputs the voice "apple" based on the voice output data transmitted from the server 30.

The user can learn the object name "apple" of the object OBJ1 closest to his or her finger from the voice output by the client device 10.

In this way, the user (for example, a visually impaired person) can receive audio guidance using the finger on which the nail cap NC is attached. In other words, it is possible to eliminate restrictions on the range in which visually impaired people can receive audio guidance.

(3) Information processing Information processing of this embodiment will be explained. FIG. 6 is a sequence diagram of information processing according to this embodiment. FIG. 7 is an explanatory diagram of S500 in FIG. 6. FIG. 8 is an explanatory diagram of S100 in FIG. 6. FIG. 9 is an explanatory diagram of S502 in FIG. 6.

The camera unit 50 performs imaging (S500).
Specifically, the image sensor 50d generates image data corresponding to the light that has passed through the lens 50a by converting the image of the light that has passed through the lens 50a into an electrical signal (FIG. 7A).
The camera controller 50e transmits image data generated by the image sensor 50d to the client device 10.

After step S500, the client device 10 executes an image analysis request (S100).
Specifically, the processor 12 determines whether the image IMG corresponding to the image data transmitted in step S500 includes the pattern formed on the nail cap NC. As an example, as shown in FIG. 8, when the user performs a gesture of raising the thumb of the right hand within the field of view of the lens 50a, the image data transmitted in step S500 is displayed on the nail attached to the thumb of the right hand. Contains an image of the pattern of the cap NC (eg, text "R1"). In this case, the processor 12 determines that the pattern formed on the nail cap NC is included.
When the processor 12 determines that the pattern formed on the nail cap NC is included, the processor 12 transmits image analysis request data to the server 30.
The image analysis request data includes image data of an image IMG including a pattern formed on the nail cap NC.

After step S100, the server 30 performs image analysis (S300).
Specifically, the processor 32 applies feature analysis to the image data included in the image analysis request data, thereby analyzing the following objects included in the image IMG (pixels of the nail cap NC and object OBJ1). ~ OBJ2 pixel) Specify the coordinates.
Based on the identified coordinates, the processor 32 identifies the object OBJ1 closest to the nail cap NC among the objects OBJ1 to OBJ2 included in the image IMG.

After step S300, the server 30 performs object estimation (S301).
Specifically, the processor 32 refers to the learning data set stored in the storage device 31 and estimates the object name corresponding to the feature amount of the pixel of the object OBJ1 identified in step S300.

After step S302, the server 30 executes text data generation (S302).
Specifically, the processor 32 generates text data of a sentence (for example, "This is an apple.") composed of the object name estimated in step S301 and a predetermined phrase (for example, a subject and a predicate). generate.

After step S302, the server 30 executes an image analysis response (S303).
Specifically, the processor 32 transmits image analysis response data to the client device 10.
The image analysis response data includes the text data generated in step S302.

After step S303, the client device 10 executes audio data generation (S101).
Specifically, the processor 12 converts text data included in the image analysis response data into audio data corresponding to the text data.
Processor 12 transmits the converted audio data to camera unit 50.

After step S101, the camera unit 50 performs audio output (S502).
Specifically, the camera controller 50e reproduces the audio corresponding to the audio data transmitted in step S101.
As shown in FIG. 9, the speaker 50b outputs the reproduced sound (for example, "This is an apple.").

According to this embodiment, the name of the object OBJ1 closest to the nail cap NC is read out in the image captured in step S500. A user (for example, a visually impaired person) can recognize the object OBJ1 closest to the nail cap NC through the sound output from the speaker 50b.

In particular, the thumb nail is the largest, and the act of holding the thumb up gives a positive impression, so when a pattern (for example, text "R1") formed on the thumb nail cap NC is recognized, step S100 is performed. It is preferable to do so.

(4) Modification Example A modification example will be explained. A modified example is an example in which information processing is performed according to a user's gesture.

(4-1) Database The database of the modified example will be explained. FIG. 10 is a diagram showing a data structure of a gesture database according to a modified example.

As shown in FIG. 10, the gesture information database of the modified example includes a "pattern" field, a "gesture" field, and an "action" field. Each field is associated with each other.

The "pattern" field stores information that identifies the pattern formed on the nail cap NC.

The "gesture" field stores information (for example, motion vector) regarding displacement of the position of the nail cap NC per unit time. When the motion vector is 0, it means that the nail cap NC is stationary. If the motion vector is not 0, it means the direction and speed of the movement of the nail cap NC.

The "action" field stores information regarding the content of information processing executed by the processor 12. Information processing includes, for example, the following.
- Sending the image sent from the camera unit 50 in step S500 to the server 30 - Sending the image sent from the camera unit 50 in step S500 to the server 30, and sending an email with the image attached - Step Sending an email to which the image sent from the camera unit 50 is attached in S500 - Storing the image sent from the camera unit 50 in step S500 in the storage device 11 (that is, saving the image)
・Start the specified application

(4-2) Information Processing Information processing in a modified example will be explained. FIG. 11 is a sequence diagram of information processing in a modified example. FIG. 12 is a diagram illustrating an example of a modified gesture.

As shown in FIG. 11, after step S500 (FIG. 6), the client device 10 executes gesture identification (S110).
Specifically, the processor 12 identifies the motion vector of the pattern formed on the nail cap NC included in the image IMG corresponding to the image data transmitted in step S500.

As an example, as shown in FIG. 12A, when the user makes a gesture of holding up the index finger and middle finger of the right hand and holding them still within the range of the angle of view of the lens 50a, the image data transmitted in step S500 is and an image of a pattern (for example, text "R2" and "R3") formed on the nail cap NC of the middle finger, and the motion vector is 0. In this case, the processor 12 determines that "the pattern of the nail cap NC attached to the index finger and middle finger of the right hand is stationary."
As shown in FIG. 12B, when the user makes a gesture of holding the index finger, middle finger, and ring finger of the right hand upright within the field of view of the lens 50a, the image data transmitted in step S500 is transmitted to the right hand. It includes images of patterns (for example, text "R2" to "R4") formed on the nail caps NC of the index finger, middle finger, and ring finger, and the motion vector is 0. In this case, the processor 12 determines that "the patterns of the nail caps NC attached to the index finger, middle finger, and ring finger of the right hand are stationary."
As shown in FIG. 12C, when the user makes a gesture of raising the index and middle fingers of the right hand and moving them from top to bottom within the field of view of the lens 50a, the image data transmitted in step S500 is transferred to the right hand. It includes an image of a pattern (for example, text "R2" and "R3") formed on the nail cap NC of the index finger and middle finger, and a motion vector indicating that the image moves from top to bottom. In this case, the processor 12 determines that "the pattern of the nail cap NC attached to the index finger and middle finger of the right hand is moving from top to bottom."
As shown in FIG. 12D, when the user makes a gesture of raising the index finger, middle finger, and ring finger of the right hand and moving them from bottom to top within the field of view of the lens 50a, the image data transmitted in step S500 is an image of a pattern (for example, text "R2" to "R4") formed on the nail cap NC of the index finger, middle finger, and ring finger of the right hand, and a motion vector indicating that the image moves from bottom to top. and, including. In this case, the processor 12 determines that "the pattern of the nail cap NC attached to the index finger, middle finger, and ring finger of the right hand is moving from the bottom to the top."

After step S110, the client device 10 executes an action (S111).
Specifically, the processor 12 refers to the gesture information database (FIG. 10) and identifies information in the "action" field associated with the motion vector identified in step S110.
The processor 12 executes processing corresponding to the information in the specified "action" field.

As an example, when the gesture shown in FIG. 12A is identified in step S110, image transmission and email transmission are executed (FIG. 10).
If the gesture shown in FIG. 12B is identified in step S110, email transmission is executed (FIG. 10).
In this way, the client device 10 executes processing according to the combination of patterns recognized in S100.

As another example, if the gesture of FIG. 12C is identified in step S110, image storage is performed (FIG. 10).
If the gesture shown in FIG. 12D is identified in step S110, a predetermined application is activated (FIG. 10).
In this way, the client device 10 executes processing according to the combination of the pattern recognized in S100 and the movement of the nail cap NC.

According to the modification, the client device 10 performs an action in response to a gesture made by a finger to which the nail cap NC is attached. Thereby, the user can give user instructions to the client device 10 using only the finger wearing the nail cap NC. This is particularly useful when the user is visually impaired, since the user can give various user instructions to the client device 10 just by moving his or her fingers without looking at the display.

(5) Summary of this embodiment This embodiment will be summarized.

The first aspect of this embodiment is
A client device 10 connected to a server 30 capable of generating audio output data regarding an object included in an image, the client device 10 comprising:
A means (for example, step S100) for acquiring image data of an image IMG including at least one attachment (for example, a nail cap NC) attached to a finger of a user (for example, a visually impaired person) and at least one object. a processor 12) that executes the processing of
comprising means for transmitting image data to the server 30 (for example, the processor 12 that executes the process of step S100),
comprising means (for example, the processor 12 that executes the process of step S101) for receiving voice output data (for example, text data) from the server 30 for outputting voice regarding the object included in the image IMG;
comprising means for outputting audio based on the audio output data (for example, a processor 12 that executes the process of step S101);
This is a client device 10.

According to the first aspect, the client device 10 transmits to the server 30 image data of an image including an object OBJ and an attachment (e.g., a nail cap NC) attached to a finger of a user (e.g., a visually impaired person). , and outputs audio related to object OBJ. This makes it possible to eliminate restrictions on the range in which visually impaired people can receive audio guidance.

In particular, since image data including the nail cap NC attached to the finger is acquired, it is possible to provide audio guidance of an image closer to the user's line of sight.

The second aspect of this embodiment is
The output audio includes the name of the object,
This is a client device 10.

According to the second aspect, the user can know the name of the object near the nail cap NC through the output audio.

The third aspect of this embodiment is
A client device 10 connected to a server 30 capable of executing image analysis,
comprising means for acquiring image data (for example, a processor 12 that executes the process of step S100),
The method includes means (for example, a processor 12 that executes the process of step S100) for determining whether or not the image data includes an image of at least one attachment (for example, a nail cap NC) worn on a user's finger. ,
If the image data includes an image of the wearable object, the device includes means for transmitting the image data to the server 30 (for example, the processor 12 that executes the process of step S100),
comprising means for receiving analysis results for image data from the server 30 (for example, the processor 12 that executes the process in step S101);
This is a client device 10.

According to the third aspect, when the client device 10 recognizes an image including an attachment (for example, a nail cap NC) attached to a finger of a user (for example, a visually impaired person), the client device 10 transmits the image data of the image to a server. Send to 30. This allows the user to easily give user instructions for transmitting image data to the server 30.

The fourth aspect of this embodiment is
The acquiring means is the client device 10 that acquires image data from a camera connected to the client device 10 (for example, the camera unit 50) or a camera placed in the client device 10.

The fifth aspect of this embodiment is
The acquisition means acquires image data when the worn object is recognized.
This is a client device 10.

According to the fifth aspect, the user can give a user instruction for generating image data simply by holding the finger wearing the nail cap NC within the field of view of the lens 50a.

The sixth aspect of this embodiment is
The acquiring means acquires image data when a pattern formed on the wearable object is recognized.
This is a client device 10.

The seventh aspect of this embodiment is
comprising means (for example, a processor 12 that executes steps S110 to S111) for executing processing according to the combination of recognized patterns;
This is a client device 10.

According to the seventh aspect, the user can give a user instruction to the client device 10 by a combination of fingers reflected in the lens 50a.

The eighth aspect of this embodiment is
The executing means executes a process according to a combination of the recognized pattern and the movement of the worn object.
This is a client device 10.

According to the eighth aspect, the user can give a user instruction to the client device 10 by the combination of fingers reflected in the lens 50a and the movement of the fingers.

The ninth aspect of this embodiment is
A server 30 connected to the client device 10,
comprising means for acquiring image data from the client device 10 (for example, a processor 32 that executes the process of step S300),
comprising means for estimating the name of an object included in an image corresponding to the acquired image data (for example, a processor 32 that executes the process of step S301),
comprising means for generating audio output data for outputting audio including the estimated name of the object (for example, a processor 32 that executes the process of step S302),
comprising means for transmitting the generated audio output data to the client device 10 (for example, the processor 32 that executes the process of step S303);
This is the server 30.

(6) Other variations

The storage device 11 may be connected to the client device 10 via the network NW. The storage device 31 may be connected to the server 30 via the network NW.

Each step of the above information processing can be executed by either the client device 10 or the server 30.

The camera unit 50 may be built into the client device 10.

The pattern formed on each nail cap NC is not limited to letters. The pattern includes: In particular, by forming a highly aesthetically pleasing pattern on the nail cap NC, it is possible to motivate the user to wear the nail cap NC and use the audio guidance of this embodiment.
・Shapes, geometric patterns, unevenness, color variations

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Moreover, various improvements and changes can be made to the embodiments described above without departing from the spirit of the present invention. Furthermore, the above embodiments and modifications can be combined.

1 : Information processing system 10 : Client device 11 : Storage device 12 : Processor 13 : Input/output interface 14 : Communication interface 30 : Server 31 : Storage device 32 : Processor 33 : Input/output interface 34 : Communication interface 50 : Camera unit 50a : Lens 50b: Speaker 50c: Clip 50d: Image sensor 50e: Camera controller

Claims (16)

A client device connected to a server configured to generate audio output data regarding an object included in an image, the client device comprising:
comprising means for acquiring image data of an image including the object ,
comprising means for determining whether the image data includes an image of at least one attachment worn on a user's finger;
If an image of the wearable object is included in the image data, comprising means for transmitting the image data to the server,
comprising means for receiving audio output data regarding an object included in the image from the server;
Client device. The client device according to claim 1, wherein the acquiring means acquires the image data from a camera connected to the client device or a camera located at the client device. The transmitting means transmits the image data when the wearable object is recognized.
The client device according to claim 1 or claim 2. 4. The client device according to claim 3, wherein the transmitting means transmits the image data when recognizing a pattern formed on the wearable object. 5. The client device according to claim 1, further comprising means for executing processing according to a combination of the recognized patterns when the patterns formed on the wearable object are recognized. 6. The client device according to claim 5, wherein the executing means executes processing according to a combination of the recognized pattern and the movement of the wearable object. The acquiring means acquires image data of a plurality of attachments attached to each finger of the user,
A client device according to any one of claims 1 to 6. The acquiring means acquires image data of wearable items worn on the right and left hands of the user,
The client device according to claim 7. The acquiring means acquires image data of an attachment attached to each finger of the right hand and each finger of the left hand of the user,
The client device according to claim 8. The plurality of attachments are formed with mutually different patterns,
The acquiring means acquires image data including the pattern,
The client device according to any one of claims 7 to 9. The acquiring means acquires image data of an attachment worn on a specific finger.
The client device according to any one of claims 7 to 10. the user is visually impaired;
A client device according to any one of claims 1 to 11. means for acquiring image data; means for determining whether an image of at least one wearable item attached to a user's finger is included in the image data; and an image of the wearable item is included in the image data. If the server is connected to a client device, the server includes means for transmitting the image data to the server and means for receiving analysis results for the image data from the server ,
comprising means for acquiring the image data from the client device,
comprising means for estimating the name of an object included in an image corresponding to the acquired image data,
comprising means for generating audio output data for outputting audio including the estimated name of the object,
comprising means for transmitting the generated audio output data to the client device;
server. comprising means for specifying the position of the wearing object and the position of the object in the image,
The estimating means estimates the name of an object closest to the attached object.
The server according to claim 13. A program for causing a computer to function as each means according to any one of claims 1 to 14. An information processing method for generating audio output data regarding an object included in an image using a computer, the method comprising:
Obtaining image data of an image including a plurality of nail caps attached to a user's fingers and at least one object,
comprising the step of transmitting the image data to a server,
comprising the step of receiving audio output data from the server for outputting audio related to the object included in the image;
comprising the step of outputting audio based on the audio output data,
comprising means for, when recognizing the patterns formed on the plurality of nail caps, executing processing according to the combination of the recognized patterns ;
Information processing method.