JP7382847B2 - Information processing method, program, and information processing device - Google Patents

Description

The present disclosure relates to an information processing method, a program, and an information processing device.

BACKGROUND ART For some time now, systems that mediate individual sales have been published on electronic commerce platforms such as CtoC (Customer To Customer) marketplaces (see, for example, Patent Document 1).

Japanese Patent Application Publication No. 2001-167163

However, with conventional technology, when it is difficult for users who are not accustomed to user interfaces such as e-commerce to understand how to set up, register, etc. to enable use, only information such as explanatory text exists. I was unable to properly understand the setting method, etc.

The purpose of the present disclosure is to provide an information processing method, a program, and an information processing device that provide a mechanism that enables appropriate guidance when settings are made using a user interface from a predetermined screen. be one of the.

An information processing method according to an embodiment of the present disclosure includes one or more processors included in an information processing device controlling the display of an image being photographed by a photographing device, and other information displayed in the image. identifying the screen of the processing device; recognizing characters displayed on the identified screen; acquiring recognition results including character information of the characters; and acquiring guide information corresponding to the recognition results. and controlling the display of the guide information by associating it with corresponding character information and superimposing it on the image being photographed.

According to the disclosed technology, it is possible to provide a mechanism that allows appropriate guidance to be provided when settings are made from a predetermined screen using a user interface.

1 is a diagram illustrating each configuration example of an information processing system 1 in an embodiment. FIG. It is a block diagram showing an example of user terminal 10A concerning an embodiment. FIG. 2 is a block diagram illustrating an example of a server 20 according to an embodiment. It is a figure showing an example of object data 233 concerning an embodiment. It is a figure showing an example of character recognition data 234 concerning an embodiment. FIG. 3 is a diagram showing an example of guide storage location data 235 according to the embodiment. FIG. 2 is a sequence diagram illustrating an example of registration processing of the information processing system 1 according to the embodiment. FIG. 2 is a sequence diagram illustrating an example of display processing of the information processing system 1 according to the embodiment. It is a figure showing an example of screen transition in user terminal 10A concerning an embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the same elements are given the same reference numerals and redundant explanations will be omitted.

[Embodiment]
In the embodiment, an image being captured by an imaging device of an information processing device is displayed, a predetermined screen displayed on a display unit of another information processing device is specified, and a user sets settings for each item included in the predetermined screen. A method, program, device, system, etc. for superimposing guide information to guide the input method of each item when doing so will be explained.

In the embodiment, description will be given using smart glasses as a wearable terminal as an example of an information processing device, and using a smartphone as an example of another information processing device, but it goes without saying that the present invention is not limited to this example. For example, the information processing device may include a built-in imaging device or may be an externally attachable device, and the other information processing device may be a device that can be operated by a user from a screen.

Since the guide information regarding the method of setting the predetermined screen is superimposed on the display unit, the user can easily understand the appropriate guide. Furthermore, since the guide information is displayed in a superimposed manner, the user does not have to read a separate explanatory text.

<System application example>
FIG. 1 is a diagram illustrating each configuration example of an information processing system 1 in an embodiment. In the example shown in FIG. 1, each information processing device 10A, 10B, . . . used by a user, and a server 20 that executes object recognition processing, character recognition processing, and stores guide information. , are connected via network N. Note that the server 20 may be configured with a plurality of servers or databases, and each function may be processed by one server, or each data may be stored in one database.

The information processing device 10A is, for example, a wearable terminal (by way of example and not limitation, a glasses-type device, etc.). A wearable terminal is an electronic device worn by a user. Wearable devices include, for example, glasses-type devices (smart glasses), contact lens-type devices (smart contact lenses), head-mounted displays using augmented reality (AR) technology, artificial eyes, brain-machine interfaces, etc. It's okay. Furthermore, the wearable terminal may be a terminal that cannot be worn by the user, such as a smart speaker or a robot. In this embodiment, as described above, the case where the wearable terminal is a glasses-type terminal (smart glasses) will be described as an example. Note that the information processing device 10A is not limited to a wearable terminal, but may be an information processing terminal such as a smartphone or a tablet terminal. Further, the information processing device 10A is also referred to as the user terminal 10A hereinafter.

The information processing device 10B is, for example, a smartphone, a mobile phone (feature phone), a computer, a PDA (Personal Digital Assistant), a ticket vending machine, a delivery locker or delivery box, a television, a remote controller for home appliances, a display device including a screen, etc. . Further, the information processing device 10B may be any device that allows the user to perform operations using a predetermined screen. Further, the information processing device 10B is also referred to as the user terminal 10B hereinafter.

The information processing device 20 is, for example, a server, and may be composed of one or more devices. Further, the information processing device 20 is a server that executes object recognition processing, character recognition processing, stores guide information, and manages an electronic commerce platform.

In the example shown in FIG. 1, the user terminal 10B displays a predetermined screen, for example, a registration screen for registering as a member of an electronic commerce platform. The user terminal 10A uses a built-in or external photographing device (for example, a camera) to photograph the registration screen displayed on the screen of the user terminal 10B.

At this time, when the user performs a gesture on the displayed registration screen or instructs by voice that the registration screen is being photographed, the user terminal 10A acquires and acquires an image including the registration screen. The image is sent to the server 20.

The server 20 recognizes an object (for example, a smartphone, a ticket vending machine, etc.) from the acquired image, and acquires object identification information (object ID) for identifying the recognized object and screen position information within the image. The server 20 transmits the acquired object ID and position information to the user terminal 10B.

The user terminal 10B specifies a region for character recognition from the image based on the position information acquired from the server 20, and performs character recognition from the image of the specified region (region image). Regarding the character recognition process, the region image may be transmitted to the server 20, and the user terminal 10B may acquire the character recognition result by the server 20. The recognition results are, for example, each input item such as the screen name of the "registration screen", "name", and "email address".

Based on the recognition result, the user terminal 10B acquires guide information corresponding to the "registration screen" from the server 20, and displays the guide information in a superimposed manner on the display unit. As a result, by superimposing and displaying appropriate guide information while specifying guide information through object recognition, the actual registration screen of the user terminal 10B is displayed as is, and the user terminal 10A, which is different from the user terminal 10B, is used to You can assist with input and settings on the registration screen.

<Example of configuration>
FIG. 2 is a block diagram showing an example of the user terminal 10A according to the embodiment. User terminal 10A typically includes one or more processing units (CPUs) 110, one or more network or other communication interfaces 120, memory 130, user interface 150, imaging device 160, and components thereof. including one or more communication buses 170 for interconnecting the .

The user interface 150 is, for example, a user interface 150 that includes a display device 151 and an input device (a keyboard and/or a mouse, or some other pointing device, a microphone capable of inputting sound, etc.) 152. Further, the user interface 150 may be a touch panel. Furthermore, when the user terminal 10A is a wearable terminal 10A, the display device 151 may be a lens, the input device 152 may be a microphone, or the like.

The photographing device 160 is a device for photographing images (including still images and moving images). For example, the photographing device 160 may include a photographing element such as a CCD image sensor, a CMOS image sensor, and a lens.

Memory 130 is, for example, a high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state storage, and may also include one or more magnetic disk storage, optical disk storage, flash memory devices, or It may also be a nonvolatile memory such as another nonvolatile solid state storage device.

Further, as another example of the memory 130, one or more storage devices installed remotely from the CPU 110 may be used. In some embodiments, memory 130 stores the following programs, modules and data structures, or a subset thereof.

Operating system 131 includes, for example, procedures for handling various basic system services and performing tasks using hardware.

The network communications module 132 may, for example, connect the user terminal 10A to another computer via one or more network communications interfaces 120 and one or more communications networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc. Used to connect over a network.

The image-related data 133 is data that can be obtained in relation to image data taken during shooting. For example, the image-related data 133 includes an image ID for identifying image data, an object ID of an object within the image, screen position information, a region image, and the like. These data can be obtained in connection with object recognition from image data.

The text-related data 134 includes text data including characters extracted by character recognition from the image being photographed, and data related to text data input by the user. Further, the text-related data may be acquired by voice recognition of the user's voice data. The information may be acquired by operating the input device 152 by the user.

The memory 130 includes a module that performs display processing to display guide information, and a module that performs registration processing to register guide information. First, a module that performs display processing will be explained.

<Display processing>
The identification module 135 identifies the screen of another information processing device (for example, the user terminal 10B) that is displayed within the image being captured. For example, the identification module 135 acquires one image from among the images being photographed, and requests object recognition from the server 20. The identification module 135 acquires screen information of the recognized object (for example, screen position information within the image) from the server 20 . As a specific example, the identification module 135 may use an object recognition API published by the server 20 to obtain the object ID and position information of the recognized object.

The acquisition module 136 recognizes a character displayed within the specified screen and acquires a recognition result including character information of this character. For example, the acquisition module 136 transmits a region image cut out based on screen position information to the server 20, a character string in the region image is recognized on the server 20 side, and a recognition result including character information of the character string is generated. is obtained from the server 20. As a specific example, the acquisition module 136 may acquire character strings (text) in the area image and positional information of the character strings (for example, name, email address, etc.) using a character recognition API published by the server 20. .

The acquisition module 136 also acquires guide information corresponding to the character recognition results. For example, the acquisition module 136 transmits the character information of the recognition result to the server 20 for each item, the guide information corresponding to the character information of each item is identified on the server 20 side, and the identified guide information is acquired from the server 20. do.

The display control module 137 controls display of the image being photographed by the photographing device 160 on the display 151 (display section). Furthermore, the display control module 137 associates the acquired guide information with the corresponding item or text information, superimposes it on the image being photographed, and controls the display. For example, the display control module 137 associates the acquired guide information with the position of the corresponding character information, and superimposes the guide information using AR (Augmented Reality) technology so as to maintain the display of the input item. Control display. In order to maintain the display of input items, the display control module 137 may make the guide information transparent or display it in a superimposed manner so as not to overlap the position of the input item. Further, when guide information is displayed in a superimposed manner below, AR technology or the like may be used.

As a result, the user can easily understand appropriate guidance because the guide information regarding how to set the predetermined screen is displayed in a superimposed manner on the display unit. In addition, by superimposing appropriate guide information on the display while identifying guide information through object recognition, the predetermined screen of the user terminal 10B can be displayed as is, and the user terminal 10A, which is different from the user terminal 10B, can be used. , can assist with input and settings on a predetermined screen.

The transaction control module 138 controls buying and selling of products on the electronic commerce platform, and controls, for example, listing and purchasing procedures. Note that the guide information may be searched from guide information stored in a database in the electronic commerce platform.

The detection module 139 detects the user's gesture from the captured image. For example, the detection module 139 detects a predetermined gesture such as a tap. Furthermore, processing corresponding to the module detected by the detection module 139 may be executed.

In addition, the display control module 137 superimposes guide information and controls the display.
identifying text information corresponding to the position of the user's hand;
identifying guide information corresponding to the identified character information;
controlling the display of the identified guide information by associating it with the identified text information and superimposing it on the image being photographed;
May include.

For example, the display control module 137 includes recognizing the user's fingertip through object recognition processing and identifying character information in which the position of the recognized fingertip is within a predetermined range. For example, the display control module 137 includes obtaining and specifying guide information associated with the specified character information from the server 20. For example, the display control module 137 includes controlling the display of the specified guide information in association with the position of the specified text information and superimposing it on the image being photographed.

This makes it possible to individually display the guide information corresponding to the position specified by the user, and to display the guide information corresponding to each piece of character information according to the user's selection order.

Further, the identification module 135 identifies the screen of the information processing device (user terminal 10B), acquires the identification information of this screen and the position information of this screen, and identifies the area for character recognition based on this position information. It's okay. For example, the identification module 135 identifies the area indicated by the position information of the screen portion of the object indicated by the object ID. The position information includes position information within the image.

Further, the acquisition module 136 may acquire a recognition result including character information of a character recognized from an image within the specified area and the character. For example, the acquisition module 136 acquires from the server 20 a character recognition result including character information of an item name (character string) corresponding to each input item or setting item.

Thereby, it is sufficient to specify a character recognition area based on the recognized object and perform character recognition on an image within the specified area, so that the processing load can be reduced.

Further, the guide information may include a video related to another user's hand operation corresponding to the text information including each item name. For example, a video related to hand operation is a video that includes hand movements showing a method of inputting or setting the same item by an expert. Further, the video related to hand operation may be an actually shot two-dimensional video, a three-dimensional video (volumetric video), or an animation in which a pseudo hand moves based on an actually shot video.

Thereby, the user can easily perform input or settings on the predetermined screen while checking the actual hand operation.

Further, the warning module 140 outputs a warning when a character input for an item is recognized and the recognition result of this character does not satisfy the input condition associated with this item. For example, the warning module 140 recognizes character information input into an email address input field, and determines whether the recognition result satisfies a predetermined email address format condition. If the predetermined conditions are not met, the warning module 140 notifies the user that the input content or setting content does not meet the conditions, using a pop-up or voice. The predetermined condition is, for example, that @ is not included in the email address.

Thereby, the user is notified of a warning when inputting or setting an item in a predetermined screen, and can notice input errors or setting errors.

The display control module 137 also superimposes and controls the display of guide information, controls the display of each character information included in the recognition result, and displays the guide information corresponding to the character information selected by the user in the image being photographed. It may also include controlling the display by superimposing the image on the image. For example, the display control module 137 recognizes all character strings displayed within a predetermined screen and controls the display of list information (list) that allows selection of character information for each recognized character string. When predetermined text information is selected by the user, the display control module 137 superimposes and controls the display of guide information corresponding to the selected text information on the image being photographed. Regarding selection, selection is determined when the audio information of the voice recognized by voice recognition and text information match, or selection is determined when a tap process is performed at the position of predetermined text information. .

This allows the user to check the recognition results by displaying a list of recognized character information, and also allows the user to select character information (item name) from the list and check the guide information. I can do it.
<Registration process>

Next, a process for registering guide information will be explained. In the case of the registration process, the user is an expert and records and registers his or her hand operations as guide information in the server 20.

The display control module 137 controls the display of the image being photographed by the photographing device 160. The identification module 135 identifies the screen of another information processing device (user terminal 10B) displayed within the image. The method of specifying the screen is as described above. The acquisition module 136 recognizes a character displayed within the specified screen and acquires a recognition result including character information of this character.

Further, the photographing device 160 photographs each piece of guide information including the user's (expert) hand operation corresponding to each piece of recognized character information. The network communication module 132 transmits each character information including each recognized item name and each guide information corresponding to each item name to the server 20. Thereby, on the server 20 side, the text information including the item name and the guide information corresponding to the text information or the item name can be stored in association with each other.

Further, the guide information may include character input by the user (expert). This allows a novice user to input items on a predetermined screen by referring to what an expert has actually input.

Furthermore, if personal information is included due to character input by an expert user, the guide information is configured not to include personal information. For example, on the server 20 side, text information such as email addresses, names, nicknames, etc. may be blurred out or replaced with other symbols. Thereby, security can be improved from the viewpoint of personal information protection.

Further, the identification module 135 may include identifying the screen of the user terminal 10B, acquiring identification information of this screen and positional information of this screen, and specifying a character recognition area based on this positional information. . Further, the acquisition module 136 may acquire a recognition result including character information of a character recognized from an image within the specified area and the character. This makes it possible to specify the character recognition area, thereby reducing the processing load and respecting privacy without analyzing unnecessary information.

Note that one or more processing units (CPUs) 110 read and execute each module from the memory 130 as necessary. For example, one or more processing units (CPUs) 110 may configure a communication unit (including a transmitting unit and a receiving unit) by executing a network communication module 132 stored in the memory 130. In addition, one or more processing units (CPUs) 110 execute a specific module 135, an acquisition module 136, a display control module 137, a transaction control module 138, a detection module 139, and a warning module 140, which are stored in the memory 130, respectively. By doing so, the identification section, the acquisition section, the display control section, the detection section, and the warning section may be configured.

In other embodiments, the identification module 135, the acquisition module 136, the display control module 137, the transaction control module 138, the detection module 139, and the alert module 140 may be standalone applications stored in the memory 130 of the user terminal 10A. . Standalone applications include, but are not limited to, specific applications, acquisition applications, display control applications, transaction control applications, detection applications, and alert applications. In yet other embodiments, the identification module 135, acquisition module 136, display control module 137, transaction control module 138, detection module 139, and alert module 140 may be add-ons or plug-ins to another application.

Each of the elements shown above may be stored in one or more of the aforementioned storage devices. Each of the modules shown above corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., sets of instructions) illustrated above need not be implemented as separate software programs, procedures or modules, and therefore various subsets of these modules may be combined in various embodiments. Alternatively, it may be reconfigured. In some embodiments, memory 130 may store a subset of the modules and data structures shown above. Furthermore, memory 130 may store additional modules and data structures not described above.

FIG. 3 is a block diagram showing an example of the server 20 according to the embodiment. Server 20 typically includes one or more processing units (CPUs) 210, one or more network or other communication interfaces 220, memory 230, and one or more processors for interconnecting these components. A plurality of communication buses 270 are included. Although the server 20 shown in FIG. 3 is described as the server 20 shown in FIG. 1, it may be configured as a separate server with at least one function.

Server 20 may optionally include a user interface 250, including a display device (not shown) and input devices such as a keyboard and/or mouse (or some other pointing device, not shown). be able to.

Memory 230 is, for example, a high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state storage, and may also include one or more magnetic disk storage, optical disk storage, flash memory devices, or It may also be a nonvolatile memory such as another nonvolatile solid state storage device.

Other examples of the memory 230 include one or more storage devices installed remotely from the CPU 210. In some embodiments, memory 230 stores the following programs, modules and data structures, or a subset thereof.

Operating system 231 includes, for example, procedures for handling various basic system services and performing tasks using hardware.

Network communications module 232 may, for example, connect server 20 to other computers through one or more communications network interfaces 220 and one or more communications networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc. Used to connect via.

The object data 233 stores information about objects that can be recognition targets. For example, the object data 233 includes an object ID for identifying the object, a name of the object, image data including the object, etc. (see, for example, FIG. 4). The image data of the object may include image data from various angles including the screen in order to improve recognition accuracy.

The character recognition data 234 stores information including character information of a screen image that can be recognized. For example, the character recognition data 234 includes an object ID to indicate which object, a screen ID to indicate which screen, a screen name, each character information including an item name included in the screen, and the position of the character information. Includes location information, etc. The position information includes, for example, character information on the screen and information indicating the position of the item.

The guide storage location data 235 stores information for users who require guidance for inputting or setting a predetermined screen. For example, the guide storage location data 235 includes a screen ID, text information (including item names), guide information including the content of the guide corresponding to the text information, and the like. The guide information includes at least one of a moving image showing the movement of an expert's hand, an animation based on the movement of the expert's hand, an audio guide, and the like. For example, when the screen ID indicates "delivery screen" and the text information indicates "delivery selection," the guide information includes a video showing how an expert selects delivery.

The object recognition module 236 acquires image data including a screen transmitted from the user terminal 10A via an object recognition API (Application Programming Interface), performs object recognition processing on this image data, and performs object recognition. The result data is transmitted to the user terminal 10A that is the source of the image data. The result data may include the object ID of the recognized object and position information indicating the position of the screen within the recognized object.

The character recognition module 237 acquires image data including a screen (for example, area image data) transmitted from the user terminal 10A via the character recognition API, performs character recognition processing on this image data, The recognition result data is transmitted to the user terminal 10A that is the source of the image data. For the character recognition process, a known OCR (Optical Character Recognition) technique may be used. At this time, the character recognition module 237 may associate the position information of the recognized character string with the recognition result.

The guide control module 238 executes processing for registering and transmitting guide information. For example, the guide control module 238 acquires various data, information, and requests from the user terminal 10A via the network communication module 232, and controls guide information.

As a specific example, the guide control module 238 obtains a request to obtain guide information, a request to upload guide information, a request to learn image data for object recognition, and the like. Further, the guide control module 238 uploads guide information to a database or acquires requested guide information from a database (DB) based on each request.

The electronic commerce module 239 executes the buying and selling process of products and services. For example, the electronic commerce module 239 executes listing processing of products and services, and executing sales processing. Further, the electronic commerce module 239 controls membership registration, registration of products to be sold, purchase and sale transactions, delivery settings, evaluation, etc. using a predetermined screen of an application activated on the user terminal 10B.

The voice recognition module 239 acquires the voice data transmitted from the user terminal 10A via the voice recognition API, performs voice recognition on this voice data, and sends the text data of the recognition result to the user who sent the voice data. It is sent to the terminal 10 or registered in the database.

Each of the elements shown above may be stored in one or more of the storage devices mentioned above. Each of the modules shown above corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., sets of instructions) illustrated above need not be implemented as separate software programs, procedures or modules, and therefore various subsets of these modules may be combined in various embodiments. Or it can be reconfigured. In some embodiments, memory 230 may store a subset of the modules and data structures shown above. Furthermore, memory 230 may store additional modules and data structures not described above.

Note that one or more processing units (CPUs) 210 read and execute each module from the memory 230 as necessary. For example, one or more processing units (CPUs) 210 may configure a communication unit (including a transmitting unit and a receiving unit) by executing a network communication module 232 stored in the memory 230. In addition, one or more processing units (CPUs) 210 execute an object recognition module 236, a character recognition module 237, a guide control module 238, an electronic commerce module 239, and a voice recognition module 240, which are stored in the memory 230, respectively. This may constitute an object recognition section, a character recognition section, a guide control section, an electronic commerce section, and a voice recognition section.

Although FIG. 3 depicts a "server," FIG. 3 is intended less as a structural overview of the embodiments described herein and more as an illustration of the various features that may be present in a set of servers. . In fact, items shown separately could be combined, as well as certain items could be made separate, as will be recognized by those skilled in the art. For example, items shown separately in FIG. 3 could be implemented on a single server, and a single item could be implemented by one or more servers.

<Example of data structure>
FIG. 4 is a diagram showing an example of object data 233 according to the embodiment. In the example shown in FIG. 4, the object data 233 includes an object name, image data of the object, etc. in association with the object ID. These data may be used as learning data for machine learning. As an example of the object data 233, the image ID "T-000100" is associated with data such as an object name "smartphone" and image data of the object "image A." In addition, position information indicating the position of the object on the screen may be associated with the object.

FIG. 5 is a diagram showing an example of character recognition data 234 according to the embodiment. In the example shown in FIG. 5, the character recognition data 234 includes data such as a screen ID, a screen name, character information 1, position information 1, and character information 2 in association with the object ID. As an example of the character recognition data 234, the object ID "T-000100" includes the screen ID "D1", the screen name "registration screen", the character information including the item name "name", and the position information indicating the position of the item " (x1, y1)" etc. are associated.

FIG. 6 is a diagram showing an example of the guide storage location data 235 according to the embodiment. In the example shown in FIG. 6, the guide storage location data 235 includes text information, guide information storage location information, and the like in association with the screen ID. As an example of the guide storage location data 235, the screen ID "D1" is associated with text information "name" and the URL "URL1" of the storage location of guide information related to input of this name.

The data structure described above is just an example, and is not limited to this example. For example, in the guide storage location data 235 shown in FIG. 6, an ID may be provided for an item related to character information, and the guide information storage location information may be associated with this item ID.

<Operation explanation>
Next, the operation of the information processing system 1 according to the embodiment will be explained using FIGS. 7 and 8. In the example shown in FIGS. 7 and 8, the server 20 is divided by function. For example, the server 20A is a server that includes an object recognition module 236, the server 20B is a server that includes a character recognition module 237, and the DB1 is a storage destination for learning data such as object image data and guide information, The DB 2 stores object data 235, character recognition data 236, guide storage location data 236, etc., and has the function of a guide control module 238. FIG. 7 is a sequence diagram illustrating an example of the registration process of the information processing system 1 according to the embodiment.

(Step S102)
The user uses the user terminal 10A to press the registration activation button of a predetermined application (hereinafter also referred to as "A application"). Further, the user speaks to the user terminal 10A, such as "Hey XXX, start the A app," and starts the A app. At this time, the user may say something like "Start the registration function of the A app" to activate the registration function of the A app.

(Step S104)
Upon activation of the A app, the user terminal 10A activates the photographing device 160 and displays the camera image being photographed on the display 151.

(Step S106)
The user taps the user terminal 10B (object) visible through the display 151. The detection module 139 of the user terminal 10A detects the tap gesture. Further, the user may notify the user terminal 10A of the existence of the user terminal 10B by voice or the like.

(Step S108)
The specific module 135 of the user terminal 10A generates learning data for object recognition based on the tapped position information in the camera image from the photographing device 160 and the camera image.

(Step S110)
The specific module 135 of the user terminal 10A uploads the learning data to the DB1. The DB1 assigns an ID to the learning data and obtains the URL of the storage location.

(Step S112)
The specific module 135 of the user terminal 10A obtains the ID of the learning data uploaded from the DB 1 and the URL indicating the storage location of the learning data.

(Step S114)
The specific module 135 of the user terminal 10A uploads the ID and URL of the learning data to the DB2.

(Step S116)
The specific module 135 of the user terminal 10A obtains a notification from the DB2 that the storage of the learning data ID and URL has been completed.

(Step S118)
The specific module 135 of the user terminal 10A sends a request to perform object recognition of the object to the server 20A based on the camera image.

(Step S120)
The object recognition module 236 of the server 20A requests the DB2 to obtain information on an object that is most similar to the object in the image data, based on the image data of the obtained camera image. Note that the DB2 holds a trained model that is trained using learning data including image data of a camera image and object ID in the image, and when the image data of a camera image is input from the server 20A, , the object ID corresponding to this image data may be output.

(Step S122)
The server 20A acquires the object ID and position information including the position of the object within the screen from the DB2.

(Step S124)
The object recognition module 236 of the server 20A transmits the object ID and position information to the user terminal 10A.

(Step S126)
The specific module 135 of the user terminal 10A crops (cuts out) a predetermined area of the camera image based on the position information of the object.

(Step S128)
The specific module 135 of the user terminal 10A sends a request to the server 20B to recognize the character string in the cropped image.

(Step S130)
The character recognition module 237 of the server 20B recognizes a character string from within the cropped image and obtains position information of the character string within the image. The character recognition module 237 of the server 20B transmits a recognition result including character information and position information of the recognized character string to the user terminal 10A.

(Step S132)
The acquisition module 136 of the user terminal 10A takes the top line (character information) of the character string included in the recognition result as the name of the screen (screen name), and the following lines (character information) as the screen contents (items). Save as (name).

(Step S134)
The display control module 137 of the user terminal 10A controls the display 151 to display that recording is in progress, and notifies the user (for example, an expert) that recording is in progress.

(Step S136)
The user uses his/her hand to perform settings, input, etc. on the screen of the user terminal 10B, and the photographing device 160 photographs the hand being worked on and the object to be recognized (user terminal 10B). .

(Step S138)
The user terminal 10A may include, for example, a creation module (not shown) that creates animation based on hand movements. The hand animation is included in the guide information.

(Step S140)
The user presses the work end button. The work end button may be a physical button or a button displayed on the display 151. In the case of a displayed button, the creation module of the user terminal 10A may detect the end of the task if a tap of the user's hand is detected on the button.

(Step S142)
The user terminal 10A uploads guide information to the DB1. DB1 acquires the URL indicating the storage location of the guide information.

(Step S144)
The acquisition module 136 of the user terminal 10A acquires the URL indicating the storage location of the guide information and the contents of the screen (item name or character information) from the DB1.

(Step S146)
The acquisition module 136 of the user terminal 10A associates the URL indicating the storage location of the guide information with the contents of the screen (item name or character information) and transmits the associated URL to the DB2. The DB 2 stores the guide information URL and storage location data as guide storage location data.

(Step S148)
The acquisition module 136 of the user terminal 10A acquires a notification from the DB2 that the storage has been completed.

FIG. 8 is a sequence diagram illustrating an example of display processing of the information processing system 1 according to the embodiment.
(Step S202)
The user uses the user terminal 10A to press the display start button for the A app. Further, the user speaks to the user terminal 10A, such as "Hey XXX, start the A app," and starts the A app. At this time, the user may say something such as "start the display function of the A app" to start the display function of the A app.

(Step S204)
Upon activation of the A app, the user terminal 10A activates the photographing device 160 and displays the camera image being photographed on the display 151.

(Step S206)
The user taps the user terminal 10B (object) visible through the display 151. The detection module 139 of the user terminal 10A detects the tap gesture. Further, the user may notify the user terminal 10A of the existence of the user terminal 10B by voice or the like.

(Step S208)
The specific module 135 of the user terminal 10A requests the server 20A to perform object recognition based on the tapped position information in the camera image from the photographing device 160 and the camera image.

(Step S210)
The object recognition module 236 of the server 20A requests the DB2 to obtain information on an object that is most similar to the object in the image data, based on the image data of the obtained camera image. Note that the DB2 holds a trained model that is trained using learning data including image data of a camera image and object ID in the image, and when the image data of a camera image is input from the server 20A, , the object ID corresponding to this image data may be output.

(Step S212)
The object recognition module 236 of the server 20A acquires the object ID and position information including the position of the object within the screen from the DB2.

(Step S214)
The object recognition module 236 of the server 20A transmits the object ID and position information to the user terminal 10A.

(Step S216)
The specific module 135 of the user terminal 10A crops (cuts out) a predetermined area of the camera image based on the position information of the object.

(Step S218)
The specific module 135 of the user terminal 10A sends a request to the server 20B to recognize the character string in the cropped image.

(Step S220)
The character recognition module 237 of the server 20B recognizes a character string from within the cropped image and obtains position information of the character string within the image. The character recognition module 237 of the server 20B transmits a recognition result including character information and position information of the recognized character string to the user terminal 10A.

(Step S222)
The acquisition module 136 of the user terminal 10A takes the top line (character information) of the character string included in the recognition result as the name of the screen (screen name), and the following lines (character information) as the screen contents (items). Save as (name).

(Step S224)
The acquisition module 136 of the user terminal 10A requests the DB2 to acquire the URL for downloading the guide information based on the name of the screen (screen name) and the contents of the screen (item name or character information).

(Step S226)
The acquisition module 136 of the user terminal 10A acquires the guide information download URL from the DB2.

(Step S228)
The acquisition module 136 of the user terminal 10A requests the DB1 to acquire the guide information file stored in the DB1 using the guide information download URL.

(Step S230)
The acquisition module 136 of the user terminal 10A downloads and acquires the guide information file from the DB1.

(Step S232)
The display control module 137 of the user terminal 10A controls the reproduction of guide information (hand animation) set in the 3D model of the hand, for example, based on the position information of the character string (character information) of the item name.

(Step S234)
The user operates a predetermined screen of the user terminal 10B viewed through the display 151 according to the guide information displayed on the display 151 of the user terminal 10A. In addition, the user can utter a voice such as "next", make a predetermined gesture to indicate going to the next step, or make the same movement with the same hand position (regardless of 2D or 3D data) as in the hand animation. By doing so, these words and actions can be recognized on the user terminal 10 side, and guide information corresponding to the character string of the item name in the next line can be displayed.

As described above, the disclosed technology can provide a mechanism that allows appropriate guidance to be provided when settings are made from a predetermined screen using a user interface. In addition, by identifying guide information on a predetermined screen through object recognition and superimposing appropriate guide information on another device, the predetermined screen of the user terminal 10B can be displayed as is, and a user terminal different from the user terminal 10B can 10A can be used to assist with input and settings on a predetermined screen.

<Screen example>
Next, an example displayed on the display 151 of the user terminal 10A will be described. FIG. 9 is a diagram showing an example of screen transitions in the user terminal 10A according to the embodiment. A screen H10 shown in FIG. 9 shows an example in which a cell phone frame G10 is displayed by the display control module 137 to notify the user of the location of the user terminal 10B. Frame G10 may not be displayed.

Next, a screen H12 shows an example in which the user brings the user terminal 10B on which a predetermined screen is displayed to the position of the frame G10. On the screen H12, when the user's hand UH taps the user terminal 10B (step S206 shown in FIG. 8), the detection module 139 detects the tap gesture, and the guide function is started in the user terminal 10A.

Next, on the screen H14, an example is shown in which object recognition and character recognition are performed, and guide information is displayed in a superimposed manner. When the display control module 137 acquires the guide information corresponding to a predetermined screen, the display control module 137 controls the display 151 to display the guide information in a superimposed manner. In the example shown in screen H14, 3D animation GH is displayed in a superimposed manner using AR technology.

Note that the disclosed technology is not limited to the embodiments described above, and can be implemented in various other forms without departing from the gist of the disclosed technology. Therefore, the above embodiments are merely illustrative in all respects, and should not be interpreted in a limited manner. For example, each of the above-mentioned processing steps can be arbitrarily changed in order or executed in parallel as long as there is no inconsistency in the processing contents.

The program of each embodiment of the present disclosure may be provided in a state stored in a computer-readable storage medium. The storage medium is a "non-temporary tangible medium" that can store a program. Programs include, by way of example and not limitation, software programs and computer programs.

[Modified example]
Further, modifications of each of the above-described embodiments are shown below.

<Modification 1>
In the first modification, the following techniques may be combined in object recognition. Examples include location information such as VPS (Visual Positioning System) and GPS (Global Positioning System), network information for determining communication status based on signal strength, etc., ultrasound, and the like. For example, if there are multiple similar objects such as a ticket vending machine, delivery locker, delivery box, etc., which object may be identified based on position information and an image.

<Modification 2>
Images used to identify and recognize objects do not have to be rectangular. Furthermore, in object recognition, learning data prepared in advance, rather than dynamically, may be utilized. For example, object recognition may be performed using Semantic Segmentation, Instance Segmentation, stereoscopic recognition, or the like.

<Modification 3>
Voice or the like may be used as a trigger when the user moves on to the next operation. In this case, the user terminal 10A is provided with a speaker and a microphone. Further, image recognition may be appropriately combined when determining the situation. Furthermore, the guidance does not need to be based on voice recognition. For example, hand gestures, analysis of items that are likely to be set by image recognition, tapping on the screen, etc. may be used.

<Modification 4>
The guide information may include not only the hand motion but also the following information. For example, the information includes audio, text, moving images, images such as still images, tactile sensations such as heat, olfactory sensations such as the smell of smoke, and tastes such as the taste of food. Further, the guide information may be moved by reflecting a hand motion stored in the user terminal 10A on the 3D model of the hand as an animation.

<Modification 5>
Information processing devices (objects) that can register guide information include smartphones, posters and notebooks with text written on them, personal computers, large screens at stadiums, multimedia terminals at convenience stores, and invoices ( Examples include Yu-Pri Touch, etc.), delivery lockers and delivery boxes (Pudo, etc.), remote controls (coolers, etc.), ticket vending machines, televisions, etc.

<Modification 6>
The method of analyzing the screen using text obtained by character recognition, including Braille, etc. used by visually impaired people, may be combined with image recognition such as icons, and the method uses the size, thickness, position, and color of the font to analyze the screen. The importance of each text may be determined, and the display order of guide information may be changed based on the importance.

<Modification 7>
Examples of devices that display guide information include smartphones, tablet terminals, smart glasses, HMDs (Head Mounted Displays), smart contact lenses, brain-invasive devices, non-brain-invasive devices, and robots.

<Modification 8>
Moreover, at least a part of each function that the server 20 has may be provided in the user terminal 10A. Moreover, at least a part of each data stored by the server 20 may be provided in the user terminal 10A.

1 Information processing system 10A, 10B Information processing device (user terminal)
20, 20A, 20B Information processing device (server)
110, 210 Processing unit (CPU)
120, 220 Network communication interface 130, 230 Memory 131, 231 Operating system 132, 232 Network communication module 133 Image related data 134 Text related data 135 Specification module 136 Acquisition module 137 Display control module 138 Transaction control module 139 Detection module 140 Warning module 150 User interface 160 Photographing device 170, 270 Communication bus 233 Object data 234 Character recognition data 235 Guide storage destination data 236 Object recognition module 237 Character recognition module 238 Guide control module 239 Electronic commerce module 240 Voice recognition module

Claims (13)

One or more processors included in the information processing device,
Displaying and controlling images being photographed by a photographing device;
identifying a screen of another information processing device displayed in the image;
each character string displayed within the specified screen is recognized and a recognition result including each character information of each character string is obtained;
acquiring each guide information corresponding to each character information included in the recognition result;
Displaying and controlling list information that allows selection of each character information included in the recognition result;
Information processing that performs display control of the guide information corresponding to one character information selected by the user from the list information by associating it with the one character information and superimposing it on the image being photographed. Method. Superimposing and controlling the display of the guide information includes:
identifying the one character information corresponding to the position of the user 's hand;
Identifying guide information corresponding to the identified one character information,
2. The information processing method according to claim 1, further comprising superimposing and controlling the display of the specified guide information on the image being photographed in association with the specified one character information. Identifying the screen includes:
A screen of the information processing device is identified, and identification information of the screen and position information of the screen are acquired;
including specifying a character recognition area based on the position information,
To obtain the recognition result of the character string ,
3. The information processing method according to claim 1, further comprising recognizing a character string from an image within the specified area and obtaining a recognition result including character information of the character string . The information processing method according to any one of claims 1 to 3, wherein the guide information includes a video related to another user's hand operation corresponding to each item related to each character information. The one or more processors,
Claim further comprising: outputting a warning when a character input to an item related to the character information is recognized and the recognition result of the character does not satisfy an input condition associated with the item. 5. The information processing method according to any one of 1 to 4. One or more processors included in the information processing device,
Displaying and controlling images being photographed by a photographing device;
identifying a screen of another information processing device displayed in the image;
each character string displayed within the specified screen is recognized and a recognition result including each character information of each character string is obtained;
acquiring each guide information corresponding to each character information included in the recognition result;
Displaying and controlling list information that allows selection of each character information included in the recognition result;
A program that causes guide information corresponding to one character information selected by a user from the list information to be displayed in association with the one character information and superimposed on the image being photographed. An information processing device including one or more processors,
The one or more processors,
Displaying and controlling images being photographed by a photographing device;
identifying a screen of another information processing device displayed in the image;
each character string displayed within the specified screen is recognized and a recognition result including each character information of each character string is obtained;
acquiring each guide information corresponding to each character information included in the recognition result;
Displaying and controlling list information that allows selection of each character information included in the recognition result;
Information processing that performs display control of the guide information corresponding to one character information selected by the user from the list information by associating it with the one character information and superimposing it on the image being photographed. Device. One or more processors included in the information processing device,
Displaying and controlling images being photographed by a photographing device;
identifying a screen of another information processing device displayed in the image;
Recognizing characters displayed within the specified screen and obtaining recognition results including character information of the characters;
photographing each guide information including the user's hand operation corresponding to each recognized character information using the photographing device;
An information processing method for transmitting each piece of character information and each piece of guide information to a server. The information processing method according to claim 8 , wherein the guide information includes character input by the user. 10. The information processing method according to claim 9, wherein if personal information is included in the character input, the guide information does not include the personal information. Identifying the screen includes:
A screen of the information processing device is identified, and identification information of the screen and position information of the screen are acquired;
including specifying a character recognition area based on the position information,
Obtaining the recognition result of the character is as follows:
The information processing method according to any one of claims 8 to 10 , comprising recognizing a character from an image within the specified area and obtaining a recognition result including character information of the character. One or more processors included in the information processing device,
Displaying and controlling images being photographed by a photographing device;
identifying a screen of another information processing device displayed in the image;
Recognizing characters displayed within the specified screen and obtaining recognition results including character information of the characters;
photographing each guide information including the user's hand operation corresponding to each recognized character information using the photographing device;
An information processing method that transmits each of the character information and each of the guide information to a server. An information processing device including one or more processors,
The one or more processors,
Displaying and controlling images being photographed by a photographing device;
identifying a screen of another information processing device displayed in the image;
Recognizing characters displayed within the specified screen and obtaining recognition results including character information of the characters;
photographing each guide information including the user's hand operation corresponding to each recognized character information using the photographing device;
An information processing device that transmits each piece of character information and each piece of guide information to a server.