JP7335221B2 - Electronic device, method, program and system for identifier information inference using image recognition model - Google Patents

Description

The present invention relates to an electronic device, method, program and system for inferring identifier information using an image recognition model.

With the development of information networks in recent years, it has become common to share information with friends using a smartphone chat application, SNS application, or the like. The content transmitted to other users for such information sharing is not limited to text messages, and can be images such as photographs and screen shots taken by users (Patent Document 1). Other users who have received the messages, images, etc. can enjoy communication between users by replying with impression messages and the like in response to these messages.

JP 2018-073070 A

Apps such as games that run on smartphones are also commonly used. The relationship between existing SNS and communication apps for information sharing and apps such as games is to share screenshots of games etc. with other users using other apps. In other words, the relationship is limited to the outbound direction from games, etc. to SNS, and the inbound direction is to send images obtained from SNS or images taken with a camera to attribute information such as characters in the application. mechanism has not been realized.

The present invention has been made in view of the above problems, and has the following features. That is, the electronic device of the present invention in one embodiment is defined by an operation accepting unit capable of accepting an operation from a user and unique learning content, and can infer at least one piece of identifier information using the learning content. a storage unit that stores a plurality of image recognition models that are unique image recognition models that are linked to unique attribute information; a destination attribute information identifying unit that identifies one piece of attribute information as a destination of image information; An image recognition model selection unit that selects from recognition models, and an identifier that inputs image information to the image recognition model selected by the image recognition model selection unit and infers identifier information from the image information based on the learning content. and an information reasoning unit.

Further, the learning contents of the plurality of image recognition models stored in the storage unit are defined by teacher data unique to each image recognition model, and the unique teacher data are pairs of labels, each of which is image information and identification information. can be defined by the set of

Further, the storage unit stores a plurality of response contents corresponding to each of a plurality of pieces of identifier information in association with each of the plurality of image recognition models, and based on the identifier information inferred by the identifier information inference unit, A response content execution unit may be provided for specifying and executing response content corresponding to the identifier information from among the plurality of response content stored in the storage unit.

a display unit for displaying information to a user, wherein the storage unit stores at least one of character information and icon image information corresponding to the attribute information; At least one of the stored character information and icon image information is displayed on the display unit, and at least one of the displayed character information and icon image information is specified by the user through the operation reception unit. At least one piece of attribute information among the attribute information stored in the storage unit may be specified as the destination of the image information.

In one embodiment of the invention, a method is one or more computer-implemented methods for identifying an image defined by unique learning content and capable of inferring at least one piece of identifier information using the learning content. A storage stage for storing a plurality of image recognition models that are recognition models and are associated with unique attribute information; an operation reception stage for receiving an operation from a user; a destination attribute information identifying step of identifying at least one piece of attribute information from the attribute information as a destination of image information; An image recognition model selection step of selecting from among a plurality of image recognition models, inputting image information to the image recognition model selected in the image recognition model selection step, and extracting identifier information from the image information based on the learning content. inferring identifier information inference stage;

The one or more computers may be user terminals.

The one or more computers can include user terminals and servers.

A program in one embodiment of the present invention can cause one or more computers to execute the above method.

A system according to one embodiment of the present invention is a system including a user terminal and a server, wherein the user terminal includes an operation accepting unit capable of accepting an operation from a user, and based on an operation accepted by the operation accepting unit, a destination attribute information identifying unit that identifies at least one piece of attribute information as a destination of image information, wherein the server is defined by unique learning content and can infer at least one piece of identifier information using the learning content. a storage unit that stores a plurality of image recognition models that are unique image recognition models and are linked to unique attribute information; input image information to an image recognition model selection unit that selects from among a plurality of image recognition models stored in the image recognition model selection unit, and the image recognition model selected by the image recognition model selection unit; an identifier information inferring unit for inferring identifier information from the information.

According to the present invention, the attribute information and the image recognition model are linked, and when the attribute information of the destination is selected, the image recognition model linked to the attribute information is selected and used for image recognition. By adopting the configuration, even when recognizing a large number of image contents due to a large number of attribute information, the target image content can be inferred using an image recognition model trained for individual attribute information. It is possible to realize high inference performance even when using a small amount of data, and achieve high response performance by implementing each image recognition model as a relatively small neural network.

1 is a system configuration diagram according to an embodiment of the present invention; FIG. 1 is a hardware configuration diagram of a server and an electronic device according to an embodiment of the present invention; FIG. 1 is a functional block diagram of a server and an electronic device according to an embodiment of the invention; FIG. 4 is a flow chart showing information processing according to one embodiment of the present invention. FIG. 4 is a diagram showing a screen display example according to one embodiment of the present invention; FIG. 4 is a diagram showing a screen display example according to one embodiment of the present invention; FIG. 4 is a diagram showing a screen display example according to one embodiment of the present invention; FIG. 4 is a diagram showing a screen display example according to one embodiment of the present invention; 1 is a functional block diagram of a server and an electronic device according to a position embodiment of the present invention; FIG.

A system 1 according to an embodiment of the present invention will be described below with reference to the drawings. In this specification, for convenience of explanation, more detailed explanation than necessary may be omitted. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially the same configurations may be omitted.

[First embodiment]
FIG. 1 shows an example of the overall configuration of a system 1 according to one embodiment of the invention. As shown in FIG. 1, the system 1 comprises a plurality of electronic devices 10 which are user terminals and a server 20 . The electronic device 10 and the server 20 are connected to a network 2 such as the Internet and can communicate with each other.

FIG. 2 is a block diagram showing the hardware configuration of the electronic device 10 and server 20 according to one embodiment of the present invention. The electronic device 10 comprises a processor 11 , a display device 12 , an input device 13 , a storage device 14 and a communication device 15 . Each of these components are connected by a bus 16 . It is assumed that an interface is interposed between the bus 16 and each constituent device as required. In this embodiment, the electronic device 10 is a smart phone. However, the electronic device 10 can be a terminal such as a tablet computer or a computer equipped with a contact input device such as a touch pad as long as it has the above configuration.

The server 20 likewise comprises a processor 21 , a display device 22 , an input device 23 , a storage device 24 and a communication device 25 . Each of these components are connected by a bus 26 . It is assumed that an interface is interposed between the bus 26 and each component as required. In this embodiment, the server 20 is implemented by a computer.

The processors 11 and 21 control the overall operation of the electronic device 10 and the server 20, and are CPUs, for example. Note that an electronic circuit such as an MPU may be used as the processors 11 and 21 . The processors 11 and 21 perform various processes by reading and executing programs and data stored in the storage devices 14 and 24 .

Display devices (displays) 12 and 22 display application screens and the like to the user of the electronic device 10 or the user of the server 20 under the control of the processors 11 and 21 . A liquid crystal display is preferable, but a display using an organic EL, a plasma display, or the like may be used.

The input devices 13 and 23 are user interfaces that receive user input to the electronic device 10 and the server 20, and are, for example, touch panels, touch pads, keyboards, or mice. In the present embodiment, the electronic device 10 is a smartphone, so the electronic device 10 includes a touch panel as the input device 13, the touch panel also functions as the display device 12, and the display device 12 and the input device 13 are integrated. . The display device 12 and the input device 13 may be separate forms arranged at different positions. Since the server 20 is a computer, it has a keyboard and a mouse as input devices, and a liquid crystal display as a display device.

The storage devices 14 and 24 include storage devices using flash memory such as volatile memory RAM and non-volatile memory eMMC, UFS, SSD, magnetic storage devices, etc., which are provided in general smartphones and computers. It is a storage device. Storage devices 14, 24 may also include external memory. For example, the storage device 14 stores applications, and the storage device 24 stores server applications. An application includes a program for executing an event of the application and various data referred to when the program is executed.

The communication devices 15 and 25 can exchange data with other devices via the network 2 (not shown in FIG. 2). For example, the communication devices 15 and 25 perform wireless communication such as mobile communication and wireless LAN, and connect to the network 2 . The electronic device 10 uses the communication device 15 to communicate with the server 20 via the network. The communication devices 15 and 25 may perform wired communication using an Ethernet (registered trademark) cable or the like.

FIG. 3 shows an example of a functional block diagram of the electronic device 10 and the server 20 according to one embodiment of the invention. The electronic device 10 includes a control unit 101, a display unit 102, an operation reception unit 103, a storage unit 104, a communication unit 105, an image information designation unit 106, a destination attribute information identification unit 107, an image recognition model selection unit 108, and an identifier information inference unit. 109 and a response execution unit 110 , and the server 20 includes a control unit 201 , a display unit 202 , an operation reception unit 203 , a storage unit 204 and a communication unit 205 . Storage units 104 and 204 store a plurality of image recognition models 150-1 to N and 250-1 to M, respectively.

In this embodiment, these functions are realized by the processors 11 and 21 executing programs. For example, the program to be executed is a program stored in the storage devices 14,24. In this way, since various functions are realized by program loading, one part (function) may be partly or wholly included in another part. These functions may be realized by hardware by configuring an electronic circuit or the like for realizing part or all of each function.

The control unit 101 of the electronic device 10 performs control processing for executing the functions of the application of the present embodiment. The display unit 102 displays application screens for controlling application functions, and displays application screens corresponding to application functions and user operations. The operation reception unit 103 receives input from the user of the electronic device 10 . In this embodiment, a touch panel including the display unit 102 and the operation reception unit 103 is used, and the operation reception unit is realized by the touch detection function.

The storage unit 104 stores information necessary for information processing executed by the control unit 101 . Furthermore, the storage unit 104 stores two or more pieces of attribute information that can be used by the user. The attribute information is information indicating the attribute to which the image is sent, and can be, for example, information identifying a character in a game application or a virtual personality.

In this embodiment, the attribute is the character of the game, and the attribute information is the character identification information. Even for the same character, if there are multiple variations due to differences in rarity, costumes, etc., different character identification information may be given as attribute information. The storage unit 104 stores character identifiers of characters owned by the user as attribute information. Further, storage unit 104 stores image recognition models 150-1 to 150-N associated with each piece of attribute information. Here, image recognition models linked to character identifiers of N characters owned by the user are stored. An image recognition model is a neural network (NN) model that is defined by unique learning content for each model and that can infer at least one piece of identifier information using the learning content.

The identifier information is information indicating image content, and can be set in advance by the system designer. In this specification, identifier information as an inference result of an image recognition model using learning content may be expressed as a class. Identifier information for defining teacher data for defining learning content may be expressed as a label. Identifier information associated with script data indicating response content to be executed based on inferred identifier information may also be expressed as a label.

Destination attribute information specifying unit 107 specifies at least one piece of attribute information from among attribute information stored in storage unit 104 as a destination of image information based on the user's input operation accepted by operation accepting unit 103 . The image information specifying unit 106 specifies image information to be sent to the destination based on the user input operation accepted by the operation accepting unit 103 . The image information may be not only image information of a still image but also image information of a moving image.

The image recognition model selection unit 108 selects an image recognition model linked to the attribute information specified by the destination attribute information specification unit 107 from among a plurality of image recognition models stored in the storage unit 104 . The identifier information inference unit 109 inputs the image information specified by the image information specification unit 106 to the image recognition model 150 selected by the image recognition model selection unit 108, and infers identifier information from the image information based on the learning contents. Based on the identifier information inferred by the identifier information inference unit 109, the response content execution unit 110 specifies and executes the response content corresponding to the identifier information from among the plurality of response contents stored in the storage unit 104. FIG.

The control unit 201 of the server 20 performs processing for applications executed in the electronic device 10 . In one example, when an application is executed in the electronic device 10 , the control unit 201 transmits and receives data regularly or as needed, and causes the electronic device 10 to implement the function of the application. The display unit 202 displays a management screen for the server administrator on the display device 22 as necessary.

The storage unit 204 stores information necessary for information processing executed by the control unit 201 . Further, the storage unit 204 stores all attribute information that can be used in the application, and image recognition models 250-1 to 250-M associated with each attribute information. In this embodiment, the server 20 prepares teacher data unique to each model for M characters, which are all characters available in the game application, and generates image recognition models 250-1 to 250-M. and stored in the storage unit 204 . Then, an image recognition model corresponding to the character owned by the user of the electronic device 10 is transmitted to the electronic device 10 and stored in the storage unit 104 .

Next, the information processing operation 400 of the system according to this embodiment will be described with reference to FIG. In this embodiment, the system is a game system, a smartphone is used as the electronic device 10, a game application is executed on the smartphone OS in the electronic device 10, and a server game application is executed in the server 20. It shall be executed on an OS for In this game, a plurality of characters are prepared as game media, and the user obtains the characters through lottery processing, for example. Then, it is assumed that the game is progressed while using the obtained plurality of characters. Here, an image obtained by the user outside the game is transmitted to the character owned by the user, and response processing determined for each character is executed according to the inferred image content of the image. A case will be described as an example. Sending to a character can include other concepts for sharing with a character.

In this embodiment, an image recognition model is modeled as a function that inputs a bitmap image as image information and outputs identifier information as a class corresponding to the content of the image at the time of inference. Here, image content means any feature that can be classified by a neural network, such as an object or background photographed in an image, or a combination of objects.

For example, when the image content is the fruit apple, the system outputs identifier information (class) and its probability, such as "apple, 90%" as an inference. Then, as inferred identifier information, it is possible to output only the identifier information for the identifier with the highest probability, or to output all the identifier information exceeding a predetermined threshold.

In the information processing operation 400 , first, learning content unique to each character is prepared, and an image recognition model is defined by this unique learning content and stored in the storage unit 204 of the server. An image recognition model linked to the owned character is downloaded and stored in the storage unit 104 (S401). This makes it possible to prepare a neural network model individualized for each character.

In this embodiment, the learning content is defined by teacher data unique to each image recognition model, and the unique teacher data is defined by a set of pairs of image information and identifier information as labels. An example of the data structure of the training data _Dc is shown in Equation (1).

Here, label _i is the label character string of the image content indicated by the i-th image. bitmap _i is the bitmap data of the i-th image. That is, in the present invention, teacher data is a set of pairs of images and their labels. In solving a multi-class classification problem by deep learning, it is possible to learn by associating multiple labels with one bitmap image, and labels and bitmaps are defined as an n×m relationship. Next, L _c , which is the set of all labels defined by D _c , can be defined as follows.

Here, label _i is the i-th label character string and has unique identification within the set L _c . p corresponds to the number of label types defined by Dc. A rendering scenario S _c corresponding to a character c can be defined as follows.

Here, label _i is the i-th type of label character string, and script _i is the script data of the effect that the system should reproduce when the i-th label is input. Note that, in the present invention, instead of using only one label, a plurality of labels may be combined to describe the image content, such as when an apple and sunglasses are photographed at the same time. Therefore, the maximum value q of the number of labels here may be larger than the number p of label types.

Next, a method for generating a neural network model individualized for each character will be described. The learn function, which is a function that performs learning using a DNN, receives teacher data learning data bucket D _c corresponding to a specific character c, and outputs a model M _c that performs multiclass classification of image content. is a function and can be defined as:

This learn function can be implemented as a method for creating a generally known image recognition NN model, or as a method for inputting training data as fine tuning to a trained model. For example, AlexNet or VGG1 can be used as a method for creating an image recognition NN model, and Inception-v3 can be used as a method for inputting training data for fine-tuning a trained model. These schemes are described, for example, in Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012, Lecun, Y, Bengio, Y & Hinton, G , "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444. 2015, Toru Ogawa, Atsushi Otsubo, Rei Narita, Yusuke Matsui, Toshihiko Yamasaki, Kiyoharu Aizawa, "Object Detection for Comics using Manga109 Annotations ," arXiv:1803.08670, 2018, Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016), and Rethinking the Inception Architecture for Computer Vision. 2016 IEEE Conference It is well known to those skilled in the art, as described in on Computer Vision and Pattern Recognition (CVPR), 2818-2826.

Thus, by preparing individualized learning contents for each character and preparing an individualized neural network model, image contents to be identified can be determined according to the character's individuality. For example, in setting a game scenario, if character A likes apples and cars, and character B dislikes spiders and frogs, an image containing apples and cars as image content is used as teacher data for character A. An image containing a spider and a frog as image contents is prepared as teacher data for the character B. These are then used to generate image recognition models for characters A and B, respectively. Thus, when the image information is input to the image recognition model of character A, the apple and the car are identified, and when the image information is input to the image recognition model of character B, the spider and frog are identified, and the image recognition model unique to the character is identified. It is possible to identify the defined image content. A personalized scenario script can then be executed according to the identified image content.

A general image recognition model using a neural network model is known, and it can solve advanced labeling problems such as labeling common nouns to objects in images such as cars, buses, handbags, and people. can. Such general image recognition models can be used in autonomous driving and advanced surveillance cameras. Although general image recognition is an excellent technology, it is difficult to customize the classification of labels according to the content of the game, or to customize the application, such as reacting only to specific signboards. In order to prepare a large number of learning models corresponding to applications using existing methods, it is necessary to prepare extremely large-scale data, so there is a problem in terms of scalability to different types of applications. In addition, it is not always easy to achieve high-speed response with electronic devices such as user terminals because it recognizes a large number of general objects, and it is also difficult to modify and extend the image recognition function by adding only small amounts of data. be.

On the other hand, the neural network model personalized for each character, as in the present embodiment, is sufficient if it can identify only specific image contents. It is easy to implement, and it can be implemented as a relatively small neural network model, so the response performance at the time of inference is high, and the image recognition function can be expanded with only a small amount of data addition.

Even if there are many characters in the game, only the image recognition models for the characters owned by each user need to be downloaded to the electronic device 10. Only the image recognition model can be updated and downloaded.

After the image recognition model for the character owned by the user is stored in the electronic device 10 , a screen 500 for specifying image information to be transmitted using the application is displayed on the display unit 102 . Then, based on the user input operation accepted by the operation accepting unit 103, the image information specifying unit 106 specifies image information to be transmitted (S402). For example, as shown in FIG. 5A, thumbnail images 501 of a plurality of photos are displayed on the screen, and the user touches the thumbnail image 501 of the image to be sent using the application, and touches the "send using application" button 502. Image information to be transmitted is specified. Two or more pieces of image information may be designated as image information to be transmitted.

Next, an application for transmitting the specified image information is specified according to the user's operation. For example, as shown in FIG. 5B, an image 512 of the image information specified in S402 is displayed, an icon 514 of an application that can be used for transmission is displayed, and an application is selected by the user touching the icon. , the "determine" button 516, the application for transmission is determined. Here, it is assumed that a game application for carrying out the present invention is selected.

In the present embodiment, the process of designating image information and the process of determining an application for transmission are assumed to be executed as functions of the OS of the smartphone, but may be part of the functions of the game application of the present invention. can also For example, the screen 500 for designating image information can be displayed by activating a function for transmitting image information to a character after activating a game application.

When a user who wishes to transmit an image to a character selects the game application for this embodiment as an application for transmission, the game application is started, and a screen for selecting character identification information as destination attribute information is displayed. is displayed on the display unit 102 . Then, the destination attribute information specifying unit 107 specifies character identification information as at least one piece of attribute information as the destination of the image information based on the user operation received by the operation receiving unit (S404). For example, as shown in FIG. 5C, an image 522 designated as a transmission target is displayed, and a destination character selection button is displayed as a button containing character information 524 indicating the character name. Character identification information as destination attribute information identification information is specified by selecting a character and touching decision button 526 . In addition to the character information of the character name, or instead of the character name, icon image information indicating the character may be displayed. As long as the character identification information can be specified, it may be specified by other methods such as character information input by the user.

Next, the image recognition model selection unit 108 selects the image recognition model linked to the character identification information, which is the specified attribute information, from among the plurality of image recognition models 150-1 to 150-N stored in the storage unit 104. (S406).

The identifier information inference unit 109 inputs the image information specified by the image information specification unit 106 to the image recognition model selected by the image recognition model selection unit 108, and infers identifier information from the image information based on the learning content ( S408).

For example, when image information d and destination character identification information c are input, infer, a function for inference about labels, can be defined as follows.

Here, label_score _i is the existence probability that the image content corresponding to the i-th label exists in the bitmap. Then, in this embodiment, the identifier information inference unit 109 assumes that image content corresponding to the label with the highest existence probability exists as the inferred identifier information, and outputs the label, which is the identifier information. A set of labels containing all labels with scores above a given threshold may be output.

Since the image 522 input to the image recognition model contains an apple as image content and does not contain a car, a high value is output for label_score ₁ and a low value for label_score ₂ , and the label corresponding to the apple is output as an inference result. ₁ is output.

Based on the identifier information output by the identifier information inferring unit 109, the response content executing unit 110 specifies and executes the response content corresponding to the identifier information from among the plurality of response contents stored in the storage unit 104. (S410).

In this embodiment, the storage unit 104 stores the production scenario Sc in association with the character identifier information in the data structure shown in Equation (3). The response content execution unit 110 identifies the character scenario Sc linked to the character identification information identified by the destination attribute information identification unit 107, and selects the identifier inferred by the identifier information inference unit 109 from among the set of character scenarios. Identify the script data linked to the label that is the information. Then, the content of the response is executed based on the specified script data. For example, as shown in FIG. 5D, a character image 530 corresponding to the character identification information specified as the destination is displayed on the display unit 102, and a message related to an apple, which is the image content of the input image, is displayed as a response message. 540.

Communication with characters in conventional games has been limited, such as by tapping the character and displaying the response. By using this embodiment, for example, when a user sends an image taken in the real world to a character, the reaction of the character is produced according to the content of the image. It enables natural communication as if you were there, and enhances the reality of the character. As a result, it is possible to enhance the sense of immersion in the game and the fusion with reality.

The image information transmitted in this embodiment does not depend on a specific camera device. Images that have been processed can also be targeted. As a result, it is possible to realize an effect in which a character reacts to an image taken while participating in a real event, or an effect in which a character reacts to an image distributed via an SNS account.

The content of the script data response is not limited to a message response, and may be anything as long as it responds to the input image information. For example, a moving image of a character may be reproduced, or a battle event with that character may be started. Execution processing of an in-game event may be started by photographing a poster put up at a real event venue and transmitting the attribute information.

As in this embodiment, when the attribute information and the image recognition model are linked and the attribute information of the destination is selected, the image recognition model linked to the attribute information is selected and used for image recognition. By adopting this configuration, for example, even in a setting where several hundred characters or more recognize tens of thousands of types of image contents in a game application, at most, the image recognition model trained for each individual character is used. It is possible to recognize the content of the target image with a single inference process, and achieve high response performance.

A learning model for recognizing an image can be, for example, about several hundred kilobytes, and can be linked to attribute information in advance and easily downloaded to an electronic device, and image recognition processing can be executed on the electronic device side. be. Therefore, there is no need to send image information to the server, and personal information can be appropriately protected, and the server load can be reduced. Further, even when the image recognition model is updated, it is only necessary to re-download the updated image recognition model to the electronic device side, thereby realizing high extensibility.

[Second embodiment]
In this embodiment, the electronic device 10 does not store the image recognition model in the storage unit 104, and instead of the image recognition model selection unit 108 and the identifier information inference unit 109 of the electronic device 10, the server 20 stores the image recognition model selection unit 208. and an identifier information inference unit 209, which is different from the first embodiment. The following description will focus on the differences from the first embodiment.

In this embodiment, the server 20 prepares unique learning content for each character, defines an image recognition model based on this unique learning content, and stores it in the storage unit 204 (S401). Do not download the image recognition model to

After that, in the electronic device 10, image information for transmission is specified for the attribute information (S402), and destination attribute information is specified (S404), by the same processing as in the first embodiment. Then, using the communication unit 105 , an identifier information inference request including the specified image information and the specified destination attribute information is transmitted to the server 20 . The image recognition model selection unit 208 of the server 20 selects the image recognition model 250 for inputting image information based on the destination attribute information included in the received identifier information inference request (S406). The identifier information inference unit 209 inputs the image information included in the identifier information inference request to the selected image recognition model 250, and executes identification information inference processing (S408). The identifier information inference unit 209 outputs identifier information corresponding to the image content of the image information, and the communication unit 205 of the server 20 transmits this to the electronic device 10 . The communication unit 105 of the electronic device 10 receives the identifier information transmitted from the server 20, and executes the content of the response based on the script data linked thereto (S410).

By using this embodiment, the electronic device 10 does not need to store an image recognition model and execute identifier information inference processing. It is possible to execute processing with high response performance.

Alternatively, the server 20 may specify the script data without specifying the script data corresponding to the identifier information in the electronic device 10, and the content of the response may be executed based on the transmitted script data. Furthermore, for example, in the case of a browser-based game application, the server 20 may execute response processing, and the executed effect may be displayed on the electronic device 10 .

In the processing or operation described above, the processing or operation can be freely changed as long as there is no contradiction. Moreover, each embodiment described above is an illustration for demonstrating this invention, and this invention is not limited to these embodiments. The present invention can be embodied in various forms without departing from the gist thereof. Moreover, the effects described in the present embodiment are merely enumerations of the most suitable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the present embodiment.

1 : System 2 : Network 10 : Electronic Device 11 : Processor 12 : Display Device 13 : Input Device 14 : Storage Device 15 : Communication Device 16 : Bus 20 : Server 21 : Processor 22 : Display Device 23 : Input Device 24 : Storage Device 25: communication device 26: bus 101: control unit 102: display unit 103: operation reception unit 104: storage unit 105: communication unit 106: image information designation unit 107: destination attribute information identification unit 108: image recognition model selection unit 109: Identifier information inference unit 110 : Response content execution unit 150 : Image recognition model 201 : Control unit 202 : Display unit 203 : Operation reception unit 204 : Storage unit 205 : Communication unit 208 : Image recognition model selection unit 209 : Identifier information inference unit 250 : image recognition model

Claims (9)

an operation receiving unit capable of receiving an operation from a user;
a storage unit that stores a plurality of image recognition models that are defined by specific learning content and that are capable of inferring at least one piece of identifier information using the learning content, and that are linked to character identification information of a character; ,
a destination attribute for specifying the character identification information of the selected character from among the character identification information stored in the storage unit as a destination of image information based on the user's operation for selecting the character received by the operation reception unit; an information identifying unit;
an image recognition model selection unit that selects an image recognition model linked to the character identification information specified by the destination attribute information specifying unit from among a plurality of image recognition models stored in the storage unit;
an identifier information inference unit that inputs image information to the image recognition model selected by the image recognition model selection unit and infers identifier information from the image information based on the learning content;
An electronic device comprising: The plurality of image recognition models stored in the storage unit have their learning contents defined by teacher data unique to each image recognition model, and the unique teacher data is a set of pairs of labels each of which is image information and identification information. The electronic device of claim 1, defined by: the storage unit stores a plurality of response contents corresponding to each of a plurality of pieces of identifier information in association with each of the plurality of image recognition models;
a response content execution unit that identifies and executes response content corresponding to the identifier information from among the plurality of response content stored in the storage unit based on the identifier information inferred by the identifier information inference unit; 3. The electronic device according to claim 1 or 2. a display for displaying information to a user;
the storage unit stores at least one of character information and icon image information corresponding to the character identification information ;
The destination attribute information specifying unit displays at least one of the character information and the icon image information stored in the storage unit on the display unit, and displays at least one of the displayed character information and icon image information. at least one of the character identification information stored in the storage unit is specified as the destination of the image information by the user specifying through the operation reception unit. 4. The electronic device according to any one of 3. 1. A method implemented by one or more computers, comprising:
a storage stage for storing a plurality of image recognition models that are defined by unique learning content and that are capable of inferring at least one piece of identifier information using the learning content, and that are linked to character identification information of a character; ,
an operation reception stage for receiving an operation from a user;
a destination attribute information specifying step of specifying the character identification information of the selected character from the stored character identification information as a destination of the image information based on the user's operation for selecting the character received in the operation receiving step; and,
an image recognition model selection step of selecting an image recognition model linked to the character identification information specified in the destination attribute information specifying step from among the plurality of stored image recognition models;
an identifier information inference step of inputting image information to the image recognition model selected in the image recognition model selection step and inferring identifier information from the image information based on the learning content;
A method characterized by performing 6. The method of claim 5, wherein the one or more computers are user terminals. 6. The method of claim 5, wherein the one or more computers comprise user terminals and servers. A program that causes one or more computers to execute the method according to any one of claims 5 to 7. A system including a user terminal and a server,
The user terminal
an operation receiving unit capable of receiving an operation from a user;
a destination attribute information identifying unit that identifies character identification information of a selected character as a destination of image information based on an operation by a user for selecting a character received by the operation receiving unit;
with
the server
a storage unit that stores a plurality of image recognition models that are defined by specific learning content and that are capable of inferring at least one piece of identifier information using the learning content, and that are linked to character identification information of a character; ,
an image recognition model selection unit that selects an image recognition model linked to the character identification information specified by the destination attribute information specifying unit from among a plurality of image recognition models stored in the storage unit;
an identifier information inference unit that inputs image information to the image recognition model selected by the image recognition model selection unit and infers identifier information from the image information based on the learning content;
A system characterized by comprising:

Images