JP7023005B2 - Information processing systems, information processing methods and programs - Google Patents

Description

The present invention relates to information processing systems, information processing methods and programs.

In recent years, a technique of displaying an object in a virtual space and changing the object has been considered. For example, Patent Document 1 discloses a technique of displaying an object in a three-dimensional virtual space and changing the display mode in response to an external input.

Japanese Unexamined Patent Publication No. 2000-207583

In the above-mentioned techniques, the objects displayed in the virtual space are irrelevant to those existing in the real space. Therefore, there is a problem that the reality and the virtual space cannot be associated with each other.

An object of the present invention is to provide an information processing system, an information processing method and a program capable of obtaining a virtual space related to a real space.

The information processing system of the present invention
Image pickup unit and
An estimation unit that estimates the subject based on an image of the subject captured by the image pickup unit and a reference image registered in advance, and an estimation unit.
An attribute reading unit that reads the attributes of the subject estimated by the estimation unit from the database,
It has a virtual space setting unit that sets the subject in the virtual space by using the attribute read by the attribute reading unit.

Further, the information processing method of the present invention is:
Processing to estimate the subject based on the image captured by the camera and the reference image registered in advance,
The process of reading the estimated subject attributes from the database and
Using the read attribute, the process of setting the subject in the virtual space is performed.

Further, the program of the present invention is
A program to run on a computer
On the computer
A procedure for estimating the subject based on an image captured by the camera and a reference image registered in advance, and a procedure for estimating the subject.
The procedure for reading the estimated subject attributes from the database and
Using the read attribute, the procedure of setting the subject in the virtual space is executed.

In the present invention, a virtual space related to the real space can be obtained.

It is a figure which shows the 1st Embodiment of the information processing system of this invention. It is a figure which shows an example of how the image pickup unit shown in FIG. 1 takes an image of a subject. It is a figure which shows an example of the internal structure of the estimation part shown in FIG. It is a figure which shows an example of the teacher data owned by the subject learning model shown in FIG. It is a figure which shows an example of the correspondence stored in the database shown in FIG. It is a sequence diagram for demonstrating an example of the information processing method in the information processing system shown in FIG. It is a figure which shows the 2nd Embodiment of the information processing system of this invention. It is a figure which shows an example of the state of operation of the moving part shown in FIG. 7. It is a figure which shows an example of the state of operation of the moving part shown in FIG. 7. It is a figure which shows an example of the internal structure of the estimation part shown in FIG. 7. It is a sequence diagram for demonstrating an example of the information processing method in the information processing system shown in FIG. 7.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First Embodiment)

FIG. 1 is a diagram showing a first embodiment of the information processing system of the present invention. As shown in FIG. 1, the information processing system in the present embodiment includes an imaging unit 110, an estimation unit 120, a database 130, an attribute reading unit 140, and a virtual space setting unit 150.

The image pickup unit 110 is a camera that captures an image of a subject to be imaged.

FIG. 2 is a diagram showing an example of how the image pickup unit 110 shown in FIG. 1 images a subject. As shown in FIG. 2, the image pickup unit 110 takes an image of the subject 300. The image pickup unit 110 may take an image of the subject 300 from a plurality of directions (angles). This is because the estimation unit 120 can easily estimate what the subject 300 is by using the image captured by the image pickup unit 110. For example, the image pickup unit 110 may image the subject 300 from a plurality of preset directions, or may image the subject 300 from a preset number of directions (for example, six directions). There may be. At this time, the imager who operates the image pickup unit 110 to take an image of the subject 300 may move the image pickup unit 110 or the subject 300 so that the image pickup unit 110 can take an image of the subject 300 from a plurality of directions. FIG. 2 shows an example in which the subject 300 is a bottle gum.

The estimation unit 120 estimates the subject based on the image captured by the image pickup unit 110 and the reference image registered in advance. At this time, the estimation unit 120 estimates the subject based on the image captured by the image pickup unit 110, the depth information of the image (distance from the image pickup unit 110 to the subject), and the reference image registered in advance. There may be. Further, the estimation unit 120 may estimate the subject based on a plurality of images captured by the imaging unit 110 from a plurality of directions and a plurality of reference images registered in advance. The estimation unit 120 inputs an image captured by the imaging unit 110 into a learned learning model that has been machine-learned to estimate the subject, and acquires and estimates the subject from the learning model. When the image pickup unit 110 can obtain the depth information, the estimation unit 120 can also use the depth information for estimating the subject, so that the estimation accuracy can be improved.

FIG. 3 is a diagram showing an example of the internal configuration of the estimation unit 120 shown in FIG. As shown in FIG. 3, the estimation unit 120 shown in FIG. 1 has a subject learning model 200. Note that FIG. 3 shows only the main components related to the present embodiment among the components included in the estimation unit 120 shown in FIG. In the subject learning model 200, in the learning phase of machine learning, image data (reference image) of various subjects and information on what image the image data represents are associated with each other and stored as teacher data. Based on the image data input in the application phase and the stored teacher data, it is estimated what image the input image data represents (subject). The subject learning model 200 outputs the estimated result. The information indicating the subject, which is the estimation result, is the subject identification information given in advance unique to the subject. Further, the output result may be fed back to the subject learning model 200 for further learning.

FIG. 4 is a diagram showing an example of teacher data owned by the subject learning model 200 shown in FIG. As shown in FIG. 4, the teacher data owned by the subject learning model 200 shown in FIG. 3 is associated with a subject name, subject identification information, and a plurality of image data (reference images). The subject name is the name of the subject. The subject name may be, for example, a product name, a manufacturer name, a product number, or the like. The subject identification information is information given in advance to each subject in order to make the subject distinguishable from other subjects. The subject identification information may be expressed using numbers or alphabets. The image data is data showing the image of the subject, and is an image of the subject taken from a plurality of directions, an image of the subject taken under various environments (for example, bright outdoors, dark indoors, etc.), and a subject. Was trained by combining images taken in various states (for example, the lid is open, closed, etc.) and 3D-CG (3Dimental Computer Graphics) created with high accuracy in combination with various lighting conditions and background images. It is a large amount of image data such as an image.

The subject learning model 200 compares the input image data with the teacher data as shown in FIG. 4, and estimates (determines) what the input image data is the subject. .. The subject learning model 200 outputs subject identification information of the subject as a result of estimation (determination). The estimation (determination) process using such teacher data may be the same as that using general AI (Artificial Intelligence).

The database 130 stores the subject identification information unique to the subject and the attributes of the subject in advance in association with each other.

FIG. 5 is a diagram showing an example of the correspondence stored in the database 130 shown in FIG. As shown in FIG. 5, the database 130 shown in FIG. 1 stores the subject identification information and the attributes in association with each other. The subject identification information is information given in advance to each subject in order to make the subject distinguishable from other subjects, and is the same as that shown in FIG. The attribute is information indicating the nature of the subject. For example, as shown in FIG. 5, the subject identification information "A001" and the attribute "bottle gum, exhilaration" are stored in association with each other. This indicates that the subject whose subject identification information is "A001" is bottle gum and has a refreshing taste (for example, mint type). Further, the subject identification information "B001" and the attribute "carbonated drink, sweet" are stored in association with each other. This indicates that the subject whose subject identification information is "B001" is a carbonated drink and its taste is sweet. Further, the subject identification information "C001" and the attribute "cooling agent, cold" are stored in association with each other. This indicates that the subject whose subject identification information is "C001" is an ice pack and is cold.

The attribute reading unit 140 reads the attributes of the subject estimated by the estimation unit 120 from the database 130. At this time, the attribute reading unit 140 reads the attribute associated with the subject identification information from the database 130 using the subject identification information given to the subject estimated by the estimation unit 120 as a search key.

The virtual space setting unit 150 sets the subject in the virtual space by using the attributes read by the attribute reading unit 140. The virtual space setting unit 150 sets the subject as an item having properties according to the attribute in the game played in the virtual space. For example, when the attribute of the subject read by the attribute reading unit 140 is the "cold" of the correspondence shown in FIG. 5, in the game played in the virtual space, if the subject is owned, it is "cold" to cool the heat. Can be used as an item. When setting the subject in the virtual space, the virtual space setting unit 150 displays (expresses) the image of the subject itself or a similar image (for example, an illustration of the image of the subject) in the virtual space. It may be an image, or an image of a subject may be changed into a predetermined shape or the like and displayed (expressed) in a virtual space. Further, the virtual space setting unit 150 may display according to the attribute of the subject set in the virtual space. For example, if the attribute of the subject set in the virtual space is an attribute that can be used as a strong item when playing against an opponent during the game, the virtual space setting unit 150 displays the level as a high-strength level. It may be.

The information processing method in the information processing system shown in FIG. 1 will be described below. FIG. 6 is a sequence diagram for explaining an example of an information processing method in the information processing system shown in FIG.

First, in the real space, the image pickup unit 110 takes an image of the subject (step S1). Subsequently, the estimation unit 120 inputs image data indicating an image of the subject captured by the image pickup unit 110 into the subject learning model 200 (step S2). The subject learning model 200 estimates the subject based on the input image data and the teacher data obtained in the learning phase (step S3). Subsequently, the attribute reading unit 140 reads the attribute associated with the subject identification information from the database 130 using the estimated subject identification information of the subject as a search key (step S4). Then, the virtual space setting unit 150 sets the subject in the virtual space using the attributes read by the attribute reading unit 140 (step S5).

As described above, in the present embodiment, the image pickup unit 110 takes an image of the owned subject, the estimation unit 120 estimates the subject based on the image data captured, and the attribute reading unit 140 determines the attributes of the estimated subject. Reading from the database 130, the virtual space setting unit 150 sets a subject having that attribute in the virtual space. Therefore, if a subject owned in the real space is imaged as a subject, the subject can be owned and used as an item in a game played in the virtual space.
(Second embodiment)

FIG. 7 is a diagram showing a second embodiment of the information processing system of the present invention. As shown in FIG. 7, the information processing system in the present embodiment includes an imaging unit 110, an estimation unit 121, a database 130, an attribute reading unit 140, a virtual space setting unit 150, and a moving unit 161. The image pickup unit 110, the database 130, the attribute reading unit 140, and the virtual space setting unit 150 are the same as those in the first embodiment.

The moving unit 161 moves the subject so that the imaging unit 110 can image the subject from a plurality of directions. For example, the moving unit 161 rotates the subject in a predetermined direction.

8 and 9 are diagrams showing an example of the operation of the moving unit 161 shown in FIG. 7. As shown in FIG. 8, a subject 300 is mounted on the moving unit 161 and the image pickup unit 110 takes an image of the subject 300. Then, as shown in FIG. 9, the moving unit 161 in the state where the subject 300 is mounted is rotated by a predetermined angle, and the image pickup unit 110 takes an image of the subject 300. The moving unit 161 and the imaging unit 110 repeat this operation. The reason why the image pickup unit 110 images a subject from a plurality of directions by moving the moving unit 161 for one subject makes it easier for the estimation unit 121 to estimate what the subject 300 is by using the captured image. Because. For example, the image pickup unit 110 and the moving unit 161 may be interlocked with each other, the moving unit 161 may rotate the subject 300 at predetermined angular intervals, and the image pickup unit 110 may take an image of the subject 300. 8 and 9 show an example in which the subject 300 is a bottle gum. In the examples shown in FIGS. 8 and 9, the moving unit 161 rotates like a rotary table on which the subject 300 is placed, but in the moving unit 161 the image pickup unit 110 displays the subject 300 from a plurality of directions. As long as it can be imaged, for example, the subject 300 may be moved (rotated) so that the top surface, the bottom surface, and the side surface of the subject 300 can be imaged, and the present invention is not limited to this.

The estimation unit 121 estimates the subject based on a plurality of images of the subject captured by the image pickup unit 110 from a plurality of directions and a plurality of reference images registered in advance by moving the subject by the moving unit 161. do. The estimation unit 121 moves the subject to the learned learning model that has been machine-learned to estimate the subject, and the image pickup unit 110 captures a plurality of images of the subject from a plurality of directions. By inputting, the subject is acquired from the learning model and estimated.

FIG. 10 is a diagram showing an example of the internal configuration of the estimation unit 121 shown in FIG. 7. As shown in FIG. 10, the estimation unit 121 shown in FIG. 7 has a subject learning model 201. Note that FIG. 10 shows only the main components related to the present embodiment among the components included in the estimation unit 121 shown in FIG. 7. The subject learning model 201 is stored as teacher data in association with image data (reference images) of various subjects and information on what image the image data represents in the learning phase of machine learning. Based on the image data input in the application phase and the stored teacher data, it is estimated what image the input image data represents (subject). The subject learning model 201 outputs the estimated result. The information indicating the subject, which is the estimation result, is the subject identification information given in advance unique to the subject. Further, the output result may be fed back to the subject learning model 201 for further learning. Further, the subject learning model 201 may also have as teacher data how the subject is moved (rotated) by the moving unit 161. In this case, the information input to the subject learning model 201 by the estimation unit 121 includes information indicating the amount and direction of movement (rotation) of the subject by the moving unit 161. The teacher data owned by the subject learning model 201 is the same as that of the first embodiment.

The information processing method in the information processing system shown in FIG. 7 will be described below. FIG. 11 is a sequence diagram for explaining an example of an information processing method in the information processing system shown in FIG. 7.

First, in the real space, the image pickup unit 110 takes an image of the subject from a predetermined direction (step S11). Then, the image pickup unit 110 and the moving unit 161 cooperate with each other to determine whether or not the imaging from all the preset directions is completed (step S12). When the imaging from all directions is not completed, the moving unit 161 rotates (moves) the subject by a predetermined angle (step S13). Then, in step S11 again, the image pickup unit 110 takes an image of the subject.

On the other hand, when the imaging from all directions is completed in step S12, the estimation unit 121 moves the subject by the moving unit 161 so that the imaging unit 110 images the subjects from a plurality of directions. The image data showing the image of is input to the subject learning model 201 (step S14). The subject learning model 201 estimates the subject based on the input plurality of image data and the teacher data obtained in the learning phase (step S15). Subsequently, the attribute reading unit 140 reads the attribute associated with the subject identification information from the database 130 using the estimated subject identification information of the subject as a search key (step S16). Then, the virtual space setting unit 150 sets the subject in the virtual space using the attributes read by the attribute reading unit 140 (step S17).

As described above, in the present embodiment, the moving unit 161 rotates (moves) the owned subject, the imaging unit 110 images images from a plurality of directions, and the estimation unit is based on the plurality of image data captured. 121 estimates the subject, the attribute reading unit 140 reads the estimated attribute of the subject from the database 130, and the virtual space setting unit 150 sets the subject having the attribute in the virtual space. Therefore, if a subject owned in the real space is imaged as a subject, the subject can be owned and used as an item in a game played in the virtual space. Further, by moving the subject using the moving unit 161 it is possible to save the trouble of moving the subject by the imager.

In the first embodiment and the second embodiment described above, the imager who holds the subject in the real space and images the held subject, and the user who uses the subject in the virtual space. Are preferably the same. Therefore, when the imager captures a subject in a real space and inputs it into the information processing system of the present invention, an authentication code such as a user code or a password is input, and the user is set in the virtual space. When starting a game or the like that uses a subject, the information processing system may input the authentication code to authenticate whether or not the user is an imager.

In the above, each component has been assigned to each function (process), but this allocation is not limited to the above. Further, the above-mentioned form is merely an example of the configuration of the constituent elements, and the present invention is not limited to this.

The processing performed by the above-mentioned information processing system may be performed by logic circuits manufactured according to the purpose. Further, a computer program (hereinafter referred to as a program) in which the processing contents are described as a procedure is recorded on a recording medium readable by a device constituting the information processing system, and the program recorded on the recording medium is recorded in the information processing system. It may be read and executed. The recording media that can be read by the device constituting the information processing system are floppy (registered trademark) disk, magneto-optical disk, DVD (Digital Versaille Disc), CD (Compact Disc), Blu-ray (registered trademark) Disc, and the like. In addition to transferable recording media such as USB (Universal Serial Bus) memory, memory such as ROM (Read Only Memory) and RAM (Random Access Memory) and HDD (Hard Disc Drive) built into the device constituting the information processing system. ) Etc. The program recorded on this recording medium is read by a CPU provided in an apparatus constituting an information processing system, and the same processing as described above is performed under the control of the CPU. Here, the CPU operates as a computer that executes a program read from a recording medium in which the program is recorded.

110 Imaging unit 120, 121 Estimating unit 130 Database 140 Attribute reading unit 150 Virtual space setting unit 161 Moving unit 200, 201 Subject learning model 300 Subject

Claims (11)

Image pickup unit and
An estimation unit that estimates the subject based on an image obtained by the imaging unit from each of a plurality of directions and a plurality of reference images registered in advance.
An attribute reading unit that reads the attributes of the subject estimated by the estimation unit from the database,
A virtual space setting unit that sets the subject in the virtual space using the attributes read by the attribute reading unit, and a virtual space setting unit .
An information processing system including a moving unit that moves the subject so that the imaging unit can image the subject from the plurality of directions . In the information processing system according to claim 1,
The estimation unit is an information processing system that estimates the subject based on an image captured by the image pickup unit, depth information of the image, and a reference image registered in advance. In the information processing system according to claim 1 or 2 .
Information processing in which the estimation unit acquires and estimates the subject from the learning model by inputting an image captured by the imaging unit into a learned learning model that has undergone machine learning for estimating the subject. system. In the information processing system according to any one of claims 1 to 3 ,
The database stores the subject identification information uniquely assigned to the subject and the attributes of the subject in advance in association with each other.
The attribute reading unit is an information processing system that reads the attributes from the database based on the subject identification information of the subject estimated by the estimation unit. In the information processing system according to any one of claims 1 to 4 ,
The virtual space setting unit is an information processing system that sets the subject to an item having a property corresponding to the attribute in a game played in the virtual space. An image captured from each of the plurality of directions using a moving unit for moving the subject so that the camera can capture the subject from a plurality of directions, and a plurality of pre-registered reference images. The process of estimating the subject based on
The process of reading the estimated subject attributes from the database and
An information processing method that uses the read attributes to perform processing for setting the subject in a virtual space. In the information processing method according to claim 6 ,
The process of inputting the captured image into the trained learning model that has been machine-learned to estimate the subject, and
An information processing method that performs a process of acquiring the subject from the learning model and estimating it. In the information processing method according to claim 6 or 7 .
An information processing method for setting a subject to an item having properties corresponding to the attributes in a game played in the virtual space. On the computer
An image captured from each of the plurality of directions using a moving unit for moving the subject so that the camera can capture the subject from a plurality of directions, and a plurality of pre-registered reference images. The procedure for estimating the subject based on
The procedure for reading the estimated subject attributes from the database and
A program for executing a procedure for setting the subject in a virtual space using the read attributes. In the program of claim 9 .
The procedure for inputting the captured image into the trained learning model that has been machine-learned to estimate the subject, and
A program for causing the computer to execute a procedure for acquiring and estimating the subject from the learning model. In the program according to claim 9 or 10 .
A program for causing the computer to execute a procedure for setting the subject to an item having a property corresponding to the attribute in the game played in the virtual space.

Images