JP7414918B1 - Image collection system, image collection method, and program - Google Patents

Abstract

The present invention efficiently collects captured images that can be used to generate a three-dimensional model. An image collection unit (102) of an image collection system (1) collects photographed images generated by a photographing unit (26) from a user's user terminal (20). A determination result acquisition unit (103) acquires a determination result as to whether or not an object for which a three-dimensional model is to be generated is shown in the photographed image. The reward giving unit (104) gives a reward to the user based on the determination result. [Selection diagram] Figure 4

Description

The present disclosure relates to an image collection system, an image collection method, and a program.

2. Description of the Related Art Conventionally, a technique for generating a three-dimensional model of an object shown in a photographed image generated by a photographing unit is known. For example, as a technique for generating a three-dimensional model from a photographed image, a method called photogrammetry described in Patent Document 1 and a method called neural rendering described in Non-Patent Documents 1 and 2 are known. In such a technique, it is necessary to photograph the object from various positions and angles, so it is very time-consuming to prepare the photographed images. Therefore, there is a need to efficiently collect captured images.

For example, Patent Document 2 discloses that among the captured images uploaded by the user, captured images that satisfy the search conditions specified by the searcher are identified, and the search conditions are set while providing the identified captured images to the searcher. It describes a technology that provides rewards to users who upload images that meet the criteria. For example, Patent Document 3 describes a technique for providing a reward to a user who takes a photograph of an accommodation facility he/she has reserved using a photographing unit and uploads the photographed image. The techniques disclosed in Patent Documents 2 and 3 can be said to be examples of techniques for collecting captured images for some purpose.

JP2019-032226A Japanese Patent Application Publication No. 2021-002147 JP2019-211833A

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, arXiv preprint arXiv:2003.08934 (https://arxiv.org/ pdf/2003.08934.pdf) Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, Daniel Duckworth, “NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections”, arXiv preprint arXiv:2008.02268 (https://arxiv. org/pdf/2008.02268)

However, the techniques of Patent Documents 2 and 3 are not intended to generate a three-dimensional model of an object in the first place. For this reason, even if the techniques of Patent Documents 2 and 3 are applied to the techniques of Patent Document 1 and Non-Patent Documents 1 and 2 that generate a 3D model from a captured image, the captured images are not suitable for generating a 3D model. cannot be collected efficiently. With conventional techniques, it has not been possible to efficiently collect captured images that can be used to generate a three-dimensional model.

One of the objectives of the present disclosure is to efficiently collect captured images that can be used to generate a three-dimensional model.

An image collection system according to the present disclosure includes an image collection unit that collects photographed images generated by a photographing unit from a user's user terminal, and whether or not an object for which a three-dimensional model is to be generated is shown in the photographed image. The present invention includes a determination result acquisition unit that acquires the determination result, and a reward granting unit that provides a reward to the user based on the determination result.

According to the present disclosure, it is possible to efficiently collect captured images that can be used to generate a three-dimensional model.

1 is a diagram showing an example of the overall configuration of an image collection system. It is a diagram showing an example of a flow in which a user uses a reward provision service. It is a figure which shows an example of an examination screen. FIG. 2 is a diagram illustrating an example of functions realized by the image collection system. FIG. 3 is a diagram showing an example of a user database. FIG. 3 is a diagram showing an example of a task database. This is an example of shooting conditions set for a task. FIG. 3 is a diagram illustrating an example of processing executed by the image collection system. It is a figure which shows an example of the function implement|achieved in a modification. FIG. 3 is a diagram showing an example of a plurality of captured images including mutually overlapping parts. FIG. 3 is a diagram illustrating an example of how photographing conditions are displayed.

[1. Overall configuration of image acquisition system]
An example of an embodiment of an image collection system according to the present disclosure will be described. FIG. 1 is a diagram showing an example of the overall configuration of an image acquisition system. For example, the image collection system 1 includes a server 10, a user terminal 20, and a judge terminal 30. Each of the server 10, user terminal 20, and judge terminal 30 can be connected to a network N such as the Internet or a LAN.

Server 10 is a server computer. For example, the server 10 includes a control section 11, a storage section 12, and a communication section 13. Control unit 11 includes at least one processor. The storage unit 12 includes volatile memory such as RAM and nonvolatile memory such as flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication.

User terminal 20 is a user's computer. A user is a person who performs a task described below. Users are sometimes called crowdworkers. For example, the user terminal 20 is a personal computer, a tablet terminal, or a smartphone. For example, the user terminal 20 includes a control section 21, a storage section 22, a communication section 23, an operation section 24, a display section 25, and a photographing section 26. The physical configurations of the control section 21, the storage section 22, and the communication section 23 are the same as those of the control section 11, the storage section 12, and the communication section 13, respectively. The operation unit 24 is an input device such as a touch panel. The display section 25 is a liquid crystal display or an organic EL display. Photographing unit 26 includes at least one camera.

The judge terminal 30 is a judge's computer. The judge is a person who judges the captured images, which will be described later. For example, the judge terminal 30 is a personal computer, a tablet terminal, or a smartphone. For example, the judge terminal 30 includes a control section 31, a storage section 32, a communication section 33, an operation section 34, and a display section 35. The physical configurations of the control unit 31, storage unit 32, communication unit 33, operation unit 34, and display unit 35 are the same as those of the control unit 11, storage unit 12, communication unit 13, operation unit 24, and display unit 25, respectively. be.

Note that the programs stored in the storage units 12, 22, and 32 may be supplied via the network N. In addition, a program stored in a computer-readable information storage medium can be used as a reading section (for example, an optical disk drive or a memory card slot) for reading the information storage medium, or an input/output section for inputting and outputting data with external equipment. (e.g., a USB port).

Furthermore, the image collection system 1 only needs to include at least one computer, and is not limited to the example shown in FIG. 1 . For example, the image collection system 1 may include only the server 10, and the user terminal 20 and judge terminal 30 may exist outside the image collection system 1. For example, the image collection system 1 may include only the server 10 and the judge terminal 30, and the user terminal 20 may exist outside the image collection system 1.

[2. Overview of image collection system]
In this embodiment, a case where a reward provision service is provided to a user will be exemplified. The reward giving service is a service that gives a reward when a predetermined task is accomplished. A task is a condition or task for acquiring a reward. For example, a task is a work or a questionnaire conducted using the user terminal 20. Rewards are compensation for completing a task. The reward may be anything that provides some kind of benefit to the user, such as points, electronic money, electronic value, coupons, tickets, tokens, crypto assets, digital content such as images, videos, or songs, cash, or goods. There may be. In this embodiment, rewards are given based on the user's account, wallet address on the blockchain network, address, etc., depending on the type of reward.

FIG. 2 is a diagram illustrating an example of a flow in which a user uses a reward provision service. In this embodiment, it is assumed that the user has already registered as a member of the reward granting service. For example, when a user operates the user terminal 20 to log in to the reward grant service, a top screen SC1 corresponding to the top page of the reward grant service is displayed on the display unit 25. A list of tasks in which the user can participate is displayed on the top screen SC1. In this embodiment, a task of uploading a captured image showing an object for which a three-dimensional model is to be generated will be taken as an example.

A three-dimensional model is a collection of polyhedra called polygons. Since a polygon is defined by a plurality of vertices, a three-dimensional model can also be called a collection of vertices. A three-dimensional model represents a virtual object. For example, a three-dimensional model is placed in a virtual space constructed using computer graphics software. The virtual space is a virtual three-dimensional space. Coordinate axes of a world coordinate system with a predetermined position as the origin are set in the virtual space. The positions of the vertices constituting the three-dimensional model are expressed in coordinates of the world coordinate system.

An object is an object that exists in real space. The object can also be referred to as a subject to be photographed. The object may be any object, such as a building, road, animal, plant, vehicle, electrical appliance, or miscellaneous goods. The photographed image is an image in which all or part of the object is shown. An object is shown in at least some pixels of the captured image. In this embodiment, a case will be described in which a photographed image is generated by the photographing section 26, but the photographed image may be generated by a camera other than the photographing section 26.

In the example of FIG. 2, a task is set to upload photographed images of Temple ABC in order to generate a three-dimensional model of Temple ABC, which exists in a fictitious tourist spot. For example, when the user selects button B10, a detailed screen SC2 showing details of the task is displayed on the display unit 25. When the user selects button B20, participation in the task is completed. When the user participates in the task, a list screen SC3 showing a list of captured images stored in the storage unit 22 is displayed on the display unit 25.

For example, when the user selects and uploads a photographed image of the ABC temple from the list screen SC3, a completion screen SC4 is displayed on the display unit 25 indicating that the upload of the photographed image has been completed. The user may take a photograph of the ABC temple with the photographing unit 26 and upload the photographed image on the spot. The user may select and upload multiple captured images at once. The user may upload the captured image after a certain amount of time has passed, rather than immediately after participating in the task. In this embodiment, it is assumed that the user can upload captured images at any time within the deadline set for the task.

For example, in a reward granting service, captured images are collected from various users. Although details will be described later, a known method can be used to generate a three-dimensional model from a photographed image. However, if the photographed image uploaded by the user does not indicate the object for which a three-dimensional model is to be generated, this photographed image cannot be used to generate a three-dimensional model. For example, a user may erroneously select and upload a captured image that does not show an object for which a three-dimensional model is to be generated.

For example, even if an object for which a three-dimensional model is to be generated is shown in a photographed image, if the photographed image uploaded by the user is unclear, there is a possibility that it cannot be used to generate a three-dimensional model. Even if the user uploads a photographed image in which an object is photographed at the same position and angle as other photographed images that have already been uploaded, it may not be useful for generating a three-dimensional model.

Therefore, in this embodiment, a judge judges whether the captured image can be used to generate a three-dimensional model. For example, the judge is an employee of a company that provides a compensation service. A reward will be given to a user who uploads a captured image that passes the review by the judge. For example, when a judge operates the judge terminal 30 to access the server 10, a screening screen showing a list of captured images uploaded by each of a plurality of users is displayed on the display unit 35.

FIG. 3 is a diagram showing an example of the examination screen. In the example of FIG. 3, photographed images of the object photographed from different positions in the front direction are displayed on the examination screen SC5, but it is assumed that photographed images of the object photographed from other angles have also been uploaded. . For example, when the judge scrolls the screening screen SC5, captured images of the object captured from other angles are also displayed. The judge selects a photographed image to be used for generating a three-dimensional model from among the photographed images displayed on the examination screen SC5.

For example, the judge selects a captured image in which the object is clearly shown. For example, the judge selects a photographed image in which the object is photographed from a different position and angle from the selected photographed image. The judge may judge the captured images from other viewpoints as long as they are useful for generating a three-dimensional model. When the judge selects a certain number of captured images and completes the examination, the server 10 acquires the judgment result by the judge from the judge terminal 30. The server 10 gives a reward to the user who uploads the photographed image selected by the judge.

As described above, the image collection system 1 of this embodiment sets, as a task of the reward provision service, to upload a photographed image showing an object for which a three-dimensional model is to be generated. The image collection system 1 gives a reward to the user who uploads the photographed image selected by the judge. The image collection system 1 can motivate the user to upload captured images that can be used to generate a three-dimensional model by providing a reward. This makes it possible to efficiently collect captured images. Hereinafter, details of this embodiment will be explained.

[3. Functions achieved by the image acquisition system]
FIG. 4 is a diagram showing an example of functions realized by the image collection system 1.

[3-1. Functions realized by the server]
For example, the server 10 includes a data storage section 100, a task setting section 101, an image collection section 102, a determination result acquisition section 103, a reward provision section 104, and a generation section 105. The data storage section 100 is realized by the storage section 12. Each of the task setting section 101 , image collection section 102 , determination result acquisition section 103 , reward giving section 104 , and generation section 105 is realized by the control section 11 .

[Data storage unit]
The data storage unit 100 stores data necessary for providing reward granting services. For example, the data storage unit 100 stores a user database DB1 and a task database DB2.

FIG. 5 is a diagram showing an example of the user database DB1. The user database DB1 is a database that stores various information regarding users. For example, the user database DB1 stores user IDs, passwords, user names, held points, and participating task information. When a certain user completes membership registration for the reward granting service, a record corresponding to this user is created in the user database DB1. The user database DB1 may store any information regarding users. The information stored in the user database DB1 is not limited to the example shown in FIG. 5.

The user ID is information that can uniquely identify a user. The user ID and password are used by the user to log into the reward provision service. The held points are points held by the user. For example, a user can use points at a physical store, an online store, or an online service. When points are given as rewards, the number of points held increases. The held points may be managed in a database other than the user database DB1. When rewards other than points are awarded, information regarding the other rewards awarded to the user may be stored in the user database DB1.

The participating task information is information regarding tasks in which the user has participated. For example, the participation task information includes the task ID of the task in which the user has participated, and the progress status of the task. The task ID is information that can uniquely identify a task. The progress status of a task is information indicating how far the task is progressing. In this embodiment, since uploading a photographed image corresponds to a task, the photographed image uploaded by the user is stored as the task accomplishment status. The user may be able to delete uploaded captured images from the task progress status.

FIG. 6 is a diagram showing an example of the task database DB2. The task database DB2 is a database that stores various information regarding tasks. For example, the task database DB2 stores a task ID, title, explanatory text, deadline, reward information, and grant information. When a new task is registered in the remuneration service, a record corresponding to the new task is created in the task database DB2. The task database DB2 may store any information regarding tasks. The information stored in the task database DB2 is not limited to the example shown in FIG.

The title and description are the contents of the task to be displayed on screens such as the top screen SC1 and the details screen SC2. The deadline is the deadline date and time of the task. A user needs to complete a task by a deadline. The remuneration information is information regarding remuneration given to the user. For example, the reward information indicates the conditions for acquiring the reward and the details of the reward. The condition for obtaining rewards can be said to be the task itself. The grant information is information regarding the user to whom the reward has been granted. For example, the grant information includes the user ID of the user to whom the reward is granted, and the details of the reward to be granted to the user.

[Task setting section]
The task setting unit 101 sets, as a task in the remuneration service, to upload a photographed image showing an object for which a three-dimensional model is to be generated. Setting a task means storing information regarding the task in the task database DB2. In other words, creating a new task corresponds to setting the task. For example, the task setting may be performed based on a request from the judge terminal 30, or based on a request from a terminal of a person involved other than the judge.

In the example of FIG. 2, a case will be explained in which the object for which a 3D model is generated is a building such as the ABC temple, but the object for which a 3D model is generated can be any object that exists in real space. It's fine. For example, the objects for which the three-dimensional model is generated may be the aforementioned buildings, roads, animals, plants, vehicles, electrical appliances, or miscellaneous goods, or may be other objects not exemplified in this embodiment. There may be. The object may be anything that generates a three-dimensional model for some purpose.

For example, when the task setting unit 101 receives a request for setting a task from the judge terminal 30 or another person's terminal, it issues a task ID so as not to overlap with other tasks. This request also includes the task title, description, deadline, and reward information. This information is specified by the judge or other party. The task setting unit 101 sets a task by creating a new record in the task database DB2 and storing the issued task ID and information such as the task title in the new record. In this embodiment, it is assumed that the assigned information is stored in the task database DB2 after being examined by a judge.

[Image collection department]
The image collection unit 102 collects captured images generated by the imaging unit 26 from the user terminal 20 of the user. Collecting photographed images means receiving image data of the photographed images. In this embodiment, since a task for the reward granting service is set, the image collection unit 102 collects captured images from the user terminal 20 when the task is set. In the example of FIG. 2, the image collection unit 102 collects captured images showing ABC Temple from the user terminal 20 after the task of uploading captured images of ABC Temple is set.

For example, as in this embodiment, when the user needs to select the button B20 to participate in a task, the image collection unit 102 collects captured images from the user terminal 20 of the user who has participated in the task. . When a task that does not particularly require the user's participation is set, the image collection unit 102 may collect captured images from the user terminal 20 without requesting the user to perform an operation such as selecting the button B20. .

Note that the image collection unit 102 only needs to collect at least one photographed image from the user terminal 20, and may collect a plurality of photographed images from the same user terminal 20. An upper limit may be set for the number of captured images that can be uploaded by the user. Conversely, a minimum number of captured images that the user must upload may be determined. For example, the task may not be considered to have been completed unless the user uploads five or more captured images. The image collection unit 102 stores the photographed images acquired from the user terminal 20 in the user database DB1 as the progress status of the task.

[Judgment result acquisition unit]
The determination result acquisition unit 103 acquires a determination result as to whether or not an object for which a three-dimensional model is to be generated is shown in the photographed image. Obtaining the determination result means obtaining data indicating whether or not the object is shown in the captured image. This determination is performed by the server 10, another computer, or manually.

In this embodiment, a visual inspection by a judge is performed, so the judgment result acquisition unit 103 acquires the judgment result from the judge terminal 30 of the judge who judges whether the object is shown in the photographed image. do. The determination result acquisition unit 103 receives, from the judge terminal 30, a determination result transmitted by a determination result transmitting unit 301, which will be described later. In this embodiment, a case will be described in which the review is performed after the task deadline has arrived, but the review may be performed before the task deadline has arrived.

In this embodiment, the determination result acquisition unit 103 not only determines whether an object is shown in a photographed image, but also determines whether an object photographed under predetermined photographing conditions is shown in a photographed image. get. The photographing conditions are the conditions under which the object was photographed. The photographing conditions can also be referred to as information indicating how the object was photographed. For example, at least one of the so-called external parameters and internal parameters corresponds to imaging conditions. The external parameters are the position and orientation of the camera in real space. Intrinsic parameters include the optical center and focal length of the camera.

Note that the photographing conditions may be any conditions at the time of photographing, and are not limited to these examples. For example, the shooting conditions may be the weather, brightness, season, or crowding at the shooting location. For example, the photographing conditions may be the ISO sensitivity, aperture value, or shutter speed of the camera. For example, the photographing conditions may be the type of camera itself such as a single-lens reflex camera or a smartphone, the manufacturer of the camera, the size of the photographed image, or the resolution. The imaging conditions may be other conditions as long as they are useful for generating a three-dimensional model.

FIG. 7 is an example of photographing conditions set for a task. For example, eight photographing conditions such as positions P1 to P8 and orientations V1 to V8 are set for the task. Hereinafter, when positions P1 to P8 are not distinguished, they will simply be referred to as position P. When directions V1 to V8 are not distinguished, they are simply referred to as direction V. In the example of FIG. 7, the number of imaging conditions is set to eight to simplify the explanation, but the number of imaging conditions required for generating a three-dimensional model may be set in the task. . For example, if tens to thousands of images taken from different positions P and directions V are required, dozens to thousands of imaging conditions may be set for the task.

For example, the position P and orientation V are set to surround the object OB existing in the real space RS. The user needs to upload a photographed image of the object OB taken from at least one position P and direction V. The actual position P and direction V may be slightly deviated from the position P and direction V set as the photographing conditions. In this embodiment, the degree of deviation to be allowed depends on the discretion of the judge. The judge allows the deviation to an extent that does not interfere with the generation of the three-dimensional model.

Note that in the example of FIG. 7, the intervals between the positions P are equal, but for a part of the object OB that has a complex shape, the intervals between the positions P may be shortened so that a part of the object OB has a complex shape can be photographed more finely. . Similarly, for parts of the object OB that have complex shapes, the difference in orientation V may be small. Photographing from the same position P in different directions V may be set as the photographing condition. Conversely, photographing from different positions P in the same direction V may be set as the photographing condition. Hereinafter, the reference numerals for objects OB will be omitted.

For example, the determination result acquisition unit 103 acquires a determination result as to whether or not the captured image shows an object that was photographed under a photographing condition that is not sufficient in the collected photographed images. Collected means that the captured images have been determined to be used for generating a three-dimensional model. As in this embodiment, when a photographed image is selected from the examination screen SC5, being selected on the examination screen SC5 corresponds to being collected.

For example, the judge visually checks and selects a captured image that satisfies any of the eight imaging conditions shown in FIG. 7 from among the selected captured images from the examination screen SC5. The judge visually checks and selects the photographed images of the photographing conditions that have not yet been selected from among the eight photographing conditions. The judge selects photographed images one after another so that all eight photographing conditions are met. The judge may select a plurality of captured images that satisfy the same imaging conditions. The determination result acquisition unit 103 acquires the determination result by acquiring information (for example, a file name) that can identify the selected captured image.

Note that in the example of FIG. 7, a case has been described in which external parameters such as position P and direction V correspond to the imaging conditions, but the determination result acquisition unit 103 may acquire determination results of other imaging conditions other than the external parameters. Good too. For example, when the internal parameter corresponds to a photographing condition, the determination result acquisition unit 103 acquires a determination result as to whether the photographed image satisfies the photographing condition indicating that the internal parameter is a predetermined internal parameter. In this embodiment, it is assumed that in this case as well, it is determined whether the photographed image meets the predetermined internal parameters by visual inspection by the judge. The same applies to other photographing conditions such as the weather at the time of photographing mentioned above.

[Remuneration Department]
The reward granting unit 104 grants a reward to the user based on the determination result acquired by the determination result acquisition unit 103. Giving a reward means storing information regarding the reward in the user database DB1, task database DB2, or other database. In this embodiment, increasing the number of held points stored in the user database DB1 or storing grant information in the task database DB2 corresponds to granting a reward. When a reward other than points is awarded, the reward granting unit 104 associates information that can identify the other reward with the user ID in the user database DB1, task database DB2, or other database. Just store it.

For example, the reward granting unit 104 refers to the determination result acquired by the determination result acquisition unit 103 and identifies the user who uploaded the captured image selected by the judge. The reward granting unit 104 increases the number of points held associated with the user ID of the identified user. The reward granting unit 104 stores grant information indicating that the identified user has received a reward in the task database DB2. The reward giving unit 104 gives a reward to the user by executing a series of these processes.

In this embodiment, a case will be described in which all users receive the same remuneration, but remuneration may be provided depending on the user. For example, the reward granting unit 104 may increase the reward given to the user as the number of uploaded images increases. For example, the reward granting unit 104 may set the reward given to the user to increase as the number of captured images used to generate similar three-dimensional models in the past increases. For example, a total reward for all users is determined, and the reward giving unit 104 may give each user a reward obtained by dividing the total reward by the number of users.

[Generation part]
The generation unit 105 generates a three-dimensional model based on the captured image determined to show the object. In the present embodiment, it is also determined whether or not the photographing conditions are satisfied, so the generation unit 105 generates a three-dimensional model based on the photographed image determined to satisfy the photographing conditions. The generation unit 105 does not use a photographed image determined not to satisfy the photographing conditions to generate a three-dimensional model, even if the photographed image is uploaded by the user. In this embodiment, a user who uploads a photographed image that is not used in generating a three-dimensional model is not given a reward, but a certain amount of reward may be given to this user.

For example, the generation unit 105 generates a three-dimensional model based on at least one captured image. The generation unit 105 may generate a three-dimensional model based on one captured image, or may generate a three-dimensional model based on a plurality of captured images. In this embodiment, two methods, a rule-based method and a machine learning method, will be described as three-dimensional model generation methods, but various methods can be used as the three-dimensional model generation methods themselves. Generation of a three-dimensional model is sometimes referred to as three-dimensional shape restoration.

For example, the generation unit 105 generates a three-dimensional model from the captured image using SfM (Structure from Motion) or SLAM (Simultaneous Localization And Mapping) as a rule-based method. SfM is a method that does not use time-series images or videos. SLAM is a method that uses time-series images or videos. The generation unit 105 may separate the foreground and background in the photographed image and then generate the three-dimensional model using a technique called photogrammetry. If the object can be the foreground, the generation unit 105 generates a three-dimensional model based on the foreground of the photographed image. If the object can be a background, the generation unit 105 generates a three-dimensional model based on the background of the photographed image. Note that a method called background subtraction method or U2Net can be used to separate the foreground and background.

For example, the generation unit 105 may generate a three-dimensional model by generating a low-density point group (simple point group) of the captured image based on SfM, which is an example of Bundle Adjustment. The generation unit 105 generates a high-density point group by executing MVS (Multi-View-Stereo) processing based on the parameters of the imaging unit 26 and the generated low-density point group. The generation unit 105 generates a three-dimensional model based on the high-density point group. A texture may be mapped to the three-dimensional model.

For example, the generation unit 105 may generate a three-dimensional model based on a learning model using a machine learning technique. The learning model is trained with a group of captured images for training as a training data set. The learning model may be a model capable of outputting an image of a three-dimensional model viewed from an arbitrary viewpoint. The learning model may be a NeRF-based model, an MLP-based model, or an Attention-based model. For example, as the machine learning method, the method described in Non-Patent Document 1 may be used. A machine learning method that generates a three-dimensional model from a single photographed image may be used.

For example, when the generation unit 105 generates a three-dimensional model from a captured image, the generation unit 105 records data of the three-dimensional model in the data storage unit 100. The three-dimensional model data may be transmitted to the user terminal 20, the judge terminal 30, or another terminal. A virtual image showing how the three-dimensional model is viewed from a virtual camera set in the virtual space may be displayed on these terminals. The position and orientation of the virtual camera may be changeable by operation by a user, a judge, or another person. The generation unit 105 may generate a texture image of a three-dimensional model based on the captured image, and may map the generated texture image onto the three-dimensional model.

[3-2. Functions realized on user terminal]
For example, the user terminal 20 includes a data storage section 200 and a captured image transmission section 201. The data storage section 200 is realized by the storage section 22. The photographed image transmission section 201 is realized by the control section 21.

[Data storage unit]
The data storage unit 200 stores data necessary for the user to use the reward provision service. For example, the data storage section 200 stores photographed images generated by the photographing section 26. The data storage unit 200 may store captured images downloaded from a website or the like instead of the captured images generated by the imaging unit 26.

[Photographed image transmission section]
The photographed image transmitting unit 201 transmits the photographed image to the server 10. For example, the photographed image transmitting unit 201 transmits the photographed image selected on the list screen SC3 to the server 10. When the object is photographed on the spot by the photographing section 26, the photographed image transmitting section 201 transmits the photographed image generated on the spot to the server 10.

[3-3. Functions realized on judge terminal]
For example, the judge terminal 30 includes a data storage section 300 and a judgment result transmitting section 301. The data storage section 300 is realized by the storage section 32. The determination result transmitter 301 is realized by the controller 31.

[Data storage unit]
The data storage unit 300 stores data necessary for determining captured images. For example, the data storage unit 300 stores a browser or a dedicated application for displaying the examination screen SC5.

[Judgment result transmitter]
The determination result transmitter 301 transmits the determination result to the server 10. In this embodiment, the determination result is a visual determination result by a judge. Transmitting the determination result means transmitting data indicating the determination result. For example, the determination result transmitting unit 301 transmits to the server 10 a determination result as to whether the object is shown in the photographed image. In this embodiment, it is determined not only whether the object is shown in the photographed image but also whether the object is photographed under predetermined photographing conditions. The determination result as to whether or not the photograph is taken under the photographing conditions is transmitted.

In this embodiment, the judge selects at least one photographed image that satisfies the photographing conditions from the examination screen SC5. The determination result transmitting unit 301 transmits to the server 10 a determination result indicating at least one captured image selected by the judge. The determination result acquisition unit 103 of the server 10 can specify that the at least one photographed image satisfies the photographing condition by receiving the determination result indicating the at least one photographed image selected by the judge.

Note that, contrary to the present embodiment, the judge may select at least one photographed image that does not satisfy the photographing conditions. In this case as well, the determination result acquisition unit 103 of the server 10 receives the determination result indicating at least one photographed image selected by the judge, so that images other than the at least one photographed image meet the photographing conditions. You can specify what you want to meet.

[4. Processing performed by the image acquisition system]
FIG. 8 is a diagram illustrating an example of processing executed by the image collection system 1. The processing in FIG. 8 is executed by the control units 11, 21, and 31 operating according to programs stored in the storage units 12, 22, and 32, respectively.

As shown in FIG. 8, a process for logging the user into the reward granting service is executed between the server 10 and the user terminal 20 (S1). In S1, the server 10 determines whether the user ID and password input by the user exist in the user database DB1. If the login process is successful, a process for displaying the top screen SC1 on the display unit 25 is executed between the server 10 and the user terminal 20 (S2). In S2, the server 10 specifies tasks within the deadline based on the task database DB2, and causes the display unit 25 to display a top screen SC1 in which the specified tasks are lined up.

When the user selects the button B10 on the top screen SC1, a process for displaying the detailed screen SC2 on the display unit 25 is executed between the server 10 and the user terminal 20 (S3). In S3, the server 10 acquires information such as a descriptive text of the task selected by the user based on the task database DB2, and causes the display unit 25 to display a detailed screen SC2 including the acquired information. When the user selects button B20 on the details screen SC2, a process for allowing the user to participate in the task is executed between the server 10 and the user terminal 20 (S4). In S4, the server 10 stores participating task information including the task ID of the task in which the user participated in the user database DB1 in association with the user ID of the user.

The user terminal 20 causes the display unit 25 to display a list screen SC3 based on the captured images stored in the storage unit 22 (S5). The user terminal 20 transmits the photographed image selected from the list screen SC3 to the server 10 (S6). When the server 10 receives the photographed image from the user terminal 20 (S7), the server 10 stores the photographed image in the user database DB1 as the progress status of the task by the user (S8). Thereafter, various users execute the processes S1 to S8 until the task deadline arrives, and captured images are accumulated in the user database DB1.

When the deadline for the task arrives, a process for displaying the screening screen SC5 on the display unit 35 is executed between the server 10 and the judge terminal 30 (S9). In S9, the server 10 refers to the user database DB1 and acquires captured images uploaded by various users. The server 10 causes the display unit 25 to display the examination screen SC5 including the acquired photographic image. The judge terminal 30 receives the judgment result by the judge based on the detection signal from the operation unit 34 (S10). In S10, the judge terminal 30 identifies the captured image selected by the judge.

The judge terminal 30 transmits the judgment result received in S10 to the server 10 (S11). In S11, the judge terminal 30 transmits to the server 10 information that allows identification of the captured image selected by the judge. The server 10 receives the judgment result by the judge from the judge terminal 30 (S12), and executes a process for awarding a reward to the user based on the received judgment result (S13). ,finish. In S13, the server 10 stores grant information including the user ID of the user who uploaded the captured image selected by the judge and predetermined points in the task database DB2. Points are awarded to the user at any time thereafter.

The image collection system 1 of this embodiment provides a reward to the user who uploaded the photographed image based on the determination result of whether or not the object for which the three-dimensional model is to be generated is shown in the photographed image. The reward can motivate the user to upload the captured image in which the object for which the 3D model is to be generated is shown, so that the captured images that can be used to generate the 3D model can be efficiently used. Can be collected well. For example, if a reward is given when the object is not shown in the photographed image uploaded by the user, the user may end up receiving the reward unfairly even though he has not contributed to the generation of the 3D model. Become. In this regard, since the reward is given based on the determination result of whether the object is shown in the photographed sand image, it is possible to prevent the user from acquiring an unfair reward. For example, with the technology in Non-Patent Document 2, it is known that the accuracy of the 3D model decreases when there are areas where the viewpoints are sparse or concentrated. By collecting images, it becomes easier to collect images of the object taken from all directions, thereby increasing the accuracy of the three-dimensional model. For example, it is possible to collect a wide variety of photographed images not only in the direction of the object but also in other photographing conditions such as the weather.

Further, the image collection system 1 collects photographed images from the user terminal 20 when uploading a photographed image showing an object is set as a task in the reward provision service. This allows users to upload captured images as one of the tasks in a reward granting service where various tasks are set, so it is possible to efficiently collect captured images that can be used to generate a 3D model. . For example, if a user who normally uses a reward service to perform other tasks happens to have a photographed image showing an object for which a 3D model is to be generated, the user may be asked to send the photographed image to the user. Since you can upload images, it becomes easier to collect captured images.

The image collection system 1 also obtains a determination result as to whether an object photographed under predetermined photographing conditions is shown in the photographed image. By making a determination that takes into account not only the object being shown but also the photographing conditions, photographed images that can be used to generate a three-dimensional model can be collected more efficiently.

The image collection system 1 also obtains a determination result as to whether or not the captured image shows an object that was photographed under a photography condition that is not sufficient in the collected captured images. As a result, instead of collecting only similar captured images, it is possible to evenly collect captured images that can be used to generate a three-dimensional model. For example, if you have already collected images of an object taken from the front, there is little need to collect further images of the object taken from the same direction, but By determining whether the object is an image, images of the object taken from various directions can be collected.

Furthermore, the image collection system 1 generates a three-dimensional model based on the captured image determined to show the object. Thereby, the image collection system 1 can generate a three-dimensional model by itself and provide it to a user or the like.

Furthermore, the image collection system 1 acquires from the judge terminal 30 a determination result as to whether or not an object for which a three-dimensional model is to be generated is shown in the photographed image. This allows the judge to exclude captured images that constitute noise that cannot be used to generate a three-dimensional model.

[5. Modified example]
Note that the present disclosure is not limited to the embodiments described above. Changes can be made as appropriate without departing from the spirit of the present disclosure.

FIG. 9 is a diagram illustrating an example of functions realized in a modified example. As shown in FIG. 9, in the modified example described below, the server 10 includes a display control section 106, an imaging condition setting section 107, an imaging determination section 108, and an image analysis section 109. These functions are realized by the control unit 11.

[Modification 1]
For example, in the embodiment, a case has been described in which the judge determines whether or not the photographed image satisfies predetermined photographing conditions. A determination result may be obtained as to whether or not an object photographed under a photographing condition with a difference from the photographing condition equal to or greater than a threshold value is shown in the photographed image. In Modification 1, as in the embodiment, a case where visual judgment is performed by a judge will be exemplified.

The collected photographed images are photographed images that have already been collected at the time of determining the photographing conditions for a certain photographed image. In Modification 1, the photographed images are selected by the judge through visual judgment, so the photographed images selected by the judge correspond to the collected images. The collected photographing conditions are the photographing conditions of the collected photographed images. For example, the difference in imaging conditions is a difference in at least one of an external parameter and an internal parameter. In Modification 1, a case will be described in which a difference in the position P and direction V, which are external parameters, corresponds to a difference in imaging conditions, but a captured image may be selected based on a difference in internal parameters.

For example, the judge selects a photographed image with a photographing condition that is different from the photographing condition of the selected photographic image on the examination screen SC5 by a threshold value or more. In the case of visual judgment by a judge, the threshold value depends on the sense of the judge. The judge selects a photographed image whose difference from the position P and orientation V of the selected photographed image is equal to or greater than a threshold value from among the photographed images that have not yet been selected. Until the judge has selected a sufficient number of captured images to generate a three-dimensional model, the judge selects from among the captured images that have not yet been selected, the difference between the position P and orientation V of the selected captured image that is greater than the threshold value. Select the captured images one after another. The judge terminal 30 transmits the judgment result indicating the photographed image selected by the judge to the server 10 in the same manner as in the embodiment.

The determination result acquisition unit 103 acquires the determination result from the judge terminal 30 in the same manner as in the embodiment. Based on the determination result acquired by the determination result acquisition unit 103, the reward granting unit 104 provides a reward to a user who uploads a photographed image under a photographing condition that differs from the photographing condition of the selected photographed image by a threshold value or more. Give. The method of giving rewards is the same as described in the embodiment. The generation unit 105 generates a three-dimensional model based on a photographed image under a photographing condition that is different from the photographing condition of the selected photographed image by a threshold value or more. The method itself for generating a three-dimensional model from a captured image is the same as described in the embodiment.

Note that, as in Modification 12, which will be described later, when the determination result is obtained by image analysis rather than visual inspection by a judge, the image analysis unit 109, which will be described later, uses the shooting conditions of the collected captured images. , and the photographing conditions of the photographed image to be determined are acquired by image analysis. The image analysis unit 109 determines whether the difference in these imaging conditions is equal to or greater than a threshold value. For example, when the position P and direction V are used as the photographing conditions, the image analysis unit 109 uses the position P and direction V obtained by performing image analysis on the collected photographed images, and the determination target. What is necessary is to determine whether the difference between the position P and the direction V obtained by performing image analysis on the photographed image is equal to or greater than a threshold value. The determination result acquisition unit 103 acquires the determination result by the image analysis unit 109.

The image collection system 1 of Modification 1 determines whether or not the captured image shows an object that was photographed under a photographing condition in which the difference from the collected photographic conditions corresponding to the collected photographed image is equal to or greater than a threshold value. get. This makes it easier to collect captured images that have not yet been collected, rather than collecting only similar captured images, making it possible to evenly collect captured images that can be used to generate a 3D model.

[Modification 2]
For example, in the embodiment, a case has been described in which the judge judges whether or not the photographed image satisfies predetermined photographing conditions. A determination result may be obtained as to whether or not an object photographed under photographing conditions satisfying the constraints is shown in the photographed image. A constraint is a condition for a captured image to be usable for generating a three-dimensional model. For example, the constraints are distance to the object, focal length, angle, degree of blur, or brightness. In modification 2, a case where the focal length corresponds to the constraint will be described. It is assumed that the focal length serving as a constraint is determined in advance.

For example, the determination result acquisition unit 103 acquires the determination result of whether the focal length of a certain captured image falls within a predetermined range as the determination result of the above constraints. In Modification 2, as in the embodiment, a case where visual judgment is performed by a judge will be exemplified. For example, the judge selects a photographed image whose focal length falls within a predetermined range from the examination screen SC5. The judge terminal 30 transmits the judgment result indicating the photographed image selected by the judge to the server 10 in the same manner as in the embodiment.

The determination result acquisition unit 103 acquires the determination result from the judge terminal 30 in the same manner as in the embodiment. Based on the determination result acquired by the determination result acquisition unit 103, the reward granting unit 104 grants a reward to the user who uploads a photographed image showing an object photographed under photographing conditions that satisfy the constraints. The method of giving rewards is the same as described in the embodiment. The generation unit 105 generates a three-dimensional model based on a photographed image showing an object photographed under photographing conditions that satisfy the constraints. The method itself for generating a three-dimensional model from a captured image is the same as described in the embodiment.

Note that, as in Modification 12, which will be described later, when the determination result is obtained by image analysis rather than visual examination by a judge, the image analysis unit 109, which will be described later, determines the photographing conditions of the photographed image by image analysis. get. For example, the image analysis unit 109 extracts feature points by performing SIFT processing or the like on each of the plurality of captured images. The image analysis unit 109 identifies photographing conditions such as focal length based on the pattern of feature points.

For example, the image analysis unit 109 determines whether the acquired imaging conditions satisfy the constraints. For example, when focal length is used as a constraint, the image analysis unit 109 may determine whether the acquired focal length is within a predetermined range by performing image analysis on the captured image. . Note that the image analysis unit 109 may match feature points between captured images. The image analysis unit 109 may specify imaging conditions such as external parameters or internal parameters of the camera by executing Bundle Adjustment based on the feature points and minimizing reprojection errors. The image analysis unit 109 may determine whether the constraints are satisfied based on the identified imaging conditions.

The image collection system 1 of the second modification obtains a determination result as to whether or not an object photographed under photographing conditions that satisfy constraints set for generating a three-dimensional model is shown in a photographed image. By predetermining constraints that are useful for generating 3D models and giving rewards to users who upload captured images that satisfy the constraints, it is possible to generate captured images that are useful for generating 3D models. Can be collected efficiently.

[Modification 3]
For example, in order to generate a three-dimensional model, it may not be possible to map a certain captured image and other captured images unless they overlap to some extent. For this reason, each of a plurality of captured images used to generate a three-dimensional model may be acquired so that they each include a certain amount of overlap.

FIG. 10 is a diagram illustrating an example of a plurality of captured images including mutually overlapping parts. In the example of FIG. 10, the left end of the object is cut off in the photographed image I1. In the photographed image I2, the right end of the object is cut off. The photographed images I1 and I2 have an overlapping portion of about 60% with each other. The ratio of the overlapping portion may be any ratio effective for generating a three-dimensional model, and is not limited to 60%. For example, the ratio of the overlapping portion may be approximately 50% to 59%, or 61% or more, or may be less than 50%. Hereinafter, the reference numerals of the photographed images I1 and I2 will be omitted.

The determination result acquisition unit 103 of Modification 3 acquires a determination result as to whether or not a photographed image shows an object photographed under photographing conditions that partially overlap with other photographed images. In modification 3, a case will be described in which, when a certain user uploads a plurality of photographed images, overlapping portions are determined among the plurality of photographed images uploaded by this user. Therefore, the image collection unit 102 collects a plurality of captured images from the user terminal 20. For example, the determination result acquisition unit 103 acquires a determination result as to whether or not an object photographed under photographing conditions that partially overlap among a plurality of photographed images is shown in the photographed image. In Modification 1, as in the embodiment, a case where visual judgment is performed by a judge will be exemplified.

For example, a list of a plurality of photographed images uploaded by a certain user is displayed on the examination screen SC5. The judge selects a plurality of photographed images that overlap each other at a predetermined ratio or more from the examination screen SC5. In the case of visual judgment by a judge, the threshold value for the ratio of overlapping portions depends on the sense of the judge. The judge terminal 30 transmits the judgment result indicating the photographed image selected by the judge to the server 10 in the same manner as in the embodiment. Note that even if the overlapped portion is 100% and a plurality of identical photographed images are uploaded, it is not useful for generating a three-dimensional model, so a threshold value may exist that is the upper limit of the overlapped portion.

The determination result acquisition unit 103 acquires the determination result from the judge terminal 30 in the same manner as in the embodiment. Based on the determination result acquired by the determination result acquisition unit 103, the reward granting unit 104 rewards a user who uploads a plurality of photographed images showing objects photographed under mutually overlapping photographing conditions. Give. The method of giving rewards is the same as described in the embodiment. The generation unit 105 generates a three-dimensional model based on a plurality of photographed images in which objects are photographed under mutually overlapping photographing conditions.

For example, the generation unit 105 performs mutual mapping based on the overlapping portions of each of the plurality of captured images, and generates a three-dimensional model. A known method can be used to map overlapping portions of photographed images to each other. The generation unit 105 determines that portions where the same feature points are arranged are overlapping portions, and maps the photographed images. The generation unit 105 generates a three-dimensional model based on the mutually mapped captured images. As a method for generating a three-dimensional model from a photographed image, various methods described in the embodiments can be used. For example, the generation unit 105 may generate a three-dimensional model by converting screen coordinates (dimensional coordinates on the photographed image) of feature points in the photographed images mapped to each other into world coordinates through geometry processing.

Note that, as in Modification 12, which will be described later, when the determination result is obtained by image analysis rather than visual inspection by a judge, the image analysis unit 109, which will be described later, analyzes the overlapping portions of each of the plurality of captured images. is obtained by image analysis. For example, the image analysis unit 109 extracts feature points from each of the plurality of captured images, and identifies portions where the arrangement of feature points is similar as mutually overlapping portions. The image analysis unit 109 determines whether these overlapping portions are at a predetermined ratio. The ratio may be a ratio to the width of the photographed image, or may be a ratio of pixels in the overlapping portion among all pixels of the photographed image.

The image collection system 1 of the third modification obtains a determination result as to whether or not a photographed image shows an object photographed under photographing conditions that partially overlap with other photographed images. In order to generate a 3D model, a certain amount of overlapping parts are sometimes required between photographed images, but by giving rewards to users who upload photographed images that include parts that overlap with other photographed images, Captured images useful for generating a three-dimensional model can be efficiently collected.

Furthermore, a determination result is obtained as to whether or not an object photographed under photographing conditions such that some of the plurality of photographed images acquired from a certain user terminal 20 overlap is shown in the photographed image. Thereby, useful captured images for generating a three-dimensional model can be efficiently collected. For example, when setting a threshold for the upper limit of the overlapped portion, it is possible to prevent a certain user from uploading only the same captured images and earning unfair rewards.

[Modification 4]
For example, in Modification 3, overlapping parts are determined among multiple captured images uploaded by a certain user, but there is a difference between captured images uploaded by a certain user and captured images uploaded by another user. An overlapping portion of may be determined. The determination result acquisition unit 103 acquires a determination result as to whether or not the photographed image shows an object photographed under photographing conditions that partially overlap with other photographed images collected from other user terminals 20. do. This differs from Modification 3 in that images uploaded by different users are subject to overlapping portion determination, but is similar to Modification 3 in other respects.

The image collection system 1 of modification 4 determines whether or not an object photographed under photographing conditions that partially overlaps with other photographed images collected from other user terminals is shown in the photographed image. get. Thereby, useful captured images for generating a three-dimensional model can be efficiently collected.

[Modification 5]
For example, when uploading a photographed image, the user may not know under what photographing conditions the photographed image is useful for generating a three-dimensional model. For this reason, the shooting conditions that the user wants to shoot may be displayed on a screen such as the detailed screen SC2. The image collection system 1 includes a display control section 106. The display control unit 106 causes the user terminal 20 to display the shooting conditions. In the fifth modification, a case is exemplified in which the photographing conditions are displayed on the detailed screen SC2, but the photographing conditions may be displayed on other screens such as the top screen SC1. For example, the display control unit 106 causes the user terminal 20 to display the shooting conditions by transmitting display data of a detailed screen SC2 including the shooting conditions.

FIG. 11 is a diagram illustrating an example of how photographing conditions are displayed. In the example of FIG. 11, the display control unit 106 displays the imaging conditions that have not yet been collected on the detailed screen SC2. The imaging conditions that have not yet been collected may be determined by a judge and registered in the data storage unit 100. The display control unit 106 causes the photographing conditions registered in the data storage unit 100 to be displayed on the detailed screen SC2. The imaging conditions that have not yet been collected may be specified by the image processing described in Modifications 1 to 4 instead of being determined by a judge. The user checks the details screen SC2 and uploads a photographed image in which the object is photographed under photographing conditions that have not yet been collected.

The image collection unit 102 collects photographed images from the user terminal after the photographing conditions are displayed on the user terminal 20. This embodiment differs from the embodiment in that the photographing conditions are displayed on the detailed screen SC2, but is otherwise similar to the embodiment. The user is not prohibited from uploading images taken under shooting conditions other than those displayed on the detailed screen SC2, but is allowed to upload images taken under any shooting conditions.

The image collection system 1 of the fifth modification collects photographed images from the user terminal 20 after the photographing conditions are displayed on the user terminal 20. Thereby, it is possible to efficiently collect photographed images under the desired photographing conditions in order to generate a three-dimensional model.

[Modification 6]
For example, imaging conditions useful for generating a three-dimensional model vary depending on the object. If the object has a simple structure, it seems possible to generate a highly accurate three-dimensional model even if the imaging conditions do not differ too much. If the object has a complex structure, it is possible that a highly accurate three-dimensional model cannot be generated unless captured images with different shooting conditions are collected. For this reason, the shooting conditions under which captured images are collected may be set depending on the object.

The image collection system 1 includes a shooting condition setting section 107. The photographing condition setting unit 107 sets photographing conditions based on the object. It is assumed that the relationship between objects and photographing conditions is stored in the data storage unit 100 in advance. This relationship may be in any data format, for example, a table format, a mathematical formula format, a model of a machine learning method, or a part of a program code. The photographing condition setting unit 107 sets photographing conditions associated with the object based on the above relationship.

For example, the photographing condition setting unit 107 sets photographing conditions according to the object by storing the photographing conditions in the task database DB2. The photographing conditions may be recorded in a location other than the task database DB2. For example, the photographing condition setting unit 107 sets more detailed photographing conditions as the shape of the object becomes more complex. Although the method of setting photographing conditions is different from the embodiment, other points are similar to the embodiment.

The image collection system 1 of Modification 6 sets photographing conditions based on the object. This makes it possible to collect captured images that are captured under imaging conditions that match the object, making it easier to collect captured images that are useful for generating three-dimensional objects.

[Modification 7]
For example, if the shooting conditions are determined in advance as in the embodiment, it is possible to specify the shooting conditions that have not yet been collected, but first a 3D model is generated using the shooting conditions that have been collected at present, and then the 3D model is Photography conditions may be set to compensate for areas with low accuracy. The photographing condition setting unit 107 of Modification 7 sets photographing conditions based on a three-dimensional model generated using collected photographed images.

The generation unit 105 of Modification 7 generates a temporary three-dimensional model based on the currently collected captured images at an arbitrary timing before the deadline of the task arrives. The method of generating the temporary three-dimensional model itself may be the same as the method of generating the three-dimensional model described in the embodiment. The photographed images used in the generation of the temporary three-dimensional model may be all the photographed images stored in the user database DB1, or some of the photographed images may be randomly selected. The judge may select a captured image to be used in generating the temporary three-dimensional model.

For example, the imaging condition setting unit 107 projects the temporary three-dimensional model onto a two-dimensional space from a plurality of viewpoints, and then evaluates the accuracy of the temporary three-dimensional model based on an evaluation index such as SNR, PSNR, or SSIM. The photographing condition setting unit 107 sets photographing conditions such that a portion of the temporary three-dimensional model with relatively low accuracy falls within the photographing range. For example, if the accuracy of the rear part of the object is relatively low in the temporary three-dimensional model, the photographing condition setting unit 107 sets the photographing condition such that the object is photographed from the rear (in the example of FIG. 7, position P5 and direction V5). Note that the temporary three-dimensional model may be displayed on the judge terminal 30, and the missing imaging conditions may be specified by the judge. In this case, the imaging condition setting unit 107 may acquire the imaging conditions designated by the judge from the judge terminal 30 and set the acquired imaging conditions.

The image collection system 1 of Modification 7 sets photographing conditions based on a three-dimensional model generated using collected photographed images. This makes it easier to collect captured images that are useful for improving the accuracy of the three-dimensional model. For example, it becomes easier to collect captured images in which parts of objects with relatively low accuracy are captured using a temporary three-dimensional model.

[Modification 8]
For example, the display control unit 106 may cause the user terminal 20 to display a three-dimensional model generated using collected captured images. This three-dimensional model is the temporary three-dimensional model described in Modification Example 7. In Modified Example 8, an example is given in which the display control unit 106 displays a temporary 3D model on the detailed screen SC2, but the display control unit 106 displays the temporary 3D model on other screens other than the detailed screen SC2. It may be displayed.

After the three-dimensional model is displayed on the user terminal 20, the image collection unit 102 collects captured images from the user terminal. This embodiment differs from the embodiment in that a temporary three-dimensional model is displayed on the detailed screen SC2, but is similar to the embodiment in other respects. The user is not prohibited from uploading images taken under shooting conditions other than those that compensate for the low accuracy of the temporary 3D model displayed on the details screen SC2, but the user may upload images under any shooting conditions. It is assumed that the photographed image can be uploaded.

The image collection system 1 of Modification 8 collects captured images from the user terminal 20 after the three-dimensional model is displayed on the user terminal 20. This allows the user to upload captured images after understanding the locations where the accuracy of the three-dimensional model is low. As a result, it becomes easier to collect useful captured images to improve the accuracy of the three-dimensional model. Furthermore, each time the user uploads a photographed image, the accuracy of the three-dimensional model displayed on the user terminal 20 gradually increases, which increases the user's interest and strongly motivates the user to upload the photographed image. Can be done.

[Modification 9]
For example, in the embodiment, a case has been described in which there is one object to be generated for a three-dimensional model, but a plurality of objects may be to be generated. In modification 9, a case will be exemplified in which each of a plurality of buildings on the premises of a temple located in a tourist spot is a three-dimensional model generation target. The determination result acquisition unit 103 acquires a determination result as to whether at least one of the plurality of objects is shown in the photographed image.

Note that this embodiment differs from the embodiment in that the object to be determined is at least one of a plurality of objects, but is similar to the embodiment in other respects. In modification 9, a case will be described in which an object shown in a photographed image is determined by visual inspection by a judge. Therefore, the determination result acquisition unit 103 acquires the determination result by acquiring the type of object determined by the judge. When image analysis is performed as in Modified Example 12 described below, the image analysis unit 109 uses a known object detection method such as pattern matching or machine learning method to detect the object shown in the captured image. All you have to do is specify what it is.

The reward giving unit 104 of Modification 9 gives a reward to the user based on the object shown in the photographed image among the plurality of objects. In modification 9, it is assumed that the relationship between objects and rewards is defined in reward information. For example, this relationship is defined so that rewards are different for each object. The reward giving unit 104 gives the user a reward associated with the object shown in the photographed image. For example, the above relationship is defined such that, among a plurality of buildings placed on a temple site, a particularly popular building receives a relatively high reward.

The image collection system 1 of modification 9 provides a reward to the user based on the object shown in the photographed image among the plurality of objects. As a result, when a plurality of objects are targets for three-dimensional model generation, it becomes easier to collect captured images by, for example, increasing the reward for objects whose captured images are particularly desired to be collected.

[Modification 10]
For example, even if an object is photographed in a photographed image, a portion of the object may be hidden by an obstacle. Since obstacles are only noise when generating a three-dimensional model, the reward giving unit 104 gives a reward to the user based on the determination result and the portion of the photographed image in which the object is shown. You may. For example, the reward giving unit 104 gives a reward to the user such that the more objects are shown in the photographed image, the more the reward is. Note that since the obstacle corresponds to the foreground of the object, the obstacle may be detected using the method of separating the foreground and background described in the embodiment. The reward giving unit 104 calculates the proportion of pixels in which an object is shown among all the pixels of the photographed image, and gives a reward to the user.

The image collection system 1 of Modification 10 provides a reward to the user based on the determination result acquired by the determination result acquisition unit 103 and the portion of the photographed image in which the object is shown. This can prevent rewards from being given to a user who has uploaded a photographed image that is full of obstacles and almost no objects.

[Modification 11]
For example, a user may download a photographed image from another person's SNS without permission and upload the photographed image falsely as a photographed image taken by the user. It is not desirable for the purpose of the reward provision service to provide a reward to the user in such a case. In order to prevent such fraud, it may be determined whether the user has photographed the object himself or not, and a reward may be given to the user who has photographed the object himself.

The image collection system 1 of Modification 11 includes an imaging determination unit 108. The photographing determination unit 108 determines whether a photographed image is generated by the user photographing the object himself or not. For example, the photographing determination unit 108 compares the uploaded photographed image with the photographed image stored in the user database DB1, and if they completely match, the user photographs the object himself. Therefore, it is not determined that the photographed image has been generated, but that the photographed image of another person has been uploaded without permission.

For example, if the name of the photographer is associated with the photographed image as incidental information of the photographed image, the photographing determination unit 108 may combine the name of the photographer of the photographed image with the name of the user stored in the user database DB1. , and if they match, it may be determined whether the captured image was generated by the user photographing the object himself or not. For example, the photographing determination unit 108 collects various photographed images from SNS, compares the uploaded photographed image and the photographed image collected from the SNS, and if they completely match, the user Instead of determining that the photographed image was generated by photographing the object by the person himself/herself, it may be determined that the photographed image was uploaded without permission by another person.

The reward giving unit 104 of Modification 11 gives a reward to the user who has photographed the object himself/herself. For example, the reward giving unit 104 does not give a reward to a user who does not photograph the object himself/herself. The reward granting unit 104 gives a reward to a user who did not photograph the object himself, but may give a lower reward than a user who photographed the object himself. This embodiment differs from the embodiment in that the reward granting unit 104 grants a reward to the user based on the determination result of the photographing determination unit 108, but the method of granting the reward itself is the same as described in the embodiment.

The image collection system 1 of Modification 11 provides a reward to a user who photographs an object by himself/herself. This makes it easier to prevent fraud such as uploading a captured image obtained from someone else's SNS without permission.

[Modification 12]
For example, as described above, the determination result acquired by the determination result acquisition unit 103 may be acquired by image analysis of a photographed image rather than by visual observation by a judge. The image collection system 1 includes an image analysis section 109. The image analysis unit 109 determines whether an object is shown in the captured image by analyzing the captured image. The determination result acquisition unit 103 acquires determination results based on image analysis. An example of the processing by the image analysis unit 109 is as described in Modifications 1 to 3.

For example, the image analysis unit 109 may determine whether or not an object for which a three-dimensional model is to be generated is shown in the photographed image based on an object detection method such as pattern matching or a machine learning method. The image analysis unit 109 acquires the external parameters or internal parameters of the camera as the imaging conditions of the captured image based on the method described in the second modification, and determines whether the imaging conditions described in the embodiment are satisfied. Good too. The image analysis unit 109 may obtain the imaging conditions using another image analysis method and determine whether the imaging conditions described in the embodiment are satisfied.

The image collection system 1 of Modification 12 obtains a determination result based on image analysis of a photographed image. This eliminates the need for a judge to make a judgment. Furthermore, since there is no judgment by a judge, the flow of processing up to the awarding of rewards becomes smoother.

[Other variations]
For example, the modifications described above may be combined.

For example, the object may be an accommodation facility that can be booked through a travel reservation service or a tourist attraction that exists around the accommodation facility. In this case, an accommodation facility or a tourist attraction may be set as a landmark tower, and a larger reward may be given to a user who provides a photographed image of the landmark tower. When collecting images taken at the accommodation facility or its surroundings, only the images showing GPS position information within a predetermined range may be accepted. Basically, objects in public spaces such as outdoor buildings or roads are assumed, but objects placed indoors or the like may also be used.

For example, in the embodiment, a case has been described in which a user is asked to upload a photographed image as a task of a reward service in which various tasks (in the example of FIG. 2, a task of sorting receipts and a task of checking radio field strength) are set. However, it is also possible to have the user upload a photographed image, rather than the task of the remuneration service. For example, a user may upload a photographed image from a website unrelated to the reward provision service.

For example, the functions described as being implemented by the server 10 may be implemented by the user terminal 20, the judge terminal 30, or another computer. For example, functions described as being implemented by one computer may be shared among multiple computers. Each function may be realized by at least one computer.

[6. Additional notes]
For example, the image acquisition system can also have the following configuration.

(1)
an image collection unit that collects captured images generated by the imaging unit from the user terminal of the user;
a determination result acquisition unit that acquires a determination result as to whether or not an object for which a three-dimensional model is to be generated is shown in the photographed image;
a reward giving unit that gives a reward to the user based on the determination result;
Image acquisition system including.
(2)
The image collection system further includes a task setting unit that sets uploading the photographed image in which the object is shown as the task in a reward granting service that grants a reward when a predetermined task is accomplished,
The image collection unit collects the photographed images from the user terminal when the task is set.
The image collection system according to (1).
(3)
The determination result acquisition unit acquires the determination result as to whether or not the object photographed under predetermined photographing conditions is shown in the photographed image.
The image collection system according to (1) or (2).
(4)
In order to generate the three-dimensional model, it is necessary to photograph the object under each of the plurality of photographing conditions,
The determination result acquisition unit acquires the determination result as to whether or not the object photographed under the photographing conditions for which the collected photographic images are insufficient is shown in the photographed image.
The image collection system described in (3).
(5)
The determination result acquisition unit determines whether or not the object photographed under the photographing conditions in which a difference from the collected photographing conditions corresponding to the collected photographic images is equal to or more than a threshold value is shown in the photographed image. Obtain the judgment results,
The image collection system according to (4).
(6)
The determination result acquisition unit acquires the determination result as to whether or not the object photographed under the photographing conditions that satisfy constraints set for generating the three-dimensional model is shown in the photographed image.
The image collection system according to any one of (3) to (5).
(7)
The determination result acquisition unit acquires the determination result as to whether or not the object photographed under the photographing conditions that partially overlap with another photographed image is shown in the photographed image.
The image collection system according to any one of (3) to (5).
(8)
The image collection unit collects the plurality of captured images from the user terminal,
The determination result acquisition unit acquires the determination result as to whether or not the object photographed under the photographing conditions such that some of the plurality of photographed images overlap is shown in the photographed image. ,
The image collection system according to (7).
(9)
The determination result acquisition unit determines whether or not the object photographed under the photographing conditions that partially overlaps the other photographed image collected from another user terminal is shown in the photographed image. obtaining the determination result;
The image collection system according to (7) or (8).
(10)
The image collection system further includes a display control unit that causes the user terminal to display the imaging conditions,
The image collection unit collects the photographed images from the user terminal after the photographing conditions are displayed on the user terminal.
The image collection system according to any one of (3) to (9).
(11)
The image collection system further includes a photographing condition setting unit that sets the photographing conditions based on the object.
The image collection system according to any one of (3) to (10).
(12)
The image collection system further includes a generation unit that generates the three-dimensional model based on the captured image determined to show the object.
The image collection system according to any one of (1) to (11).
(13)
The image collection system further includes a photographing condition setting unit that sets the photographing conditions based on the three-dimensional model generated using the collected photographed images.
The image collection system according to any one of (1) to (12).
(14)
The image collection system further includes a display control unit that causes the user terminal to display the three-dimensional model generated using the collected captured images,
The image collection unit collects the photographed images from the user terminal after the three-dimensional model is displayed on the user terminal.
The image collection system according to any one of (1) to (13).
(15)
A plurality of objects are the generation targets,
The determination result acquisition unit acquires a determination result as to whether at least one of the plurality of objects is shown in the photographed image,
The reward giving unit gives the reward to the user based on the object shown in the captured image among the plurality of objects.
The image collection system according to any one of (1) to (14).
(16)
The reward giving unit gives the reward to the user based on the determination result and a portion of the photographed image in which the object is shown.
The image collection system according to any one of (1) to (15).
(17)
The image collection system further includes a photographing determination unit that determines whether the photographed image is generated by the user photographing the object himself/herself,
The reward giving unit gives the reward to the user who has photographed the object by himself.
The image collection system according to any one of (1) to (16).
(18)
The image collection system further includes an image analysis unit that determines whether the object is shown in the captured image by analyzing the captured image,
The determination result acquisition unit acquires the determination result based on the image analysis.
The image collection system according to any one of (1) to (17).
(19)
The determination result acquisition unit acquires the determination result from a judge terminal of a judge who determines whether the object is shown in the photographed image.
The image collection system according to any one of (1) to (18).

1 image collection system, 10 server, 11, 21, 31 control unit, 12, 22, 32 storage unit, 13, 23, 33 communication unit, 20 user terminal, 24, 34 operation unit, 25, 35 display unit, 26 photography unit, 30 judge terminal, N network, 100 data storage unit, 101 task setting unit, 102 image collection unit, 103 judgment result acquisition unit, 104 reward giving unit, 105 generation unit, 106 display control unit, 107 photographing condition setting unit , 108 photography determination unit, 109 image analysis unit, 200 data storage unit, 201 photography image transmission unit, 300 data storage unit, 301 determination result transmission unit, B10, B20 button, DB1 user database, DB2 task database, SC1 top screen, SC2 details screen, SC3 list screen, SC4 completion screen, SC5 review screen.

Claims (22)

an image collection unit that collects captured images generated by the imaging unit from the user terminal of the user;
Image analysis of the photographed image showing the object that is a target for generation of a three-dimensional model, and for which it is necessary to photograph the object under each of a plurality of photographing conditions in order to generate the three-dimensional model. an image analysis unit that acquires the photographing conditions of the photographed image and determines whether or not the object photographed under the photographing conditions that are insufficient in the collected photographic images is shown in the photographed image;
a reward granting unit that grants a reward to the user based on the determination result of the image analysis unit ;
Image acquisition system including . The image collection system further includes a task setting unit that sets uploading the photographed image in which the object is shown as the task in a reward granting service that grants a reward when a predetermined task is accomplished,
The image collection unit collects the photographed images from the user terminal when the task is set.
The image acquisition system according to claim 1. The image analysis unit determines whether or not the object photographed under the photographing condition in which a difference from the collected photographing condition corresponding to the collected photographic image is equal to or more than a threshold is shown in the photographed image. do,
The image acquisition system according to claim 1 or 2. an image collection unit that collects captured images generated by the imaging unit from the user terminal of the user;
Determine whether or not the object for which a three-dimensional model is to be generated, which was photographed under predetermined photographing conditions that satisfy constraints set for generating the three-dimensional model, is shown in the photographed image. , an image analysis unit that makes a determination by image analysis of the photographed image ;
a reward granting unit that grants a reward to the user based on the determination result of the image analysis unit ;
Image acquisition system including. an image collection unit that collects captured images generated by the imaging unit from the user terminal of the user;
The photographed image determines whether or not the object for which a three-dimensional model is to be generated, which was photographed under predetermined photographing conditions that partially overlap with other photographed images, is shown in the photographed image. an image analysis unit that makes a determination by image analysis of the image ;
a reward granting unit that grants a reward to the user based on the determination result of the image analysis unit ;
Image acquisition system including. The image collection unit collects the plurality of captured images from the user terminal,
The image analysis unit determines whether or not the object photographed under the photographing conditions such that some of the plurality of photographed images overlap is shown in the photographed image.
The image acquisition system according to claim 5. The image analysis unit determines whether the object photographed under the photographing conditions that partially overlaps the other photographed image collected from another user terminal is shown in the photographed image. do,
The image acquisition system according to claim 5 or 6. The image collection system further includes a display control unit that causes the user terminal to display the imaging conditions,
The image collection unit collects the photographed images from the user terminal after the photographing conditions are displayed on the user terminal.
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. The image collection system further includes a photographing condition setting unit that sets the photographing conditions based on the object.
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. The image collection system further includes a generation unit that generates the three-dimensional model based on the captured image determined to show the object.
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. The image collection system projects the temporary three-dimensional model generated using the collected captured images onto a two-dimensional space from multiple viewpoints, and calculates the temporary three-dimensional model based on the evaluation index of SNR, PSNR, or SSIM. further comprising a photographing condition setting unit that evaluates the accuracy of the dimensional model and sets the photographing conditions such that a portion of the temporary three-dimensional model with relatively low accuracy falls within the photographing range;
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. The image collection system further includes a display control unit that causes the user terminal to display the three-dimensional model generated using the collected captured images,
The image collection unit collects the photographed images from the user terminal after the three-dimensional model is displayed on the user terminal.
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. A plurality of objects are the generation targets,
The image analysis determines whether at least one of the plurality of objects is shown in the captured image,
The reward giving unit gives the reward to the user based on the object shown in the captured image among the plurality of objects.
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. The reward giving unit gives the reward to the user based on the determination result and a portion of the photographed image in which the object is shown.
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. The image collection system further includes a photographing determination unit that determines whether the photographed image is generated by the user photographing the object himself/herself,
The reward granting unit grants the reward to the user who has photographed the object himself.
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. The image analysis unit determines whether the object is shown in the captured image by performing image analysis on the captured image .
The image acquisition system according to claim 1 or 2 or 4 or 5 or 6. The computer is
an image collection step of collecting captured images generated by the imaging unit from the user's user terminal;
Image analysis of the photographed image showing the object that is a target for generation of a three-dimensional model, and for which it is necessary to photograph the object under each of a plurality of photographing conditions in order to generate the three-dimensional model. an image analysis step of acquiring the photographing conditions of the photographed image and determining whether or not the object photographed under the photographing conditions, which is insufficient in the collected photographic images, is shown in the photographed image;
a reward granting step of granting a reward to the user based on the determination result of the image analysis step ;
Image collection method to perform . The computer is
an image collection step of collecting captured images generated by the imaging unit from the user's user terminal;
Determine whether or not the object for which a three-dimensional model is to be generated, which was photographed under predetermined photographing conditions that satisfy constraints set for generating the three-dimensional model, is shown in the photographed image. , an image analysis step of determining by image analysis of the photographed image ;
a reward granting step of granting a reward to the user based on the determination result of the image analysis step ;
Image collection method to perform. The computer is
an image collection step of collecting captured images generated by the imaging unit from the user's user terminal;
The photographed image determines whether or not the object for which a three-dimensional model is to be generated, which was photographed under predetermined photographing conditions that partially overlap with other photographed images, is shown in the photographed image. an image analysis step of determining by image analysis of the image ;
a reward granting step of granting a reward to the user based on the determination result of the image analysis step ;
Image collection method to perform. an image collection unit that collects captured images generated by the imaging unit from the user terminal of the user;
Image analysis of the photographed image showing the object that is a target for generation of a three-dimensional model, and for which it is necessary to photograph the object under each of a plurality of photographing conditions in order to generate the three-dimensional model. an image analysis unit that acquires the photographing conditions of the photographed image and determines whether or not the object photographed under the photographing conditions that are insufficient in the collected photographic images is shown in the photographed image;
a reward granting unit that grants a reward to the user based on the determination result of the image analysis unit ;
A program that makes a computer function as an image collection unit that collects captured images generated by the imaging unit from the user terminal of the user;
Determine whether or not the object for which a three-dimensional model is to be generated, which was photographed under predetermined photographing conditions that satisfy constraints set for generating the three-dimensional model, is shown in the photographed image. , an image analysis unit that makes a determination by image analysis of the photographed image ;
a reward granting unit that grants a reward to the user based on the determination result of the image analysis unit ;
A program that allows a computer to function as a computer. an image collection unit that collects captured images generated by the imaging unit from the user terminal of the user;
The photographed image determines whether or not the object for which a three-dimensional model is to be generated, which was photographed under predetermined photographing conditions that partially overlap with other photographed images, is shown in the photographed image. an image analysis unit that makes a determination by image analysis of the image ;
a reward granting unit that grants a reward to the user based on the determination result of the image analysis unit ;
A program that allows a computer to function as a computer.

Images