JP7360854B2 - Program, information processing method, and information processing device - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Publication date: July 16, 2020 Publication location: Melchari application [Publications, etc.] Publication date: July 18, 2020 Publication address: https://apps ．． apple. com/jp/app/id1341611829

The present disclosure relates to a program, an information processing method, and an information processing device.

In recent years, ride sharing services have been increasing, and systems are being developed to manage the rental and return of shared vehicles. For example, a technology has been disclosed that manages a rental vehicle by installing a transmitter in the vehicle and communicating with a user's mobile terminal (for example, see Patent Document 1).

Japanese Patent Application Publication No. 2019-75090

Here, in order to manage the location and unlocking of the shared vehicle, it is conceivable to manage the vehicle by assigning an information code (bar code, QR code (registered trademark), NFC, etc.) to the vehicle. In this case, when the vehicle is returned, there is a need to manage whether the vehicle has been returned to a predetermined location, whether there are any defects in the vehicle, and a need to reduce the management cost. Therefore, when returning the vehicle, it is possible to send the image taken of the vehicle to the company providing the sharing service, but if the image is not taken properly, the operator will have to check and contact you, and the management will not be able to manage it. There is a problem that costs increase.

One of the objects of the present disclosure is to provide a program, an information processing method, and an information processing device that can reduce the increase in management costs when returning a vehicle in a sharing service.

A program according to an embodiment of the present disclosure causes one or more processors included in an information processing device to recognize a predetermined marker from image data that is being captured, and to create a three-dimensional space based on the image data that is being captured. and a self-position estimation process to estimate the position of the imaging device in the three-dimensional space, and the method includes: performing a surrounding environment mapping process to create an image, and a self-position estimation process to estimate the position of the imaging device in the three-dimensional space; A correspondence relationship is specified, and based on the correspondence relationship, the position or orientation of the imaging device is guided to a position or orientation where the object including the predetermined marker can be photographed.

According to the disclosed technology, it is possible to reduce the increase in management costs when returning a vehicle in a sharing service.

1 is a diagram illustrating each configuration example of an information processing system 1 in an embodiment. FIG. FIG. 1 is a block diagram showing an example of a user terminal 10 according to an embodiment. FIG. 2 is a block diagram illustrating an example of a server 20 according to an embodiment. 2 is a diagram illustrating an example of processing data 134 processed by the user terminal 10 according to the embodiment. FIG. It is a figure showing an example of the 1st data of management data 233 concerning an embodiment. It is a figure showing an example of the 2nd data of management data 233 concerning an embodiment. 2 is a flowchart illustrating an example of processing of the information processing system 1 according to the embodiment. FIG. 3 is a sequence diagram regarding image data processing according to the embodiment. FIG. 6 is a diagram illustrating an example of screen transition according to the embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the same elements are given the same reference numerals and redundant explanations will be omitted.

[Embodiment]
In the embodiment, a system will be described in which, when a vehicle is returned in a sharing service, a photograph of the vehicle is taken and transmitted to a server, so that the server side manages whether the vehicle has been returned appropriately. In the embodiment, an information code is given to the vehicle as an example of a predetermined marker for managing location information, unlocking, etc., and when the user reads this information code with a mobile terminal etc., the server side , the user unlocks the vehicle, and manages the location from which the user starts using the vehicle. The vehicle will be described below using a bicycle as an example, but it may also be a vehicle such as a car or a motorcycle. Further, the information code given to a vehicle includes an ID for identifying the vehicle, and may be given solely for the purpose of identifying the vehicle.

<System application example>
FIG. 1 is a diagram illustrating each configuration example of an information processing system 1 in an embodiment. In the example shown in FIG. 1, information processing devices 10A and 10B used by each user, bicycles 30A and 30B that can be used by each user, and a server 20A that manages a sharing service are connected via a network N. Note that information codes 40A and 40B are given to the bicycles 30A and 30B, respectively.

The information processing devices 10A and 10B are, for example, a smartphone, a mobile phone (feature phone), a computer, a PDA (Personal Digital Assistant), and have a built-in or external imaging device. Note that when the information processing apparatuses 10A and 10B are not distinguished, they are referred to as information processing apparatus 10. The information processing device 10 is also referred to as a user terminal 10 because it is a terminal used by a user.

The information processing device 20A is, for example, a server, and may be composed of one or more devices. The information processing device 20A also executes processing for managing the sharing service. Hereinafter, the information processing device 20A will also be referred to as the server 20.

Bicycles 30A, 30B are basically located at predetermined stations, used by users, and returned to the same or different stations. Note that when the bicycles 30A and 30B are not distinguished, they are also referred to as bicycle 30. It is sufficient that the information codes 40A and 40B are predetermined markers that can be recognized during image capture. Below, when the information codes 40A and 40B are not distinguished, they will also be referred to as information code 40. Note that any number of bicycles 30 may be prepared.

In the example shown in FIG. 1, the user terminal 10 reads the information code 40 of the bicycle 30 that the user will use from now on, and transmits the read information to the server 20. For example, the server 20 uses the bicycle ID included in the read information to transmit an unlock signal for unlocking the key part of the bicycle 30 corresponding to the bicycle ID. The key unit unlocks the key upon receiving the unlock signal. This allows the user to use the bicycle 30.

The user finishes using the bicycle 30, returns the bicycle 30 to a predetermined station, and locks the key. At this time, the user reads the information code 40 given to the bicycle 30 using the user terminal 10, and is prompted by the user terminal 10 to take a photograph of the returned bicycle 30. The user moves the user terminal 10 as guided by the user terminal 10 and photographs the bicycle 30. Image data captured by this photography is transmitted to the server 20. Thereby, a company providing a sharing service can acquire desired image data through guidance, and can prevent an increase in management costs due to acquisition of inappropriate image data.

Note that the configuration may not be such that the user is prompted to take a picture of the bicycle 30 every time the bicycle 30 is returned, but may be configured to be prompted when a predetermined condition is met. The predetermined conditions include, for example, when the return process is not performed correctly (the location where the bicycle 30 is returned is different from the predetermined station, etc.), when the time period when the bicycle 30 is returned is within a specific time period, or when the user who returned the bicycle This is the case when a specific user is free.

<Example of configuration>
FIG. 2 is a block diagram showing an example of the user terminal 10 according to the embodiment. User terminal 10 typically includes one or more processing units (CPUs) 110, one or more network or other communication interfaces 120, memory 130, user interface 150, imaging device 160, and components thereof. including one or more communication buses 170 for interconnecting the .

The user interface 150 is, for example, a user interface 150 that includes a display device 151 and a touch panel 152. The touch panel 152 may also be a keyboard and/or a mouse (or some other input device such as a pointing device).

Memory 130 is, for example, a high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state storage, and may also include one or more magnetic disk storage, optical disk storage, flash memory devices, or It may also be a nonvolatile memory such as another nonvolatile solid state storage device.

Further, as another example of the memory 130, one or more storage devices installed remotely from the CPU 110 may be used. In some embodiments, memory 130 stores the following programs, modules and data structures, or a subset thereof.

Operating system 131 includes, for example, procedures for handling various basic system services and performing tasks using hardware.

The network communications module 132 may, for example, connect the user terminal 10 to another computer via one or more network communications interfaces 120 and one or more communications networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc. Used to connect over a network.

The imaging data 133 includes image data being captured by the imaging device 160. The image data may be either moving image data or still image data.

Processed data 134 includes data processed by an image processing module 136, which will be described later, using image data being captured. For example, the processed data 134 may include feature point data of image data, a three-dimensional space model created based on the feature point data, or a self-image in which the position of the imaging device 160 within the created three-dimensional space model is estimated. It includes position information and the like, and will be described later using FIG.

The recognition module 135 recognizes the information code 40 included in the image data being photographed, and reads the information included in the information code 40. For example, the information code 40 includes a code ID, a bicycle ID for identifying a bicycle, and the like. The read information is sent to the server 20 using the network communication module 132.

The image processing module 136 performs surrounding environment mapping processing (environmental map generation processing) that creates a three-dimensional space based on image data being captured, and self-position estimation processing that estimates the position of the imaging device in the three-dimensional space. Execute. For example, the image processing module 136 uses technology such as SLAM (Simultaneous Localization and Mapping) to perform surrounding environment mapping processing that creates a space by recognizing planes and objects based on image data, and the image processing that is being taken. Self-position estimation processing for estimating where the device 160 is located in space is simultaneously performed.

The image processing module 136 also performs surrounding environment mapping processing and self-position estimation processing based on VIO (Visual-Inertial Odometry) using ARKit provided by Apple (registered trademark), ARCore provided by Google, etc. You may do so. In the VIO technology, sensing information from the acceleration sensor and gyro sensor of the user terminal 10 is also used to estimate the self-position.

Based on the recognized information code 40, the image processing module 136 identifies the correspondence between the position of the information code 40 and the position of the imaging device 160 in the three-dimensional space. For example, the image processing module 136 associates the position of the information code 40 recognized by the recognition module 135 with the position of the imaging device 160 estimated by the self-position estimation process in the generated three-dimensional space. This makes it possible to specify the positional relationship between the information code 40 and the imaging device 160 in the three-dimensional space.

Based on the identified correspondence, the guidance module 137 guides the imaging device 160 to a position or orientation (angle) where it can photograph an object (for example, the bicycle 30) that includes the information code 40. For example, the distance between the information code 40 and the imaging device 160 is set in the three-dimensional space, and the guidance module 137 guides the information code 40 and the imaging device 160 so that the distance is equal to or greater than the set distance. The guidance may be performed by displaying guidance information on the display 151, or by using sound or vibration. Here, an example in which guidance information is displayed will be described, but the present invention is not limited to this example.

Further, the guidance module 137 may also guide the direction of the imaging device 160 using a gyro sensor or the like. Even if the positional relationship between the bicycle 30 and the imaging device 160 satisfies the conditions, the bicycle 30 cannot be properly imaged unless the orientation is appropriate. Therefore, the guiding module 137 may guide the position and/or orientation of the imaging device 160 so that the position and/or orientation of the imaging device 160 satisfies the imaging conditions.

This makes it possible to guide the entire object (for example, bicycle 30) to fit within the field of view of the imaging device 30, and as a result of the guidance, appropriate image data is sent to the server 20, which allows management This makes it possible to reduce cost increases. In addition to the bicycle 30, the target object may also include a station port and the like. In addition, the user only needs to read the information code 40 and move the imaging device 160 (including the user terminal 10) according to the guidance, making it possible, for example, to take appropriate images in a series of steps. Furthermore, it is not necessary to image the entire object, and in some cases, guidance may be provided to image a predetermined position of the object.

Further, the display control module 138 superimposes guidance information on the image data being photographed and controls the display on the screen. For example, the display control module 138 adds guidance information such as text to image data displayed on the screen of the display 151 based on the correspondence between the position of the information code 40 and the position of the imaging device 160 in the three-dimensional space. Superimpose. The guidance information to be superimposed includes text, images (for example, arrows, etc.), and the like.

As a result, the user terminal 10 controls the display of guidance information for appropriately photographing the bicycle 30, etc., so that the image capturing device 160 can use technology such as augmented reality (AR). It becomes possible to visually guide the position of the As a result, it becomes possible to quickly capture an appropriate image and improve the processing efficiency of the user terminal 10.

Furthermore, the display control module 138 may superimpose guidance information on the image data that varies depending on the correspondence between the position of the information code 40 and the position of the imaging device 160 in the three-dimensional space. For example, the display control module 138 changes the size, orientation, etc. of the guidance information according to the above-mentioned correspondence. As a specific example, a predetermined image can be used as guidance information, and the display control module 138 uses AR technology to direct the size and orientation of this image to a predetermined size and orientation, thereby controlling the image capturing device 160. The position and orientation guide the bicycle 30 to a position and orientation where photography can be performed. For example, the image to be superimposed may be superimposed on the detection position of the information code 40.

Thereby, the user terminal 10 can guide the position of the imaging device 160 more visually by controlling the display while changing the guidance information for appropriately photographing the bicycle 30 etc. using AR technology. becomes possible. As a result, it becomes possible to take appropriate images more quickly and further improve the processing efficiency of the user terminal 10.

Further, the guidance module 137 performs control to permit the imaging device 160 to take a picture if the position or orientation of the imaging device 160 satisfies a predetermined condition regarding the position or orientation in which the bicycle 30 or the like including the information code 40 can be photographed. It's okay. For example, as a result of the guidance, the guidance module 137 determines whether or not the positional relationship between the information code 40 and the imaging device 160 in the three-dimensional space satisfies the conditions for photographing the bicycle 30. If the determination result is affirmative (if the conditions are met), the guidance module 137 allows the imaging device 160 to take an image.

Furthermore, when photography is permitted, the display control module 138 may display on the screen a UI component (for example, a photography button) that enables photography. When the user presses this shooting button, the imaging device 160 images the scenery appearing on the screen and acquires image data. The display control module 138 can also display the shooting button from the screen during guidance, and if the shooting conditions are not met, the shooting button can be disabled (disabled) and the shooting conditions are met. If this happens, the shooting button may be changed to a state where it can be pressed (enabled). Furthermore, the display control module 138 may control the image capturing device 160 to automatically capture images when the capturing conditions are satisfied.

Thereby, the user terminal 10 can notify the user of appropriate photographing timing. As a result, the user can capture an appropriate image of the bicycle 30 and the like without losing the timing for photographing.

Further, the network communication module 132 may transmit image data including the bicycle 30 and the like captured by the imaging device 160 to another information processing device (for example, the server 20). For example, the network communication module 132 transmits image data captured by the imaging device 160 to the server 20 under the control of the guidance module 137.

This allows the user terminal 10 to transmit image data including the bicycle 30 and the like to the server 20, and the server 20 to understand the status of the bicycle 30 at the time of return. By increasing the possibility that an appropriate image will be transmitted through guidance, the management at the time of return at the server 20 becomes easier, and an increase in management costs can be reduced. Note that the condition of the bicycle 30 includes, for example, whether there is a breakdown or abnormality, and these determinations can be made using image recognition.

Further, the network communication module 132 may transmit the first location information of the user terminal 10 to another information processing device (for example, the server 20) when the information code 40 is recognized. And/or the network communication module 132 may transmit the second position information of the user terminal 10 to another information processing device (for example, the server 20) when image data including the bicycle 30 etc. is captured. For example, the location information of the user terminal 10 is latitude and longitude information that can be obtained by a GPS (Global Positioning System) function included in the terminal.

This makes it possible for another information processing device (for example, the server 20) to acquire the position information of the bicycle 30 at the time of reading the information code 40, and to determine whether the bicycle 30 has been returned to a predetermined station. Further, if the bicycle 30 has a communication function, a position measurement function, etc., the bicycle 30 may transmit position information when the key is locked. In this case, the position information transmitted from the bicycle 30 and the The location information transmitted from the user terminal 10 may be compared to determine whether they each indicate the same station.

Furthermore, by transmitting the second location information together with the image data transmission, the user terminal 10 can determine whether or not the second location information at the time of image data transmission indicates a predetermined station. . As a result, when reading the information code 40 and transmitting image data, it becomes possible on the server 20 side to determine whether or not the item was returned at an appropriate location.

Further, the guidance module 137 may acquire warning information when another information processing device (for example, the server 20) determines that the first position information is not at a predetermined position. In this case, the display control module 138 may control display of the warning information on the screen. For example, if the server 20 side determines that the vehicle has been abandoned without being returned to a predetermined station based on the first location information, warning information is sent to the user terminal 10 to request the user to return the vehicle to a predetermined station. do.

Thereby, the user terminal 10 can prompt the user to return the bicycle 30 to an appropriate location by receiving the warning information. Further, it is possible to inform the warning information of the position of a predetermined station near the first position information, and it is also possible to improve the return rate to the predetermined station.

Further, the guidance module 137 hides the warning information when it is determined by another information processing device (for example, the server 20) that the second position information is at a predetermined position (the position of a predetermined station set in advance). You may also obtain request information. In this case, the display control module 138 may hide the warning information according to the request information. For example, even if the first location information indicates a drop-off location, if the second location information indicates the location of an appropriate station, the warning information is controlled to be hidden.

Thereby, the server 20 can grasp the position of the bicycle 30 by acquiring position information at a predetermined timing, and can appropriately manage the position of the bicycle 30 and the like.

Further, the warning information may include time information that urges the bicycle 30 to be moved to a predetermined position within a predetermined time. For example, a countdown from a predetermined time (for example, 10 minutes) is displayed on the screen.

Thereby, it is possible to prompt the user to quickly return the vehicle to an appropriate location. Furthermore, if the device is not returned to the appropriate location within a predetermined time, it becomes possible to impose a penalty on the user.

The guidance module 137 may recognize the bicycle 30 based on the image data and further determine whether the bicycle 30 satisfies the photographing conditions. For example, the guidance module 137 performs object recognition from the image data and recognizes that the object is the bicycle 30. In this case, the guidance module 137 may determine whether the entire bicycle 30 is within the angle of view (determination of the photographing conditions), and may permit photographing if the photographing conditions are met. Note that object recognition may be performed by the image processing module 136.

This makes it possible to increase the possibility of obtaining a more appropriate image by recognizing objects not only based on positional relationships in three-dimensional space but also from image data being captured.

Note that one or more processing units (CPUs) 110 read and execute each module from the memory 130 as necessary. For example, one or more processing units (CPUs) 110 may constitute a communication unit by executing a network communication module 132 stored in memory 130. In addition, one or more processing units (CPUs) 110 execute an image processing module 136, a guidance module 137, and a display control module 138 stored in the memory 130, respectively, thereby controlling the image processing section, the guidance section, and the display control module 138. A control unit may also be configured.

In other embodiments, image processing module 136, guidance module 137, and display control module 138 may be standalone applications stored in memory 130 of user terminal 10. Standalone applications include, but are not limited to, image processing applications, guidance applications, and display control applications. In yet other embodiments, the guidance module 137, display control module 138 may be an add-on or plug-in to another application.

Each of the elements shown above may be stored in one or more of the aforementioned storage devices. Each of the modules shown above corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., sets of instructions) illustrated above need not be implemented as separate software programs, procedures or modules, and therefore various subsets of these modules may be combined in various embodiments. Alternatively, it may be reconfigured. In some embodiments, memory 130 may store a subset of the modules and data structures shown above. Furthermore, memory 130 may store additional modules and data structures not described above.

FIG. 3 is a block diagram showing an example of the server 20 according to the embodiment. Server 20 typically includes one or more processing units (CPUs) 210, one or more network or other communication interfaces 220, memory 230, and one or more processors for interconnecting these components. A plurality of communication buses 270 are included.

Server 20 may optionally include a user interface 250, including a display device (not shown) and input devices such as a keyboard and/or mouse (or some other pointing device, not shown). be able to.

Memory 130 is, for example, a high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state storage, and may also include one or more magnetic disk storage, optical disk storage, flash memory devices, or It may also be a nonvolatile memory such as another nonvolatile solid state storage device.

Other examples of the memory 230 include one or more storage devices installed remotely from the CPU 210. In some embodiments, memory 230 stores the following programs, modules and data structures, or a subset thereof.

Operating system 231 includes, for example, procedures for handling various basic system services and performing tasks using hardware.

Network communications module 232 may, for example, connect server 20 to other computers through one or more communications network interfaces 220 and one or more communications networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc. Used to connect via.

The management data 233 includes first data regarding the usage status of the bicycle 30 and second data including station position information. The first data may include, for example, an ID for identifying the bicycle 30, an ID for identifying the information code 40, usage time, usage start position, return location, etc. The second data may include, for example, an ID for identifying the station, location information, and a name. The management data 233 will be described later using FIGS. 4 and 5.

The management module 234 executes processing for managing the use of the bicycle 30. For example, the management module 234 manages unlocking and locking of each bicycle 30, manages the position information of each bicycle 30, manages the usage time of each bicycle 30, and monitors the usage status of each user's bicycle 30. manage.

Management module 234 includes an alert module 235. The warning module 235 determines whether the location information transmitted from the user terminal 10 is within a predetermined range from the location of a predetermined station. If the position information is not within a predetermined range, control is performed to send warning information to the user terminal 10. At this time, the warning module 235 may include countdown information, nearby station information, or penalty information in the warning information.

The warning module 235 determines whether the location information transmitted again from the user terminal 10 is within a predetermined range from the location of the predetermined station. Even if the previous position information is not within the predetermined range, if the subsequent position information is within the predetermined range, the warning information is controlled to be hidden. For example, the warning module 235 may stop sending warning information.

Note that the warning module 235 may acquire image data from the user terminal 10 and control whether or not to transmit warning information based on the image data. For example, the warning module 235 analyzes the image data, determines whether the image data includes an image of the bicycle and the station, and determines whether the bicycle has been returned correctly, and transmits warning information based on the determination result. You can control whether or not to do so. Further, the warning module 235 may output image data so that the administrator can visually confirm the image data.

Each of the elements shown above may be stored in one or more of the storage devices mentioned above. Each of the modules shown above corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., sets of instructions) illustrated above need not be implemented as separate software programs, procedures or modules, and therefore various subsets of these modules may be combined in various embodiments. Or it can be reconfigured. In some embodiments, memory 230 may store a subset of the modules and data structures shown above. Furthermore, memory 230 may store additional modules and data structures not described above. One or more processing units (CPUs) 210 may constitute a management section and a warning section by respectively executing a management module 234 and a warning module 235 stored in the memory 230.

Although FIG. 3 depicts a "server," FIG. 3 is intended less as a structural overview of the embodiments described herein and more as an illustration of the various features that may be present in a set of servers. . In fact, items shown separately could be combined, as well as certain items could be made separate, as will be recognized by those skilled in the art. For example, items shown separately in FIG. 3 could be implemented on a single server, and a single item could be implemented by one or more servers.

<Example of data structure>
FIG. 4 is a diagram illustrating an example of processing data 134 processed by the user terminal 10 according to the embodiment. In the example shown in FIG. 4, the processed data 134 includes, for example, each feature point extracted using AR technology, a three-dimensional space model generated from each feature point, and the position of the imaging device 160 in the three-dimensional space. includes information on the estimated self-position and the position of the information code 40 in three-dimensional space. Note that the processing data 134 may further include sensor information (camera tilt, etc.) acquired by the user terminal 10.

The image processing module 136 associates the position of the information code 40 recognized by the recognition module 135 with the position of the imaging device 160 estimated by the self-position estimation process in the generated three-dimensional space. This allows the guidance module 137 to determine whether this correspondence relationship satisfies predetermined imaging conditions. The photographing conditions include, for example, that the position of the information code 40 and the position of the imaging device 160 estimated by self-position estimation processing are separated by a predetermined distance, and that the angle of view of the imaging device 160 includes the predetermined code 40. Things like that.

FIG. 5 is a diagram showing an example of first data of the management data 233 according to the embodiment. In the example shown in FIG. 5, the first data includes code ID, status, location information 1, location information 2, usage time, and the like.

The code ID is an ID that can identify an object such as the bicycle 30, and the code ID is an ID for identifying the information code 40. The status indicates whether the lock is unlocked or locked. Location information 1 indicates the usage start location, and location information 2 indicates the return location. The usage time indicates the usage time of the sharing service. Note that the first data is an example, and the current status of each bicycle 30 and usage history information may be managed separately. For example, the current status includes the current location and status, and the usage history information includes the usage start location, return location, and usage time.

Note that the first data is not limited to the configuration shown in FIG. 5. For example, the first data may be managed by being divided into a usage table and a status table. In this case, the usage table includes, for example, the ID of the user using the bicycle, the code ID, location information 1, location information 2, and usage time, and the status table includes the code ID, the status, and the usage time of the bicycle 30. The location information may also be included.

FIG. 6 is a diagram illustrating an example of second data of the management data 233 according to the embodiment. In the example shown in FIG. 6, the second data includes a station ID and station position information. The station location information may be used to determine whether the bicycle 30 has been returned to an appropriate location.

The data structure described above is just an example, and is not limited to this example. For example, the management data 233 may hold usage history information of the bicycle 30 for each user.

<Operation explanation>
Next, the operation of the information processing system 1 according to the embodiment will be explained. FIG. 7 is a flowchart illustrating an example of processing of the information processing system 1 according to the embodiment.

(Step S102)
The recognition module 135 of the user terminal 10 guides the user to capture a predetermined marker (for example, the information code 40) from the screen being captured. For example, the recognition module 135 displays an information code frame on the screen.

(Step S104)
The recognition module 135 detects and recognizes the information code 40 included in the image data being photographed.

(Step S106)
The image processing module 136 executes surrounding environment mapping processing and self-position estimation processing based on the image data being captured. The image processing module 136 associates the position of the recognized information code 40 with the position of the imaging device 160. Based on this correspondence, the guidance module 137 guides the imaging device 160 to a position or orientation where the object including the information code 40 can be photographed. At this time, the display control module 138 may display the AR content to guide the user.

(Step S108)
If the correspondence relationship satisfies a predetermined shooting condition, the guidance module 137 activates the shooting button or controls the display on the screen.

(Step S110)
The imaging device 160 captures an image at the timing when the user presses the shooting button, and obtains image data including the object. Note that the imaging device 160 may automatically perform imaging when predetermined imaging conditions are met.

(Step S112)
As a result of the guidance, the network communication module 132 transmits image data captured by the imaging device 160 to the server 20. This makes it possible to acquire appropriate image data and transmit the appropriate image data to the server 20.

FIG. 8 is a sequence diagram regarding image data processing according to the embodiment. The example shown in FIG. 8 includes processing performed after image data is captured by guidance of the guidance module 137.

(Step S202)
The guidance module 137 of the user terminal 10 acquires the captured image data.

(Step S204)
The guidance module 137 uses the GPS function or the like of the user terminal 10 to obtain current position information.
(Step S206)

Network communication module 132 transmits image data and location information to server 20.

(Step S208)
The management module 234 of the server 20 determines whether the bicycle 30 should be returned based on the acquired image data and position information. For example, the management module 234 determines whether the bicycle 30 has been returned to an appropriate position or determines whether the bicycle 30 is malfunctioning or the like based on the image data. Furthermore, the management module 234 (warning module 235) determines whether the acquired position information is within a predetermined range from the position of a predetermined station. Here, if the determination result is affirmative (determined that the bicycle 30 is in the predetermined position), the process proceeds to step S210, and if the determination result is negative (determined that the bicycle 30 is not in the predetermined position), the process proceeds to step S220. .

(Step S210)
Warning module 235 sends warning information to user terminal 10 . The warning information may include, for example, countdown information, penalty information, or information prompting the user to return the vehicle to a nearby predetermined station.

(Step S212)
The display control module 138 of the user terminal 10 controls the display of warning information on the screen. For example, the warning information is notified on the screen as a pop-up or the like, or the display is controlled by switching screens in an application.

(Step S214)
It is assumed that the user returns the bicycle 30 to a predetermined station and photographs the bicycle 30 again. At this time, the guidance module 137 of the user terminal 10 acquires image data.

(Step S216)
The guidance module 137 uses the GPS function or the like of the user terminal 10 to obtain current position information.

(Step S218)
Network communication module 132 transmits image data and location information to server 20. After this, the management module 234 of the server 20 performs the return determination again. Here, it is assumed that the determination result is positive, but if the determination result is negative, step S210 is executed again.

(Step S220)
The network communication module 232 of the server 20 transmits return report information to the user terminal 10.

(Step S222)
The display control module 138 of the user terminal 10 controls the display of return report information on the screen. For example, the report information may include the usage start location, return location, usage time, cost, etc.

As described above, by transmitting appropriate image data to the server 20 side, the disclosed technology can reduce the increase in management costs when returning a vehicle in a sharing service. Furthermore, the disclosed technology makes it possible to obtain appropriate image data through guidance from the process of reading the information code 40, which is a predetermined marker.

<Screen example>
Next, an example displayed on the screen of the user terminal 10 will be described. FIG. 9 is a diagram showing an example of screen transition according to the embodiment. The upper left screen shown in FIG. 9 is an example of a screen that guides photographing of the information code 40. For example, "Please copy the QR code" is displayed on the screen.

The screen at the top center is an example of a screen where the information code 40 is recognized and the AR content A10 is displayed on the screen. This AR content A10 is information that guides the position of the imaging device 160. In this screen example, since the AR content A10 is not contained within the screen, the user is guided to keep a distance from the bicycle 30.

The upper right screen is an example of a screen when the user is leaving the bicycle 30. As the user moves away from the bicycle 30, the entire AR content A10 becomes visible, and this AR content A10 guides the user to move away from the bicycle 30 a little more.

The lower right screen is an example of a screen in which the photographing button B10 is activated when the distance between the imaging device 160 and the bicycle 30 becomes a predetermined distance. In all of the upper screen examples, the shooting button B10 is disabled. This allows the user to easily understand that it is only necessary to take a photograph at this position.

The screen at the center of the bottom is an example of a screen for confirming the transmission of image data acquired by photography. When the user presses "Send" displayed on this confirmation screen, the image data taken and acquired by the guidance can be transmitted to the server 20.

Note that the AR content A10 is preferably an image that changes depending on the positional relationship between the information code 10 and the imaging device 160, and is preferably represented by an image that changes the distance and direction.

Note that the disclosed technology is not limited to the embodiments described above, and can be implemented in various other forms without departing from the gist of the disclosed technology. Therefore, the above embodiments are merely illustrative in all respects, and should not be interpreted in a limited manner. For example, each of the above-mentioned processing steps can be arbitrarily changed in order or executed in parallel as long as there is no inconsistency in the processing contents.

The program of each embodiment of the present disclosure may be provided in a state stored in a computer-readable storage medium. The storage medium is a "non-temporary tangible medium" that can store a program. Programs include, by way of example and not limitation, software programs and computer programs.

[Modified example]
Further, modifications of each of the above-described embodiments are shown below.

<Modification 1>
In the first modification, in the above embodiment, the bicycle 30 used in a sharing service is used as an example of the object to be imaged. good. For example, the object to be photographed may be any object that falls within the field of view of the imaging device 160, such as furniture, clothing, stationery, electrical appliances, daily necessities, and vehicles. Furthermore, the technology of the present disclosure can be applied to any object that is placed at a predetermined position. Note that the system configuration and the like in Modification 1 are similar to the configuration of the embodiment.

<Modification 2>
In the second modification, the bicycle 30 is the only object to be photographed in the above embodiment, but this is not intended to exclude photographing other objects. For example, in the case of a bicycle 30, the photographing distance and direction (angle) may be set in advance as photographing conditions so that the port installed at a predetermined station and the bicycle 30 are photographed together.

1 Information processing system 10, 10A, 10B Information processing device (user terminal)
20, 20A Information processing device (server)
110, 210 Processing unit (CPU)
120, 220 Network communication interface 130, 230 Memory 131, 231 Operating system 132, 232 Network communication module 133 Imaging data 134 Processing data 135 Recognition module 136 Image processing module 137 Guidance module 138 Display control module 150 User interface 160 Imaging device 170 Communication bus 233 Management data 234 Management module 235 Warning module

Claims (8)

One or more processors included in the information processing device,
Recognizing a predetermined marker from image data being captured;
executing a surrounding environment mapping process for creating a three-dimensional space and a self-position estimation process for estimating the position of the imaging device in the three-dimensional space, based on the image data being captured;
identifying a correspondence relationship between the position of the predetermined marker and the position of the imaging device in the three-dimensional space;
Based on the correspondence relationship, the position or orientation of the imaging device guides the object including the predetermined marker to a position or orientation where it can be photographed ;
transmitting image data including the object captured by the imaging device to another information processing device, and transmitting first position information of the information processing device to the other information processing device when the predetermined marker is recognized; transmitting, including transmitting to an information processing device;
acquiring warning information when the other information processing device determines that the first location information is not at a predetermined location;
controlling the display of the warning information on a screen;
transmitting second position information of the information processing device to the other information processing device when image data including the target object is captured;
acquiring request information for hiding the warning information when the other information processing device determines that the second location information is at the predetermined position;
Hiding the warning information according to the request information;
A program to run. the one or more processors;
2. The program according to claim 1, further causing guidance information to be superimposed on the image data being photographed and to be displayed on a screen. The display control includes:
3. The program according to claim 2, further comprising superimposing guidance information that varies depending on the correspondence relationship on the image data. Said inducing:
4. The method according to claim 1, further comprising: permitting photographing by the imaging device when the position or orientation of the imaging device satisfies a predetermined condition regarding a position or orientation in which an object including the predetermined marker can be photographed. The program described in Section. The program according to claim 1 , wherein the warning information includes time information urging the object to be moved to the predetermined position within a predetermined time. the one or more processors;
6. The program according to claim 1, wherein the program recognizes the object based on the image data and further determines whether the object satisfies a photographing condition. One or more processors included in the information processing device,
Recognizing a predetermined marker from image data being captured;
executing a surrounding environment mapping process for creating a three-dimensional space and a self-position estimation process for estimating the position of the imaging device in the three-dimensional space, based on the image data being captured;
identifying a correspondence relationship between the position of the predetermined marker and the position of the imaging device in the three-dimensional space;
Based on the correspondence relationship, the position or orientation of the imaging device guides the object including the predetermined marker to a position or orientation where it can be photographed;
transmitting image data including the object captured by the imaging device to another information processing device, and transmitting first position information of the information processing device to the other information processing device when the predetermined marker is recognized; transmitting, including transmitting to an information processing device;
acquiring warning information when the other information processing device determines that the first location information is not at a predetermined location;
controlling the display of the warning information on a screen;
transmitting second position information of the information processing device to the other information processing device when image data including the target object is captured;
acquiring request information for hiding the warning information when the other information processing device determines that the second location information is at the predetermined position;
Hiding the warning information according to the request information;
An information processing method that performs. An information processing device including one or more processors,
The one or more processors,
Recognizing a predetermined marker from image data being captured;
executing a surrounding environment mapping process for creating a three-dimensional space and a self-position estimation process for estimating the position of the imaging device in the three-dimensional space, based on the image data being captured;
identifying a correspondence relationship between the position of the predetermined marker and the position of the imaging device in the three-dimensional space;
Based on the correspondence relationship, the position or orientation of the imaging device guides the object including the predetermined marker to a position or orientation where it can be photographed;
transmitting image data including the object captured by the imaging device to another information processing device, and transmitting first position information of the information processing device to the other information processing device when the predetermined marker is recognized; transmitting, including transmitting to an information processing device;
acquiring warning information when the other information processing device determines that the first location information is not at a predetermined location;
controlling the display of the warning information on a screen;
transmitting second position information of the information processing device to the other information processing device when image data including the target object is captured;
acquiring request information for hiding the warning information when the other information processing device determines that the second location information is at the predetermined position;
Hiding the warning information according to the request information;
An information processing device that executes.

Images