JP2021039586A - Program, information processing method, and information processing apparatus - Google Patents

Abstract

To provide a program, an information processing method, and an information processing apparatus for reducing an increase in management costs when returning a vehicle in a sharing service.SOLUTION: A program causes one or more processors included in an information processing apparatus to execute processing of: recognizing a predetermined marker from image data being photographed; executing, based on the image data being photographed, an ambient environment mapping process that creates three-dimensional space and a self-position estimation process that estimates a position of an imaging apparatus in the three-dimensional space; identifying correspondence relationship between a position of the predetermined marker and the position of the imaging apparatus in the three-dimensional space; and guiding an object including the predetermined marker to a position or orientation in which the object can be photographed, depending on the position or orientation of the imaging apparatus based on the correspondence relationship.SELECTED DRAWING: Figure 7

Description

The present disclosure relates to programs, information processing methods, and information processing devices.

In recent years, the number of services for sharing vehicles has increased, and systems for managing the lending and returning of shared vehicles are being developed. For example, a technique for managing a vehicle by mounting a transmitter on a rental vehicle and communicating with a user's mobile terminal has been disclosed (see, for example, Patent Document 1).

JP-A-2019-75090

Here, in order to manage the position and unlocking of the shared vehicle, it is conceivable to assign an information code (bar code, QR code (registered trademark), NFC, etc.) to the vehicle to manage the vehicle or the like. In this case, when the vehicle is returned, there is a need to manage whether the vehicle has been returned to a predetermined place, whether the vehicle has any defects, and a need to reduce the management cost. Therefore, when returning the vehicle, it is conceivable to send the image of the vehicle to the company that provides the sharing service, but if proper shooting is not done, confirmation and contact by the operator will occur and management will be done instead. There is a problem that the cost increases.

One of the purposes of the present disclosure is to provide a program, an information processing method, and an information processing device capable of reducing an increase in management cost when returning a vehicle in a sharing service.

The program according to the embodiment of the present disclosure recognizes a predetermined marker from the image data being photographed by one or a plurality of processors included in the information processing apparatus, and is based on the image data being photographed in a three-dimensional space. The surrounding environment mapping process for creating the above and the self-position estimation process for estimating the position of the image device in the three-dimensional space are executed, and the position of the predetermined marker and the position of the image pickup device in the three-dimensional space are determined. Identifying the correspondence relationship, and based on the correspondence relationship, the position or orientation of the imaging device guides the object including the predetermined marker to a position or orientation that can be photographed.

According to the disclosed technology, it is possible to reduce the increase in management cost when returning a vehicle in the sharing service.

It is a figure which shows each configuration example of the information processing system 1 in embodiment. It is a block diagram which shows an example of the user terminal 10 which concerns on embodiment. It is a block diagram which shows an example of the server 20 which concerns on embodiment. It is a figure which shows an example of the processing data 134 processed by the user terminal 10 which concerns on embodiment. It is a figure which shows an example of the 1st data of the management data 233 which concerns on embodiment. It is a figure which shows an example of the 2nd data of the management data 233 which concerns on embodiment. It is a flowchart which shows an example of the processing of the information processing system 1 which concerns on embodiment. It is a sequence diagram concerning the processing of the image data which concerns on embodiment. It is a figure which shows the transition example of the screen which concerns on embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The same elements are designated by the same reference numerals, and duplicate description will be omitted.

[Embodiment]
In the embodiment, when returning a vehicle in the sharing service, a system for managing whether or not the vehicle has been properly returned on the server side by taking a picture of the vehicle and transmitting it to the server will be described. In the embodiment, an information code is given to the vehicle as an example of a predetermined marker for managing position information, unlocking, etc., and the user reads the information code with a mobile terminal or the like on the server side. , Unlock the vehicle and manage from which position the user started using it. Hereinafter, the vehicle will be described by taking a bicycle as an example, but the vehicle may be a vehicle such as a car or a motorcycle. Further, the information code given to the vehicle also includes an ID for identifying the vehicle, and may be given only for the purpose of identifying the vehicle.

<System application example>
FIG. 1 is a diagram showing each configuration example of the information processing system 1 in the embodiment. In the example shown in FIG. 1, the information processing devices 10A and 10B used by each user, the bicycles 30A and 30B available to each user, and the server 20A that manages the sharing service are connected via the network N. Information codes 40A and 40B are assigned to the bicycles 30A and 30B, respectively.

The information processing devices 10A and 10B are, for example, a smartphone, a mobile phone (feature phone), a computer, a PDA (Personal Digital Assistant), and the like, and have a built-in or external imaging device. When the information processing devices 10A and 10B are not distinguished, they are referred to as the information processing device 10. Since the information processing device 10 is a terminal used by the user, it is also referred to as a user terminal 10.

The information processing device 20A is, for example, a server, and may be composed of one or more devices. Further, the information processing device 20A executes a process of managing the sharing service. Hereinafter, the information processing device 20A is also referred to as a server 20.

The bicycles 30A and 30B are basically located at a predetermined station, used by the user, and returned to the same or another station. When the bicycles 30A and 30B are not distinguished, they are also referred to as the bicycle 30. The information codes 40A and 40B may be predetermined markers that can be recognized during image capture. In the following, when the information codes 40A and 40B are not distinguished, they are also referred to as the information code 40. Any number of bicycles 30 may be prepared.

In the example shown in FIG. 1, the user terminal 10 reads the information code 40 of the bicycle 30 to be used by the user, and transmits the read information to the server 20. For example, the server 20 uses the bicycle ID included in the read information to transmit an unlocking signal for unlocking the key to the key portion of the bicycle 30 corresponding to the bicycle ID. When the key unit receives the unlock signal, it unlocks the key. This allows the user to use the bicycle 30.

The user finishes using the bicycle 30, returns the bicycle 30 to a predetermined station, and locks the key. At this time, the user reads the information code 40 given to the bicycle 30 by using the user terminal 10, and the user terminal 10 prompts the user to take a picture of the returned bicycle 30. The user moves the user terminal 10 and photographs the bicycle 30 as guided by the user terminal 10. The image data captured by this shooting is transmitted to the server 20. As a result, the company that provides the sharing service can acquire desired image data by guidance, and can prevent an increase in management cost due to acquisition of inappropriate image data.

It should be noted that the photography of the bicycle 30 does not have to be prompted every time it is returned, and it may be such that it is prompted when a predetermined condition is satisfied. The predetermined conditions are, for example, when the return process is not performed correctly (the position where the bicycle 30 is returned is different from the predetermined station, etc.), when the returned time zone is a specific time zone, or when the returned user Is vacant for a specific user, etc.

<Example of configuration>
FIG. 2 is a block diagram showing an example of the user terminal 10 according to the embodiment. The user terminal 10 typically has one or more processing units (CPUs) 110, one or more networks or other communication interfaces 120, memory 130, user interfaces 150, imaging devices 160, and components thereof. Includes one or more communication buses 170 for interconnecting.

The user interface 150 is, for example, a user interface 150 including a display device 151 and a touch panel 152. Further, the touch panel 152 may be a keyboard and / or a mouse (or an input device such as some other pointing device).

The memory 130 is, for example, a high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid-state storage, and one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or It may be a non-volatile memory such as another non-volatile solid-state storage device.

Further, as another example of the memory 130, one or a plurality of storage devices installed remotely from the CPU 110 may be used. In certain embodiments, the memory 130 stores the following programs, modules and data structures, or a subset thereof.

Operating system 131 includes, for example, procedures for processing various basic system services and performing tasks using hardware.

The network communication module 132 connects the user terminal 10 to another computer, for example, one or more network communication interfaces 120, and one or more communications such as the Internet, another wide area network, a local area network, and a metropolitan area network. Used to connect over the network.

The image pickup data 133 includes image data being photographed by the image pickup apparatus 160. The image data may be either moving image data or still image data.

The processing data 134 includes data processed by the image processing module 136, which will be described later, using the image data during shooting. For example, the processed data 134 is a self in which the position of the image pickup device 160 is estimated in the feature point data of the image data, the three-dimensional space model created based on the feature point data, or the created three-dimensional space model. It includes position information and the like, and will be described later with reference to FIG.

The recognition module 135 recognizes the information code 40 included in the image data being photographed and reads the information contained in the information code 40. For example, the information code 40 includes a code ID, a bicycle ID that identifies a bicycle, and the like. The read information is transmitted to the server 20 by using the network communication module 132.

The image processing module 136 includes an ambient environment mapping process (environment map generation process) that creates a three-dimensional space based on the image data being photographed, and a self-position estimation process that estimates the position of the imaging device in the three-dimensional space. To execute. For example, the image processing module 136 uses a technique such as SLAM (Simultaneous Localization and Mapping) to perform ambient environment mapping processing that recognizes a plane or an object based on image data to create a space, and imaging that is being photographed. The self-position estimation process for estimating where the position of the device 160 is in the space is performed at the same time.

Further, the image processing module 136 uses ARKit provided by Apple (registered trademark), ARCore provided by Google, etc., and performs ambient environment mapping processing and self-position estimation processing based on VIO (Visual-Inertial Odometry). May be done. In VIO technology, self-position estimation is performed using sensing information from the acceleration sensor and gyro sensor of the user terminal 10.

The image processing module 136 identifies the correspondence between the position of the information code 40 and the position of the image pickup apparatus 160 in the three-dimensional space based on the recognized information code 40. For example, the image processing module 136 associates the position of the information code 40 recognized by the recognition module 135 with the position of the image pickup apparatus 160 estimated by the self-position estimation process in the generated three-dimensional space. This makes it possible to specify the positional relationship between the information code 40 and the image pickup apparatus 160 in the three-dimensional space.

The guidance module 137 guides the position or orientation (angle) of the image pickup apparatus 160 to a position or orientation in which an object (for example, a bicycle 30) including the information code 40 can be photographed, based on the specified correspondence. For example, the guidance module 137 has a set distance between the information code 40 and the image pickup device 160 in the three-dimensional space, and guides the guide module 137 so that the distance is equal to or greater than the set distance. As a method of guiding, guidance information may be displayed on the display 151, or guidance may be performed by sound or vibration. Here, the display of guidance information will be described as an example, but the present invention is not limited to this example.

Further, the guidance module 137 may also guide the orientation of the image pickup apparatus 160 by using a gyro sensor or the like. Even if the positional relationship between the bicycle 30 and the image pickup device 160 satisfies the condition, the bicycle 30 cannot be properly imaged unless the orientation is appropriate. Therefore, the guidance module 137 may guide the position and / or orientation of the image pickup apparatus 160 so that the position and / or orientation of the image pickup apparatus 160 satisfies the imaging condition.

This makes it possible to guide the entire object (for example, the bicycle 30) so that it fits within the angle of view of the image pickup device 30, and as a result of the guidance, appropriate image data is transmitted to the server 20 for management. It becomes possible to reduce the increase in cost. The object may include a station port or the like in addition to the bicycle 30. Further, the user only needs to read the information code 40 and move the image pickup apparatus 160 (including the user terminal 10) according to the guidance, and can take an appropriate image in a series of flows, for example. Further, it is not always necessary to image the entire object, and in some cases, guidance may be made to image a predetermined position of the object.

Further, the display control module 138 superimposes guidance information on the image data during shooting and controls the display on the screen. For example, the display control module 138 converts guidance information such as text into image data displayed on the screen of the display 151 based on the correspondence between the position of the information code 40 and the position of the image pickup device 160 in the three-dimensional space. Superimpose. The superimposed guidance information includes text, images (for example, arrows, etc.) and the like.

As a result, the user terminal 10 displays and controls guidance information for appropriately photographing the bicycle 30 and the like, thereby using a technology (AR technology) such as Augmented Reality (AR) to display and control the image pickup device 160. It becomes possible to visually guide the position of. As a result, it is possible to quickly take an appropriate image and improve the processing efficiency of the user terminal 10.

Further, the display control module 138 may superimpose guidance information that fluctuates according to the correspondence between the position of the information code 40 and the position of the image pickup apparatus 160 in the three-dimensional space on the image data. For example, the display control module 138 changes the size and direction of the guidance information according to the correspondence relationship described above. As a specific example, a predetermined image can be used as guidance information, and the display control module 138 uses AR technology to direct the size and orientation of this image to a predetermined size and orientation, so that the image pickup apparatus 160 can be used. The position and orientation guide the bicycle 30 to a position and orientation in which the image can be taken. For example, the image to be superimposed may be superimposed on the detection position of the information code 40.

As a result, the user terminal 10 more visually guides the position of the image pickup device 160 by controlling the display of the guidance information for appropriately photographing the bicycle 30 or the like while changing it by using the AR technology. Becomes possible. As a result, it is possible to take an appropriate image more quickly and further improve the processing efficiency of the user terminal 10.

Further, the guidance module 137 is controlled so as to allow the image pickup device 160 to take a picture when the position or the direction of the image pickup device 160 satisfies a predetermined condition regarding the position or the direction in which the bicycle 30 or the like including the information code 40 can be photographed. You may. For example, the guidance module 137 determines, as a result of guidance, whether or not the correspondence relationship between the information code 40 and the image pickup device 160 in the three-dimensional space satisfies the condition for photographing the bicycle 30. When the determination result is affirmative (when the condition is satisfied), the guidance module 137 permits the imaging device 160 to take a picture.

Further, the display control module 138 may display and control a UI component (for example, a shooting button) that enables shooting on the screen when shooting is permitted. When the user presses this shooting button, the image pickup device 160 captures the scenery reflected on the screen and acquires the image data. In addition, the display control module 138 can display the shooting button from the screen during guidance, and if the shooting condition is not satisfied, the shooting button cannot be pressed (disabled) and the shooting condition is satisfied. If this happens, you may change to a state where you can press the shooting button (enabled). Further, the display control module 138 may control the image pickup apparatus 160 to automatically take a picture when the shooting conditions are satisfied.

As a result, the user terminal 10 can inform the user of an appropriate shooting timing. As a result, the user can acquire an appropriate image of the bicycle 30 or the like without losing the shooting timing, and the user terminal 10 can acquire an appropriate image of the bicycle 30 or the like.

Further, the network communication module 132 may transmit image data including the bicycle 30 and the like imaged by the image pickup device 160 to another information processing device (for example, the server 20). For example, the network communication module 132 transmits the image data captured by the image pickup device 160 to the server 20 under the control of the guidance module 137.

As a result, the user terminal 10 can transmit image data including the bicycle 30 and the like to the server 20, and the server 20 can grasp the status of the bicycle 30 at the time of return. By increasing the possibility that an appropriate image is transmitted by the guidance, management at the time of return on the server 20 becomes easy, and an increase in management cost can be reduced. The situation of the bicycle 30 is, for example, whether or not there is a failure or abnormality, and these determinations can be made by using image recognition.

Further, the network communication module 132 may transmit the first position information of the user terminal 10 to another information processing device (for example, the server 20) when the information code 40 is recognized. And / or, the network communication module 132 may transmit the second position information of the user terminal 10 to another information processing device (for example, the server 20) when the image data including the bicycle 30 or the like is captured. For example, the position information of the user terminal 10 is latitude and longitude information that can be acquired by the GPS (Global Positioning System) function or the like provided in the terminal.

As a result, another information processing device (for example, the server 20) can acquire the position information at the time of reading the information code 40 of the bicycle 30 and determine whether or not the bicycle 30 has been returned to a predetermined station. Further, when the bicycle 30 has a communication function, a position measurement function, or the like, the bicycle 30 may transmit the position information when the key is locked. In this case, the position information transmitted from the bicycle 30 and the position information are used. The location information transmitted from the user terminal 10 may be compared to determine whether or not each indicates the same station.

Further, the user terminal 10 can determine whether or not the second position information at the time of transmitting the image data indicates a predetermined station by transmitting the second position information together with the transmission of the image data. .. As a result, when the information code 40 is read and when the image data is transmitted, it becomes possible for the server 20 to grasp whether or not the information code 40 has been returned at an appropriate place.

Further, the guidance module 137 may acquire warning information when it is determined by another information processing device (for example, the server 20) that the first position information is not at a predetermined position. In this case, the display control module 138 may display and control the warning information on the screen. For example, when the server 20 determines that the user has been abandoned without being returned to the predetermined station based on the first location information, warning information for requesting the user to return to the predetermined station is transmitted to the user terminal 10. To do.

As a result, the user terminal 10 can urge the user to return the bicycle 30 to an appropriate place by receiving the warning information. Further, it is possible to inform the warning information of the position of a predetermined station close to the first position information, and it is also possible to improve the return rate to the predetermined station.

Further, the guidance module 137 hides the warning information when it is determined by another information processing device (for example, the server 20) that the second position information is at a predetermined position (a preset predetermined station position). You may get the request information to make. In this case, the display control module 138 may hide the warning information according to the request information. For example, even if the first position information indicates the drop-off position, the warning information is controlled so as to be hidden if the second position information indicates an appropriate station position.

As a result, the server 20 can grasp the position of the bicycle 30 by acquiring the position information at a predetermined timing, and can appropriately manage the position of the bicycle 30 and the like.

Further, the warning information may include time information for prompting the bicycle 30 to move to a predetermined position within a predetermined time. For example, a countdown is displayed on the screen from a predetermined time (for example, 10 minutes).

This can encourage the user to quickly return to the appropriate location. It is also possible to impose a penalty on the user if it is not returned to the proper position within the specified time.

The guidance module 137 may recognize the bicycle 30 based on the image data and further determine whether or not the bicycle 30 satisfies the photographing condition. For example, the guidance module 137 recognizes an object from image data and recognizes that the object is a bicycle 30. In this case, the guidance module 137 may determine whether the entire bicycle 30 is within the angle of view (determination of shooting conditions), and if the shooting conditions are satisfied, shooting may be permitted. The object recognition may be performed by the image processing module 136.

As a result, it is possible to increase the possibility of acquiring a more appropriate image by recognizing the object not only from the positional relationship in the three-dimensional space but also from the image data during shooting.

The one or more processing devices (CPUs) 110 read and execute each module from the memory 130 as needed. For example, one or more processing units (CPUs) 110 may configure a communication unit by executing the network communication module 132 stored in the memory 130. Further, one or a plurality of processing devices (CPUs) 110 execute the image processing module 136, the guidance module 137, and the display control module 138 stored in the memory 130, respectively, to execute the image processing unit, the guidance unit, and the display. A control unit may be configured.

In another embodiment, the image processing module 136, the guidance module 137, and the display control module 138 may be a stand-alone application stored in the memory 130 of the user terminal 10. Stand-alone applications include, but are not limited to, image processing applications, guidance applications, and display control applications. In yet another embodiment, the guidance module 137 and the display control module 138 may be add-ons or plug-ins to another application.

Each of the elements shown above may be stored in one or more of the aforementioned storage devices. Each of the modules shown above corresponds to a set of instructions for performing the functions described above. The modules or programs (ie, sets of instructions) shown above do not need to be implemented as separate software programs, procedures or modules, so various subsets of these modules may be combined or combined in different embodiments. Alternatively, it may be reconstructed. In certain embodiments, the memory 130 may store a subset of the modules and data structures shown above. In addition, memory 130 may store additional modules and data structures not described above.

FIG. 3 is a block diagram showing an example of the server 20 according to the embodiment. The server 20 typically has one or more processing units (CPUs) 210, one or more networks or other communication interfaces 220, memory 230, and one or more for interconnecting these components. Includes a plurality of communication buses 270.

The server 20 may optionally include a user interface 250, including a display device (not shown) and an input device such as a keyboard and / or mouse (or some other pointing device, etc. not shown). be able to.

The memory 130 is, for example, a high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid-state storage, and one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or It may be a non-volatile memory such as another non-volatile solid-state storage device.

Further, another example of the memory 230 may include one or a plurality of storage devices installed remotely from the CPU 210. In certain embodiments, the memory 230 stores the following programs, modules and data structures, or a subset thereof.

Operating system 231 includes, for example, procedures for processing various basic system services and performing tasks using hardware.

The network communication module 232 connects the server 20 to another computer, for example, one or more communication network interfaces 220 and one or more communication networks such as the Internet, other wide area networks, local area networks, and metropolitan area networks. Used to connect via.

The management data 233 includes the first data regarding the usage status of the bicycle 30 and the second data including the position information of the station. The first data may include, for example, an ID for identifying the bicycle 30, an ID for identifying the information code 40, a usage time, a usage start position, a return position, and the like. The second data may include, for example, an ID for identifying the station, location information, a name, and the like. The management data 233 will be described later with reference to FIGS. 4 and 5.

The management module 234 executes a process of managing the use of the bicycle 30. For example, the management module 234 manages the unlocking and locking of the key of each bicycle 30, manages the position information of each bicycle 30, manages the usage time of each bicycle 30, and displays the usage status of each user's bicycle 30. Manage it.

The management module 234 includes a warning module 235. The warning module 235 determines whether or not the position information transmitted from the user terminal 10 is within a predetermined range from the position of the predetermined station. If the position information is not within the predetermined range, the warning information is controlled to be transmitted to the user terminal 10. At this time, the warning module 235 may include countdown information, information on nearby stations, and penalty information in the warning information.

The warning module 235 determines whether or not the position information retransmitted from the user terminal 10 is within a predetermined range from the position of the predetermined station. Even if the previous position information is not within the predetermined range, if the later position information is within the predetermined range, the warning information is controlled to be hidden. For example, the warning module 235 stops transmitting warning information.

The warning module 235 may acquire image data from the user terminal 10 and control whether or not to transmit the warning information based on the image data. For example, the warning module 235 analyzes the image data, determines whether the image data is correctly returned based on whether or not the image data includes the image of the bicycle and the station, and transmits the warning information based on the determination result. It is possible to control whether or not to do so. Further, the warning module 235 may output image data so that the administrator can visually confirm the image data.

Each of the elements shown above may be stored in one or more of the storage devices described above. Each of the modules shown above corresponds to a set of instructions for performing the functions described above. The modules or programs (ie, sets of instructions) shown above do not need to be implemented as separate software programs, procedures or modules, and thus various subsets of these modules are combined or combined in various embodiments. Alternatively, it can be reconstructed. In certain embodiments, the memory 230 may store a subset of the modules and data structures shown above. In addition, memory 230 may store additional modules and data structures not described above. One or a plurality of processing units (CPUs) 210 may form a management unit and a warning unit by executing the management module 234 and the warning module 235 stored in the memory 230, respectively.

Although FIG. 3 shows a “server”, FIG. 3 is intended to describe various features that may be present in a set of servers, rather than as a structural outline of the embodiments described herein. .. In practice, as will be appreciated by those skilled in the art, items shown separately could be combined and some items could be separated. For example, the items shown separately in FIG. 3 could be implemented on a single server, and a single item could be implemented by one or more servers.

<Example of data structure>
FIG. 4 is a diagram showing an example of processing data 134 processed by the user terminal 10 according to the embodiment. In the example shown in FIG. 4, the processed data 134 is, for example, each feature point extracted by using AR technology, a three-dimensional space model generated from each feature point, and the position of the image pickup device 160 in the three-dimensional space. Includes each information of the estimated self-position and the position of the information code 40 in the three-dimensional space. The processed data 134 may further include sensor information (camera tilt, etc.) acquired by the user terminal 10.

The image processing module 136 associates the position of the information code 40 recognized by the recognition module 135 with the position of the image pickup apparatus 160 estimated by the self-position estimation process in the generated three-dimensional space. This makes it possible for the guidance module 137 to determine whether or not this correspondence satisfies a predetermined imaging condition. The shooting conditions include, for example, that the position of the information code 40 and the position of the imaging device 160 estimated by the self-position estimation process are separated by a predetermined distance, and that the predetermined code 40 is included in the angle of view of the imaging device 160. And so on.

FIG. 5 is a diagram showing an example of the first data of the management data 233 according to the embodiment. In the example shown in FIG. 5, the first data includes a code ID, a status, a position information 1, a position information 2, a usage time, and the like.

The code ID is an ID that can identify an object such as a bicycle 30, and the code ID is an ID for identifying the information code 40. The status indicates unlocking or locking. The position information 1 indicates the use start position, and the position information 2 indicates the return position. The usage time indicates the usage time of the sharing service. The first data is an example, and the current status of each bicycle 30 and the usage history information may be managed separately. For example, the current status includes the current position and the status, and the usage history information includes the usage start position, the return position, and the usage time.

The first data is not limited to the configuration shown in FIG. For example, the first data may be divided into a usage table and a status table and managed. In this case, the usage table includes, for example, the ID of the user who is using the user, the code ID, the location information 1, the location information 2, and the usage time, and the status table includes the code ID, the status, and the bicycle 30. It may include the position information of.

FIG. 6 is a diagram showing an example of the second data of the management data 233 according to the embodiment. In the example shown in FIG. 6, the second data includes the station ID and the station position information. The location information of the station may be used to determine whether the bicycle 30 has been returned to the proper location.

The above-mentioned data structure is merely an example, and is not limited to this example. For example, the management data 233 may hold the usage history information of the bicycle 30 for each user.

<Operation explanation>
Next, the operation of the information processing system 1 according to the embodiment will be described. FIG. 7 is a flowchart showing an example of processing of the information processing system 1 according to the embodiment.

(Step S102)
The recognition module 135 of the user terminal 10 guides the user terminal 10 to copy a predetermined marker (for example, the information code 40) from the screen during shooting. For example, the recognition module 135 displays an information code frame on the screen.

(Step S104)
The recognition module 135 detects and recognizes the information code 40 included in the image data being photographed.

(Step S106)
The image processing module 136 executes the ambient environment mapping process and the self-position estimation process based on the image data during shooting. The image processing module 136 associates the position of the recognized information code 40 with the position of the image pickup apparatus 160. Based on this correspondence, the guidance module 137 guides the position or orientation of the image pickup apparatus 160 to a position or orientation in which the object including the information code 40 can be photographed. At this time, the display control module 138 may display and guide the AR content.

(Step S108)
The guidance module 137 activates the shooting button or controls the display on the screen if the correspondence relationship satisfies a predetermined shooting condition.

(Step S110)
The image pickup device 160 takes an image at the timing when the user presses the shooting button, and acquires image data including the object. The imaging device 160 may automatically perform imaging when a predetermined imaging condition is satisfied.

(Step S112)
As a result of the guidance, the network communication module 132 transmits the image data captured by the image pickup device 160 to the server 20. As a result, appropriate image data can be acquired, and appropriate image data can be transmitted to the server 20.

FIG. 8 is a sequence diagram relating to the processing of image data according to the embodiment. The example shown in FIG. 8 includes a process performed after the image data is captured by the guidance of the guidance module 137.

(Step S202)
The guidance module 137 of the user terminal 10 acquires the captured image data.

(Step S204)
The guidance module 137 acquires the current position information by using the GPS function or the like of the user terminal 10.
(Step S206)

The network communication module 132 transmits the image data and the location information to the server 20.

(Step S208)
The management module 234 of the server 20 determines the return of the bicycle 30 based on the acquired image data and the position information. The management module 234 determines, for example, whether the bicycle 30 has been returned to an appropriate position based on the image data, or determines a failure of the bicycle 30 or the like. Further, the management module 234 (warning module 235) determines whether or not the acquired position information is within a predetermined range from the position of the predetermined station. Here, if the determination result is affirmative (determined that the bicycle 30 is in the predetermined position), the process proceeds to step S210, and if the determination result is negative (determined that the bicycle 30 is not in the predetermined position), the process proceeds to step S220. ..

(Step S210)
The warning module 235 transmits warning information to the user terminal 10. The warning information may include, for example, countdown information, penalty information, or information prompting the user to return the information to a predetermined station nearby.

(Step S212)
The display control module 138 of the user terminal 10 controls the display of warning information on the screen. For example, the warning information is notified on the screen by a pop-up or the like, or the display is controlled by switching the screen in the application.

(Step S214)
It is assumed that the user returns the bicycle 30 to a predetermined station and photographs the bicycle 30 again. At this time, the guidance module 137 of the user terminal 10 acquires the image data.

(Step S216)
The guidance module 137 acquires the current position information by using the GPS function or the like of the user terminal 10.

(Step S218)
The network communication module 132 transmits the image data and the location information to the server 20. After that, the management module 234 of the server 20 makes a return determination again. Here, it is assumed that the determination result is affirmative, but if the determination result is negative, step S210 is executed again.

(Step S220)
The network communication module 232 of the server 20 transmits the return report information to the user terminal 10.

(Step S222)
The display control module 138 of the user terminal 10 controls the display of the return report information on the screen. For example, the report information may include a usage start position, a return position, a usage time, a cost, and the like.

As described above, the disclosed technology can reduce the increase in management cost when returning a vehicle in the sharing service by transmitting appropriate image data to the server 20 side. Further, the disclosed technology makes it possible to acquire appropriate image data from the process of reading the information code 40, which is a predetermined marker, through guidance.

<Screen example>
Next, an example displayed on the screen of the user terminal 10 will be described. FIG. 9 is a diagram showing a transition example of the screen according to the embodiment. The upper left screen shown in FIG. 9 is an example of a screen that guides the shooting of the information code 40. For example, "Please copy the QR code" is displayed on the screen.

The upper center screen is an example of a screen in which the information code 40 is recognized and the AR content A10 is displayed on the screen. The AR content A10 is information for guiding the position of the image pickup apparatus 160. In this screen example, since the AR content A10 does not fit in the screen, the user is guided to take a distance from the bicycle 30.

The screen on the upper right is an example of a screen while the user is leaving the bicycle 30. When the user moves away from the bicycle 30, the entire AR content A10 is visible, and the AR content A10 guides the user to move away from the bicycle 30 a little more.

The lower right screen is an example of a screen in which the shooting button B10 is activated when the distance between the image pickup device 160 and the bicycle 30 is a predetermined distance. In the upper screen example, the shooting button B10 is disabled. As a result, the user can easily understand that the image should be taken at this position.

The screen in the lower center is an example of a screen for confirming the transmission of image data acquired by shooting. When the user presses "Send" displayed on the confirmation screen, the image data captured and acquired by the guidance can be transmitted to the server 20.

The AR content A10 is preferably an image that fluctuates according to the positional relationship between the information code 10 and the image pickup apparatus 160, and may be represented by an image that fluctuates in distance and direction.

The disclosed technology is not limited to each of the above-described embodiments, and can be implemented in various other forms within a range that does not deviate from the gist of the disclosed technology. For this reason, each of the above embodiments is merely an example in all respects and is not to be construed in a limited manner. For example, the above-mentioned processing steps can be arbitrarily changed in order or executed in parallel within a range that does not cause a contradiction in the processing contents.

The program of each embodiment of the present disclosure may be provided stored in a computer-readable storage medium. The storage medium can store the program in a "non-temporary tangible medium". Programs include, but are not limited to, software programs and computer programs as examples.

[Modification example]
Further, a modification of each of the above-described embodiments is shown below.

<Modification example 1>
In the first modification, in the above embodiment, the bicycle 30 used for the sharing service is taken as an example of the imaging target, but any one having a predetermined marker is not limited to the sharing service. Good. For example, the object to be photographed may be an object that falls within the angle of view of the image pickup apparatus 160, and may be, for example, furniture, clothing, stationery, electrical appliances, daily necessities, vehicles, and the like. Further, the technique of the present disclosure can be applied as long as the object is arranged at a predetermined position. The system configuration and the like in the first modification are the same as the configuration of the embodiment.

<Modification 2>
In the second modification, the subject to be photographed in the above embodiment has been described by taking only the bicycle 30 as an example, but it is not intended to exclude photographing other objects. For example, in the case of the bicycle 30, the shooting distance and the direction (angle) may be set as shooting conditions in advance so that the port installed at the predetermined station and the bicycle 30 are imaged together.

1 Information processing system 10, 10A, 10B Information processing device (user terminal)
20, 20A Information processing device (server)
110, 210 processing unit (CPU)
120, 220 Network communication interface 130, 230 Memory 131, 231 Operating system 132, 232 Network communication module 133 Imaging data 134 Processing data 135 Recognition module 136 Image processing module 137 Guidance module 138 Display control module 150 User interface 160 Imaging device 170 Communication bus 233 Management data 234 Management module 235 Warning module

Claims (12)

For one or more processors included in the information processing device
Recognizing a predetermined marker from the image data during shooting,
To execute the ambient environment mapping process for creating a three-dimensional space and the self-position estimation process for estimating the position of the imaging device in the three-dimensional space based on the image data during shooting.
Identifying the correspondence between the position of the predetermined marker and the position of the imaging device in the three-dimensional space.
A program that causes the position or orientation of the imaging device to guide an object including the predetermined marker to a position or orientation that can be photographed based on the correspondence. To the one or more processors
The program according to claim 1, wherein the guidance information is superimposed on the image data during shooting to further execute display control on the screen. The display control is
The program according to claim 2, further comprising superimposing guidance information that fluctuates according to the correspondence relationship on the image data. The induction is
Any one of claims 1 to 3, including permitting photography by the imaging device when the position or orientation of the imaging device satisfies a predetermined condition regarding a position or orientation in which an object including the predetermined marker can be photographed. The program described in the section. To the one or more processors
The program according to claim 4, wherein the image data including the object captured by the image pickup apparatus is further transmitted to another information processing apparatus. The transmission is
When the predetermined marker is recognized, the first position information of the information processing device is transmitted to the other information processing device, and / or when the image data including the object is imaged, the first position information of the information processing device is captured. 2. The program according to claim 5, which comprises transmitting location information to the other information processing apparatus. To the one or more processors
When it is determined by the other information processing device that the first position information is not in the predetermined position, the warning information is acquired.
The program according to claim 6, further executing the display control of the warning information on the screen. When the other information processing device determines that the second position information is in the predetermined position, the request information for hiding the warning information is acquired.
The program according to claim 7, which hides the warning information in response to the request information. The program according to claim 7 or 8, wherein the warning information includes time information for prompting the movement of the object to the predetermined position within a predetermined time. To the one or more processors
The program according to any one of claims 1 to 9, which recognizes the object based on the image data and further determines whether or not the object satisfies the photographing condition. One or more processors included in the information processing device
Recognizing a predetermined marker from the image data during shooting,
To execute the ambient environment mapping process for creating a three-dimensional space and the self-position estimation process for estimating the position of the imaging device in the three-dimensional space based on the image data during shooting.
Identifying the correspondence between the position of the predetermined marker and the position of the imaging device in the three-dimensional space.
An information processing method in which the position or orientation of the imaging device guides an object including the predetermined marker to a position or orientation in which an object can be photographed based on the correspondence. An information processing device that includes one or more processors.
The one or more processors
Recognizing a predetermined marker from the image data during shooting,
To execute the ambient environment mapping process for creating a three-dimensional space and the self-position estimation process for estimating the position of the imaging device in the three-dimensional space based on the image data during shooting.
Identifying the correspondence between the position of the predetermined marker and the position of the imaging device in the three-dimensional space.
An information processing device that executes the position or orientation of the imaging device to guide an object including the predetermined marker to a position or orientation that can be photographed based on the correspondence.

Images