JP7311461B2 - Collaborative device, method and program - Google Patents

Description

The present invention relates to a cooperation device, method, and program that enable efficient server selection when a plurality of VPS servers exist.

Various techniques exist for providing map information and the like to users.

In Patent Document 1, when there are multiple maps, a map is selected and displayed based on movement history and landmarks. In Patent Document 2, in position estimation by car, a map information selection unit is provided to provide a mechanism for selecting a map based on position and angle. Patent Document 3 has an area estimation algorithm based on geographical rules. Location information, user posted information, and the like are used for these.

JP-A-2007-249221 JP 2011-209845 A Japanese Patent Application Laid-Open No. 2016-161828

However, in the conventional technology as described above, it is assumed that there are no particular restrictions on selecting and using a suitable system when there are multiple systems for providing map information and the like. No consideration was given to making appropriate use of

As a case where there are such restrictions, users can provide rough location information by GNSS (Global Navigation Satellite System) etc. to VPS (Visual Positioning System) service providers with different business entities, An image of the surrounding environment of the user is sent, and the service provider compares the image with the 3D point cloud to return the detailed shooting position to the user.

In particular, when there are a plurality of service providers who have maps corresponding to the general location information of the user, it is difficult for the user to select an appropriate service provider in advance. When there are multiple such service providers, it may not be possible to know which VPS service provider is in charge of which map area, such as indoor/outdoor boundaries, from the user's rough location information. That is, it is difficult to select a VPS system based on location if the user is in such a boundary.

For example, if a user exists in a certain local area, and this local area has a distinction between indoors and outdoors, and operator A is in charge of indoor maps, and operator B is in charge of outdoor maps. In addition, if the user exists at this border position, it is impossible to determine which of the operators a and b is appropriate from only the general positional information. Even though the user is indoors, if the user were to inquire about outdoor information provider b, not only would the user not be able to obtain appropriate location information, but the cost of the inquiry would be wasted. I had a problem. Although it is conceivable that the user himself/herself selects which of the companies a and b to use, there is a problem that each time, trouble occurs.

In view of the problems of the prior art as described above, the present invention provides a linking device, method and method that enable efficient server selection when there are a plurality of VPS servers that can cover general positioning information of a user. The purpose is to provide a program.

To achieve the above object, the present invention provides a method for receiving positioning information obtained by a terminal and a photographed image photographed by the terminal, and storing each of a plurality of VPS servers, The positioning information range covered is referred to, the VPS server to which the received positioning information corresponds is selected as a candidate, and from the selected VPS servers, the VPS server is determined as the target of the VPS positioning request. , a determination unit that transmits the captured image; and a result of estimating the position and orientation of the captured image estimated by the determined VPS server by comparing the captured image with the point cloud, and success or failure of the estimation result. and a determination result storage unit that associates and stores the captured image, the success or failure of the estimation result, and the determined VPS server, wherein the determining unit performs the determination The decision is made by referring to information about the captured image, the success or failure of the estimation result of the captured image, and the VPS server, which are accumulated and stored in the result storage unit. Moreover, it is characterized by being the method and program corresponding to the said cooperation apparatus.

According to the present invention, as information on the result of a request for VPS positioning, the captured image, the success or failure of the estimation result of the captured image, and the information on the VPS server are accumulated and stored, and this information is referred to. By determining the VPS server as the target of the VPS positioning request by using the above method, it is possible to efficiently determine the appropriate VPS server as the target of the VPS positioning request by considering the actual performance.

1 is a diagram showing a configuration example of a VPS system according to one embodiment; FIG. 2 is a diagram showing a schematic example of a configuration of a local area where the three servers of FIG. 1 provide positioning services; FIG. 1 is a functional block diagram of a VPS system according to one embodiment; FIG. 4 is a flowchart of the operation of the VPS system according to one embodiment; FIG. 5 is a diagram showing an operation example when the flow of FIG. 4 is performed in the local area L having the configuration of FIG. 1; It is a figure which shows the example of the hardware constitutions in a common computer.

FIG. 1 is a diagram showing a configuration example of a VPS system 100 according to one embodiment. The VPS system 100 includes a terminal 10 (mobile terminal such as a smart phone) used by a user who desires to perform its own positioning, a cooperation device 20, and a plurality of VPS servers 30 (described later in FIG. 3) as an example. There are three servers 30A, 30B, 30C, which are configured to communicate with each other via a network.

The user of the terminal 10 can make a request for positioning by selecting a server that provides an appropriate positioning service from among the three servers 30A, 30B, and 30C by himself/herself. , may be wrong. In this embodiment, the user does not need to make such a selection, and requests positioning to the cooperation device 20, and the cooperation device 20 selects an appropriate server from among the three servers 30A, 30B, and 30C. The request can be sent to the selected server, the positioning result can be received as a reply from the server, and the positioning result can be returned to the terminal 10 .

Here, the user's terminal 10 exists in the local area L, has map information corresponding to this local area L, and is an example of the plurality of servers 30 capable of providing the positioning service by VPS, the three servers in FIG. They are 30A, 30B, and 30C. The number and types of such servers can generally vary for each local area where the terminal 10 is located.

FIG. 2 is a diagram showing a schematic example of the configuration of the local area L where the three servers 30A, 30B, and 30C of FIG. 1 provide positioning services. Example EX1 shows the state of local area L viewed from above, and example EX2 shows the state of local area L viewed horizontally as a sectional view along line LN of example EX1. In the local area L, there are two buildings BA, BC (for example, buildings such as shopping malls) and an outdoor area LB between these buildings BA, BC. Indoor area LA (1st floor area A1, 2nd floor area A2 and 3rd floor area A3) exists in building BA, and indoor area LC (1st floor area C1 and 2nd floor area C2) exists in building BC. .

The three servers 30A, 30B, and 30C in FIG. 1 are, for example, VPS servers having maps of the indoor area LA, the outdoor area LB, and the indoor area LC in FIG. It is configured to be able to provide an appropriate positioning service. However, since the terminal 10 exists within the local area L, even if the terminal 10 exists in any of the areas LA, LB, and LC, positioning by the terminal 10 using GNSS or the like may cause multipath errors and the like. Since it is not possible to accurately determine in which of the areas LA, LB, and LC the terminal 10 is located, the rough positioning information by GNSS or the like as prior information for receiving the VPS service does not include three servers 30A, Unable to accurately determine whether to receive VPS positioning service at 30B or 30C. In this embodiment, the linking device 20 relays the VPS positioning request from the terminal 10, and from among the three servers 30A, 30B, and 30C (as will be described later, a certain degree of positioning service provision result (i.e., positioning result After the information on the success or failure of ) is accumulated, it is possible to automatically select an appropriate server.

FIG. 3 is a functional block diagram of the VPS system 100 according to one embodiment. The VPS system 100 includes a terminal 10, a cooperation device 20 and a VPS server 30. FIG. The VPS server 30 has map information of the local area where the terminal 10 exists, and is any of a plurality of servers capable of providing positioning services, as outlined with reference to FIGS. It represents one.

The terminal 10 includes a positioning unit 11, an imaging unit 12 and a UI (user interface) unit 13. The cooperation device 20 includes an area storage unit 21, a determination unit 22, a determination result storage unit 23, and a determination unit . A VPS server estimation unit 31 and a point cloud storage unit 32 are provided. The function outline of each part is as follows.

The positioning unit 11 is configured with GNSS or the like as hardware, performs positioning with three-dimensional coordinates or two-dimensional coordinates as a rough position of the terminal 10, and sends positioning information (which may include errors) to the determining unit 22. and send. The positioning unit 11 uses WiFi (registered trademark) positioning and Bluetooth (registered trademark) as methods that enable indoor positioning where positioning is difficult with GNSS, etc. that use positioning satellites, instead of or in addition to GNSS, etc. beacons for positioning. The positioning unit 11 may also be configured to output positioning information corrected by combining a plurality of these positioning methods, and output positioning information that may contain errors by any existing method other than VPS positioning. It is sufficient if it is configured as The photographing unit 12 is configured by a camera as hardware, photographs the surrounding environment (field) of the terminal 10 , and transmits the obtained image to the determining unit 22 .

The determination unit 22 (1) compares the positioning information obtained by the positioning unit 11 and possibly including errors with the area storage unit 21, and maps the map information to which this positioning information corresponds (after considering the margin due to the error in the positioning information). information of the VPS server 30 (information such as the IP address for accessing the server via the network) as a candidate that has the map information covering the position where the positioning is performed) and (2) when two or more such VPS servers 30 are acquired, by comparing the image obtained from the imaging unit 12 with the determination result storage unit 23, two or more Determine one of the VPS servers 30 that is suitable for requesting the positioning service, and (3) send the image obtained from the imaging unit 12 to the estimating unit 31 of the determined server (VPS server 30). to send.

Among the processes (1) to (3) of the determination unit 22, the schematic example of (1) is the example described in FIGS. That is, when the positioning information obtained by the positioning unit 11 is within the local area L, the information of the three servers 30A, 30B, and 30C as the candidate VPS servers 30 is obtained.

In addition, regarding the processing of (2) above, the details thereof will be further described in the description of step S2 in FIG. 4, which will be described later. Regarding the above process (3), in addition to the image obtained from the photographing unit 12 , the positioning information obtained from the positioning unit 11 may also be transmitted to the estimation unit 31 .

The area storage unit 21 stores applicable positioning information (positioning information obtained by the positioning unit 11) for each of the plurality of VPS servers 30 in advance so that the determination unit 22 can perform the processing of (1) above. ) is memorized. Area It is stored in the storage unit 21 in advance. Therefore, as a predetermined range R2 (R2⊃R1) wider than the range R1 stored in the point cloud storage unit 32 (predetermined range in which the positioning service can be provided) considering the margin due to the error of the positioning information of the positioning unit 11, The area storage unit 21 stores the range R2. As an embodiment different from the embodiment in which the range R2 is stored, the area storage unit 21 stores only the range R1 similar to that stored in the point cloud storage unit 32 for each of the VPS servers 30. When the determination unit 22 performs the above process (1), the positioning information obtained by the positioning unit 11 is determined to fall within the range R1 by considering a predetermined error margin. You may make it judge whether there exists. That is, in the positioning information obtained by the positioning unit 11, the proximity range of the positioning information considering a predetermined error margin (range of true values that can fluctuate with the error margin centering on the position of the positioning information) overlaps the range R1. If so, the VPS server 30 may be determined as a candidate.

The determination result storage unit 23 stores the determination result (and related information) obtained by the determination unit 24, which will be described later, so that the determination unit 22 can perform the above process (2). Specifically, a positioning service request is transmitted to the VPS server 30, and the following data D1 is linked and stored as information acquired each time the result is obtained.
(Data D1)
● Positioning information obtained from the positioning unit 11 ● Image obtained by the imaging unit 12 (and/or image feature information extracted from this image)
Common time information (time stamp) between the positioning information of the positioning unit 11 and the image of the imaging unit 12
● ID (identifier) of VPS server 30 determined by determination unit 22 in process (2)
●Determination result determined by the determination unit 24 (determination success/failure, information on the position and orientation estimated by the estimation unit 31 in the case of success)

The estimating unit 31 analyzes the image (the image captured by the imaging unit 12) transmitted from the determining unit 22, and compares it with the 3D point cloud stored in the point cloud storage unit 32 to determine the position of this image. and orientation (the position and orientation of the camera constituting the imaging unit 12 in the three-dimensional world coordinates) are estimated, and the position and orientation of the estimation result are transmitted to the determination unit 24 .

The estimation unit 31 can estimate the position and orientation of the image by an existing VPS technique (a technique using SfM (Structure from Motion) or the like). In SfM, for example, feature points (two-dimensional image coordinates) and local feature amounts are obtained from an image using SIFT features or the like. The local feature values are registered in advance as 3D world coordinates), and the correspondence between the feature points of the image and the feature points of the 3D point cloud (the correspondence between the feature points determined to match the local feature values) is calculated. It is possible to estimate the position and orientation of an image by obtaining information on the three-dimensional world coordinates of each feature point of the image by using principles such as triangulation.

In the estimation unit 31, the degree of matching between the 3D point cloud stored in the point cloud storage unit 32 and the feature points extracted from the image obtained by the determination unit 22 is low in threshold determination (for example, error in point correspondence is large in the threshold determination), it is determined that the position and orientation corresponding to this image do not exist, and information to the effect that positioning has failed is transmitted to the determination unit 24.

In the point cloud storage unit 32, predetermined 3D point cloud information (three-dimensional coordinates of each feature point and local feature amount information) is stored in advance so that the estimation unit 31 can estimate the position and orientation as described above. Remember. As described above regarding the determination unit 22 and the area storage unit 21, the range of the 3D point cloud stored by the point cloud storage unit 32 corresponds to the range of the map in which the VPS server 30 can provide the positioning service. .

The determination unit 24 stores the position and orientation estimation results obtained from the estimation unit 31 as determination results in the determination result storage unit 23 and transmits the position and orientation estimation results to the UI unit 13 . By storing the determination result in the determination unit 24, the determination result storage unit 23 accumulates and stores the determination result and related information in the form of the data D1 described above.

If the estimation result obtained from the estimation unit 31 means that the positioning has failed, the determination unit 24 determines that the determination has failed and stores the determination result in the determination result storage unit 23. If it does not mean that the position and orientation have been estimated and the position and orientation estimation results have been obtained, the determination result may be stored in the determination result storage unit 23 assuming that the positioning has been successful.

The above-described embodiment of the determination unit 24 uses the estimation result obtained from the estimation unit 31 as it is. After that, if the estimation result of the estimation unit 31 is successful, it is possible to save the determination result in the determination result storage unit 23 after confirming whether it is actually successful from the user's point of view. good. That is, when the estimation result obtained by the estimation unit 31 does not mean that the estimation result has failed, the UI unit 13 presents the obtained position and orientation estimation result to the user of the terminal 10, and the user and information on whether or not the direction estimation result is actually correct from the user's point of view is received as user feedback, and after the UI unit 13 returns it to the determination unit 24, if the user feedback is also determined to be correct. , the determination result may be stored in the determination result storage unit 23 as being successful. (That is, even if the estimation result is obtained as a success in the estimation unit 31, if it is determined as failure by the user feedback, the determination result may be stored in the determination result storage unit 23 as failure. .)

As described above, the UI unit 13 has a role of providing a user interface to the user of the terminal 10, and performs functions such as plotting the position and orientation estimation results transmitted from the determination unit 24 on a predetermined map. Provide it to the user in a form. (In the case of failure in estimation, the user may be notified of the failure by means of a text message or the like.) In the case of the above-described embodiment in which the determination unit 24 requests user confirmation, the UI unit 13 further confirms the estimation result. Information as to whether or not the estimation result is actually correct based on the user's judgment is received as user feedback in the form of a menu input or the like from the user, and this user feedback is transmitted to the determination unit 24 .

FIG. 4 is a flowchart of the operation of the VPS system 100 according to one embodiment, and explains the processing of each unit of the functional blocks of FIG. 3 described above from the viewpoint of processing timing.

When the flow is started, in step S1, the terminal 10 determines whether or not positioning and image capturing have been performed for the user's request for VPS positioning. If yes, step S2. If the result is negative, the process returns to step S1 and waits until an affirmative determination is obtained. In the case of an affirmative determination in step S1, the imaging unit 12 obtains an image by imaging the field where the user exists (the field where the terminal 10 exists) by the user's operation, and at the same time as this imaging time, the positioning is performed. The positioning information is acquired by the unit 11 performing positioning, and these images and the positioning information are transmitted to the determination unit 22 of the cooperation device 20 .

In step S2, the determination unit 22 of the linking device 20 receives the positioning information and the image transmitted with the positive determination in step S1, and performs the processes (1) to (3) described above to perform VPS positioning. One VPS server 30 determined to be optimal for sending the request is determined, and the image (the image captured by the imaging unit 12 with a positive determination in step S1) is sent to this VPS server 30 (and step After transmitting the positioning information obtained by the positioning unit 11 after obtaining a positive determination in S1, the process proceeds to step S3.

The details of the process (2) of the determination unit 22, which will be described later, are as follows. The determination unit 22 refers to the determination result storage unit 23, and determines whether positioning is successful among the information stored in the determination result storage unit 23 for each of the plurality of VPS servers 30 obtained as candidates in the process (1). Get information to be judged. By this acquisition, for each of the candidate VPS servers 30, a plurality of images (successful image clusters) when positioning is successful are obtained. Therefore, the determining unit 22 compares the requested image (captured image) for which an affirmative determination was obtained in step S1 with the successful image cluster of each VPS server 30, and determines that it is most similar to the captured image. The VPS server 30 corresponding to the successful image cluster can be determined as the server that should request the positioning service. Any existing method can be used to determine the similarity between an image and a successful image cluster, and the degree of similarity can be evaluated by a method such as BoVW (Bug of Visual Words). For successful image clusters, one or a plurality of BoVW representative vectors may be determined to evaluate similarity with images.

In order for such similarity evaluation to function properly, for example, in the example of FIGS. It is assumed that the images are dissimilar to each other at least to some extent. It should be noted that this premise normally holds because the exact same image cannot normally be captured in different areas. Also, even if somewhat similar images are taken in different areas, it is usually impossible for completely identical images to be taken. If the number of images in the successfully localized image cluster is large, the similarity evaluation can work properly.

In addition, when obtaining successful image clusters for each of the candidate VPS servers 30 in the processing (2) of the determination unit 22, the information stored in the determination result storage unit 23 indicates that positioning has been successful, and , the positioning information stored in the format of the data D1 described above is determined to be close to the positioning information transmitted with a positive determination in step S1 by threshold determination, and the successful image cluster is acquired. may

In step S3, the estimation unit 31 of the VPS server 30 determined in step S2 receives the image (and positioning information) transmitted from the determination unit 22, extracts feature points and local feature amounts from the image, and extracts the points. After matching with the point cloud stored in the group storage unit 32 and transmitting the position and orientation of the image as a matching result to the determination unit 24 of the cooperation device 20, the process proceeds to step S4.

In step S3, when the estimating unit 31 has received the positioning information in addition to the image, the three-dimensional position of the positioning information in the entire point cloud stored in the point cloud storage unit 32 is determined by the threshold value. Only a part of the point cloud determined to be close may be limited as an object of collation.

In step S4, the judgment unit 24 of the cooperation device 20 judges whether or not the estimation result of the position and orientation transmitted from the estimation unit 31 in step S3 succeeds or fails, and then proceeds to step S5.

As already explained with regard to the determination unit 24, the success or failure of the estimation result is based only on whether or not the estimation result of the position and orientation obtained by the estimation unit 31 is determined to be unsuccessful by the threshold value determination. The (automatic) estimation result may be used as it is as the determination result. A determination result that also reflects the user's determination obtained by

In step S5, the determination result of the determination unit 24 in step S4 is newly stored in the determination result storage unit 23 in the format of the data D1 described above, so that the storage contents of the determination result storage unit 23 (accumulation of data D1 ) is updated, and the process proceeds to step S6.

In step S6, cases are classified according to the success or failure of the estimation result determined in step S4. If the result is yes (estimation has failed), the process proceeds to step S8.

In step S7, the user of the terminal 10 is notified of the estimation result (position and orientation determined to be successful) via the UI unit 13, and then the process returns to step S1. (Note that in step S4, if the determination unit 24 has already notified the user of the estimation result from the UI unit 13 in order to obtain user feedback, the notification in step S7 will be duplicated. may be omitted.)

In step S8, since the estimation result was a failure, the determination unit 24 determines whether additional estimation is possible in the VPS server 30 in order to perform estimation again, and then proceeds to step S9. Whether or not additional estimation is possible may be determined by setting a predetermined upper limit number of times of estimation processing corresponding to one positioning and photographing operation in step S1. Depending on whether or not the upper limit number of estimation processes by the user of the terminal 10 has been reached, or whether the billing amount when the user of the terminal 10 uses the VPS server 30 has reached a predetermined upper limit value, etc. It may be judged on a rule basis.

In step S9, cases are divided according to whether the determination of the possibility of additional estimation in step S8 was affirmative or negative. is impossible), proceed to step S10.

In addition, in step S2 after returning from step S9, the VPS server 30 whose estimation result is a failure is excluded from the candidates, and the determination unit 22 selects another VPS server 30 by processing (1) to (3). should be determined.

In step S10, the user of the terminal 10 is notified of the estimation result (failure) via the UI unit 13, and then the process returns to step S1. (In step S4, if the UI unit 13 has already notified the user of the estimation result (estimation result determined to be successful by the estimation unit 31) in order for the determination unit 24 to obtain user feedback (in this case, the user If a failure is returned as feedback), the notification in step S10 may be omitted because it will be a duplicate notification.)

The above flow of FIG. 4 focuses on one terminal 10, but in reality, the flow of FIG. Become. Therefore, in the determination result storage unit 23, a large amount of information on the VPS servers 30 that have succeeded in VPS positioning are accumulated in the form of the aforementioned data D1 corresponding to various local areas. By using the sufficiently accumulated information in the determination result storage unit 23, the cooperation device 20 can accurately determine the VPS server 30 capable of performing VPS positioning.

FIG. 5 is a diagram showing an operation example when the flow of FIG. 4 is performed in the local area L having the configuration of FIG. It shows the flow of information transmission and reception. As a procedure p1 (an example in which step S1 is affirmative), the terminal 10 performs positioning and photographing, and transmits the result to the cooperation device 20 . In step pA1, the linking device 20 determines the server 30A. In step pA2, the server 30A estimates the position and orientation. (In procedure pA3, a two-way arrow is drawn between the terminal 10 and the server 20, but this is the embodiment in which the estimation result of the server 30A is successful, but the user is asked to confirm, as described above. It represents the exchange of information when applying , and can be omitted in embodiments that do not require confirmation, and is not an essential process.The same applies to the two-way arrow in the next step pB3.) This is In this example, the server 30A is determined in step S3, and then steps S4->S5->S6->S8->S9->S2 are performed.

Again, in procedure pB1, the cooperation device 20 determines the server 30B (determined by excluding the failed server 30A), estimates the position and orientation in the server 30B in procedure pB2, and determines the success or failure of this estimation result in procedure pB3. It is determined that the determination was a failure. This is an example in which the server 30B is determined in step S3, and then steps S4→S5→S6→S8→S9→S2 are performed.

Again, as procedure pC1, the linking device 20 determines server 30C (determined by excluding failed servers 30A and 30B), estimates the position and orientation in server 30C as procedure pC2, and estimates the result of this estimation in procedure pC3. Success or failure is determined, it is determined to have been successful, and the user of the terminal 10 is notified of the estimation result.

The example of FIG. 5 is an example in which the server 30C is the correct answer (which can correctly estimate the position and orientation) of the three servers 30A, 30B, and 30C, but the server 30C is incorrectly determined twice. However, as described above, as correct cases are accumulated in the determination result storage unit 23 of the cooperation device 20, the cooperation device 20 can more accurately determine the VPS server 30 capable of performing VPS positioning. It becomes possible. (When the number of correct cases is small, there is a possibility of making mistakes as in the example of Fig. 5, but such errors are reduced by accumulating correct cases.)

As described above, according to the present embodiment, there are a plurality of VPS servers 30 capable of VPS positioning for adjacent or partially overlapping local areas, and GNSS and the like acquired with errors on the user side. Even in a situation where the appropriate VPS server 30 cannot be automatically determined with the positioning information, the appropriate VPS server 30 can be automatically determined by accumulating correct examples in such a situation in the cooperation device 20. .

In particular, when using the VPS server 30, the automatic determination according to the present embodiment is suitable in a situation where a usage fee is imposed on the user's terminal 10 side, or in a case where there is an upper limit to the number of uses. Also, on the VPS server 30 side, it is possible to reduce wasteful processing by reducing the number of times of receiving inappropriate requests that do not have map information (3D point group information). becomes. In addition, if each VPS server 30 is operated by a different operator and the details of the internal processing and map information of the VPS server 30 are unknown (part or all of the internal processing of the VPS server 30, map information, etc. such information is closed to the user of the terminal 10 and the operator of the cooperating device 20), or the map information is complicated even if it can be disclosed. However, the automatic determination according to the present embodiment makes it possible to suppress generation of useless requests.

Various supplementary and additional matters are described below.

(A) As a result of checking the area storage unit 21 in the process (1) of the determination unit 22, only one VPS server 30 having map information corresponding to the positioning information obtained by the positioning unit 11 exists. In this case, the one VPS server 30 may be determined as a single candidate server without checking the determination result storage unit 23 in the process of (2). As a result of collating the area storage unit 21 in the processing of (1) of the determination unit 22, if there is no VPS server 30 having map information corresponding to the positioning information obtained by the positioning unit 11, ( It suffices to omit the processes of 2) and (3) and notify the user of the terminal 10 that it is impossible to perform VPS positioning due to the lack of an appropriate VPS server 30. . In any of these cases, the process of newly storing the determination result in the determination result storage unit 23 (the process of step S5 in FIG. 4) may be omitted.

(B) In the above description, it is assumed that only one appropriate VPS server 30 is determined in the process (2) of the determination unit 22 . This assumption is based on the assumption that the map information ranges covered by the plurality of VPS servers 30 are close to each other in the local area but do not overlap, as shown in the schematic example of FIG. If this premise is not provided, for example, in the schematic example of FIG. When map information of area LC and outdoor area LB is provided, that is, when all of the three servers 30A, 30B, and 30C regarding outdoor area LB have map information, and positioning information can be provided. Embodiments are also possible.

In this case, there are a plurality of VPS servers 30 in which at least a part of the VPS positioning range (range of map information held) overlaps, and there are a plurality of VPS servers 30 that are equally suitable. Therefore, the following embodiment is possible. First, the determination unit 22 determines a plurality of appropriate VPS servers 30 within a predetermined number, and transmits the captured image to all or part of these VPS servers 30 to determine the position and orientation. inquire. Next, the determination unit 24 stores the position and orientation estimation results received from the plurality of VPS servers 30 as determination results in the determination result storage unit 23, Statistical processing (e.g., averaging) may be performed on the results of the determination, and the results may be transmitted to the UI unit 13 of the terminal 10 as final position and orientation estimation results. Alternatively, only one result with the highest estimation accuracy in the VPS server 30 may be transmitted to the terminal 10 (and stored in the determination result storage unit 23) from among a plurality of estimation results that have resulted in successful estimation. The results weighted and averaged according to accuracy may be transmitted to the terminal 10 (and stored in the determination result storage unit 23). This enables the user to obtain highly accurate estimation results in a short period of time.

As described above, an embodiment that can be applied even when map information may overlap is made possible by allowing not only one but also a plurality of VPS servers 30 determined by the determination unit 22 . Specifically, in the processing of (2) of the determination unit 22, the degree of similarity between the successful image cluster of each candidate VPS server 30 and the captured image is determined, and one VPS server 30 that is the most similar is determined. Instead, all or some of the VPS servers 30 whose similarity is greater than or equal to the threshold value may be determined. (Also, if there is no VPS server 30 with a degree of similarity equal to or greater than the threshold, the determination unit 22 may determine that there is no match, as in the case of (A).)

(C) In the processing of (2) of the determination unit 22, the degree of similarity between the successful image cluster of each candidate VPS server 30 and the captured image is determined. The dissimilarity between each failure image cluster and the captured image of each VPS server 30 may also be determined, and the similarity with the success image cluster (the higher the similarity, the higher the value) and the dissimilarity with the failure image cluster The degree (the less similar, the higher the value) and the weighted score can be used for determination in the same way as the similarity alone. Each image that constitutes the failed image cluster can be obtained as an image for which the determination result of the determination unit 24 is a failure among the information stored in the data D1 format.

(D) In the VPS system 100 of each embodiment of the present invention, the internal processing of each VPS server 30, which can be operated by various business operators, is a black box when viewed from the terminal 10 or the cooperation device 20 side. It is possible to operate even if it is modified. (However, it is necessary to store information about the range of map information covered by each VPS server 30, which is stored in the area storage unit 21.) (and positioning information), and return the position and orientation estimation results (and estimation accuracy, etc.). Individual estimation algorithms in the estimation unit 31 and detailed 3D point cloud information stored in the point cloud storage unit 32 may be black boxes, and different algorithms and the like are implemented for each VPS server 30. may be

(E) After the determination unit 22 determines the VPS server 30 to which the image is transmitted and the positioning request is made based on the similarity determination with the successful image cluster, the load of the positioning processing on this VPS server 30 is reduced. Additional processing such as the following (E1) and (E2) may be performed so that it is possible to do so.

(E1) The determining unit 22 selects the image received from the photographing unit 11 (referred to as the "current image") from the success image cluster (belonging to the cluster) stored in the determination result storage unit 23 for the determined VPS server 30. each image to be displayed is referred to as a "past image"), one or more past images determined to be similar to the current image are determined, and these past images are linked as the aforementioned data D1. Acquire position and orientation information (position and orientation information stored as having been successfully estimated by the estimation unit 31 of the VPS server 30 at a past point in time), and send the current image to the estimation unit 31 of the VPS server 30. (and the general positioning information of the current positioning unit 11), it also transmits the position and orientation information of this past image.

Note that the similarity determination between the current image and the past image may be performed using the same image features as when the VPS server 30 was determined.

(E2) The estimation unit 31 uses the position and orientation information of the past image additionally transmitted from the determination unit 22 to convert the entire 3D point cloud stored in the point cloud storage unit 32 into the current image. Instead of estimating the position and orientation of the current image, only a portion of the 3D point cloud determined to be close to the position and orientation of the previous image (at least one previous image if there are more than one) is used as the position and orientation of the current image. And it is used as a direction estimation target.

That is, with this additional processing, the estimating unit 31 can limit not all of the 3D point cloud stored in the point cloud storage unit 32 but only a part of the 3D point cloud to be matched with the current image. It becomes possible to reduce the load of the positioning processing of the unit 31 . As described in (D), the details of the internal processing of each VPS server 30 may be black boxed, but when performing this additional processing, the position of the terminal 10 and By explicitly receiving the instruction information indicating that the orientation is in the vicinity of the position and orientation of the past image transmitted by the processing of (E1), the load reduction processing of (E2) is performed in the estimation unit 31 of the VPS server 30. can be done.

(F) FIG. 7 is a diagram showing an example of hardware configuration in a general computer device 70. As shown in FIG. Each of the terminal 10, the cooperation device 20, and the VPS server 30 in the VPS system 100 can be implemented as one or more computer devices 70 having such a configuration. When the terminal 10, the cooperation device 20, or the VPS server 30 is implemented by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, and a GPU (graphics processing unit) as a dedicated processor that executes part or all of the execution instructions of the CPU 71 instead of the CPU 71 or in cooperation with the CPU 71. ) 72, RAM 73 as a main storage device that provides a work area to CPU 71 (and GPU 72), ROM 74 as an auxiliary storage device, communication interface 75, display 76, input interface 77 that accepts user input by mouse, keyboard, touch panel, etc. It includes a camera 78 that configures the imaging unit 12 as hardware, a sensor 79 that configures the positioning unit 11 as hardware, and a bus BS for exchanging data therebetween.

Each functional unit of the terminal 10, the cooperation device 20, and the VPS server 30 can be realized by the CPU 71 and/or the GPU 72 that reads and executes a predetermined program corresponding to the function of each unit from the ROM74. Both the CPU 71 and the GPU 72 are a kind of arithmetic unit (processor). Here, when display-related processing is performed, the display 76 further operates in conjunction, and when communication-related processing relating to data transmission/reception is performed, the communication interface 75 further operates in conjunction. The processing results and the like by the point cloud processing system 100 may be displayed on the display 76 and output. The input interface 77 may constitute the UI unit 13 as hardware.

100... VPS system, 10... terminal, 11... positioning unit, 12... imaging unit, 13... UI unit, 20... cooperation device, 21... area storage unit, 22... determination unit, 23... determination result storage unit, 24... determination Part, 30... VPS server, 31... Estimation part, 32... Point cloud storage part

Claims (7)

receiving positioning information acquired by a terminal and a captured image captured by the terminal;
referring to the positioning information range covered by the VPS server, which is stored for each of a plurality of VPS servers, and selecting the VPS server corresponding to the received positioning information as a candidate;
a determination unit that determines a VPS server as a target for requesting VPS positioning from among the selected VPS servers and transmits the captured image;
a determination unit that receives an estimation result of the position and orientation of the captured image estimated by the determined VPS server by matching the captured image with the point cloud, and determines success or failure of the estimation result;
a determination result storage unit that associates and stores the captured image, the success or failure of the estimation result, and the determined VPS server,
The determination unit makes the determination by referring to the captured image, the success or failure of the estimation result of the captured image, and information on the VPS server, which are accumulated and stored in the determination result storage unit. Device. The determination unit refers to the captured image, the success or failure of the estimation result of the captured image, and the information of the VPS server, among the information accumulated and stored in the determination result storage unit. 2. The linkage according to claim 1, wherein a VPS server determined to store a photographed image similar to the photographed image photographed by the terminal is determined as a target for requesting VPS positioning. Device. The determination unit transmits positioning information acquired by the terminal in addition to the captured image to the determined VPS server,
The determination result storage unit associates and stores the captured image, the success or failure of the estimation result, the determined VPS server, and the positioning information acquired by the terminal,
The determination unit refers to information that is determined to be close to the positioning information acquired by the terminal, among the information accumulated and stored in the determination result storage unit. , is determined. 2. When the determination unit determines that the estimation result is successful, the terminal is notified of the position and orientation in the estimation result as the position and orientation of the terminal. 4. The cooperation device according to any one of 1 to 3. In the decision unit, after making the decision,
Similar to the photographed image from past photographed images for which the estimation result is successful, which are accumulated and stored in the determination result storage unit as past images in association with the determined VPS server Searching for what is determined to be the case, obtaining information on the position and orientation in the estimation result linked to the searched past captured image,
In the VPS server determined above,
5. The range in which the photographed image is collated with the point cloud is limited to a portion determined to be close to the position and orientation in the estimation result linked to the past photographed image. The cooperation device according to any one of the above. receiving positioning information acquired by a terminal and a captured image captured by the terminal;
referring to the positioning information range covered by the VPS server, which is stored for each of a plurality of VPS servers, and selecting the VPS server corresponding to the received positioning information as a candidate;
a determination step of determining a VPS server as a target for requesting VPS positioning from among the selected VPS servers and transmitting the captured image;
a determination step of receiving an estimation result of the position and orientation of the captured image estimated by the determined VPS server by matching the captured image with the point cloud, and determining success or failure of the estimation result;
A determination result storage step of storing the captured image, the success or failure of the estimation result, and the determined VPS server in association with each other , wherein the linkage method is executed by a computer ,
In the decision step, the decision is made by referring to the captured image, the success or failure of the estimation result of the captured image, and information on the VPS server, which are accumulated and stored in the determination result storage step. Method. A program that causes a computer to function as the cooperation device according to any one of claims 1 to 5.

Images