JP7039535B2 - Generator, generation method, and generation program - Google Patents

Description

The present invention relates to a generator, a generator, and a generator.

Conventionally, a technique for guiding a user to information indicating a current position or a route to a destination has been known. As an example of such a technology, there is a technology that provides information indicating a user's current position in three dimensions by photographing a two-dimensional bar code indicating the current position using a terminal device used by the user. Are known.

Japanese Patent No. 4015632

However, with the above-mentioned technology, it may not be possible to properly guide the user.

For example, in the above-mentioned technology, since the current position of the user in a certain facility is only estimated, it is shown in which direction the user should move when moving from the current position to the destination. I can't.

The present application has been made in view of the above, and an object thereof is to appropriately guide the user.

The generation device according to the present application includes an acquisition unit that acquires a photographed image of an object previously installed in a predetermined area, and the object of the photographing device that photographs the photographed image based on the photographed image. An estimation unit that estimates a relative position that is a position with respect to an object and an estimation unit that estimates the orientation of the imaging device with respect to the object, and the imaging device within the predetermined region based on the relative position and the relative orientation. It is characterized by having a generation unit that generates guidance information indicating a position and an orientation.

According to one aspect of the embodiment, the user can be appropriately guided.

FIG. 1 is a diagram showing an example of an information providing system according to an embodiment. FIG. 2 is a diagram showing an example of the effect of the generation process according to the embodiment. FIG. 3 is a diagram showing an example of a process in which the information providing system according to the embodiment corrects the position and orientation. FIG. 4 is a first diagram showing an example of a flow of processing executed by the information providing device regarding the embodiment. FIG. 5 is a second diagram showing an example of a flow of processing executed by the information providing device regarding the embodiment. FIG. 6 is a diagram showing an outline of the processing executed by the information providing device according to the embodiment at the time of correcting the guidance information. FIG. 7 is a diagram showing a configuration example of the information providing device according to the embodiment. FIG. 8 is a diagram showing an example of information registered in the drawing database according to the embodiment. FIG. 9 is a diagram showing an example of information registered in the landmark database according to the embodiment. FIG. 10 is a diagram showing a configuration example of the terminal device according to the embodiment. FIG. 11 is a diagram showing an example of a screen displayed by the terminal device according to the embodiment. FIG. 12 is a flowchart showing an example of a flow of processing executed by the information providing device according to the embodiment. FIG. 13 is a diagram showing an example of a hardware configuration.

Hereinafter, a generation device according to the present application, a generation method, and an embodiment for implementing the generation program (hereinafter referred to as “embodiment”) will be described in detail with reference to the drawings. It should be noted that this embodiment does not limit the generator, the generation method, and the generation program according to the present application. In addition, each embodiment can be appropriately combined as long as the processing contents do not contradict each other. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description is omitted.

[Embodiment]
[1. Information provision system]
First, with reference to FIG. 1, an example of an information providing system 1 having an information providing device 10 and a terminal device 100, which are examples of a generating device, and providing guidance information for guiding a user will be described. FIG. 1 is a diagram showing an example of an information providing system according to an embodiment. As shown in FIG. 1, the information providing system 1 includes an information providing device 10 and a plurality of terminal devices 101 and 102 (hereinafter, may be collectively referred to as "terminal device 100").

The information providing device 10 is an information processing device capable of communicating with the terminal device 100 via a predetermined network N (see, for example, FIG. 7) such as the Internet, and is realized by, for example, a server device, a cloud system, or the like. To. The information providing device 10 may be able to communicate with an arbitrary number of terminal devices 100 via the network N, or may be able to communicate with various external servers and external systems (not shown).

The terminal device 100 is a terminal device used by a user, and is realized by, for example, a PC (Personal Computer), a server device, various smart devices, and the like. Further, the terminal device 100 has a camera capable of shooting various subjects, and also has a function of transmitting captured images such as still images and moving images captured by the camera to the information providing device 10. Further, when the terminal device 100 receives the guidance information generated by the information providing device 10 by the generation process described later, the terminal device 100 has a function of guiding the user by displaying the received guidance information on the screen. ..

In the following description, it is assumed that the terminal device 101 is a terminal device used by the user U1 and the terminal device 102 is a terminal device used by the user U2. Further, in the following description, an example of a process of providing individual guidance information to the terminal devices 101 and 102 by one information providing device 10 will be described, but the embodiment is not limited to this. .. For example, the information providing system 1 may have a plurality of information providing devices 10, and each information providing device 10 may individually execute a generation process. Further, the generation process described below may be realized, for example, by the terminal device 100 in a stand-alone format.

[1-1. About user guidance]
Here, guidance information for appropriately guiding the user to the destination will be described. For example, when the position of a user is measured using various positioning systems using satellites such as GPS (Global Positioning System) and information indicating the measured position is provided, it is possible to indicate where the user is. can. In addition, the current position of the user can also be indicated by describing information indicating the position where the map is installed on the map.

However, when the user is trying to move to a destination different from the current position, the user identifies the direction in which he / she is facing, and then moves in the direction of the destination. Therefore, simply providing the user with information indicating the current position may cause the user to move in a direction different from the destination. In addition, it is not possible to save the trouble of specifying the direction in which the user is facing only by providing the user with the information indicating the current position.

A more specific example will be described. For example, when an icon indicating the position of a user is provided on a map, it is possible to indicate where the user is. However, when the user moves to the destination, the user identifies the direction in which he / she is facing by comparing the surrounding situation with the information on the map, and the purpose is based on the specified direction. Identify the direction to the ground. In this way, if only the position of the user is shown, it is not possible to reduce the time and effort required to specify the direction in which the user is facing. In addition, if the user mistakenly identifies the direction in which he / she is facing, there is a high possibility that the user will move in the wrong direction.

In this way, when guiding the user, information indicating the direction of the user may be important. Therefore, it is conceivable to provide information indicating not only the position of the user but also the direction of the user. For example, a method is conceivable in which a positioning system is used to measure the position and orientation of a user, and an icon indicating the measured position and an icon indicating the orientation are superimposed and displayed on a map. However, in various closed areas (hereinafter, simply referred to as "areas") such as underground malls, subway station platforms, department stores, shopping malls, indoor parking lots, etc., positioning signals transmitted from satellites. May not be received.

In order to guide the user in such an area, for example, a method of guiding using geomagnetism or guiding using a wireless signal such as a beacon has been proposed. However, when estimating the position of the user using the geomagnetism, it is necessary to measure the geomagnetism in the region in detail in advance, which is troublesome. Further, when the magnetic force generated from the line, the electric wire, various wirings, etc. installed in the area changes, the magnetic force measured by the terminal device changes, so that there is a possibility that accurate guidance cannot be performed.

On the other hand, when the position of the user is estimated using a wireless signal such as a beacon, it may not be possible to say that the position of the user is estimated accurately. For example, when a plurality of terminal devices located at different positions receive radio signals transmitted from the same beacon, it may be presumed that they are located at the same position. Moreover, when a wireless signal is used, the direction of the user cannot be estimated.

[1-2. About generation processing]
Therefore, the information providing device 10 executes the generation process described below and provides the guidance information generated by the generation process, thereby reducing the time and effort of the user and realizing the guidance.

For example, the information providing device 10 acquires a photographed image in which an object previously installed in a predetermined area is photographed. Subsequently, the information providing device 10 estimates, based on the captured image, a relative position which is a position of the photographing device that captured the photographed image with respect to the object and a relative orientation which is the direction of the photographing device with respect to the object. Then, the information providing device 10 generates guidance information indicating the position and orientation of the photographing device in a predetermined region based on the relative position and the relative orientation.

For example, the information providing device 10 acquires a photographed image of a drawing installed in a predetermined area as an object and in which a drawing schematically showing the area is taken. More specifically, the information providing device 10 is a movement route within various areas such as a drawing (for example, a premises map) installed inside a facility such as a station or a building, a map installed on a street corner, or the like. Acquires captured images of drawings that conceptually show various facilities, sections, etc.

Subsequently, the information providing device 10 uses the image of the drawing included in the captured image to determine the relative position, which is the position of the photographing device based on the drawing, and the relative orientation, which is the orientation of the photographing device based on the drawing. To estimate. For example, the information providing device 10 estimates the relative position and the relative orientation from the image of the map included in the captured image by using various camera calibration techniques.

Then, the information providing device 10 generates guidance information in which information indicating a position in the area of the photographing device in which the photographed image is photographed is superimposed on the drawing included in the photographed image. For example, the information providing device 10 may acquire data of the same drawing as the drawing included in the captured image, and generate guidance information in which an icon indicating the position and orientation of the photographing device is superimposed on the acquired drawing. .. Further, the information providing device 10 converts the drawing included in the captured image into a drawing viewed from the front, and generates guidance information in which an icon indicating the position and orientation of the photographing device is superimposed on the converted drawing. You may.

Then, the information providing device 10 provides the generated guidance information to the user. As a result of such processing, the information providing device 10 can provide the user with information indicating the current position of the user and the orientation of the user, so that the user can specify his / her current orientation. You can save the trouble of doing it. As a result, the user can immediately start moving to the destination.

Further, since the information providing device 10 provides guidance information in which information indicating the current position of the user is superimposed on the photographed drawing, it is necessary to provide information that maintains the same as the drawing that the user is viewing. Can be done. As a result, the information providing device 10 can easily recognize which direction the user is facing, and as a result, the user can be appropriately guided.

[1-3. About an example of generation processing]
Hereinafter, an example of the generation process executed by the information providing apparatus 10 will be described with reference to FIG. 1. In the following description, an outline of the flow of the generation process executed by the information providing device 10 will be described, and then detailed means for realizing the generation process will be described. For example, in the example shown in FIG. 1, it is assumed that a drawing M showing information on a passage or a store of a shopping mall is installed in an area such as a shopping mall. Further, it is assumed that the installation position MP of the drawing M is shown in the drawing M.

In such a case, the information providing device 10 uses the feature of the drawing M as a reference feature as a preparatory step, and sets the installation position of the drawing and the position of the photographing device in which the drawing is photographed (hereinafter, “reference photographing position”, or simply “reference”. In some cases, it is associated with and held (step S1). For example, the information providing device 10 acquires an image of the drawing M taken from the front by using a predetermined photographing device such as a camera or a smartphone. In such a case, the information providing device 10 uses various feature quantity extraction techniques to extract feature points, which are characteristic points, from the captured image, and the coordinates of the extracted feature points (hereinafter, "feature points"). It is described as "coordinates") and the feature amount indicating the feature of the image around the feature point is extracted.

The above-mentioned features include GIST (http://ilab.usc.edu/siagian/Research/Gist/Gist.html), color histograms, wide-area features such as color distribution, and SIFT (Scale-Invariant Feature Transform). , SURF (Speed-Up Robust Features), local feature amount, and any other technique for acquiring the feature amount of the image can be adopted. The features generated by the neural network can also be used. In addition to these existing feature amount extraction techniques, the information providing device 10 may use various arbitrary feature amount extraction techniques.

Further, the information providing device 10 acquires information indicating the installation position MP of the drawing M and the direction in which the front surface of the drawing M is facing (that is, the orientation of the drawing) in the area. Then, the information providing device 10 registers the extracted feature point coordinates and feature quantities, the installation position of the drawing, and the reference photographing position in a predetermined recording device in association with each other. In the following description, the feature points extracted in advance are referred to as reference feature points, the feature point coordinates of the reference feature points are used as reference coordinates, and the feature quantities of the reference feature points are collectively referred to as reference feature quantities. Further, in the following description, the reference feature point, the reference coordinate, and the reference feature amount may be collectively referred to as reference information.

Subsequently, the user U1 photographs the drawing M using his / her own terminal device 101 (step S2). For example, it is assumed that the user U1 photographs the drawing M diagonally from a direction shifted clockwise from the front of the drawing M. In such a case, the terminal device 101 transmits the captured image of the drawing M to the information providing device 10 (step S3).

On the other hand, it is assumed that the user U2 uses the terminal device 102 to take an oblique image of the drawing M from a direction deviated from the front of the drawing M in a counterclockwise direction (step S4). In such a case, the terminal device 102 transmits the captured image of the drawing M to the information providing device 10 (step S5).

In such a case, the information providing device 10 estimates the relative positions and orientations of the users U1 and U2 with respect to the drawing by using the captured images received from each terminal device 100. More specifically, the information providing device 10 is relative to the photographing device (that is, the terminal device 100) with respect to the drawing M based on the comparison result between the reference information registered in the storage device and the feature of the captured image. Estimate the position and orientation (step S6). That is, the information providing device 10 stores the reference information based on the reference image which is an image of the object captured from a predetermined position, and is based on the comparison result between the image information based on the captured image and the stored reference information. To estimate the relative position and the relative orientation. More specifically, the information providing device 10 estimates the relative position and the relative orientation by using the information about the feature points included in the reference image and the information about the feature points extracted from the captured image.

For example, the information providing device 10 uses the same algorithm as when extracting the reference information, and extracts feature points as comparative feature points from the captured image taken by the terminal device 100. Further, the information providing device 10 extracts the feature amount at the comparative feature point as the comparative feature amount, and extracts the coordinates of the comparative feature point as the comparative coordinate.

Subsequently, the information providing device 10 mutually compares the comparative feature amount and the reference feature amount, and identifies a set of the comparative feature point and the reference feature point whose feature amounts are similar to each other. In the following description, a comparative feature point similar to any of the reference feature points will be referred to as a similar comparative feature point, and a reference feature point similar to any of the comparative feature points will be referred to as a similar reference feature point. Further, in the following description, it is assumed that a plurality (predetermined number or more) of similar comparative feature points are extracted from the captured image. Then, the information providing device 10 generates a transformation matrix that converts the feature point coordinates of the similar comparison feature points into the feature point coordinates of the similar reference feature points.

Such a transformation matrix indicates how much the terminal device 101, which is a photographing device, moves in parallel with respect to the reference photographing position (that is, in front of the drawing M) and how much is rotated from the reference photographing position with respect to the drawing M. It is an index to show. Therefore, the information providing device 10 estimates the position and orientation of the terminal device 101 with respect to the drawing M, that is, the relative position and the relative orientation, based on the transformation matrix. Similarly, the information providing device 10 estimates the relative position and the relative orientation of the terminal device 102 with respect to the front surface of the drawing M from the captured image received from the terminal device 102. The detailed method for generating the transformation matrix and estimating the relative position and the relative orientation will be described later.

Then, the information providing device 10 generates guidance information in which information indicating the specified relative position and relative orientation is superimposed on the drawing M included in the captured image captured by each terminal device 100 (step S7). For example, the information providing device 10 uses the above-mentioned transformation matrix to convert the image of the drawing M included in the captured images taken by the terminal devices 101 and 102 into an image when the drawing M is viewed from the front. In the following description, the image of the drawing M after conversion is collectively referred to as a correction drawing. Further, various known image conversion techniques can be adopted for such image conversion.

Subsequently, the information providing device 10 specifies the installation position of the drawing M on the correction drawing. For example, the information providing device 10 specifies a projection installation position, which is a position on the drawing M indicating the installation position MP, from the pre-registered drawing M. It should be noted that such identification of the projection installation position may be realized by an image identification technique for specifying a predetermined image (for example, a predetermined icon or character indicating the installation position MP) included in the image. For example, the drawing M may be realized. When registering the image of, it may be done manually. Subsequently, the information providing device 10 converts the projection installation position on the pre-registered drawing M into the projection installation position on the correction drawing.

Then, the information providing device 10 specifies the position and orientation of the terminal device 100 on the correction drawing based on the projection installation position on the correction drawing and the estimated relative position and relative orientation. For example, the information providing device 10 specifies the relative position and the relative orientation on the correction drawing with reference to the projection installation position on the correction drawing. Then, the information providing device 10 arranges a round icon at a relative position on the correction drawing, and generates guidance information in which an arrow, a fan shape, or a triangle icon indicating a relative direction on the correction drawing is arranged.

In this way, the information providing device 10 generates a correction drawing in which the drawing included in the photographed image is corrected as if the drawing was photographed from a predetermined direction, and information indicating the position of the photographing device is superimposed on the generated correction drawing. Generate guidance information. For example, the information providing device 10 estimates the position of the photographing device in the predetermined area based on the position where the object is installed in the predetermined area and the relative position, and provides guidance information indicating the estimated position. Generate. Further, the information providing device 10 estimates the orientation of the photographing device in the predetermined region based on the orientation of the object in the predetermined region and the relative orientation, and generates guidance information indicating the estimated position. Further, the information providing device 10 converts the position and orientation of the photographing device into coordinates on the drawing based on the relative position and the relative orientation, and generates a guide drawing using the converted coordinates.

After that, the information providing device 10 transmits the generated guidance information to the terminal devices 101 and 102 (steps S8 and 9). As a result, the terminal device 101 displays the guidance information DM1 in which the icon UI1 indicating that the icon UI1 is facing the direction of the drawing M from the position rotated clockwise from the front of the drawing M is arranged. On the other hand, the terminal device 102 will display the guidance information DM2 in which the icon UI2 indicating that the icon UI2 is facing the direction of the drawing M from the position rotated counterclockwise from the front of the drawing M is arranged.

For example, FIG. 2 is a diagram showing an example of the effect of the generation process according to the embodiment. For example, in the example shown in FIG. 2, the drawing M is arranged on the signboard MB arranged in the facility. Further, in the example shown in FIG. 2, the user U1 is located at a place moved clockwise around the drawing M (or the signboard MB) from the front of the drawing M, and is half centered on the drawing M (or the signboard MB). The user U2 is located at the place where it has moved clockwise. Further, the users U1 and U2 are assumed to be located within a predetermined range NA from the signboard MB, respectively.

Here, when the drawing M is not installed on the signboard MB and, for example, when a radio signal indicating a position is transmitted by a beacon, the terminal device 101 used by the user U1 and the terminal device 102 used by the user U2 are The same radio signal will be received. Therefore, although the terminal devices 101 and 102 are located at different positions, they output guidance information indicating that they are at the same position.

On the other hand, it is considered that the photographed image taken by the user U1 and the photographed image taken by the user U2 have different aspects of the drawing M. That is, it is considered that the captured image contains the image of the drawing M in a different manner depending on the relative position and orientation of the photographing device with respect to the drawing M. Therefore, the information providing device 10 according to the embodiment generates guidance information indicating the position and direction of each user U1 and U2 from the captured images of the drawing M taken by the users U1 and U2 from different positions.

As a result of such processing, for example, in the terminal device 101, the user U1 is located at a place where the user U1 moves clockwise around the drawing M (or the signboard MB) from the front of the drawing M, and the user U1 is used. It is possible to provide the guidance information DM1 indicating that the person U1 is facing the direction of the drawing M. Further, in the terminal device 102, the user U2 is located at a place where the user U2 moves counterclockwise around the drawing M (or the signboard MB) from the front of the drawing M, and the user U2 is shown in the drawing M. Guidance information DM2 indicating that the user is facing a direction can be provided.

Here, the terminal device 100 may only provide the guidance information generated by the generation process, but may further display the information for guiding the user after that.

For example, when the terminal device 100 detects the movement or change of the direction of the user by using a positioning system such as GPS, a beacon, or the like, the terminal device 100 collectively refers to the icon UI1 and UI2 (hereinafter, referred to as “icon UI”) on the drawing. The position of.) May be moved or rotated. Further, the terminal device 100 detects the movement or change in orientation of the user by using an acceleration sensor, a gyro sensor, a vibration sensor, or the like, and applies the detected movement or change in orientation to the position or orientation of the icon UI. May be good. It should be noted that such processing can be realized by, for example, providing information indicating the relationship between the position on the drawing and the actual distance (for example, information indicating the scale of the drawing or map) together with the guidance information. be.

Further, the terminal device 100 and the information providing device 10 may provide guidance information indicating a guidance direction to the destination when the destination of the user is set in advance. For example, the terminal device 100 may provide guidance information in which a solid line or a dotted line indicating a guidance route to a destination, an arrow indicating the direction of the destination, or the like is superimposed on a drawing.

In addition to these, the information providing device 10 and the terminal device 100 may provide information for guiding the user to the destination by combining various publicly known techniques related to the user's guidance.

[1-4. About updating the position and orientation using landmarks]
Here, when the position and orientation of the user after movement are superimposed on the drawing, the position and orientation of the user may not be correctly indicated depending on the detection accuracy of the position and orientation. For example, it is possible that the user's position is located in the store even though the user is moving along the aisle, or the user's direction is different from the correct direction.

Therefore, the information providing device 10 generates guidance information in which the position and orientation of the user are corrected by using photographed images of various arbitrary equipment such as signboards, doors, stores and objects installed in the area. You may. For example, the information providing device 10 acquires a photographed image in which a new object is photographed after the guidance information is provided, estimates the relative position and the relative orientation of the photographing device with respect to the new object, and makes a new object. Based on the relative position and relative orientation with respect to the object, new guidance information may be generated in which the position and orientation of the photographing apparatus within a predetermined region are corrected. To give a more specific example, the information providing device 10 acquires a photographed image taken by a predetermined facility installed in a predetermined area as a new object, and is relative to a position relative to the predetermined facility. Correction may be made based on the orientation.

Hereinafter, with reference to FIG. 3, an example of a process of generating guidance information corrected for the position and orientation of the user based on the relative position and the relative orientation with respect to the predetermined equipment will be described. FIG. 3 is a diagram showing an example of a process in which the information providing system according to the embodiment corrects the position and orientation. In the example shown in FIG. 2, an example of the process of correcting the position and orientation of the user U1 by using the photographed image of the door LM installed in the facility is described. Further, in the following description, various facilities used for correction are collectively referred to as landmarks.

For example, the information providing device 10 uses the feature of the image in which the landmark is photographed as reference information, and holds the position on the map of the image in which the landmark is photographed, the installation position of the landmark, and the reference photographing position in association with each other. (Step S10). Subsequently, it is assumed that the user U1 moves on the passage after receiving the guidance information DM1 generated by the generation process shown in FIG. 1 (step S11). In such a case, the terminal device 101 updates the current position and orientation using various sensor information (step S12).

Here, it is assumed that the terminal device 101 erroneously estimates the current position and orientation, and displays the guidance information DM3 indicating that the terminal device 101 has moved into the store. In such a case, the user U1 photographs the landmark using the terminal device 101 (step S13) and transmits the captured image to the information providing device 10 (step S14).

In such a case, the information providing device 10 estimates the relative position and the relative direction of the user with respect to the landmark by performing the same processing as the generation processing shown in FIG. 1, and provides new guidance information. May be good. However, unlike maps, landmark equipment is often a three-dimensional object. In order to accurately estimate the relative position and the relative orientation with respect to such a three-dimensional object, the information providing device 10 executes the following processing at the time of correction.

For example, the information providing device 10 estimates the relative position and the relative orientation of the user from the estimation result (step S15). For example, the information providing device 10 uses the camera calibration technique to generate a basic matrix that converts the reference shooting position where the landmark was shot in step S10 to the position where the landmark was shot in step S13. ..

Subsequently, the information providing device 10 estimates the three-dimensional shape of the landmark from the captured image by using various known techniques for restoring (estimating) the three-dimensional shape of the object captured from the two-dimensional image (step). S16). For example, the information providing device 10 has a point that satisfies a predetermined condition among the estimated three-dimensional shapes of the landmark (for example, the center and the center of gravity of the three-dimensional shape, the point that first intersects the line of sight of the photographing device among the three-dimensional shapes, and the like. ) Is specified, and the relative position and orientation of the photographing device with respect to the specified point are estimated.

Then, the information providing device 10 generates guidance information corrected for the position and orientation of the user in the facility based on the estimated relative position and relative direction (step S17), and transmits the generated guidance information (step S17). Step S18). For example, the information providing device 10 converts the relative position and the relative direction into the position and direction on the drawing of the transmitted guidance information DM1, and transmits the information indicating the converted position and direction to the terminal device 101. As a result, the terminal device 101 can display the guidance information DM4 in which the position and orientation of the user are correctly corrected.

[1-5. Relationship between estimation using drawings and estimation using landmarks]
In the above example, in the initial stage, the relative position and relative orientation of the user are estimated from the captured image of the drawing schematically showing the inside of the area such as the facility, and the relative position and the relative orientation in the area are estimated based on the estimated relative position and the relative orientation. Provided information indicating the position and orientation of the user. On the other hand, at the time of correction, the relative position and relative orientation of the user were estimated from the captured image of the landmark, and the position and orientation of the user in the guidance information were corrected based on the estimated relative position and relative orientation. However, the embodiments are not limited to this.

For example, when the information providing device 10 acquires a captured image of a landmark in the initial stage, the information providing device 10 estimates the three-dimensional shape of the landmark from the acquired captured image, and the user's relative to the landmark is based on the estimated three-dimensional shape. The position and relative orientation may be estimated, and guidance information indicating the user's position and orientation may be generated based on the estimated relative position and relative orientation. Further, the information providing device 10 estimates the relative position and the relative orientation with respect to the drawing from the captured image of the drawing, and corrects the position and the orientation of the user in the guidance information based on the estimated relative position and the relative orientation. good.

That is, regardless of whether the object is a two-dimensional object such as a drawing or a three-dimensional object such as various facilities, the information providing device 10 can be used to obtain the object and the photographing device from the photographed image of the object. Photographing any object as long as it can estimate the relative position and orientation of the image and provide guidance information indicating the position and orientation of the imaging device within the region based on the estimated relative position and orientation. Guidance information may be generated or corrected from the device.

As will be described later, in the process using the drawing, the size and the aspect ratio of the drawing are used when acquiring the reference information in order to improve the estimation accuracy. However, it is troublesome to acquire such an aspect ratio for various landmark facilities such as doors and stores. Therefore, the information providing device 10 estimates the three-dimensional shape of the landmark from the captured image, and considers that the gazing point in the estimated three-dimensional shape is the same at the time of acquiring the reference information and at the time of acquiring the captured image. Estimate the relative position and orientation of the imager with respect to the mark. As a result of such processing, the information providing device 10 can accurately estimate the relative position and the relative orientation of the photographing device without acquiring the aspect ratio of the landmark.

Also, in order to secure the opportunity for correction, it is better to prepare many landmarks that can be used for correction, but it is not realistic to install many drawings in the facility. However, the information providing device 10 estimates the three-dimensional shape of the landmark from the captured image, and estimates the relative position and the relative orientation based on the three-dimensional shape. Therefore, for example, a photograph of the facility that becomes the landmark is taken and photographed. By simply registering the position, it is possible to add a position where the guidance information can be corrected.

In the above-mentioned example, the information providing device 10 acquires the position of the photographing device in which the landmark is photographed when acquiring the reference information, but the embodiment is not limited to this. For example, the information providing device 10 reduces the trouble of registering the position of the photographing device when collecting reference information by acquiring an image of a landmark taken at a predetermined distance from a predetermined direction. You may.

As described above, the information providing device 10 generates guidance information using drawings in the initial stage, and corrects the guidance information using the captured image of the landmark in the correction stage. Therefore, the information providing device 10 can accurately provide the current position and orientation of the user while suppressing the time and effort of preparation for realizing the generation process.

[1-6. Detailed processing flow of estimation using drawings]
Next, with reference to FIG. 4, an example of a detailed flow of the process of estimating the relative position and the relative orientation using the drawings will be described. FIG. 4 is a first diagram showing an example of a flow of processing executed by the information providing device regarding the embodiment.

First, the processing in the preparatory stage for acquiring the reference information of the drawing will be described. First, the information providing device 10 acquires a photographed image obtained by photographing a drawing from a predetermined distance. Here, it is assumed that the aisles and stores of a predetermined facility are described in the drawings at a predetermined scale.

Here, it is desirable that the captured image is an image captured from the front of the drawing (for example, on a line passing through the center of the drawing and perpendicular to the drawing). However, when the image is taken manually, it is difficult to take the drawing from the front completely. Therefore, the information providing device 10 acquires a photographed image of the drawing and information indicating the aspect ratio (aspect ratio) of the drawing. Then, the information providing device 10 corrects the drawing to a front image which is an image taken from the front based on the information indicating the aspect ratio. For example, the information providing device 10 specifies a range in which the drawing is captured in the captured image, and corrects the captured image so that the aspect ratio of the specified range matches the acquired aspect ratio.

Further, the information providing device 10 extracts the feature point coordinates and the feature amount from the generated front image. Then, the information providing device 10 stores the feature point coordinates, the feature amount, the information indicating the installation position of the drawing in the facility, and the information indicating the reference photographing position as reference information. For example, when the information providing device 10 superimposes an icon indicating the position and orientation of the photographing device on the photographed drawing, the information providing device 10 acquires information indicating the installation position of the drawing and the reference photographing position on the drawing. When the icon is superimposed on a drawing different from the captured drawing, the information providing device 10 may hold, for example, the latitude and longitude indicating the installation position and the reference imaging position of the drawing as the installation position.

Subsequently, in the generation processing stage, the information providing device 10 acquires a captured image. In such a case, the information providing device 10 acquires the feature point coordinates and the feature amount as feature information from the captured image. Then, the information providing device 10 compares the extracted feature information with the reference information and generates a homography matrix. For example, the information providing device 10 generates a homography matrix as a matrix that mutually converts the feature point coordinates of the similarity comparison feature point and the feature point coordinates of the similarity reference feature point. That is, the information providing device 10 stores information indicating the position of the feature point included in the reference image and the feature amount indicating the feature at the feature point. Then, the information providing device 10 generates a transformation matrix that converts the positions of the feature points included in the captured image into the positions of the feature points having similar feature amounts among the feature points included in the reference image.

Such a homography matrix is a relative positional relationship between the position of the photographing device when the photographed image is taken and the front surface of the drawing, and the orientation of the photographing device when the photographed image is taken and the front direction of the drawing. The relative orientation relationship (rotational relationship) with and is reflected. Therefore, the information providing device 10 estimates the relative position and the relative orientation based on the generated transformation matrix, that is, the homography matrix.

For example, the information providing device 10 uses the homography matrix as a parallel matrix (first matrix) that transforms the positional relationship between the position of the photographing device when the captured image is captured and the front surface of the drawing, and the captured image. Generates a rotation matrix that transforms the orientation of the imaging device at the time of imaging and the front orientation of the drawing. Then, the information providing device 10 estimates the relative position from the generated parallel matrix, and estimates the relative direction from the generated rotation matrix.

On the other hand, the information providing device 10 generates a correction drawing when the drawing is photographed from the front from the photographed image. For example, the information providing device 10 uses a homography matrix (or its inverse matrix) to convert a captured image into a captured image when the drawing is captured from the front. Further, the information providing device 10 uses various image recognition techniques to extract a range including the drawing from the converted captured image as a correction drawing.

Then, the information providing device 10 generates guidance information from the installation position serving as reference information, the reference shooting position, the estimated relative position, and the relative orientation. For example, the information providing device 10 specifies the position and orientation of the photographing device on the front image based on the estimated relative position, the installation position serving as the reference information, and the reference photographing position.

For example, the information providing device 10 identifies the position of the photographing device in the area based on the installation position and the relative position. Further, the information providing device 10 estimates the orientation of the imaging position in the region based on the reference imaging position, the installation position, and the relative orientation. Then, the information providing device 10 estimates the installation position and orientation of the photographing device on the front screen based on the estimated photographing position and orientation. The information providing device 10 estimates the orientation of the photographing device on the front drawing based on the installation position of the drawing on the front image, the reference photographing position on the front drawing, and the relative position and the relative orientation on the front drawing. You may.

Further, the information providing device 10 estimates the distance between the photographing device and the drawing when the drawing is photographed in the processing stage based on the relative position, and sets the estimated distance and the distance when the drawing is photographed in the reference stage. Based on this, the scale relationship between the correction drawing and the front drawing is estimated. Then, the information providing device 10 converts the position and orientation of the photographing device on the front drawing into the position and orientation of the photographing device on the correction drawing based on the estimated scale relationship. After that, the information providing device 10 generates guidance information in which an icon indicating the position and orientation of the photographing device is superimposed on the correction drawing.

The information providing device 10 holds information indicating the reference position and the reference direction with the position of the photographing device with respect to the drawing when the drawing is photographed in the preparation stage as the reference position and the orientation as the reference direction. Then, the information providing device 10 estimates the relative position and the relative orientation with respect to the reference position and the reference orientation by comparing the feature information of the captured image acquired in the processing stage with the feature information of the drawing acquired in the preparation stage. .. Then, the information providing device 10 specifies the position and orientation of the photographing device on the drawing to be displayed based on the scale of the drawing to be displayed, the installation position and the reference photographing position of the drawing, and the relative position and the relative orientation. May be good.

That is, the information providing device 10 may adopt the installation position and the installation direction of the drawing as the reference position and the reference direction in the preparation stage, and the position of the photographing device is based on the reference position and the relative position and the relative direction with respect to the reference direction. And the orientation may be specified. Further, the information providing device 10 is a captured image captured in the preparatory stage in advance or based on arbitrary information that can be acquired from the captured image (for example, the scale of the drawing or the distance between the imaging device and the drawing). The relationship between the coordinates on the drawing and the coordinates on the drawing in the captured image captured in the processing stage may be specified, and information indicating the position and orientation of the specified imaging device may be displayed based on the specified relationship.

[1-7. Detailed processing flow of correction using landmarks]
Next, an example of a detailed flow of the process of correcting the guidance information using the landmark will be described with reference to FIG. FIG. 5 is a second diagram showing an example of a flow of processing executed by the information providing device regarding the embodiment.

As described above, when making corrections using landmarks, it is troublesome to register the aspect ratio for each facility that becomes a landmark. Therefore, in the preparation stage, the information providing device 10 acquires an image with the central portion of the landmark as a gaze point as a photographed image as a photographed image of the landmark. Then, the information providing device 10 extracts the information of the feature points and holds the information indicating the installation position of the landmark and the reference photographing position as the reference information.

Subsequently, the information providing device 10 acquires an image with the central portion of the landmark as the gazing point as a captured image in the correction step of correcting the guidance information. Then, the information providing device 10 extracts the feature information of the feature point from the captured image, compares it with the reference information, and corrects the position and orientation of the imaging device when the landmark is captured in the preparation stage. Generates a basic matrix that converts the position and orientation of the photographing device when the landmark is photographed.

For example, the information providing device 10 acquires the internal parameters of the camera of the photographing device that captured the landmark in the preparation stage, and acquires the internal parameters of the camera of the photographing device that captured the landmark in the correction stage. Then, the information providing device 10 estimates the basic matrix using the position of the similar reference feature point coordinates, the position of the similar comparison feature point coordinates, and the internal parameters of each photographing device, and rotates and translates the basic matrix. Decompose into external parameters of. Then, the information providing device 10 restores the three-dimensional shape of the landmark by using the external parameter.

As for the specific generation method of the basic matrix, known generation techniques such as the generation method using epipolar geometry (for example, "Visual Geometry 3 Camera Calibration" Wakayama University Internet <http: // web.wakayama-u.ac.jp/~wuhy/CV11.pdf> Search on August 26, 1st year of Wakayama), and detailed explanation is omitted. As for the 3D shape restoration method, known restoration technology (for example, "Understanding the deformation of the structure by the 3D point cloud data restored from the photographic image" Koji Ichii Hiroshima University Internet <http: // www. jacic.or.jp/josei/pdf/2014_01.pdf> August 26, 1st year of Reiwa Search, technology using various deep learning, etc.) can be adopted, and detailed explanation is omitted.

Subsequently, the information providing device 10 identifies the center point of the landmark from the three-dimensional shape of the generated landmark. For example, the information providing device 10 estimates the line-of-sight direction of the image-taking device from the captured image in the correction stage, and sets the position where the estimated line-of-sight direction and the three-dimensional shape of the landmark intersect as the center point of the landmark. Then, the information providing device 10 estimates the relative position and the relative direction of the photographing device with respect to the center point. That is, the information providing device 10 assumes that the positions of the landmarks set by the photographer as the center points are almost the same in the preparation stage and the correction stage, and the relative positions and the relative orientations with respect to the center points of the landmarks. To estimate.

For example, the information providing device 10 estimates the scale of the distance between the photographing device and the center point of the landmark in the preparatory stage based on the basic matrix based on the distance between the reference photographing position and the center point of the landmark. Further, the angle between the line connecting the information providing device 10 and the reference photographing position and the center point of the landmark and the line connecting the photographing device and the center point of the landmark in the correction stage is calculated. Then, the information providing device 10 estimates the relative position and the relative orientation of the photographing device in the correction stage from the calculated distance and angle with reference to the photographing position in the preparation stage.

For example, FIG. 6 is a diagram showing an outline of processing executed by the information providing device according to the embodiment at the time of correction of guidance information. For example, a photographer who shoots a landmark in the preparation stage shoots an image from the position PU with the position considered to be the center of the landmark LM as the gazing point. Subsequently, in the correction stage, the user U1 takes an image from the position AP with the position considered to be the center of the landmark LM as the gazing point.

Here, when the positions to be the gazing points in each image are the same position CP, the generated basic matrix converts the position PP and the position AP, and at the same time, from the position PP to the position CP with the position CP as the center. It can be regarded as a matrix that transforms the direction of and the direction from the position AP to the position CP. Therefore, the information providing device 10 estimates the central position CP from the captured image taken from the position AP. Then, the information providing device 10 assumes that the position CP is the same position in both the preparation stage and the correction stage, and from the basic matrix, the relative position of the position AP with respect to the position PP and the position PP. The relative direction from the position AP to the position CP is estimated based on the direction from the position CP.

Here, the information providing device 10 reduces the information in the Y-axis direction of the basic matrix when the direction perpendicular to the ground is the Y-axis direction and the plane parallel to the ground is the XZ plane. The amount of calculation may be further reduced by estimating the relative position and the relative direction on the XZ plane. That is, the information providing device 10 may estimate the relative position and the relative orientation on the assumption that the height of the photographing device with respect to the ground is substantially the same in the preparation stage and the correction stage.

Returning to FIG. 5, the explanation will be continued. The information providing device 10 estimates the position and orientation of the terminal device 100 with respect to the center of the landmark from the landmark installation position, the reference shooting position, and the estimated relative position and relative orientation. For example, the information providing device 10 estimates the position and orientation of the terminal device 100 with respect to the center of the landmark from the reference shooting position and the estimated relative position and orientation.

For example, the information providing device 10 assumes that the imaging device has photographed the landmark from a predetermined position with respect to the landmark in the preparatory stage, and is the center of the landmark from the estimated relative position and relative orientation. The position and orientation of the terminal device 100 may be estimated with reference to. That is, the information providing device 10 sets the relative position of the position AP with respect to the position PP and the relative direction from the position AP to the position CP with respect to the direction from the position PP to the position CP with respect to the center of the landmark. Convert to relative position and relative orientation as a reference.

Then, the information providing device 10 specifies the position and orientation of the photographing device on the drawing provided in the processing stage based on the relative position and the relative orientation after conversion and the installation position and the reference photographing position in the reference information. , Provide position and orientation on the identified drawing. As a result of such processing, the terminal device 100 can provide the corrected guidance information.

In the above-mentioned example, the information providing device 10 has acquired an image with the center of the landmark as the gaze point, but the embodiment is not limited to this. For example, if an object that becomes a landmark has an object (for example, a guide such as a seal or an object such as a doorknob) that has been notified in advance to be the gazing point, an image with these as the gazing point. Should be acquired as a captured image.

[1-8. Variations on generation processing]
In the above-mentioned example, the information providing device 10 provides the position and orientation of the photographing device using the photographed image. Here, the above description is merely an example, and the embodiment is not limited thereto. Hereinafter, variations of the processing executed by the information providing device 10 will be described.

[1-8-1. Automatic judgment of processing]
In the above description, the information providing device 10 acquires a photographed image in which the drawing is photographed at the time of starting the guidance in the initial state, and acquires an image in which the landmark is photographed at the time of correction. Here, the information providing device 10 may determine whether the captured image contains either a drawing or a landmark, and may change the process to be executed among the above-mentioned processes according to the determination result. ..

For example, the terminal device 100 displays a UI (User Interface) such as a shooting button that is different when shooting a drawing and when shooting a landmark, and the shot image is according to the UI used by the user for shooting. The information providing device 10 may be notified whether the drawing is photographed or the landmark is photographed. Then, when the information providing device 10 receives the notification that the drawing is taken in the captured image, the information of the feature points extracted from the drawing as the reference information and the information of the feature points extracted from the captured image. From, a homography matrix may be generated, and the generated homography matrix may be used to generate guidance information indicating the position and orientation of the user. Further, when the information providing device 10 receives the notification that the landmark is photographed in the photographed image, the information of the feature points extracted from the drawing as the reference information and the feature points extracted from the photographed image. A basic matrix may be generated from the information, and the generated basic matrix may be used to generate guidance information indicating the position and orientation of the user.

Further, the information providing device 10 compares the feature points included in the captured image with the feature points extracted from the drawing and the feature points extracted from the landmark to check whether the drawing is captured in the captured image or the land. It may be determined whether the mark is photographed. For example, the information providing device 10 searches for feature points that are most similar to the feature points included in the captured image among the feature points extracted in advance from the drawings and landmarks. Then, the information providing device 10 may target a drawing or a landmark determined to have the most similar features in the captured image. That is, the object captured in the image provided by the information providing device 10 is a drawing installed in a predetermined area and schematically shows the predetermined area, or is installed in the predetermined area. It is determined whether it is a predetermined facility, and if it is determined that the object is a drawing, guidance information indicating the position and orientation of the photographing device in the predetermined area is generated, and the object is the predetermined facility. If it is determined to be present, new guidance information may be generated in which the position and orientation of the photographing apparatus in a predetermined area are corrected.

Further, the information providing device 10 further determines whether the object is an object other than the drawing and the predetermined equipment, and if it is determined that the object is an object other than the drawing and the predetermined equipment, the drawing or the object is determined. Suggestion information suggesting imaging of a given facility may be generated. For example, the information providing device 10 searches for the feature amount most similar to the feature amount extracted from the captured image among the feature amounts of the drawings and landmarks registered in the database in advance as the similar feature amount. Subsequently, the information providing device 10 counts the number of similar feature quantities for each drawing and landmark registered in the database in advance. Then, the information providing device 10 may determine that the object is an object other than the drawing and the predetermined equipment when there is no drawing or landmark whose number of similar feature quantities exceeds a predetermined threshold value. Further, the information providing device 10 calculates, for example, the variance of the number of similar features extracted from each drawing and landmark, and when the calculated variance exceeds a predetermined threshold, the object is a drawing and a predetermined equipment. It may be determined that the object is other than the above.

In such a case, the information providing device 10 may display a message such as "Please take a picture of a drawing or a landmark object" on the terminal device 100. Further, the information providing device 10 may display a message indicating the name of an object to be a landmark (for example, "signboard", "door", etc.) on the terminal device 100.

[1-8-2. About drawings]
In the above-mentioned example, the information providing device 10 provides guidance information in which information indicating the position and orientation of the photographing device is superimposed on the drawing included in the captured image. However, the embodiments are not limited to this.

For example, the information providing device 10 may generate a text message indicating the position and orientation of the photographing device and deliver the generated text message. For example, the information providing device 10 may specify the latitude and longitude of the photographing device and the direction in which the photographing device is oriented from the relative position and the relative direction, and display the specified information on the terminal device 100.

Further, the information providing device 10 specifies a store or facility located in the vicinity of the user from the position of the user, and also specifies a store or facility located in the direction of the user. Then, the information providing device 10 may output a text or voice such as "currently facing the direction of the candy shop near the movie theater".

Further, the information providing device 10 may generate guidance information in which information schematically indicating the position and orientation of the photographing device is superimposed on the drawing registered in advance instead of the photographed drawing. For example, the information providing device 10 holds the coordinates of the installation position of the object and the reference photographing position on the drawing registered in advance and the scale of the drawing as reference information. Then, the information providing device 10 estimates the relative position and the relative orientation of the photographing apparatus from the captured image, and from the estimated relative position and the relative orientation and the scale of the drawing, the position of the photographing apparatus and the position of the photographing apparatus on the pre-registered drawing and the drawing. The orientation may be specified, and guidance information may be distributed by superimposing information indicating the specified position and orientation on a pre-registered drawing.

The drawing registered in advance may be the same drawing as the drawing installed as the object, or may be a different drawing. For example, the information providing device 10 captures a drawing (for example, a bird's-eye view, a bird's-eye view, a panoramic view, a projection drawing, etc., hereinafter collectively referred to as a “three-dimensional drawing”) showing a region three-dimensionally, such as a subway yard map. When the image was acquired, the position and orientation of the photographing apparatus were superimposed on a drawing (for example, a top view, etc., hereinafter collectively referred to as "two-dimensional drawing") in which the area was schematically shown two-dimensionally from above. Guidance information may be provided.

For example, the information providing device 10 holds the feature points extracted from the three-dimensional drawing and the reference information indicating the installation position and the reference shooting position of the three-dimensional drawing on the two-dimensional drawing to be displayed. Subsequently, when the information providing device 10 receives the captured image in which the three-dimensional drawing is captured, the relative position and relative orientation are estimated from the captured image, the estimated relative position and relative orientation, and the installation position and reference of the reference information. From the shooting position, the position and orientation of the shooting device on the two-dimensional drawing to be displayed are estimated. Then, the information providing device 10 provides guidance information in which information indicating the position and orientation of the photographing device is superimposed on the two-dimensional drawing. When the information providing device 10 receives the captured image captured by the two-dimensional drawing, the information providing device 10 may provide information in which information indicating the position and orientation of the photographing device is superimposed on the three-dimensional drawing.

That is, the information providing device 10 holds information on the feature points of the object to be photographed and information indicating the installation position and the reference photographing position of the object on the drawing to be displayed as reference information. Then, the information providing device 10 estimates the relative position and relative orientation of the photographing device from the comparison result between the feature points extracted from the captured image and the feature points of the reference information, and the estimated relative position and relative orientation and the reference. The position and orientation of the imaging device on the drawing to be displayed may be estimated from the information installation position and the reference imaging position.

When the latitude, longitude, and direction are registered as the installation position and the reference photographing position, the information providing device 10 specifies the latitude, longitude, and direction of the photographing device from the relative position and the relative direction. Then, the information providing device 10 may superimpose and display information indicating the latitude, longitude, and direction of the photographing device on, for example, a map or a drawing.

In addition, although train and subway platforms often have station yard maps, some users may want to know information about facilities located outside the station. For example, at a subway station, the east exit and the west exit are located, but when heading to a facility on the ground, there may be a desire to know which direction to go, the east exit or the west exit. Therefore, the information providing device 10 acquires, as an object, an image in which a drawing of a facility located underground is taken as a photographed image, and is a drawing of a facility located in the same area as the facility and located on the ground. Guidance information may be generated on which information schematically indicating the position and orientation of the photographing apparatus is superimposed.

The information providing device 10 holds the latitude and longitude in which the object is installed as the installation position and the reference photographing position. Further, the information providing device 10 holds information indicating the latitude, longitude and direction at each position of the drawing together with a drawing schematically showing the facility on the ground as a drawing to be displayed. Subsequently, the information providing device 10 estimates the relative position and the relative orientation of the photographing device from the photographed image obtained by photographing the underground premises map, and based on the estimated relative position and the relative orientation, the installation position, and the reference photographing position. , Estimate the latitude, longitude and direction of the imaging device. Then, the information providing device 10 may provide guidance information in which information indicating the latitude, longitude, and direction of the photographing device is superimposed on the drawing schematically showing the facility on the ground.

Further, the information providing device 10 may, for example, superimpose and display information indicating the position and orientation of the photographing device on a drawing on a floor different from the photographed drawing. For example, the information providing device 10 receives a photographed image of a drawing installed on the first floor of a facility and also receives information on a store intended by the user. In such a case, the information providing device 10 identifies the floor where the store is located, which is the target of the user, and displays guidance information on which information indicating the position and orientation of the photographing device is superimposed on the drawing of the specified floor. You may.

When displaying the photographed drawing, the information providing device 10 may trim the photographed drawing using the feature points, or may change the display size. Further, when the photographed drawing is a bird's-eye view, the information providing device 10 may appropriately modify the scale or the like in each direction. For such various corrections of drawings, for example, information indicating correction contents and correction methods is stored in association with information of drawings taken in the preparation stage, and when the drawings are taken, the correspondence is associated. It can be realized by applying the corrected content and correction method.

[1-8-3. Processing when using a photographed drawing]
When displaying the drawing included in the captured image, the information providing device 10 does not have to execute the process based on the relative position and the relative orientation of the user. For example, the information providing device 10 generates a homography matrix by comparing the feature points extracted from the captured image with the feature points of the reference information, and the generated homography matrix is included in the captured image. Change the drawing to a correction drawing. Then, the information providing device 10 may display information indicating the position and orientation of the user specified by using GPS or the like on the correction drawing.

That is, if the information providing device 10 can correct the drawing included in the captured image as seen from the front and superimpose the current position on the corrected drawing, the position of the user can be obtained from the captured image. It is not necessary to make an estimation or the like. That is, the information providing device 10 may generate and display an image as if it were map data by performing a projective transformation of a map area in a photograph taken.

Further, for example, the information providing device 10 includes a drawing in the captured image based on a comparison result between the feature amount included in the captured image and the feature amount of the reference information (that is, the feature amount of the previously captured drawing). If it is determined that it is included, the drawing included in the captured image or its correction drawing is displayed, and information indicating the position and orientation of the user positioned by various positioning systems is superimposed. May be displayed. Further, the information providing device 10 may superimpose and display only the information indicating the position of the user.

[1-8-4. Applicable target]
In the above-mentioned example, the information providing device 10 estimates the relative position and the relative orientation from the captured images of the objects installed in various facilities. However, the embodiments are not limited to this.

For example, the information providing device 10 may estimate the relative position and the relative orientation not only from a facility such as a shopping mall but also from a photographed image of an object installed in, for example, a multi-story parking lot or a one-story parking lot. good. Further, the information providing device 10 estimates the relative position and orientation of the in-vehicle device from the captured image taken by the in-vehicle device such as a navigation system or a drive recorder, and obtains information indicating the position and orientation of the in-vehicle device on the map. It may be displayed.

Further, the information providing device 10 may execute a process of correcting the position and orientation of the photographing device based on the photographed image, for example, during guidance using various navigation systems. That is, the information providing device 10 may only estimate the position and orientation of the user using the object, or may only correct the position and orientation.

[1-8-5. Information]
The information providing device 10 may provide guidance information indicating the position and orientation of the photographing device, and then guide the user using GPS or the like. At this time, the information providing device 10 may guide the user by using the photographed drawing or the correction drawing. Such processing uses, for example, information indicating the correspondence between the position on the pre-photographed drawing and the actual position, and the correspondence between the pre-photographed drawing and the drawing extracted from the photographed image. , Feasible.

For example, the information providing device 10 may display images of different floors when the user moves to the position of the stairs. For example, the information providing device 10 provides guidance using a drawing or a correction drawing extracted from a photographed image, and displays a drawing of a different floor prepared in advance when the user moves to the stairs. You may do it.

[1-8-6. About the object]
In the above example, the relative position and orientation of the user were estimated using drawings or landmarks. Here, the information providing device 10 estimates the relative position and the relative orientation using a three-dimensional object by managing the information of the images of one object taken from a plurality of directions as individual reference information. You may.

For example, the information providing device 10 uses an image of an object taken from the north side as a first front image, and stores information on feature points, an installation position, and a reference shooting position on the north side of the object in association with each other. Further, the information providing device 10 stores an image of an object taken from the south side as a second front image, and stores information on feature points in association with a reference shooting position on the south side of the object together with an installation position. When such information is registered, when the information providing device 10 receives a photographed image of an object taken from the north side, it is presumed that the photographing device is located on the north side of the object and faces the south side. However, when a photographed image of an object taken from the south side is received, it can be presumed that the photographing device is located on the south side of the object and faces the north side.

[1-8-7. others〕
The above-mentioned processing using various matrices is merely an example, and the information providing device 10 uses various techniques such as camera calibration in addition to the above-mentioned method to obtain the relative position of the photographing device from the photographing drawing and the relative position of the photographing device. Relative orientation estimation may be performed. Further, the information providing device 10 may execute the above-mentioned various processes from the captured images taken not only in the facility but also in an arbitrary space such as outdoors.

Further, the information providing device 10 not only captures the feature points in the range in which the object is captured, but also captures the feature points of the object captured as the background among the captured images captured in the preparation stage, the processing stage, and the correction stage. The processing used may be performed. That is, the information providing device 10 does not need to limit the range of the captured image from which the feature points are extracted to the range including the object, and extracts and compares the feature points including the background. , Relative position and relative orientation may be estimated based on these comparison results.

In this way, the information providing device 10 calculates the positional relationship between the photographing terminal and the space from the data acquired in advance and the captured image, and acquires the position of the photographing device. For example, the information providing device 10 holds the reference data in which the multidimensional vector data obtained by the local feature amount calculation and the position of the spatial reference point are linked from an arbitrary spatial image taken in advance. Further, the information providing device 10 holds the actual size and scale ratio of the map to be superimposed. Then, the information providing device 10 determines the position of the photographing device from the relative position and the relative orientation, the map installation coordinates in the captured map image, and the coordinates on the map image at the captured location in an arbitrary space photographed in advance. And the orientation may be estimated.

[2. Configuration of information providing device]
Hereinafter, an example of the functional configuration of the above-mentioned information providing device 10 will be described. FIG. 7 is a diagram showing a configuration example of the information providing device according to the embodiment. As shown in FIG. 7, the information providing device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 20 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the terminal device 100.

The storage unit 30 stores reference information based on a reference image, which is an image of an object taken from a predetermined position. For example, the storage unit 30 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. Further, the storage unit 30 stores the drawing database 31 and the landmark database 32. The information registered in the drawing database 31 and the landmark database 32 may be stored, for example, in a state of being integrated on the same database as the object database.

Hereinafter, an example of the information registered in the storage unit 30 will be described with reference to FIGS. 8 and 9. In the drawing database 31, various reference information such as information on feature points of drawings taken in advance and installation positions of drawings are registered. For example, FIG. 8 is a diagram showing an example of information registered in the drawing database according to the embodiment. As shown in FIG. 8, the drawing database 31 contains "area ID (Identifier)", "area information", "drawing ID", "installation position", "reference", "feature point coordinates", and "feature amount". Information such as "is registered in association with each other.

Here, the "area ID" is an identifier for identifying an area such as a facility in which a drawing is installed. Further, the "area information" is information such as the name of the area in which the drawing is installed. The "drawing ID" is an identifier for identifying the drawing. Further, the "installation position" is information indicating the installation position of the drawing. Further, the "reference" is information indicating the reference shooting position of the drawing. Further, the "feature point coordinates" are information indicating the coordinates of the feature points extracted from the image obtained by taking the drawing. Further, the "feature amount" is information indicating the feature amount corresponding to the feature point.

For example, in the example shown in FIG. 8, the drawing database 31 has the area ID “AID # 1”, the area information “area # 1”, the drawing ID “M # 1”, the installation position “position # M1”, and the reference “reference”. Information such as "# M2", feature point coordinates "MFC # 1-1", and feature amount "MC # 1-1" is registered. As for such information, the information of the area indicated by the area ID "AID # 1" is the "area # 1", and the drawing indicated by the drawing ID "M # 1" is located at the position indicated by the installation position "position # M1". , Refers to the effect that the reference "Reference # M2" is installed in the direction indicated. Further, such information indicates that the feature amount indicated by the feature amount "MC # 1-1" is extracted from the position indicated by the feature point coordinates "MFC # 1-1" in the image in which the drawing is taken in advance. show.

In the landmark database 32, various reference information such as information on the feature points of the landmarks taken in advance and the installation position of the landmarks are registered. For example, FIG. 9 is a diagram showing an example of information registered in the landmark database according to the embodiment. As shown in FIG. 9, the landmark database 32 includes "area ID", "area information", "equipment ID", "installation position", "reference", "feature point coordinates", and "feature amount". Information is associated and registered. The "equipment ID" is an identifier for identifying a landmark.

In the examples shown in FIGS. 8 and 9, conceptual values such as "region # 1", "position # M1", "reference position # M2", "MFC # 1-1", and "MC # 1-1" are used. However, in reality, the character string indicating the name of the area, the numerical value indicating the latitude and longitude that is the installation position and the reference shooting position, the coordinates on the drawing that is the installation position and the reference shooting position, and the coordinates of the feature points are described. Numerical values and multidimensional quantities (vectors) that are feature quantities are registered. Further, the drawing database 31 and the landmark database 32 can be integrated by integrating the drawing ID and the equipment ID as the "object ID".

Returning to FIG. 7, the explanation will be continued. The control unit 40 is a controller, and for example, various programs stored in a storage device inside the information providing device 10 by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) are RAM or the like. Is realized by executing as a work area. Further, the control unit 40 is a controller, and may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 7, the control unit 40 includes a reception unit 41, a correction unit 42, a registration unit 43, an acquisition unit 44, a determination unit 45, an estimation unit 46, a generation unit 47, and a provision unit 48. The reception unit 41 receives an image of an object registered in advance. For example, the reception unit 41 receives an image of the drawing taken from the front and information indicating the size and aspect ratio of the drawing. In addition, the reception unit 41 receives an image of the landmark. Further, the reception unit 41 accepts registration of information indicating the installation position of the object and the reference shooting position, and information necessary for executing the various processes described above (for example, the scale of the drawing).

The correction unit 42 corrects the image obtained by photographing the object to the image obtained by photographing the object from a predetermined position. For example, the correction unit 42 corrects an image of the object based on the information indicating the aspect ratio of the object. More specifically, the correction unit 42 identifies the range in which the drawing is taken from the image, and seems to have taken the drawing from the front based on the shape of the specified range and the information indicating the aspect ratio of the drawing. In addition, the image is corrected.

The registration unit 43 registers the feature amount of the image obtained by capturing the object. For example, the registration unit 43 extracts a plurality of feature points from an image corrected by the correction unit 42 or an image in which a landmark is taken (hereinafter, referred to as a “reference image”), and the extracted feature points. For each feature point coordinates and feature quantities are extracted. Then, the registration unit 43 registers the extracted feature point coordinates and feature quantities in the drawing database 31 or the landmark database 32 in association with the installation position where the object is installed and the reference shooting position.

The acquisition unit 44 acquires a photographed image in which an object previously installed in a predetermined area is photographed. For example, the acquisition unit 44 is a photographed image taken as an object, which is a drawing installed in a predetermined area such as a drawing of a facility located underground as a photographed image and schematically shows a predetermined area. To get. Further, when correcting the position and orientation of the photographing device, the acquisition unit 44 acquires a photographed image in which a new object is photographed after the guidance information is provided. For example, the acquisition unit 44 acquires a predetermined facility installed in a predetermined area, that is, a photographed image in which a landmark is photographed.

The determination unit 45 is a drawing in which the object captured in the captured image is a drawing installed in a predetermined area and schematically shows a predetermined area, or a predetermined object installed in the predetermined area. Judge whether it is the equipment of. For example, the determination unit 45 provides information on a captured image obtained by capturing a drawing or an image captured by a landmark (that is, an image used for correction of position and orientation) together with a captured image from the terminal device 100. When is accepted, it may be determined whether the drawing is photographed or the landmark is photographed in the photographed image based on the accepted information.

Further, for example, the determination unit 45 extracts the feature point coordinates and the feature amount from the captured image, and compares the extracted feature amount with the feature amount registered in each of the databases 31 and 32. Then, the determination unit 45 may determine that the object having the largest number of similar feature quantities among the objects is the object to be photographed. Further, in the determination unit 45, when there is no object whose number of similar feature quantities exceeds a predetermined threshold value or when the variance exceeds a predetermined threshold value, the photographed object is a drawing or a landmark. It may be determined that there is no such thing.

Based on the captured image, the estimation unit 46 estimates the relative position, which is the position of the photographing device that captured the captured image with respect to the object, and the relative orientation, which is the orientation of the photographing device with respect to the object. For example, the estimation unit 46 estimates the relative position and the relative orientation by using the information about the feature points extracted from the captured image and the information stored in the storage unit 30. That is, the estimation unit 46 estimates the relative position and the relative orientation based on the comparison result between the image information based on the captured image and the information of the reference image stored in the storage unit 30.

For example, when the determination unit 45 determines that the object to be photographed is a drawing, the estimation unit 46 executes the following processing. First, the estimation unit 46 extracts a plurality of feature points from the captured image, and extracts the feature point coordinates and the feature amount of the extracted feature points. Subsequently, the estimation unit 46 estimates the drawing captured in the captured image based on the comparison result between the extracted feature quantity and the feature quantity registered in the drawing database 31. The process of estimating the drawing captured in the captured image may be realized by using, for example, the result at the time of determination by the determination unit 45.

Subsequently, the estimation unit 46 compares the feature amount extracted from the reference image of the drawing estimated as the object to be photographed with the feature amount in the photographed image, and extracts similar feature points having similar feature amounts to each other. In addition, the estimation unit 46 is a feature amount extracted from the image in which the photographed drawing was photographed, and is a comparison target not only the feature amount extracted from the range in which the drawing was photographed but also the feature amount in which the background was photographed. May be. Then, the estimation unit 46 generates a homography matrix based on the positional relationship between the similar feature points in the reference image and the positional relationship between the similar feature points in the captured image. That is, the estimation unit 46 generates a transformation matrix that converts the positions of the feature points included in the captured image into the positions of the feature points having similar feature amounts among the feature points included in the reference image.

Then, the estimation unit 46 estimates the relative position and the relative orientation based on the generated homography matrix. For example, the estimation unit 46 generates from the homography matrix a first matrix that converts the installation position of the drawing into a relative position and a second matrix that converts the front surface of the drawing into a relative direction. Then, the estimation unit 46 estimates the relative position and the relative orientation from the generated first matrix and second matrix.

On the other hand, when the determination unit 45 determines that the imaging target is a landmark, the estimation unit 46 executes the following processing. First, the estimation unit 46 extracts a plurality of feature points from the captured image, and extracts the feature point coordinates and the feature amount of the extracted feature points. Subsequently, the estimation unit 46 estimates the landmark captured in the captured image based on the comparison result between the extracted feature quantity and the feature quantity registered in the landmark database 32. The process of estimating the landmarks captured in the captured image may be realized by using, for example, the result at the time of determination by the determination unit 45.

Subsequently, the estimation unit 46 compares the feature amount extracted from the reference image of the landmark estimated to be the object to be photographed with the feature amount in the photographed image, and extracts similar feature points having similar feature amounts to each other. .. Then, the estimation unit 46 generates a basic matrix based on the positional relationship between the similar feature points in the reference image and the similar feature points in the captured image, and based on the generated basic matrix, the landmark Restore the three-dimensional shape.

Subsequently, the estimation unit 46 identifies the intersection of the line of sight in the captured image and the three-dimensional shape of the landmark, for example, the center point of the landmark, and determines the relative position and the relative direction with respect to the center point of the identified landmark. presume. More specifically, the estimation unit 46 uses the center point of the landmark and the elementary matrix to be used, and the relative position of the shooting position reference in which the reference image is taken and the relative position of the shooting orientation reference in which the reference image is taken. Estimate the orientation. Then, the estimation unit 46 determines the shooting position and shooting direction when the reference image is shot, the relative position of the shooting position reference for shooting the reference image, and the relative orientation of the shooting orientation reference for shooting the reference image. Estimates the relative position and orientation of the landmark's center reference based on. The estimation unit 46 may adopt information that can be estimated or assumed for the shooting position and shooting direction when the reference image is shot, instead of the measured values.

That is, the estimation unit 46 restores the three-dimensional shape of the new object based on the captured image of the object, and estimates the relative position and the relative orientation using the restored three-dimensional shape. For example, the estimation unit 46 estimates the position and orientation of the photographing apparatus with respect to a predetermined position in the restored three-dimensional shape of the object as a relative position and a relative orientation.

The generation unit 47 generates guidance information indicating the position and orientation of the photographing apparatus within a predetermined region based on the relative position and the relative orientation. For example, the generation unit 47 estimates the position of the photographing apparatus in the predetermined area based on the position where the object is installed in the predetermined area and the relative position, and generates guidance information indicating the estimated position. do. Further, the generation unit 47 estimates the orientation of the imaging device in a predetermined region based on the reference imaging position and the relative orientation, and generates guidance information indicating the estimated position. Here, the position and orientation of the photographing apparatus within a predetermined area is a concept including, for example, the position and orientation on the drawing to be displayed, in addition to the information such as latitude, longitude and orientation.

For example, the generation unit 47 is the position of the photographing apparatus on the map to be displayed based on the relative position and the relative orientation estimated by the estimation unit 46, the installation position of the drawing to be photographed, and the reference photographing position. And specify the orientation. Then, the generation unit 47 generates guidance information in which information schematically indicating the position and orientation of the photographing apparatus is superimposed on the drawing.

The generation unit 47 may generate guidance information in which information schematically indicating the position and orientation of the photographing device is superimposed on the drawing included in the photographed image. For example, the generation unit 47 generates a correction drawing in which the drawing included in the captured image is corrected as if the drawing was captured from a predetermined direction.

To give a more specific example, the generation unit 47 is based on a comparison result between a feature amount extracted from a reference image, which is an image of a drawing taken from a predetermined position, and a feature amount extracted from the shot image. , A homography matrix may be generated, and the generated homography matrix may be used to generate a correction drawing. That is, the generation unit 47 may generate a correction drawing based on the comparison result between the information registered in the drawing database 31 and the information based on the captured image. The generation unit 47 may generate a correction drawing based on, for example, an estimated relative position and relative orientation.

Further, the generation unit 47 converts the position and orientation of the photographing apparatus into the coordinates on the correction drawing based on the estimated relative position and relative orientation. Then, the generation unit 47 may generate guidance information on which information schematically indicating the position and orientation of the photographing apparatus is superimposed on the correction drawing.

The generation unit 47 may generate guidance information by superimposing information schematically indicating the position and orientation of the photographing device on a drawing different from the drawing included in the captured image. For example, the generation unit 47 may generate guidance information by superimposing information schematically indicating the position and orientation of the photographing apparatus on a pre-registered drawing.

Further, for example, when the drawing included in the captured image is not a drawing schematically showing a predetermined area from above, the generation unit 47 is placed on a drawing schematically showing a predetermined area from above. Guidance information may be generated by superimposing information schematically indicating the position and orientation. Further, the generation unit 47 generates guidance information in which information schematically indicating the position and orientation of the photographing apparatus is superimposed on the drawing of the facility located on the ground, which is a facility located in the same area as the facility. You may. Further, the generation unit 47 may generate guidance information indicating the guidance direction to the destination preset by the user who uses the photographing device, as well as the position and orientation of the photographing device.

Further, the generation unit 47 may generate new guidance information in which the position and orientation of the photographing apparatus in a predetermined region are corrected based on the relative position and the relative orientation with respect to the landmark. For example, the generation unit 47 identifies the position and orientation of the photographing apparatus on the delivered drawing based on the relative position and the relative orientation with respect to the landmark, and generates information indicating the specified position and orientation as guidance information. You may.

When it is determined that the object is an object other than the drawing and the predetermined equipment, the generation unit 47 generates the drawing or suggestion information suggesting photography of the predetermined equipment. For example, the generation unit 47 may generate suggestive information including a message such as "Please take a picture of a drawing or a landmark."

The providing unit 48 provides the generated guidance information and suggestion information to the terminal device 100. As a result of such processing, the providing unit 48 can display the guidance information indicating the position and direction of the user on the terminal device 100.

[3. Configuration of terminal device]
Next, an example of the configuration of the terminal device 100 will be described with reference to FIG. FIG. 10 is a diagram showing a configuration example of the terminal device according to the embodiment. As shown in FIG. 10, the terminal device 100 includes a communication unit 120, a storage unit 130, a camera 140, a screen 150, a GPS antenna 160, a sensor 170, and a control unit 180.

The communication unit 120 is realized by, for example, a NIC or the like, is connected to the network N by wire or wirelessly, and transmits / receives information to / from the information providing device 10. The storage unit 130 is realized by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk, and data of various applications executed by the terminal device 100 and various data used in the application are registered. Will be done.

The camera 140 is a so-called photographing device, and is realized by a digital camera or the like using various semiconductor light receiving elements. The screen 150 is a screen capable of displaying various information, and is realized by, for example, various liquid crystal displays. When the screen 150 is realized by the touch panel, the screen 150 also operates as an input device for receiving an operation from the user. The GPS antenna 160 is an antenna for receiving GPS signals. Further, the sensor 170 is various sensor devices such as an acceleration sensor and a gyro sensor.

The control unit 180 is a controller, and is realized by, for example, using a processor such as a CPU or MPU to execute various programs stored in a storage device inside the terminal device 100 using RAM or the like as a work area. Further, the control unit 180 is a controller, and may be realized by an integrated circuit such as an ASIC or FPGA.

Here, the control unit 180 realizes various functions related to the terminal device 100 by executing a plurality of applications. For example, in the example shown in FIG. 10, the control unit 180 is executing the first application 181 and the second application 182. In addition to the applications shown in FIG. 10, the control unit 180 may execute an arbitrary number of applications for exerting an arbitrary function. For example, the control unit 180 may execute an application called an OS (Operating System) as an application for receiving an operation input to the terminal device 100 and displaying various information on the screen 150. Further, the control unit 180 may execute an application for providing various services.

Here, the control unit 180 executes the guidance application 183 as an application for displaying the guidance information generated by the above-mentioned generation process and updating the guidance information. For example, when the guidance application 183 is executed, the control unit 180 has functions such as a photographing unit 184, a display control unit 185, a position specifying unit 186, and an updating unit 187.

The photographing unit 184 uses the camera 140 to photograph a photographed image of an object such as a drawing or a landmark. For example, the photographing unit 184 displays the object being photographed by the camera 140 on the screen 150 and displays the photographing button when the application is started or the like. Then, when the user selects the shooting button, the shooting unit 184 causes the camera 140 to shoot an image, transmits the shot image as a shot image to the information providing device 10, and the object is a drawing. Send a notification to that effect.

When the user selects the update button displayed on the screen 150, for example, the photographing unit 184 performs the same processing as when the drawing is photographed. Then, when the user selects the shooting button, the shooting unit 184 causes the camera 140 to shoot an image, transmits the shot image as a shot image to the information providing device 10, and the object is a landmark. Send a notification to that effect.

The display control unit 185 performs various display controls. For example, the display control unit 185 causes the screen 150 to display a display of a shooting button, an image of an object shot by the camera 140, and the like. When the display control unit 185 receives the guidance information from the information providing device 10, the display control unit 185 displays the received guidance information on the screen 150, and responds to the position and orientation notified from the position specifying unit 186 and the updating unit 187. Then, the information indicating the position and orientation of the photographing device (that is, the terminal device 100) is updated. That is, the display control unit 185 updates the position and orientation of the icon superimposed on the drawing.

The position specifying unit 186 specifies the position and orientation of the terminal device 100. For example, the position specifying unit 186 estimates the direction and distance traveled by the terminal device 100, the direction of the terminal device 100, and the like according to the acceleration and direction detected by the sensor 170, and displays the estimated information in the display control unit 185. Notify to. In such a case, the display control unit 185 updates the position and orientation of the icon superimposed on the drawing based on the received information. For such processing, for example, various known techniques for updating the position and orientation of the terminal device 100 on the map based on the positioning signal and the sensing results of the various sensors 170 can be adopted.

The update unit 187 acquires the position and orientation of the terminal device 100 updated by the update process. For example, the update unit 187 acquires the position and orientation of the terminal device 100 estimated by the information providing device 10 based on the relative position and the relative orientation with respect to the landmark. In such a case, the updating unit 187 notifies the display control unit 185 of the acquired position and orientation, and updates the position and orientation of the icon superimposed on the drawing.

Here, an example of the content to be displayed on the screen 150 of the terminal device 100 by the guidance application 183 will be described with reference to FIG. FIG. 11 is a diagram showing an example of a screen displayed by the terminal device according to the embodiment. For example, the terminal device 100 displays a shooting button SB together with an image shot by the camera 140 (for example, an image of a drawing or a signboard on which a drawing is arranged) taken by the camera 140 at the shooting stage of shooting a drawing. do. Subsequently, when the shooting button SB is tapped, the terminal device 100 transmits the image shot by the camera 140 to the information providing device 10 as a shooting image, and the position of the terminal device 100 on the shot drawing. And acquire guidance information indicating the direction.

Then, the terminal device 100 indicates the position and orientation of the terminal device 100 to the user by displaying the received guidance information at the guidance stage for guiding the user. Further, the terminal device 100 sequentially indicates the position and orientation of the terminal device 100 to the user by updating the position and orientation of the icon superimposed on the drawing based on the positioning signal and various sensing results.

Here, in the guidance stage, the terminal device 100 displays the update button UB for updating the position and orientation of the terminal device 100 together with the guidance information. Then, when the user selects the update button UB, the terminal device 100 shifts to the update stage for updating the position and orientation of the terminal device 100. More specifically, the terminal device 100 displays the shooting button SB together with the image shot by the camera 140 (for example, the image of the door that becomes a landmark) as in the shooting stage. Then, when the user selects the shooting button SB, the terminal device 100 transmits the shot image to the information providing device 10 and acquires the updated position and orientation information of the terminal device 100. After that, the terminal device 100 shifts to the guidance stage, and updates the position and orientation of the icon superimposed on the drawing according to the updated position and orientation information of the terminal device 100.

[4. An example of the flow of processing executed by the information providing device]
Subsequently, the flow of processing executed by the information providing device 10 will be described with reference to FIG. FIG. 12 is a flowchart showing an example of a flow of processing executed by the information providing device according to the embodiment.

First, the information providing device 10 determines whether or not the captured image has been acquired from the terminal device 100 (step S101), and if not (step S101: No), waits until the captured image is acquired. Then, when the information providing device 10 acquires the captured image (step S101: Yes), the information providing device 10 determines whether or not the captured image includes a drawing (step S102).

Subsequently, when the captured image includes a drawing (step S102: Yes), the information providing device 10 of the information providing device 10 of the photographing device with respect to the drawing based on the comparison result between the feature amount of the captured image and the feature amount of the basic information. The relative position and the relative orientation are estimated (step S103). Further, the information providing device 10 corrects the drawing included in the captured image to the drawing viewed from the front (step S104). Then, the information providing device 10 generates guidance information in which information indicating the position and orientation of the photographing device is superimposed on the correction drawing (step S105). The information providing device 10 may execute steps S103 and S104 in any order. Then, the information providing device 10 provides the generated guidance information to the terminal device 100 (step S106), and ends the process.

On the other hand, when the captured image does not include the drawing (step S102: No), the information providing device 10 is a photographing device for the landmark based on the comparison result between the feature amount of the captured image and the feature amount of the basic information. The relative position and the relative orientation of the above are estimated (step S107). Then, the information providing device 10 generates new guidance information corrected for the position and orientation of the photographing device on the map (step S108), and executes step S106.

[5. Modification example]
In the above, an example of the processing by the information providing apparatus 10 has been described. However, the embodiments are not limited to this. Hereinafter, variations of the processing executed by the information providing device 10 will be described.

[5-1. Execution subject]
In the above-mentioned example, the information providing device 10 executed the generation process using the captured image acquired from the terminal device 100. However, the embodiments are not limited to this. For example, the above-mentioned generation process may be executed standalone by the terminal device 100. For example, the terminal device 100 has the drawing database 31 and the landmark database 32 shown in FIG. 7 in the storage unit 130 shown in FIG. 10, and is shown in FIG. 7 by executing the guidance application 183 shown in FIG. The same functions as those of the acquisition unit 44, the determination unit 45, the estimation unit 46, and the generation unit 47 may be exhibited.

[5-2. others〕
Further, the information providing device 10 may use information indicating the orientation of the object (for example, the orientation of the drawing perpendicular to the drawing, the orientation of the landmark, etc.) as reference information instead of the reference shooting position. good. Further, if the information providing device 10 photographs the object from the front of the object in the preparation stage, the information providing device 10 estimates the direction of the object from the reference photographing position and the installation position of the object, and from the estimated direction, The orientation and position of the imaging device may be estimated.

Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, and conversely, the processes described as being manually performed. It is also possible to automatically perform all or part of the above by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in any unit according to various loads and usage conditions. Can be integrated and configured.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

[5-3. program〕
Further, the information providing device 10 according to the above-described embodiment is realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 13 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to the output device 1010 and the input device 1020, and the arithmetic unit 1030, the primary storage device 1040, the secondary storage device 1050, the output IF (Interface) 1060, the input IF 1070, and the network IF 1080 are connected by the bus 1090. Have.

The arithmetic unit 1030 operates based on a program stored in the primary storage device 1040 or the secondary storage device 1050, a program read from the input device 1020, or the like, and executes various processes. The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various operations such as RAM. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, such as a ROM (Read Only Memory), an HDD (Hard Disk Drive), and a flash memory. Is realized by.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), and the like. It is realized by a connector of a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, a scanner, and the like, and is realized by, for example, USB.

The input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Further, the input device 1020 may be an external storage medium such as a USB memory.

The network IF 1080 receives data from another device via the network N and sends it to the arithmetic unit 1030, and also transmits the data generated by the arithmetic unit 1030 to the other device via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040.

[6. effect〕
As described above, the information providing device 10 acquires a photographed image in which an object (for example, various drawings such as a map or a landmark such as equipment) installed in advance in a predetermined area is photographed. In addition to the drawings and landmarks, the information providing device 10 may acquire a photographed image including a background such as a floor, a ceiling, and trees arranged far away from the object. In such a case, the information providing device 10 estimates, based on the captured image, the relative position of the captured image with respect to the object of the photographing device and the relative orientation of the photographing device with respect to the object. .. For example, the information providing device 10 estimates a relative position and a relative orientation by using various camera calibration methods. Then, the information providing device 10 generates guidance information indicating the position and orientation of the photographing device in a predetermined region based on the relative position and the relative orientation.

For example, the information providing device 10 estimates the position of the photographing device in the predetermined area based on the position where the object is installed in the predetermined area and the relative position, and provides guidance information indicating the estimated position. Generate. Further, the information providing device 10 estimates the orientation of the photographing device in the predetermined area based on the position of the photographing device in which the object is photographed in advance in the predetermined area and the relative orientation, and indicates the estimated position. Generate guidance information.

As a result of such processing, the information providing device 10 estimates the position and orientation of the photographing device such as the terminal device 100, and by extension, the user who uses the photographing device only by photographing the object, and the estimated position and orientation. Can be provided. Therefore, the information providing device 10 can estimate not only the current position of the user but also the direction of the current user with high accuracy and indicate the estimated position and direction, so that the user is appropriately guided. be able to.

Further, in general, even if the user does not have the habit of launching an application for providing guidance in a subway platform or a shopping mall, the information providing device 10 uses the terminal device 100 to photograph an object such as a drawing. Since the position and orientation of the user can be indicated only by doing so, the time and effort of the user can be reduced. Further, the information providing device 10 can suggest the direction to the destination to the user by showing the user the current direction. Therefore, for example, an act of moving the home while continuing to look at the terminal device 100. Therefore, the safety of the user can be ensured.

Further, the information providing device 10 stores reference information based on a reference image which is an image of an object captured from a predetermined position, and a comparison result between the image information based on the captured image and the reference information stored in the storage unit. Estimates the relative position and relative orientation based on. Therefore, the information providing device 10 can estimate the relative position and relative orientation of the user with high accuracy, and improve the accuracy of the position and orientation of the photographing device estimated from these relative positions and relative orientations. be able to.

Further, the information providing device 10 corrects the image obtained by photographing the object to the image obtained by photographing the object from a predetermined position, and stores the information based on the image corrected by the correction unit as the reference information. For example, the information providing device 10 corrects an image of an object based on information indicating an aspect ratio of the object. Therefore, the information providing device 10 can further improve the estimation accuracy of the relative position and the relative orientation.

Further, the information providing device 10 stores information about the feature points included in the reference image as the reference information, and uses the information about the feature points extracted from the captured image and the information stored in the storage unit to be relative to each other. Estimate the position and relative orientation. For example, the information providing device 10 stores information indicating the position of a feature point included in the reference image and a feature amount indicating the feature at each feature point, and sets the position of the feature point included in the captured image as the reference image. A transformation matrix that transforms the included feature points into the positions of feature points with similar feature quantities is generated, and the relative position and relative orientation are estimated based on the generated transformation matrix. For example, the information providing device 10 generates, from the transformation matrix, a first matrix that converts a predetermined position into a relative position and a second matrix that converts the direction from the predetermined position to the object in the relative direction. The relative position and relative orientation are estimated from the first matrix and the second matrix.

In this way, the information providing device 10 estimates the relative position and the relative orientation of the photographing device by using the technique of camera calibration using the feature amount extracted from the image. Therefore, the information providing device 10 can appropriately estimate the relative position and the relative orientation.

Further, the information providing device 10 acquires an image taken as an object of a drawing installed in a predetermined area as a photographed image and schematically shows a predetermined area, and the drawing of the photographing device. Estimate the relative position and relative orientation. Therefore, the information providing device 10 can indicate the position and orientation of the user in the facility only by taking, for example, a premises map or a map of the facility installed in various facilities.

Further, the information providing device 10 acquires a photographed image in which a new object is photographed after the guidance information is provided, estimates the relative position and the relative orientation of the photographing device with respect to the new object, and newly obtains a new object. Based on the relative position and relative orientation with respect to the object, new guidance information is generated in which the position and orientation of the photographing apparatus in a predetermined area are corrected. For example, the information providing device 10 acquires a photographed image of a predetermined facility such as a door installed in a predetermined area as a new object. Therefore, the information providing device 10 can realize the correction of the position and orientation of the user using the new object.

Further, the information providing device 10 restores the three-dimensional shape of the new object based on the captured image of the new object, and uses the restored three-dimensional shape to determine the relative position and the relative orientation. presume. Then, the information providing device 10 estimates the position and orientation of the photographing device with respect to a predetermined position in the three-dimensional shape of the restored new object as the relative position and the relative orientation. Therefore, for example, the information providing device 10 can increase the number of points where the position and orientation of the user can be corrected only by collecting captured images of the object used for correction in the facility from a predetermined position. ..

Further, the information providing device 10 is a drawing in which the object captured in the captured image is a drawing installed in a predetermined area and schematically shows the predetermined area, or is installed in the predetermined area. It is determined whether the equipment is a predetermined facility, and if it is determined that the object is a drawing, guidance information indicating the position and orientation of the photographing apparatus in the predetermined area is generated, and the object is a predetermined object. If it is determined that the equipment is installed, new guidance information may be generated in which the position and orientation of the photographing apparatus in a predetermined area are corrected. As a result of such processing, the information providing device 10 can automatically execute appropriate processing, such as new generation of guidance information or update of guidance information, depending on the object to be photographed.

Further, the information providing device 10 further determines whether the object is an object other than the drawing and the predetermined equipment, and if it is determined that the object is an object other than the drawing and the predetermined equipment, the drawing or the object is determined. Generates suggestive information suggesting photography of a given facility. Therefore, the information providing device 10 can urge the user to shoot an object when the captured image does not properly capture the object such as a drawing or a landmark.

Further, the information providing device 10 may generate guidance information in which information schematically indicating the position and orientation of the photographing device is superimposed on a drawing schematically showing a predetermined area. Further, the information providing device 10 acquires, as a captured image, an image in which a drawing schematically showing a predetermined area is captured as an object, and sets the position and orientation of the photographing device on the drawing included in the captured image. Guidance information may be generated by superimposing information schematically shown. Further, the information providing device 10 acquires, as a captured image, an image in which a drawing schematically showing a predetermined area is captured as an object, and positions the imaging device on a drawing different from the drawing included in the captured image. Guidance information may be generated by superimposing information schematically indicating the direction and the direction.

For example, when the drawing included in the captured image is not a drawing schematically showing a predetermined area from above, the information providing device 10 positions the photographing device on a drawing schematically showing a predetermined area from above. Guidance information may be generated by superimposing information schematically indicating the direction and the direction. Further, the information providing device 10 acquires an image in which a drawing of a facility located underground is taken as an object, and is a facility located in the same area as the facility and is a drawing of a facility located on the ground. Guidance information may be generated on which information schematically indicating the position and orientation of the photographing apparatus is superimposed.

In this way, the information providing device 10 superimposes and provides information indicating the position and orientation of the user on various drawings that can be presumed to be appropriate for guidance according to the situation of the user. Therefore, it is possible to appropriately guide the user.

Further, the information providing device 10 may generate guidance information indicating the guidance direction to the destination preset by the user who uses the photographing device, as well as the position and orientation of the photographing device. As a result of such processing, the information providing device 10 can appropriately guide the user to the destination.

Further, the information providing device 10 acquires a photographed image of a drawing installed in a predetermined area and in which a drawing schematically showing the area is photographed, and photographs the photographed image in the drawing included in the photographed image. Guidance information is generated by superimposing information indicating the position of the image pickup device in the area. As a result of such processing, the information providing device 10 can easily superimpose and provide information indicating the user's position on the drawing viewed and photographed by the user, so that the user's current position can be easily provided. Can be recognized. Further, when the information indicating the position of the user is superimposed on such a drawing, the user can specify his / her own direction more easily than a map or a drawing distributed at an arbitrary timing. Therefore, the information providing device 10 can appropriately guide the user.

Further, as described above, the information providing device 10 generates guidance information in which information indicating a position and an orientation in the area of the photographing device is superimposed. Further, the information providing device 10 generates a correction drawing in which the drawing included in the photographed image is corrected as if the drawing was photographed from a predetermined direction, and the guidance information superimposing the information indicating the position of the photographing device on the generated correction drawing. To generate. Further, the information providing device 10 stores reference information based on a reference image which is an image of a drawing taken from a predetermined position, and includes image information based on the drawing included in the shot image and reference information stored in a storage unit. A correction drawing is generated based on the comparison result of.

Further, the information providing device 10 corrects the image obtained by photographing the object to the image obtained by photographing the object from a predetermined position, and stores the information based on the image corrected by the correction unit as the reference information. Further, the information providing device 10 corrects an image of the object based on the information indicating the aspect ratio of the object. Further, the information providing device 10 has a relative position which is a position with respect to the drawing of the photographing apparatus which captured the photographed image and a relative orientation which is an orientation with respect to the drawing of the photographing apparatus based on the comparison result between the image information and the reference information. Is estimated, and a correction drawing is generated based on the relative position and the relative orientation. Further, the information providing device 10 converts the position and orientation of the photographing device into coordinates on the drawing based on the relative position and the relative orientation, and generates a guide drawing using the converted coordinates.

As a result of each of these processes, the information providing device 10 can provide a drawing to be photographed in an appropriate manner and indicate the position and orientation of the user on such a drawing. Can guide you appropriately.

Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other modified forms.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the imparting unit can be read as a specific means or a specific circuit.

10 Information provider 20, 120 Communication unit 30, 130 Storage unit 31 Drawing database 32 Landmark database 40, 180 Control unit 41 Reception unit 42 Correction unit 43 Registration unit 44 Acquisition unit 45 Judgment unit 46 Estimating unit 47 Generation unit 48 Providing unit 100 Terminal device 140 Camera 150 Screen 160 GPS antenna 170 Sensor 183 Guidance application 184 Imaging unit 185 Display control unit 186 Positioning unit 187 Update unit

Claims (24)

An acquisition unit that acquires a captured image of an object installed in advance in a predetermined area, and an acquisition unit.
Based on the captured image, an estimation unit that estimates the relative position of the imaging device that captured the captured image with respect to the object and the relative orientation of the imaging device with respect to the object . When,
A generation unit that generates guidance information in which information indicating the relative position and information indicating the relative orientation are superimposed on a drawing schematically showing the inside of the predetermined region based on the relative position and the relative orientation. A generator characterized by having and. The generation unit estimates the position of the photographing apparatus in the predetermined area based on the position where the object is installed in the predetermined area and the relative position, and guides the estimated position. The generator according to claim 1, wherein the generator is characterized by generating information. The generation unit estimates the orientation of the photographing apparatus in the predetermined region based on the position of the photographing apparatus in which the object is photographed in advance and the relative orientation, and generates guidance information indicating the estimated position. The generator according to claim 1 or 2, wherein the generator is characterized by the above. It has a storage unit that stores reference information based on a reference image, which is an image of the object taken from a predetermined position.
Claim 1 is characterized in that the estimation unit estimates the relative position and the relative orientation based on a comparison result between the image information based on the captured image and the reference information stored in the storage unit. The generator according to any one of 3 to 3. It has a correction unit that corrects an image obtained by photographing the object to an image obtained by photographing the object from the predetermined position.
The generator according to claim 4, wherein the storage unit stores information based on an image corrected by the correction unit as the reference information. The generation device according to claim 5, wherein the correction unit corrects an image obtained by photographing the object based on information indicating an aspect ratio of the object. The storage unit stores information about feature points included in the reference image as the reference information, and stores the information.
4. The estimation unit is characterized in that it estimates the relative position and the relative orientation by using the information about the feature points extracted from the captured image and the information stored in the storage unit. The generator according to any one of 6 to 6. The storage unit stores information indicating the position of the feature point included in the reference image and a feature amount indicating the feature at each feature point.
The estimation unit generates a transformation matrix that converts the positions of the feature points included in the captured image into the positions of the feature points having similar feature amounts among the feature points included in the reference image, and the generated transformation matrix. The generator according to claim 7, wherein the relative position and the relative orientation are estimated based on the above. From the conversion matrix, the estimation unit generates a first matrix that converts the predetermined position to the relative position and a second matrix that converts the direction from the predetermined position to the object to the relative direction. The generator according to claim 8, wherein the relative position and the relative orientation are estimated from the first matrix and the second matrix. The acquisition unit acquires, as the photographed image, an image of a drawing installed in the predetermined area, which is a drawing schematically showing the predetermined area as the object.
The generator according to any one of claims 1 to 9, wherein the estimation unit estimates a relative position and a relative orientation of the photographing apparatus with respect to the drawing. After the guidance information is provided, the acquisition unit acquires a photographed image in which a new object is photographed.
The estimation unit estimates the relative position and the relative orientation of the photographing apparatus with respect to the new object.
The generation unit is characterized in that it generates new guidance information that corrects the position and orientation of the photographing apparatus in the predetermined region based on the relative position and the relative orientation with respect to the new object. The generator according to any one of claims 1 to 10. The generation device according to claim 11, wherein the acquisition unit acquires a photographed image taken by a predetermined facility installed in the predetermined area as the new object. The estimation unit restores the three-dimensional shape of the new object based on the captured image of the new object, and uses the restored three-dimensional shape to determine the relative position and the relative orientation. The generator according to claim 11 or 12, characterized in that The claim is characterized in that the estimation unit estimates the position and orientation of the photographing apparatus with respect to a predetermined position in the restored three-dimensional shape of the new object as the relative position and the relative orientation. Item 13. The generator according to item 13. The object captured in the captured image is a drawing installed in the predetermined area, which is a drawing schematically showing the predetermined area, or a predetermined area installed in the predetermined area. It has a judgment unit that determines whether it is equipment, and it has a judgment unit.
When it is determined that the object is the drawing, the generation unit generates guidance information indicating the position and orientation of the photographing apparatus in the predetermined region, and the object is the predetermined equipment. If it is determined to be, one of claims 1 to 14, characterized in that new guidance information is generated by correcting the position and orientation of the photographing apparatus in the predetermined area. The generator described. The determination unit further determines whether the object is an object other than the drawings and the predetermined equipment.
The generation unit is characterized in that, when it is determined that the object is an object other than the drawing and the predetermined equipment, the generation unit generates suggestive information suggesting that the drawing or the predetermined equipment is photographed. The generator according to claim 15. Claims 1 to 16 are characterized in that the generation unit generates guidance information in which information schematically indicating the position and orientation of the photographing apparatus is superimposed on a drawing schematically showing the predetermined region. The generator according to any one of the above. The acquisition unit acquires, as the object, an image in which a drawing schematically showing the predetermined region is captured as a captured image.
The generator according to claim 17, wherein the generation unit generates guidance information in which information schematically indicating the position and orientation of the imaging device is superimposed on a drawing included in the captured image. .. The acquisition unit acquires, as the object, an image in which a drawing schematically showing the predetermined region is captured as a captured image.
17. The generation unit is characterized in that it generates guidance information in which information schematically indicating the position and orientation of the photographing apparatus is superimposed on a drawing different from the drawing included in the photographed image. The generator described. When the drawing included in the photographed image is not a drawing schematically showing the predetermined area from above, the generation unit is placed on a drawing schematically showing the predetermined area from above. The generation device according to claim 19, wherein guidance information is generated by superimposing information schematically indicating a position and an orientation. The acquisition unit acquires, as the object, an image in which a drawing of a facility located underground is taken as a photographed image.
The generation unit is a facility located in the same area as the facility, and generates guidance information in which information schematically indicating the position and orientation of the photographing apparatus is superimposed on the drawing of the facility located on the ground. 19. The generator according to claim 19 or 20. The generation unit is characterized in that, together with the position and orientation of the photographing apparatus, the generation unit generates guidance information indicating a guiding direction to a destination preset by a user using the photographing apparatus. The generator according to any one of them. It is a generation method executed by the generation device.
The acquisition process of acquiring a photographed image of an object pre-installed in a predetermined area, and
Based on the captured image, an estimation step of estimating the relative position of the imaging device that captured the captured image with respect to the object and the relative orientation of the imaging device with respect to the object . When,
A generation step of generating guidance information in which information indicating the relative position and information indicating the relative orientation are superimposed on a drawing schematically showing the inside of the predetermined region based on the relative position and the relative orientation. A generation method characterized by including and. An acquisition procedure for acquiring a photographed image of an object pre-installed in a predetermined area, and
Based on the captured image, an estimation procedure for estimating the relative position of the photographing device that captured the captured image with respect to the object and the relative orientation of the photographing device with respect to the object . When,
A generation procedure for generating guidance information in which information indicating the relative position and information indicating the relative orientation are superimposed on a drawing schematically showing the inside of the predetermined region based on the relative position and the relative orientation. A generator to make a computer execute and.

Images