JP5843288B2 - Information presentation system - Google Patents

Description

  The present invention relates to an information presentation system that presents information, and more particularly, to an information presentation system that can control display information on a display unit by changing a relative positional relationship between an imaging unit and an imaging target.

  An apparatus that presents information according to a relative positional relationship with an imaging target can intuitively change the information to be presented, and can improve user convenience.

  Examples of conventional techniques for realizing the above are disclosed in Patent Documents 1 and 2, and the following methods are disclosed here.

  In Patent Document 1, operation data is acquired from an input device including a gyro sensor, an acceleration sensor, and an imaging unit, and after the attitude of the input device is calculated based on an angular velocity detected by the gyro sensor, the acceleration sensor detects There has been proposed a method for correcting the attitude of an input device based on acceleration data to be captured and an image of a predetermined imaging target captured by an imaging unit.

  In Patent Document 2, an image is acquired from an input device including an imaging unit, and feature information that is invariant to conversion such as enlargement / reduction or rotation is calculated from the periphery of a corner such as an intersection of straight lines, and then based on a combination of feature information. A method for estimating the attitude of the input device has been proposed.

JP 2010-5332 A JP 2010-517129 A

  In the posture calculation device of Patent Document 1, first, the current posture is calculated by integrating the angular velocities sequentially output from the gyro sensor and calculating the amount of change in posture from the initial state from the integration result. At this time, in order to solve the problem that the error of the gyro sensor is accumulated, the correction is performed on the assumption that the direction indicated by the acceleration sensor is the gravitational acceleration direction.

  However, since it is not assumed that acceleration occurs in a direction other than the gravitational acceleration direction while the apparatus is moving depending on the movement, and an error in the posture calculated from the acceleration sensor is not assumed, the posture is not accurate in Patent Document 1. May not be able to be calculated. Moreover, since some acceleration sensors cannot measure gravitational acceleration, the available acceleration sensors are limited. On the other hand, in the embodiment in which the imaging means is used for correction, it is necessary to separately install a device that irradiates infrared rays. Therefore, there is a problem that places where the device of Patent Document 1 can be used are limited.

  In the apparatus of Patent Document 2, the relative posture between the planar image and the imaging unit is calculated by evaluating a combination of feature amounts extracted from the planar image. If a common plane image is used as an image to be acquired, the trouble of installing in advance can be omitted, so that the problem of Patent Document 1 can be partially solved.

  However, when it is desired to correspond to a plurality of different planar images, the number of evaluations related to the correspondence relationship of the feature information increases according to the number of recognition objects, which causes a problem that processing time is required. In particular, when processing with a weak terminal, it is difficult to maintain a practical speed even when the number of recognition targets is small.

  An object of the present invention is to solve the above-described problems, recognize even which recognition target is the recognition target at a high speed even when the number of recognition targets is large, and determine the position and orientation of the terminal using the imaging unit. An object of the present invention is to provide an information presentation system capable of accurately calculating and controlling information displayed on a display unit.

  In order to achieve the above object, the present invention is an information presentation system that includes a terminal including an imaging unit that images an imaging target and a server that communicates with the terminal, and presents additional information related to the imaging target at the terminal. The terminal includes a positioning unit that acquires positioning information including the position of the terminal, and a calculation that calculates a predetermined feature amount that is invariant to rotation and enlargement / reduction and its feature point coordinates from the captured image. A terminal-side transmitting unit that transmits the acquired positioning information and the calculated feature amount and feature point coordinates to the server, a position and orientation of the terminal recognized by the server, and additional information , A terminal-side receiving unit that receives from the server, an updating unit that updates the recognized position and orientation, and the updated position and appearance of additional information received from the server. And a control unit that controls the additional information that is controlled and an output unit that superimposes and displays the controlled additional information on the imaging target in a predetermined arrangement. The server side receiving unit that receives the positioning information and the feature amount and the feature point coordinates transmitted from the device, the feature amount and the feature point coordinates when the image is captured in a predetermined arrangement for each of a plurality of predetermined imaging targets, and the positioning information And a storage unit that holds the additional information in association with each other, and by searching the storage unit, an imaging target having positioning information within which a distance from the acquired positioning information is within a predetermined range As an imaging target candidate being imaged, an extraction unit that extracts the feature amount and feature point coordinates together with the feature amount and feature point coordinates, and the received feature amount and feature point coordinates in each of the extracted candidates and The recognition unit recognizes which candidate corresponds to the imaging target imaged by the imaging unit and the position and orientation of the imaging target with respect to the terminal by comparing with the point coordinates. And a server-side transmission unit that transmits the additional information held in the storage unit with respect to the imaging target recognized to be applicable to the terminal.

  According to the present invention, the position and orientation are recognized and updated only by a user having a terminal changing the relative position and orientation of the imaging unit and the imaging target, and the position and orientation are added to the imaging target in the captured image. It is possible to display additional information controlled based on the above. Therefore, the user can control the display information in the output unit by an intuitive operation.

  Further, when controlling the display information, the server recognizes the imaging target, not the terminal, based on the feature amount and the feature point coordinates obtained by analyzing the image input to the imaging unit with the terminal. The number of imaging objects can be increased without being restricted by performance, and the recognition accuracy can be maintained high.

  Furthermore, since the extraction unit performs the search using the positioning information in advance, the candidate imaging targets can be limited, and the recognition speed and recognition accuracy in the recognition unit can be improved at the same time. In addition, by updating the posture information in the terminal, the processing speed can be increased without delay, and thus the display information can be reliably controlled.

It is a figure for demonstrating the structure outline | summary of an information presentation system. It is a functional block diagram of the terminal and server with which an information presentation system is provided. It is a figure which shows the example which controls display information with the presumed attitude | position of an imaging target. It is a figure which shows the example which controls display information by the estimated position of an imaging target. An example of information held in the storage unit is shown in a table format. It is a figure for demonstrating the feature information notionally. It is a figure which illustrates notionally using positioning information when searching the candidate of an imaging target. It is a figure for demonstrating the display information in an output part. It is a figure explaining the various aspects of the timing of the exchange of information with the terminal relevant to an update process, and a server. It is a functional block diagram of an update part.

  FIG. 1 is a diagram for explaining a configuration outline of an information presentation system according to an embodiment of the present invention. The information presentation system 3 includes a plurality of terminals 1-1, ..., 1-N and a server 2. Hereinafter, an arbitrary one representing the plurality of terminals 1-1,..., 1-N is referred to as a terminal 1. The terminal 1 and the server 2 communicate with each other via a network N such as the Internet.

  FIG. 2 is a functional block diagram according to an embodiment of the terminal 1 and the server 2 provided in the information presentation system 3. The terminal 1 includes an imaging unit 11, a calculation unit 12, a positioning unit 13, a control unit 14, an update unit 15, an output unit 16, a (terminal side) transmitting unit 17, and a (terminal side) receiving unit 18. The server 2 includes a recognition unit 21, a storage unit 22, an extraction unit 23, a (server side) transmission unit 24, and a (server side) reception unit 25.

  For example, the terminal 1 having such a configuration can be realized by a portable terminal, but may be another computer. In addition, the configuration of the server 2 is not limited to a physically single server by such a configuration, and is separated into a plurality of servers according to the functions of the respective units of the recognition unit 21 to the reception unit 25 (may be a plurality of units). May be.

  Note that the lines connecting the parts in FIG. 2 indicate that there is an exchange between the parts, but the lines show only the main exchanges, and lines are drawn to explain the details as appropriate. There may be interactions even if not.

  3 and 4 are diagrams illustrating examples of display information controlled in the output unit 16 realized by the present invention. FIG. 3 illustrates an example of control based on posture, and FIG. 4 illustrates an example of control based on position. . Note that the control is performed simultaneously by both the posture and the position, but for the sake of explanation, both will be described separately. In addition, precisely, the control unit 14 controls the additional information according to the position and orientation in conjunction with the input image from the imaging unit 11, and then superimposes the display information on the input image in the output unit 16. This will be described later with reference to FIG. 8 and the like, and will be described here as control of display information.

  In the present invention, for example, the terminal 1 captures a substantially flat and stationary imaging target such as a signboard, and the captured image is analyzed in cooperation with the server 2 to recognize what the imaging target is. In addition, the posture and position of the imaging target with respect to the terminal 1 that is changing, for example, when the user moves the terminal 1 by hand, and the display information displayed on the terminal 1 is controlled based on the estimated posture and position. To do.

  Specifically, the estimation of the position and orientation links the image captured when the recognized imaging target is in a predetermined arrangement with respect to the terminal 1 and the actually captured image in a plane projective transformation relationship. Thus, the position is estimated from the translation component in the plane projective transformation in relation to the position when the plane is in the predetermined arrangement, and the posture is determined from the rotation component in the plane projective transformation and the relation to the attitude in the predetermined arrangement Estimated by The display information can be controlled by using the planar projective transformation relationship directly or by applying a predetermined relationship such as a constant multiple for each translational component and rotational component.

  For example, in FIG. 3, as shown in FIG. 3A, a signboard is imaged by the terminal 1 as an example of an imaging target. Here, it is assumed that a flat pattern as an example of display information is in a state as indicated by P10 in (B) when the terminal 1 images a signboard from the front in a predetermined arrangement indicated by A10.

  P11, P12, P13, and P14 shown in (B) are examples in which display information is controlled when the terminal 1 is rotated in the directions A11, A12, A13, and A14 shown in the respective axes from the arrangement A10. Here, the display information is controlled in conjunction with how the signboard to be imaged looks on the image.

  That is, according to the rotation A11, the right side of the sign can be seen approaching, so that the display information is similarly controlled by rotating from the state of P10 and rotating in the direction of P11. The rotation A12 is the reverse of the rotation A11, and the display information is rotated from the state of P10 and rotated as indicated by P12.

  In the rotation A13, since the signboard can be seen from the upper side, the display information similarly changes from the state of P10 and is rotated and controlled in the direction of P13. The rotation A14 is the reverse of the rotation A13, and the display information changes from the state of P10 and rotates as indicated by P14.

  Also, for example, in FIG. 4, as in FIG. 3, an example of display information when a signboard as an example of an imaging target is imaged on the terminal 1 from the front as the predetermined arrangement A20 as shown in (A) is (B ) Of the flat pattern shown on P20. Further, a virtual plane on the terminal 1 in a plane parallel to the signboard in the predetermined arrangement A20 is indicated by the xy axis.

  When the terminal 1 moves in the direction indicated by A21 on the xy plane, the imaged signboard appears to move to the lower left side, so that the display information similarly moves from the state of P20 to P21. As shown in the figure, the movement is controlled to move to the lower left. When the terminal 1 moves in the direction perpendicular to the xy plane and in the depth direction A22 away from the signboard, the picked-up signboard appears to be reduced, so the display information similarly changes from the state of P20 to P22 As shown in FIG.

  Returning to FIG. 2, each part of the terminal 1 and the server 2 that realizes such control will be described sequentially. First, each part of the terminal 1 is as follows.

  The imaging unit 11 continuously captures an imaging target at a predetermined sampling period, and outputs the captured image to the calculation unit 12 and the control unit 14. As the imaging unit 11, a digital camera provided as a standard in a portable terminal can be used.

  The calculation unit 12 calculates feature information from the image input from the imaging unit 11, and outputs the feature information to the transmission unit 17 and the update unit 15. The feature information is a local feature that is invariant to rotation and scaling, such as the well-known SIFT feature or SURF feature, and is calculated based on the relative brightness gradient in the local region of the image. An amount can be used. Alternatively, a known FERN feature having the same property may be used.

Thus, the feature information is obtained in the following format, for example.
(d _i , x _i ) [i = 1,2, ..., n]
Here, d _i is a feature quantity and is a quantity such as a vector, x _i is a position (feature point coordinates) on the image where the feature quantity d _i is calculated, and n is a value from the image. This is the number of pairs of calculated feature point coordinates x _i and feature amounts d _i .

  The calculation of the feature information by the calculation unit 12 is performed for the purpose of reducing the amount of data transmitted from the terminal 1 to the server 2 and shortening the response accompanying the reduction. Therefore, when the data amount of the feature information exceeds the data amount of the input image, the calculation unit 12 transmits the input image instead of the feature information from the transmission unit 17 to the server 2, and the server 2 performs the same process as the calculation unit 12. Thus, the feature information may be calculated.

  In this case, it is assumed that the server 2 includes functional blocks (not shown) similar to those of the calculation unit 12. Furthermore, in this case, the calculation unit 12 on the terminal 1 side calculates feature information when estimating the position and orientation of the terminal 1 alone described later.

  SIFT features are listed in `` DGLowe, ―Distinctive image features from scale-invariant key points, Proc. Of Int.Journal of Computer Vision (IJCV), 60 (2) pp.91-110 (2004) ''. The feature is `` H. Bay, T. Tuytelaars, and LVGool, SURF: Speed Up Robust Features, Proc. Of Int. Conf. Of ECCV, (2006) '', and the FERN feature is `` M. Ozuysal, M. Calonder, V. Lepetit, P. Fua, “Fast Keypoint Recognition using Random Ferns, IEEE PAMI, 2009”.

  The positioning unit 13 acquires the coordinates, the azimuth or the elevation angle of the terminal 1, or a combination of some or all of them to obtain positioning information, and outputs the positioning information to the transmission unit 17. The coordinates can be configured to include latitude, longitude, altitude, floor number of the building, and the like. Note that the azimuth and the elevation angle are acquired as an azimuth and an elevation angle for specifying the front direction when the imaging unit 11 in the terminal 1 captures an image.

  Positioning unit 13 is standard equipment on mobile terminals, and GPS (Global Positioning System) to acquire coordinates, IMES (Indoor Messaging System) and NFC (Near Field Communication) for indoor positioning, An electronic compass or the like that acquires the azimuth and elevation angle can be used. What is necessary is just to acquire about the building and its floor number from information from specific use AP (access point), such as IMES and NFC.

  The transmission unit 17 transmits the feature information obtained by the calculation unit 12 and the positioning information obtained by the positioning unit 13 to the server 2. Further, terminal information for specifying the terminal 1 can be transmitted as necessary. The terminal information may include a usage history of the terminal 1 by the user.

  The receiving unit 18 is the recognition information of the imaging target (information of the result of recognizing what the imaging target is) as a response to the feature information and the positioning information and the terminal information transmitted by the transmission unit 17, and the position of the recognized imaging target And posture information and additional information are received from the server 2.

  The update unit 15 updates the information on the position and orientation of the terminal 1 with respect to the imaging target to the information at the current time point, and displays the display information as described in the examples of FIGS. 3 and 4 based on the updated position and orientation information. The updated information is output to the control unit 14.

  In addition, when new data is transmitted / received to / from the server 2, the update unit 15 obtains a recognition result indicating what the imaging target is based on the recognition information, and captures an image different from the imaging target up to that point. If the recognition result indicates that the object is a target, the position and orientation are updated after that time by changing the method according to the newly recognized imaging target. Details of the update will be described later.

  The control unit 14 controls the additional information received from the server 2 in accordance with the position and orientation information obtained by the updating unit 15, and outputs the controlled additional information to the output unit 16. As the control process, planar projective transformation can be used, and display processing control in the output unit 16 of additional information is performed. The output unit 16 superimposes the image obtained by the imaging unit 11 on the image obtained by the imaging unit 11 as additional information that takes the same posture as the imaging target or a predetermined change at the position of the imaging target or a position shifted by a predetermined amount from the position. This control is performed so that That is, the control is performed so that the display acquired by superimposing the additional information in a predetermined arrangement on the imaging target in the image acquired by the imaging unit 11 is performed. By the superimposition process, the display information in the output unit 16 is controlled as if the additional information is pasted on the planar imaging target from the beginning.

  The output unit 16 outputs, from the terminal 1, display information in which the controlled additional information is superimposed on the imaging target in a predetermined arrangement in the captured image. As the output unit 16, a display provided as a standard in a portable terminal can be used.

  On the other hand, each part of the server 2 is as follows.

  The receiving unit 25 transmits the information transmitted by the transmitting unit 17, that is, the feature information obtained by the calculating unit 12 of the terminal 1, the positioning information obtained by the positioning unit 13 of the terminal 1, and the usage transmitted as necessary. Terminal information including history and the like. The received information is output to the extraction unit 23.

  The storage unit 22 is information provided for controlling display information based on the position and orientation on the terminal 1, and includes feature information for recognizing various predetermined imaging targets, and coordinates indicating the coordinates where the imaging targets exist And information. Note that the positioning information held by the storage unit 22 is configured in the same format as the positioning information acquired by the positioning unit 13.

  The storage unit 22 also includes terminal information including the same type of usage history received by the receiving unit 18 as information provided for performing control specialized for the user in the display information on the terminal 1, and the imaging target And additional information related to the terminal information.

  FIG. 5 shows an example of information held in the storage unit 22 in a table format. The storage unit 22 holds characteristic information, positioning information, additional information, and terminal information for each of a plurality (M) of imaging targets_1 to imaging target_M that are predetermined imaging targets. Imaging object_1 to imaging object_M are, for example, billboards of various stores, and are generally flat.

  For example, for imaging object_1, a feature quantity and feature point coordinate pair (feature quantity_1_i, feature point coordinates_1_i) of the same type as the feature information calculated by the calculation unit 12 is index i = 1,2 , ..., n1, n1 are stored in advance. Further, information for specifying the arrangement of the plurality of pairs of the feature amount and the feature point coordinates of the image pickup target_1 obtained from the image when the image pickup target_1 is picked up. As described above, the reference arrangement_1 is also included in the feature information and stored.

  FIG. 6 is a diagram conceptually showing the feature information. B1 is a signboard as an example of the imaging object _1 in FIG. 5, and the local regions R1, R2, R3, ..., Rn1 are feature quantity and feature point coordinate pairs (feature quantity_1_i, feature point coordinates_1_i ) [i = 1, 2, 3,..., n1] are examples of calculated areas. The sign B1 is in a state of being imaged at a predetermined distance from the front as the predetermined arrangement_1. In particular, the feature point coordinates are given as coordinates on the captured image.

  Note that the feature amount calculated by the calculation unit 12 and also held in the storage unit 22 has a property that is theoretically invariant with respect to rotation and enlargement / reduction, so that the imaging target_1 can be seen. Whatever the reference arrangement_1 is, it is almost constant. Information such as the reference arrangement_1 is used by the recognition unit 21 when obtaining the position and orientation from the set of feature point coordinates.

  Returning to FIG. 5, for imaging object_1, positioning information_1 is stored as positioning information thereof, that is, information on the location of the imaging object_1 on the field, additional information_1 is stored as additional information, and the terminal Terminal information_1 is stored as information. Although the details are not shown in FIG. 5, the additional information_1 is stored as information having contents according to the terminal information including the usage history and the like.

  For example, in the case where the imaging target_1 is a signboard of a coffee shop, and when the selection menu history of the user at the coffee shop is obtained from the usage history of the terminal 1 as terminal information, a specific type of menu is determined in the history The additional information_1 may be configured to include coupon information of a specific type of the coffee shop corresponding to being present in the period. As a result, the user is presented at the output unit 16 with additional information whose content corresponds to his / her history.

  Also, in FIG. 5, terminal information such as terminal information_1 includes user attributes, preferences, operation history, number of uses, etc. in addition to information for identifying the terminal, and only when the terminal information corresponds to the terminal information. The imaging target is used to be a search target in the extraction unit 23. A specific example is as described later as “when there is a dance in the preference (hobby) in the terminal information”.

  The extraction unit 23 searches the storage unit 22 for an imaging target having positioning information close to the positioning information of the terminal 1 obtained from the receiving unit 25 in a predetermined range, and as a candidate for the imaging target actually captured by the terminal 1 Extract. For example, an imaging target that is obtained from a position included in the positioning information and within which a distance between the terminal 1 and the imaging target is within a predetermined range is extracted as a candidate.

  Alternatively, when the obtained positioning information includes information related to a direction, in addition to the condition that the positioning information is close within a predetermined range, it is within a range that fits within the angle of view of the imaging unit 11 of the terminal 1 around the direction. The imaging target having the positioning information is extracted from the storage unit 22 as a candidate.

  In addition, if the number of floors of a given building is included in the positioning information, after narrowing down candidates to those with the same number of floors in the given building in advance, within the floor of the floor of the building from the latitude and longitude Candidates that fall within a predetermined range may be extracted by obtaining a planar distance.

  FIG. 7 is a diagram conceptually showing a search based on the positioning information. In (A), the search target is a circle C having a predetermined radius r centered on the position P specified by the positioning information of the terminal 1, and the imaging target having the positioning information in the circle C, that is, the circle C is present. The imaging target to be extracted is extracted as a candidate. In (B), in addition to the condition of (A), a condition that the orientation of the terminal 1 is within the angle of view θ is imposed, so that the sector F is searched.

  The angle of view θ can be included in the terminal information and notified to the server 2 so that the condition search can be performed for the imaging units 11 having various configurations in the various terminals 1.

  The extraction unit 23 may further perform the narrowing down using the terminal information obtained from the receiving unit 25 in addition to the narrowing down based on the positioning information when extracting the candidate from the storage unit 22 as described above.

  For example, if there is a sign of a dance school or a dance equipment store in the imaging target, only users who have the preference can be extracted as candidates only when there is a dance in the preference (hobby) in the terminal information. For example, it is possible to provide services by controlling display information. In this way, even when there are a large number of imaging targets, only the imaging targets useful for each user can be used for the control of display information.

  Further, instead of using terminal information as a specific item such as preference as described above, a contractor list of a predetermined service is converted into a list of terminal IDs, and stored as terminal information in an imaging target related to the service The information may be stored in the unit 22 so that display information can be presented only to related users.

  The extraction unit 23 outputs each candidate of the imaging target thus extracted to the recognition unit 21 together with the feature information. Furthermore, after the recognizing unit 21 specifies which of the imaging target candidates the imaging unit 11 actually captures on the image as described later, the extracting unit 23 obtains from the receiving unit 25. The additional information on the specified imaging target is selected from the storage unit 22 according to the specified terminal information, and the additional information is output to the transmission unit 24.

  Note that the additional information on the specified imaging target may be selected from the storage unit 22 by the transmission unit 24 instead of the extraction unit 23, and the additional information may be transmitted.

  The terminal information includes the user's attributes, preferences, operation history, number of times of use, etc. in addition to the information for specifying the terminal as described above, and the additional information includes various information according to the terminal information as described above. Since the content is configured to be included, it is possible to superimpose display information on the output unit 16 while changing the content of the additional information according to the terminal information or the like.

  For example, in the above-described example of the imaging target_1, coupon information that is different depending on the usage history or preference of the user between the user A and the user B who are using different terminals 1 even if they are signs of the same coffee shop Can be superimposed.

  The recognizing unit 21 compares the feature information in each candidate of the imaging target obtained from the extracting unit 23 with the feature information obtained from the actual imaging target in the terminal 1 and calculated by the calculating unit 12 in the terminal 1. Then, it recognizes which of the candidates is the actual imaging target, and further recognizes the relative positional relationship (position and orientation) of the recognized imaging target with respect to the terminal 1.

  Although the recognition process will be described later, since the extraction unit 23 narrows down the candidates for imaging in advance based on the positioning information and terminal information transmitted from the terminal 1, the search space for the feature information is limited, and the recognition process is accelerated. Can be achieved. Recognition information as a recognition result of the imaging target and information on the position and orientation of the recognized imaging target are output to the extraction unit 23 and the transmission unit 24. Note that when the transmission unit 24 selects additional information, the output is directed to the transmission unit 24 only.

  The transmission unit 24, the recognition information obtained by the recognition unit 21, the information of the position and orientation, the additional information corresponding to the recognition information and the terminal information etc. selected by the extraction unit 23 or the transmission unit 24 from the storage unit 22, To the terminal.

  FIG. 8 is a diagram for explaining display information in the output unit 16 on which the additional information processed by the control unit 14 is obtained, which is obtained by the processing of each unit as described above. Here, as shown in (1), the imaging target example B10 is a signboard, and shows an image taken from the front. An example of display information in which the additional information is superimposed in the state (1) is (2), and the signboard is extended to a predetermined area B11 on the same plane as the signboard B10.

  On the predetermined area B11, additional information of each content according to the terminal information such as the above coupon information may be described. Since the predetermined area B11 may have contents for each terminal information, the arrangement and / or shape of the predetermined area may be changed according to the terminal information. Further, the example B11 of the predetermined area exists outside the signboard B10, but B11 may exist inside the signboard B10, or B11 may exist inside and outside.

  As shown in (2), the additional information includes information on the relative position and orientation with respect to the imaging target and information on the occupied area so that a predetermined area B11 is defined with respect to the imaging target area B10. It shall be. However, since the imaging target and the additional information are basically configured on the same plane on the image, in this case, the relative posture is the same. Alternatively, in order to call the attention of the user viewing the display information, the relative posture of the additional information with respect to the imaging target may be a predetermined posture other than the same posture, such as planes being perpendicular to each other.

  In FIG. 8, additional information when the state of (1) is changed to (3) due to changes in the position and orientation of the signboard that is the imaging target with respect to the terminal 1 and the appearance in the image changes. The display information superimposed with (4) is (4). In other words, according to the signboard from the state of B10 seen from the front to the state of B20 seen from the front, the additional information also changed from the state of B11 seen from the front to the state of B21 seen from the front, Display information is obtained.

  At this time, B21 is obtained by applying the relationship of the position and orientation of B20 to B10, that is, the relationship of plane projective transformation, to B11, that is, by performing the plane projective transformation on region B11. For the user who views the display information, the signboard B10 is expanded to the area of B11 and appears to exist from the beginning at an arbitrary position and posture where the signboard B10 can be viewed. Thus, even in the control based on the posture and position as shown in FIGS. 3 and 4, the display information is controlled by replacing the pattern P10 in FIG. 3 and the pattern P20 in FIG. 4 with the areas B10 and B11. The Rukoto.

  Details of the processing of the recognition unit 21 are as follows. For comparison processing of feature information, RANSAC or PROSAC, which are well-known methods, can be used.For example, in the case of RANSAC, the recognition unit 21 performs comparison and comparison as follows (1) to (3). Recognition can be performed.

(1) Correspondence of feature amount The feature amount is associated between the feature amount calculated from the actual imaging target and the feature amount of the candidate c in the imaging target. The feature quantity calculated from the actual imaging target is d _i (i = 1, 2, ..., n), and the feature quantity stored in the storage unit 22 for the imaging target candidate c is D _cj ( If j = 1, 2,..., m), the feature quantity d _{i [i]} corresponding to each feature quantity D _cj is determined as the distance between the feature quantities being minimized. Further, when there remains a feature quantity d _i for which the corresponding feature quantity D _cj is not determined, the feature quantity D _i is associated with the feature quantity D _{cj [i]} having a minimum distance from the feature quantity d _i .

  In the association, a condition that the distance between the feature amounts is within a predetermined threshold may be imposed on the condition that the distance between the feature amounts is minimum. As another embodiment, all the features whose distances between the feature amounts are within a predetermined threshold may be associated with each other.

(2) Calculation of planar projective transformation relationship by exclusion of miscorresponding part and correct correspondence part Of the feature amount calculated from the actual imaging target, it was miscalculated in the part different from the feature point in the imaging target etc. For the reason, the correspondence determined in (1) that is determined to be an incorrect correspondence is excluded, and the relationship of the planar projective transformation is obtained from the feature points with respect to the correct correspondence feature amount.

Specifically, using the feature quantity d _i and the feature quantity D _cj associated in (1), a predetermined number from the feature quantity D _cj (however, at least four are required to obtain planar projective transformation) For all combinations of feature quantities di corresponding thereto, a set x of feature point coordinates in the corresponding feature quantity di from a set X of feature point coordinates consisting of the feature point coordinates of a predetermined number of feature quantities D _cj Is calculated so as to minimize the error e of the conversion expressed by the following equation.
e = | HX − x |

The conversion relationship is obtained for all combinations in which a predetermined number is selected from the feature value D _cj . Among all the combinations for which the conversion relationship H is obtained, the feature point coordinates X ′ and x ′ in the feature amount D _cj ′ and the corresponding feature amount d _i ′ not used when obtaining the conversion relationship are obtained. The maximum number of pairs of feature point coordinates X ′ and x ′ is obtained so that the error E shown in the following equation when combined with the conversion relation H is within a predetermined threshold.
E ＝ | HX '− x' |

  A pair of feature point coordinates X and x for which H is obtained when the number of pairs is maximum, and a pair of feature point coordinates X ′ and x ′ in which the error E is within the predetermined threshold with respect to X and x And the feature point coordinates corresponding to those for which the feature amount correspondence in (1) was correctly performed with respect to the candidate c (correct feature point coordinates) and The plane projective transformation H is obtained again using the feature point coordinates in the feature amount, and the position and orientation estimation results for the candidate c are obtained.

  Note that the plane projective transformation H is a conversion from the feature point coordinates held in the storage unit 22 to the feature point coordinates in the input image, and therefore further considers the reference arrangement information held in the storage unit 22. Thus, the estimation result of the position and orientation of the imaging target with respect to the terminal 1 can be obtained.

(3) Determining which of the candidates is the actual imaging target Perform (1) and (2) above for each candidate c, and feature point coordinates in the feature values correctly associated in (2) When the plane projective transformation H is obtained in step (2), the candidate c that minimizes the error resulting from the transformation of the feature point coordinates corresponding to the positive in H is determined to correspond to the actual imaging target. In addition, the estimation result of the position and orientation in (2) for the candidate c (that is, the estimation result corresponding to the plane projection transformation H obtained again) becomes the estimation result of the actual position and orientation of the imaging target.

  Details of the update process by the update unit 15 will be described together with related processes. The update unit 15 performs an update process according to various aspects of the timing of information exchange between the terminal 1 and the server 2. FIG. 9 is a diagram for explaining various aspects of the timing as <A> and <B>.

  In <A> and <B>, (1) to (6) and the like sequentially indicate processing on the server 2 side, and [1] to [6] and the like sequentially indicate processing on the terminal 1 side. In particular, each of [1] to [6] includes a position and orientation update process that is sequentially repeated at a predetermined timing interval on the terminal 1 side, and display information control based on the update. In FIG. 5, it is assumed that each processing such as (1) and [1] includes a plurality of steps and is performed with a certain time width.

  In the case of <A>, each time [i] (i = 1, 2, ...) when the terminal 1 updates the position and orientation, the server 2 communicates with the server 2, and the server 2 obtains by each processing (i). The latest recognition information and other information are always used by the terminal 1.

  That is, for example, if it is the first time, the terminal 1 passes the feature information and the positioning information obtained at the time point to the server 2 via the transmission unit 17 and the reception unit 25 as indicated by a1 in the process [1]. The terminal information may be passed to the server 2 when the terminal 1 and the server 2 start exchanges.

  The server 2 performs the series of processes described above using the feature information and the positioning information in the corresponding process (1). That is, the extraction unit 23 extracts imaging target candidates from the storage unit 22, and the recognition unit 21 recognizes the actual imaging target from the candidates as recognition information and recognizes the position and orientation thereof. To do.

  Then, as shown in b1, the recognition information, the position and orientation information, and the content corresponding to the terminal information among the additional information held in the storage unit 22 for the recognized imaging target A reply is made to the terminal 1 via the transmitter 24 and the receiver 18. Based on the returned information, the terminal 1 controls the display information at the time [1]. The same applies to the second and subsequent times.

  As described above, in the case of <A>, the display information on the terminal 1 is always controlled using the latest information corresponding to the time point from the server 2. When each process of the terminal 1 and the server 2 including transmission / reception of data exchanged with each other can be performed at a sufficiently high speed, display information with a predetermined accuracy can be controlled by <A>.

  On the other hand, when a part of the processing speed cannot be ensured, and there is a bottleneck in controlling the display information following the predetermined sampling rate imaged by the imaging unit 11 and / or output by the output unit 16, For example, as shown in <B>, display information can be controlled at a speed following the sampling rate while ensuring a predetermined accuracy.

  That is, in the case of <B>, the process [1] of the terminal 1 and the process (1) of the server 2 and the data transmission / reception a1 and b1 in the first time are the same as described in the case of <A>. On the other hand, in order to cope with the bottleneck, the display information control process [2] of the terminal 1 for the second time uses the server 2 between the information already obtained by b1 and the processes [1] to [2]. First, it is performed by the information c2 acquired independently by the terminal 1 itself. As will be described later, various kinds of information c2 that can be acquired following a predetermined sampling rate are used. As a result, the process [2] becomes possible without waiting for the data reception b2 from the server 2 after the data transmission a2 as in the case <A>.

  Similarly, the display information control process [3] for the third time is performed using the information c3 (same type information as c2) and b1 acquired during the processes [1] to [3], and the display information for the fourth time is displayed. The control process [4] is performed using information c4 (same type as c2) and b1 acquired during the processes [1] to [4].

  On the other hand, the information c2 to c4 can be acquired at high speed, but it is difficult (or may be difficult) to ensure accuracy over a long period of time. Maintain accuracy by using. As an example of the predetermined timing, for example, the fifth display information control process is the same as that for the first <B> or <A> in the process [5] of the terminal 1, the process (5) of the server 2, and This is performed by data transmission / reception a5 and b5.

  Thus, the display information control process [6] for the sixth time is the information obtained by b5, and the information c6 (same type as c2) obtained by the terminal 1 alone between the processes [5] to [6], Is done by. Although not shown, the same applies.

  Specifically, the information c2 to c6 and the like indicate the change in the position and posture of the terminal 1 from the most recent past time point to the current time point when the server 2 recognizes the position and posture and notifies the terminal 1 of the recognition result. This is estimated information. By adding the estimated change to the position and orientation of the imaging target with respect to the terminal 1 that the server 2 has recognized and notified to the terminal 1 in the most recent past, the position and orientation of the imaging target at each time point can be determined. can get. Given the relationship of the plane projection change in the form of a matrix, the plane projection transformation matrix of the position and orientation recognized by the server 2 in the past past is multiplied by the matrix of the plane projection transformation for the estimated change. Then, a plane projection transformation matrix corresponding to the position and orientation at each time point is obtained.

  Alternatively, the information c2 to c6 and the like may be information obtained by directly estimating the position and orientation of the imaging target at the current time point. In this case, too, the position and orientation recognized by the server 2 in the latest past are used as the reference. By using the position and orientation, the position and orientation at the current time can be estimated.

  Even in the case of <B>, only the recognition process may be executed in parallel at a predetermined frequency on the server 2 side. That is, for example, the information a2 such as <A> is transmitted in [2] of <B> and the recognition process is performed in (2), but the information of b2 may not be transmitted. In this case, when it is determined in the recognition process on the server 2 side that the imaging target has been switched, the terminal 1 may be immediately notified to shift to control for switching the imaging target.

  FIG. 10 is a functional block diagram of the updating unit 15. The update unit 15 includes a calculation update unit 51, a sensor update unit 52, another calculation update unit 53, an extrapolation update unit 54, and a camera work update unit 55. Each of the units is responsible for each embodiment for estimating information c2 to c6 and the like. The sensor update unit 52 and the camera work update unit 55 estimate changes in position and orientation, and all others estimate the position and orientation itself.

  The update unit 15 can estimate the position and orientation by using any of the units. The update unit 15 may be configured to be able to change which one of the units is used in accordance with an instruction from the user.

  The calculation update unit 51 estimates the position and orientation of the imaging target from the same type of feature information that is calculated by the calculation unit 12. That is, since it is possible to grasp the correct correspondence portion that captures the portion to be imaged in the feature information by exchanging with the server 2, the calculation update unit 51 thereafter only extracts the correct correspondence portion of the feature information. By following the image, it is possible to estimate the position and orientation of the imaging target with respect to the terminal 1 while reducing the amount of calculation compared to the case of calculating feature information from the entire image.

  For example, if a certain feature value corresponding to a certain positive point is calculated at a certain feature point coordinate, the feature amount is calculated by searching a predetermined area near the feature point coordinate at the next time point. do it. The vicinity may be determined based on an optical flow or the like.

  The sensor update unit 52 includes an acceleration sensor and an inclination sensor that respectively acquire the acceleration applied to the terminal 1 and the inclination of the terminal 1, and estimates the cumulative (integral) output of the acceleration sensor as a translational movement amount of the terminal 1, that is, a change in position. Then, the rotation amount of the terminal 1, that is, the change in posture is estimated from the change in the output of the tilt sensor. Each sensor output has the property that errors accumulate with time instead of ensuring a sufficient sampling rate. Each sensor output is converted to a planar projective conversion format and then output to the control unit 14.

  The separate calculation update unit 53 estimates the position and orientation by calculating from the image another type of feature amount having a calculation load smaller than that of the feature information calculated by the calculation unit 12. For example, a predetermined marker is provided in advance on the imaging target, and the position and orientation are estimated by detecting the marker.

  For example, four or more markers that can be distinguished from each other by performing edge detection that generally has a low calculation load on the inside of the region after making it possible to narrow down the region surrounding the marker with a predetermined color feature on the image The feature line segment and / or the feature point (four or more feature locations distinguishable from each other) can be detected. By setting the number of feature locations to four or more, plane projective transformation corresponding to the position and orientation can be calculated. If the imaging target is a signboard, such a marker can be provided in the outer frame portion or inside. For example, if the signboard is rectangular, the marker can be configured as an outer frame portion made up of four feature lines.

  The marker information is included as additional data in the data held in the storage unit 22 as described in FIG. 5, and the terminal 1 obtains the marker information when the server 1 interacts with the server 2. This makes it possible to detect the marker in the recognized imaging target. Further, the coordinates of the feature point when the marker is in a predetermined position and orientation are also stored in the storage unit 22 as reference arrangement information, and a planar projective transformation is obtained with the feature point detected by the terminal 1. It shall be made available.

  The extrapolation update unit 54 applies a predetermined fitting function to a predetermined history of the position and posture recognized by the server 2 by the exchange between the terminal 1 and the server 2, so that the position and the current point The posture is estimated by the extrapolated value.

  For example, the position and orientation recognized in the past in the past and the position and orientation recognized in the past in the past are used as a history, and the position and orientation at that time are obtained by fitting with a linear function on the time axis. presume. A fitting function can be provided for each position and orientation. According to the linear fitting function of the posture, the posture is estimated so as to move at an equal angular velocity around the fixed rotation axis.

  The camera work update unit 55 performs a camera work estimation process disclosed in Japanese Patent Laid-Open No. 2007-087049 on the image, and estimates a position and orientation by extracting predetermined camera work parameters. In other words, camera work can be classified into pan, tilt, zoom, roll, etc. that can be extracted as parameters, and camera work composed of these predetermined camera work parameters is modeled by affine transformation, which is a special case of planar projective transformation. Therefore, plane projection transformation, that is, position and orientation can be estimated from the parameters.

  In addition, each example of the predetermined timing using the recognition process of the server 2 is as follows. As described with reference to FIG. 9, the updating by the updating unit 15 as described above is continuously performed for each section divided at the predetermined timing.

  In one embodiment, the result of recognition processing of the server 2 is used every time the number of updates of the position and orientation by the update unit 15 reaches a predetermined number. When the update unit 15 estimates the change in position and orientation by the sensor update unit 52, it is preferable that the accumulation of sensor errors can be eliminated at every predetermined timing according to the embodiment. Further, when the updating unit 15 estimates the position and orientation by the extrapolation updating unit 54, it is preferable because the extrapolation accuracy can be maintained by the one embodiment.

  In one embodiment, the update unit 15 estimates the position and orientation by the calculation update unit 51 or the separate calculation update unit 53, and when the error in obtaining the planar projective transformation used for the estimation exceeds a certain value, Use the recognition result of server 2. Note that the next processing exceeding the certain value may be used as the recognition result of the server 2.

  In one embodiment, the update unit 15 estimates the position and orientation by the calculation update unit 51 (or another calculation update unit 53), and it is impossible to detect a predetermined number or more of feature points (or feature points of markers) used for the estimation. When it becomes, the recognition result of server 2 is used. Note that the next processing that becomes undetectable may be used as the recognition result of the server 2. Note that the calculation update unit 51 and the separate calculation update unit 53 perform planar projective conversion using detectable feature points (however, four or more points) while fewer than a predetermined number of feature points (or marker feature points) cannot be detected. And the position and orientation are estimated.

  Especially in the above two embodiments, when the imaging target that has been captured on the image has disappeared from the image due to the change or movement of the orientation of the imaging unit 11, the server 2 is accessed again and the current In this input image, another imaging target can be recognized.

  3 ... Information presentation system, 1 ... Terminal, 2 ... Server, 11 ... Imaging unit, 12 ... Calculation unit, 13 ... Positioning unit, 14 ... Control unit, 15 ... Update unit, 16 ... Output unit, 17 ... (Terminal side) Transmission unit, 18 ... (terminal side) reception unit, 21 ... recognition unit, 22 ... storage unit, 23 ... extraction unit, 24 ... (server side) transmission unit, 25 ... (server side) reception unit, 51 ... calculation update unit 52 ... Sensor update unit, 53 ... Separate calculation update unit, 54 ... Extrapolation update unit, 55 ... Camera work update unit

Claims (10)

An information presentation system that includes a terminal including an imaging unit that images an imaging target and a server that communicates with the terminal, and presents additional information related to the imaging target at the terminal,
The terminal
A positioning unit that acquires positioning information including the position of the terminal;
A calculation unit that calculates a predetermined feature amount that is invariant to rotation and enlargement / reduction from the captured image and its feature point coordinates;
A terminal-side transmitter that transmits the obtained positioning information and the calculated feature amount and feature point coordinates to the server;
A terminal-side receiving unit that receives from the server the position and orientation of the terminal recognized by the server with respect to the imaging target, and additional information;
An update unit for updating the recognized position and orientation;
A control unit for controlling the additional information received from the server according to the updated position and orientation;
An output unit that superimposes and displays the controlled additional information in a predetermined arrangement with respect to the imaging target in the captured image;
The server
A server-side receiving unit that receives positioning information and feature quantities and feature point coordinates transmitted from the terminal-side transmitting unit;
For each of a plurality of predetermined imaging targets, a storage unit that holds the feature amount and its feature point coordinates, positioning information, and additional information when imaged in a predetermined arrangement in association with each other;
By searching the storage unit, an imaging target having positioning information whose distance from the acquired positioning information is within a predetermined range is selected as a candidate for the imaging target captured by the imaging unit, and its feature amount and feature An extractor for extracting together with point coordinates;
By comparing the feature quantity and feature point coordinates in each extracted candidate with the received feature quantity and feature point coordinates, which candidate corresponds to the imaging target being imaged by the imaging unit, the imaging A recognition unit for recognizing the position and orientation of the target with respect to the terminal;
A server-side transmission unit configured to transmit the recognized position and orientation and the additional information held in the storage unit to the imaging object recognized as corresponding to the terminal; Information presentation system.   The said positioning part acquires the said positioning information including the combination which consists of longitude, the latitude, the altitude, the floor number of a building, an azimuth | direction, an elevation angle, or a part or all of these. Information presentation system. The terminal-side transmitting unit transmits terminal information including information for identifying the terminal, the server-side receiving unit receives the terminal information, and the storage unit has the content corresponding to the terminal information. Hold and
The storage unit holds additional information by a configuration having each content according to terminal information,
The server-side transmission unit transmits, to the terminal, content corresponding to the received terminal information among the contents of the additional information held in the storage unit with respect to the imaging target recognized to be applicable. The information presentation system according to claim 1, wherein the system is an information presentation system. The storage unit further associates and holds terminal information for each of the predetermined plurality of imaging targets,
The information extraction system according to claim 3 , wherein the extraction unit performs the extraction from an imaging target in which the received terminal information exists in the terminal information held in the storage unit. . The positioning unit includes the orientation imaged by the imaging unit and acquires the positioning information,
The extraction unit searches the storage unit so that the distance from the acquired positioning information is within a predetermined range, and the imaging unit captures an image from a position in the positioning information toward an orientation in the positioning information. An imaging target having positioning information that falls within a predetermined angle of view at the time is extracted as a candidate for the imaging target being imaged by the imaging unit together with the feature amount and the feature point coordinates. 5. The information presentation system according to any one of 4.   The recognizing unit associates similar features between the extracted feature quantities and the received feature quantities, and temporarily uses RANSAC or PROSAC between the feature point coordinates for the associated feature quantities. The relationship between the plane projection changes of the two is calculated and the erroneous correspondence is excluded. Then, the relationship between the plane projection changes between the corresponding feature point coordinates is obtained again and the conversion error is obtained, and the conversion error is minimized among the candidates. The candidate is recognized as corresponding to the imaging target being imaged by the imaging unit, and the position and orientation of the imaging target with respect to the terminal are recognized based on the relationship of the plane projection change obtained again with respect to the candidate. The information presentation system according to claim 1, wherein the information presentation system is an information presentation system. The update unit
The recognized position and orientation are updated by following the correct corresponding feature amount and feature point coordinates obtained by the recognition unit, and updating the relation of the planar projective transformation at the time of recognition. A calculation update unit;
Including an acceleration sensor for acquiring an acceleration applied to the terminal and an inclination sensor for acquiring an inclination of the terminal, estimating a change in the position of the terminal based on a cumulative acceleration acquired by the acceleration sensor, and acquiring the inclination sensor A sensor update unit that updates the recognized position and orientation by estimating a change in the posture of the terminal from the change in the tilt,
Another calculation update unit for updating the recognized position and orientation by detecting a predetermined marker provided in advance in the imaging target by edge detection;
An extrapolation update unit that updates the recognized position and posture by applying a predetermined fitting function to the predetermined history of the recognized position and posture and obtaining the position and posture at the current time point by extrapolation;
The information presentation according to claim 6, further comprising: a camera work update unit that updates the recognized position and orientation by extracting predetermined camera work parameters from the captured image. system. The recognition unit recognizes the imaging target and its position and orientation at every predetermined timing,
The predetermined timing is
The updating unit updating the predetermined number of times;
An error in obtaining the position and orientation in order to update the position and orientation in the calculation update unit or the separate calculation update unit exceeds a certain value;
The feature point coordinates corresponding to the positive corresponding feature point calculated by the calculation updating unit or the predetermined marker detected by the separate calculation updating unit are determined depending on whether a predetermined number or more of feature points cannot be calculated or detected. The information presentation system according to claim 7.   9. The information presentation system according to claim 1, wherein the calculation unit calculates the feature amount and the feature point coordinate based on a relative luminance gradient in a local region of the image.   When the feature amount calculated by the calculation unit and the data amount of the feature point coordinates are larger than the data amount of the captured image, the terminal-side transmission unit replaces the positioning information, the feature amount, and the feature point coordinates. 10. The information presentation system according to claim 1, wherein the positioning information and the captured image are transmitted to the server, and the server functions as the calculation unit.