JP2015031922A - Information terminal equipment and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide information terminal equipment capable of generating a peripheral map from a plurality of images by using the prior knowledge of an imaging object.SOLUTION: An extraction part 3 extracts an imaging object whose shape and whose arrangement on a plane are already known from a pickup image continuously imaged by an imaging part 2 on the basis of the shape. An estimation part 4 estimates a positional relation between the extracted imaging object and the imaging part 2 on the basis of the already known shape of the imaging object. A storage part 6 stores map information. A creation part 5 converts the pickup image into a correction image viewed from a predetermined direction with respect to the imaging object on the basis of the estimated positional relation, and synthesizes the converted correction image at a current point of time with the stored map information at the current point of time, to update the stored map information.

Description

  The present invention relates to an information terminal device, and in particular, an information terminal device capable of recognizing a change in video and creating a surrounding map and measuring its own position at the time of the creation, and a program thereof About.

  The following methods are disclosed as methods for realizing positioning and map creation related to the positioning.

  Patent Document 1 discloses a technique that enables an autonomous mobile robot to autonomously move a room by generating an environment map that represents a room situation based on object information recognized by a distance sensor. Yes.

  In Non-Patent Document 1, a method of capturing an image while moving a camera and analyzing a trajectory of a feature point included in a plurality of captured images to create a surrounding map and estimating a position and orientation of the camera is proposed. Disclosure.

JP 2011-209203 A

Video-rate Localization in Multiple Maps for Wearable Augmented Reality (ISWC 2008)

  In Patent Document 1, an environment map is generated by estimating a self-position using a driving amount of a moving mechanism and an acquired distance measurement value. Therefore, an actuator that rotates, an encoder that detects rotation, a laser range finder, and the like There is a problem that special hardware is required.

  Non-Patent Document 1 has a problem that it cannot operate at a sufficient speed on a terminal with scarce computing resources because the processing load of map creation and position / orientation estimation is large.

  The present invention has been proposed in view of the above problems, and an information terminal device that can generate a peripheral map from a plurality of images without requiring special hardware and using a small processing load by using prior knowledge of an imaging target. And to provide a program.

  In order to achieve the above object, the present invention is an information terminal device having an imaging unit that continuously performs imaging, and is arranged on a shape and a plane from a captured image captured by the imaging unit. An extraction unit that extracts a known imaging target based on the shape, an estimation unit that estimates a positional relationship between the extracted imaging target and the imaging unit based on the known shape, and map information are held. Based on the storage unit and the estimated positional relationship, the captured image is converted into a corrected image viewed from a predetermined direction with respect to the imaging target, and the converted corrected image is currently held at the current time. And a creating unit that updates the held map information by combining with the map information.

  Further, the present invention is a program for causing a computer to function as the information terminal device.

  According to the present invention, map information is created by simple image processing by using, as known information, continuously-captured captured images that are arranged in a shape and a plane for each imaging target. It becomes possible.

It is a functional block diagram of the information terminal device which concerns on one Embodiment of this invention. It is a figure which shows the example of the scenery used as the image imaged or imaging object. It is a figure which shows the example of the scenery used as the image imaged or imaging object. It is a figure which shows the example of the map information produced from the example of FIG. It is a figure for demonstrating the 1st method of making an area division range and an area division start coordinate correspond to the latest past extraction result. It is a figure for demonstrating the 2nd method of determining an area division range and an area division start coordinate according to the inclination of an imaging part. It is a figure for demonstrating a series of numerical formulas in the process which improves the precision of conversion precision, when there exist two extraction results about the same imaging target. It is a figure for demonstrating the process which obtains a correction image. It is a figure which shows the conceptual example by which map information is updated by the correction | amendment image.

  FIG. 1 is a functional block diagram of an information terminal device according to an embodiment of the present invention. The information terminal device 1 includes an imaging unit 2, an extraction unit 3, an estimation unit 4, a creation unit 5, a storage unit 6, and a positioning unit 7. The information terminal device 1 may be configured as a portable information terminal such as a smartphone.

  The information terminal device 1 captures images continuously captured by the image capturing unit 2, processes each unit, processes the image of the “landscape” captured by the image capturing unit 2 in the creation unit 5, and connects them. Map information is created as a combination. The positioning unit 7 estimates the position of the imaging unit 2 at each time point, that is, the position of the information terminal device 1 in the generated map information.

  Note that the information terminal device 1 may have a function of displaying the map information. When the display is performed, the estimated position may be displayed on the map information. If the display function is used, the user can grasp his / her position in real time.

  2 and 3 are examples of images captured by the imaging unit 2 or scenery to be captured. Depending on the imaging mode, the illustrated scene may be obtained as the illustrated image as it is, or an image of each part in the scene may be obtained. "Image" or "target landscape".

  In FIG. 2, a brick (or tile or the like) laid on the ground is shown as an example. FIG. 3 shows an example of the ground where blocks are spread in a station premises or the like. In the present invention, an object to be imaged is assumed to be arranged on the same plane with a certain frequency and whose shape is known, such as a brick or a block.

  FIG. 4 is an example of map information created by the creation unit 5 from the example of FIG. 3, and FIGS. 3 and 4 show common blocks B1, B2, and B3. The outline of the operation of the present invention will be described with reference to FIGS. 3 and 4 as the “scenery”. In FIG. 4, numbers “0 to 7” and “0 to 9” are attached to the lower side and the left side in the boundary rectangle of the region R100, respectively. The numbers are drawn to indicate the coordinates of blocks arranged in a grid pattern, and will be referred to in the following detailed description.

  The imaging unit 2 performs imaging while always capturing the ground on which the block is arranged in the image by the user operation, and the location where the image is captured moves with the movement of the user of the information terminal device 1. The creation unit 5 sequentially updates and captures information corresponding to the captured location into the map information, and the range of the map information is expanded as the user moves.

  For example, in the landscape of FIG. 3 (hereinafter described as “station premises”), when the vicinity including the block B1 is imaged, as shown in FIG. 4, the region R1 that is the vicinity including the block B1 Is captured in the map information. The region R1 and the like are drawn in a rectangle in FIG. 4, but are conceptual examples and are not necessarily captured as a rectangle.

  Subsequently, when the user moves and the vicinity including the block B2 is imaged in the station yard of FIG. 3, the area R2 including the block B2 is captured in the map information as shown in FIG. It is. Similarly, when the user moves and the vicinity including the block B3 is imaged in the station yard shown in FIG. 3, an area R3 including the block B3 is captured in the map information as shown in FIG. In this way, the imaged locations are sequentially captured in the map information, and when the user moves through the station premises in FIG. 3 and completes the image, the map information of the entire station premises including the region R100 in FIG. 4 is obtained. Will be obtained.

  In addition, when the image is captured, the map information is preliminarily corrected to a format captured from the sky so that the map information is meaningful as a normal map, and then captured. Thus, even if the imaging unit 2 performs imaging with a time-varying inclination with respect to the ground under a normal manual camera shooting operation by the user, as in the example of the station premises of FIG. As shown in the example of FIG. 4, the map information is obtained in such a form that the ground side is scanned in parallel while sequentially moving on the ceiling side.

  In addition, when sequentially capturing captured locations in map information, it is necessary to match coordinates (including direction) when overlaying the location captured at the present time on map information captured in the past. The coordinate alignment can be performed by using an imaging target whose blocks B1, B2, B3 and the like are known in advance.

  The operation outline of the present invention has been described above. In the following, details of each part in FIG. 1 for realizing the operation will be described.

  The imaging unit 2 continuously captures an imaging target at a predetermined sampling period, and outputs the captured image to the extraction unit 3 and the creation unit 5. As the imaging unit 2, a digital camera provided as a standard in a portable terminal can be used. Alternatively, a stereo camera or an infrared camera may be used.

  A plurality of identically shaped objects or patterns arranged on the same plane are used as the imaging target. The shape is known in advance, and information on the shape is set in the information terminal device 1. Also, it is assumed that they are arranged on the same plane, and the processing of the present invention is performed based on the assumption.

  As specific examples, imaging objects include tiles and mats laid on the floor, road bricks (see Fig. 3 above), station blocks (see Fig. 4 above), ceiling and wall panels, window frames, etc. Others that can be seen in the public are available. The block, panel, or the like may be one that emits light by having a light source such as a fluorescent lamp inside (for example, an advertising panel). The light source may be arranged on the surface instead of inside. In addition to the actual object, a printed matter, a similar pattern drawn with a laser or the like on the floor, a similar pattern displayed on the display, or the like may be used as an imaging target. As the plane on which the imaging target is arranged, any one of the ground, the wall, the ceiling, and the like can be used.

  In general, any shape that can be represented by a geometric model, such as a rectangle, a polygon, a circle, an ellipse, or a combination thereof can be adopted as the shape of the imaging target, but the processing of the estimation unit 4 described later is possible. Therefore, it is necessary to be configured so that four or more predetermined feature points can be detected. For example, if it is a rectangle, its four vertices may be set as feature points. When combining, a part of the straight line may be replaced with a curve, or vice versa. For example, a shape such as a jigsaw puzzle may be adopted as an imaging target.

  The extraction unit 3 inputs a captured image from the imaging unit 2, and extracts an imaging target from the captured image by specifying its boundary. During the extraction, feature points such as the four vertices in the case of the rectangle are also specified on the boundary. The extracted boundary information of the imaging target (including information on the identified feature points) is output to the estimation unit 4 as imaging target information.

  Here, when the inside of the imaging target is composed of a single color, one or more boundaries of the imaging target are extracted by area division based on color information (referred to as “first processing”), and based on a known shape Identify feature points on the boundary. In the case of a plurality of line segments, one or more imaging target boundaries are extracted by segmentation (referred to as “second processing”) by line segment detection, and the boundary is based on a known shape. Identify the feature points above.

  Depending on whether the imaging target is set how advance, may be applied to any of the applicable first processing or second processing, it is possible to improve the extraction accuracy in combination of both . In addition, when applying the first process, in addition to the shape information regarding the imaging target, the color feature information is also set in advance in the information terminal device 1 as known, and the division start coordinates are set in the image. A point close to the color feature is used.

  Furthermore, you may make it improve the robustness of an extraction process by applying an approximate shape as an additional process after extracting a boundary by a 1st process and / or a 2nd process. For example, by the application, the boundary may be obtained as a circumscribed polygon or other predetermined geometric model including the boundary. As the predetermined geometric model, a model that matches a known imaging target may be set in advance, but a model that takes into account that the imaging target appears to tilt is set. For example, if the imaging target is a square, a general square is applied as an approximate shape.

  In addition, when the region is divided in the first process and / or the second process, the application range of the division in the input image may be limited. For example, it may be limited to a predetermined range near the center of the image. In addition, when the size in the image of the imaging target extracted by specifying the boundary is determined to be small by the threshold determination, the extraction result is not used (the result is not output to the estimation unit 4). Also good.

  Further, in the first process, the processing load can be reduced by making the division start coordinates at the time of division based on the color features correspond to the appearance that is assumed in the imaging target. There are the following first method and second method as a method corresponding to the assumed appearance.

  In the first method, the processing range can be limited to a part of the image instead of the entire image by associating the region division range and the region division start coordinates with the latest past extraction results.

  For example, as conceptually shown in FIG. 5, with respect to the image G (t) at the current time t, which is the extraction target, the imaging targets O1, O1 in the extraction result of the most recent past image G (t-1). Using the O2, ..., On area (area surrounded by the extracted boundary), start dividing the predetermined points of each area, for example, the centroids g1, g2, ..., gn at the current time t It is good as a coordinate. Further, the area division range may be a predetermined vicinity range of the areas O1, O2,.

  In addition, in the image G (t) at the current time t, after performing the region division using the extraction result in the past image G (t-1), the region to be imaged is divided into regions that are not detected. By performing the past extraction result, it is possible to reduce the amount of calculation and simultaneously extract the boundary of the imaging target that newly appears at the current time t. When the undetected region is targeted, region segmentation based on known color features is performed in the same manner as the processing for the entire image G (0) at the imaging start time t = 0 for which no past detection result exists. This method may be applied.

  In the second method, when the information terminal device 1 includes an inclination sensor, the area division range and the area division start coordinates may be determined according to the inclination of the imaging unit 2. FIG. 6 is a diagram for explaining the second method. FIG. 6 shows an example in which the imaging target is arranged horizontally and is on the plane P (for example, the ground), but the angle θ is also applied when the imaging target is arranged perpendicular to a wall or the like, or arranged on a slope. You can do the same by changing the setting.

  As shown in the case (1) in FIG. 6, when the imaging unit 2 is directly facing the imaging target (the tilt sensor indicates an angle θ = 0 °), the area division range is the entire image, and the start coordinate is the center of the image. Set to. On the other hand, as illustrated in the case (2), when the imaging unit 2 is oriented parallel to the imaging target (the tilt sensor indicates an angle θ = 90 °), the region division range is one of the images in the imaging target direction. The part (for example, a predetermined ratio on the lower side) and the start coordinate are set at the image end (for example, the lower end) in the imaging target direction. Further, in the case (3), when the positional relationship between the imaging unit 2 and the imaging target is in the middle of the two states (0 ° <θ <90 °), the region division range and the angle θ indicated by the tilt sensor are set. By setting the area division start coordinates in proportion, setting is made as an intermediate between case (1) and case (2).

  As described above, the first method and the second method have been described as the method for limiting the region division range and the region division start coordinates at the time of the first processing (region division based on color features), but the region division range is limited in these methods. The method can be used in the same way for the second processing (region division by line segment).

  Note that a plurality of types of imaging targets may be set in advance. In this case, the extraction unit 3 executes each type of imaging target extraction process based on different color features and / or shapes. For example, a plurality of imaging targets having the same shape and different only in color characteristics may be provided, as in the example of FIGS.

  The estimation unit 4 estimates the relative positional relationship of the imaging unit 2 with respect to the imaging target by comparing the imaging target information extracted by the extraction unit 3 with the known shape information of the imaging target.

  Note that the known shape information of the imaging target is also used when the extraction unit 3 performs the extraction, but in this case, the known shape information (and color characteristics) is taken into consideration that the shape is distorted by appearing to be tilted. Were used in the form of color features and / or line segments. For example, if the object to be imaged is a square, a general rectangle (rectangle) that allows distortion is selected as an extraction object. On the other hand, the estimation unit 4 uses the square, which is the original shape, as known shape information and compares it with the imaging target information extracted in a tilted and distorted form, thereby obtaining the imaging target of the imaging unit 2. The relative positional relationship with respect to is estimated.

Specifically, the estimation unit 4 estimates a conversion matrix that matches both based on a preset conversion formula (a conversion formula for plane projection conversion). That is, the specific coefficient of the transformation matrix of the planar projection transformation that matches is estimated. M feature point coordinates (e.g. distorted coordinates of four vertexes of the rectangle) X, the reference feature point coordinates in the known shape information of the imaging target (e.g. square four vertexes coordinates) and X _T of the extracted object's information When the planar projection transformation matrix H 2 that matches the two is used as a preset transformation equation, the feature point coordinates are expressed by the following equation (1).

  When a plurality of pieces of imaging target information are extracted temporally or spatially, or when it is known that imaging targets are regularly arranged as shown in FIGS. 2 and 3, the information is combined and converted. Matrix calculation accuracy can be improved.

For example, as shown in FIG. 7, it is assumed that two pieces of imaging target information A and B are extracted for the same imaging target O. Here, the feature point coordinates of the imaging target information A and the imaging target information B (that is, the coordinates on the captured image) are given by X _A and X _B , respectively, and the known reference feature point coordinates X for the imaging target O in advance. _When the transformation matrix to _T is calculated by H _A and H _B , the following equations (2) and (3) should hold, but generally no equal sign is established due to errors or the like. FIG. 7 is a diagram for explaining the following equations (2) to (6).

Here, as shown in the following equation (4), one of the feature point coordinates of the imaging target information A and the imaging target information B, which are individual extraction results for the same shape O, is temporarily placed as a reference feature point coordinate. Then, a conversion matrix H _{AB of} both is obtained.

  At this time, the following equations (5) and (6) are obtained.

  The estimation unit 4 uses a well-known least square method when obtaining the transformation matrix H as shown in Equation (1). Here, based on the above relational expressions (2) to (6), when there are multiple pieces of imaging target information for the same imaging target and a conversion matrix can be calculated for each, the cost function e of the least square method is By including the relationship with the transformation matrix, it is possible to improve accuracy. In general, since feature point coordinates that are spatially separated can be calculated with higher accuracy, a higher accuracy can be expected by incorporating a higher accuracy conversion matrix.

For example, after obtaining the transformation matrix H _A from Equation (2) and obtaining the transformation matrix H _AB from Equation (4), the transformation matrix H _B is obtained so as to minimize the cost function of Equation (7). Α represents an arbitrary weight variable, and ‖ and ‖ represent matrix norms.

Further, the transformation matrix H _A may be obtained again by the following equation (8) using the transformation matrix H _B obtained from the equation (7). Note that equation (8) corresponds to that interchanging the H _A and H _B in the formula (7). Further in the same manner, the calculation of H _A according to equation (7) may be repeated and the calculation of H _B according to equation (8) iteratively. Here, the weight variable α may be changed according to the number of iterations i. The number of iterations may be a predetermined number, or the end of iteration may be determined by threshold determination for error e.

Note that with respect to the imaging target information A and the imaging target information B described in FIG. 6 and the equations (2) to (8), the same imaging target O from the captured images with different times such as A being the past and B being the present. Or a plurality of the same imaging object O extracted as A and B from captured images of the same time, and in either case, the accuracy can be improved. . Also, A, is the "same" imaging target O per B, type the same, i.e., may be a reference feature point coordinates X _T is the same, always A as an actual object such as the B must have the same Absent.

  For example, in the case of imaging the ground or the like in which blocks each having a known shape are spread as shown in FIGS. 2 and 3, a plurality of blocks can be used as imaging targets from captured images of the same time. For convenience of explanation, a method of using a plurality of imaging objects of the same type in a captured image at the same time will be described assuming that the imaging target is the block.

If two blocks are obtained in the captured image at the same time, such as _A and _B , obtain _HA and HB as explained in the above equations (2) to (8). Can do. (Note that the calculation for improving accuracy may or may not be used.) In this case, in H _A and H _B , applying the decomposition of terms in the well-known planar projective transformation matrix, the terms other than the translation term are the same. Therefore, it is possible to generate and synthesize a corrected image for each. That is, the corrected image may be obtained by separately obtaining the block A portion and the block B portion and then combining them by parallel movement. The same applies to the case of three or more.

  In addition, as long as the part other than the block in the captured image is on the same plane as the block, the corrected image is viewed from the front in the same manner as the block by the plane projection transformation matrix obtained from the block. It is corrected. When matrixes are obtained individually for multiple blocks, there is a relationship of mutual translation as described above, so the matrix obtained from any block may be applied, and viewed from the front based on each block Corrected to the state as well. That is, when obtaining a corrected image, one matrix obtained in any block may be applied to the entire captured image. (It is preferable to apply a high-precision matrix if high-precision is achieved in the matrix calculation.)

In addition, when there are three or more blocks in the captured image at the same time, as a generalization of the above equations (7) and (8), a matrix for generating a corrected image by weighted linear sum is used. It can be obtained with high accuracy. For example, assuming that the imaging target information of the target block is Z, and the matrix obtained for this is H _Z , the following equation (9) may be used instead of equation (7). Here, i taking the sum represents three or more blocks (excluding Z), and i = A, B,... α _i is a weighting coefficient.

  As another example of handling a plurality of blocks within the same time, a plurality of imaging targets may be expanded as one imaging target.

First, (S1) a matrix is calculated from a captured image using a single block, and a temporary overhead image is generated by applying the matrix to the captured image. Next, (S2) feature points in the temporary bird's-eye view image (such as vertices in the case of blocks) are created map information (or information if it is known that they are regularly arranged). Good), it is made to correspond with the feature point which exists in the vicinity. Subsequently, the whole feature points in the (S3) correspondence as a new X _T, back to the first (S1), the entire feature point is regarded as a single block, the same process is repeated. In (S3), the process may be terminated when the difference between the feature point distances is equal to or smaller than the threshold value or when a predetermined number of iterations are completed.

  Furthermore, as another example of handling a plurality of blocks within the same time, the feature point coordinates may be corrected for each iteration in equation (9). For example, the feature point coordinates are in pixel units, but the true value is not necessarily in pixel units. Therefore, accuracy can be improved by setting the feature point coordinates in decimal pixel units. Specifically, a random perturbation term X ′ as in the following equation (10) is added to the cost function to obtain a matrix at X ′ that is the minimum. X ′ may be changed according to the number of iterations.

  Heretofore, each method for handling a plurality of blocks (imaging targets) within the same time has been described. On the other hand, in the same manner as described above, it is possible to increase the calculation accuracy of the matrix H using the same block at different times, but in this case, it is necessary to track the same block. From the viewpoint of improving tracking accuracy, it is desirable to use a block in a captured image that is closest in time (and thus a block that is spatially close in the image at the most recent time).

  As described above, the conversion equation used for estimating the conversion coefficient in the estimation unit 4 and the estimated conversion coefficient are output to the creation unit 5 and the positioning unit 7.

  The creation unit 5 applies the conversion coefficient input from the estimation unit 4 to the captured image captured by the imaging unit 2, thereby converting the image into a bird's-eye view from specific coordinates to obtain a corrected image. FIG. 8 is a diagram for explaining processing for obtaining the corrected image. In FIG. 8, the conversion from the imaging position by the imaging unit 2 shown in (1) to the overhead position shown in (2) may be performed as follows, for example.

  That is, as a premise, the relative positional relationship between the imaging position of the imaging unit 2 in (1) and the imaging target on the plane P is estimated in the form of a conversion equation and a conversion coefficient by the processing of the estimation unit 4. ing. Therefore, the predetermined point on the imaging target is defined as P3, and its coordinates are defined as P3 (a, b, 0) in spatial coordinates (x, y, z) where z = 0 on the horizontal plane P, and The position P1 (p, q, r) of the imaging unit 2 shown in (1) can be obtained at the spatial coordinates (x, y, z).

  With the spatial coordinates (x, y, z), as shown in (2), the overhead position is a point P2 (a distance from the predetermined point P3 (a, b, 0) by a predetermined distance h in the direction perpendicular to the plane P ( a, b, h). The relationship for performing conversion to the corrected image viewed from the overhead position P2 can be obtained in the form of a conversion formula for plane projection conversion from the positional relationship of the three points P1, P2, and P3.

In addition, when the estimation unit 4 estimates the transformation matrix H using Equation (1) or the like, an image viewed from the point P (a, b, h) separated by the predetermined distance h is used as a corrected image, and the transformation matrix H is used. as determined by the conversion, the reference feature point coordinates X _T may be determined in advance. Further, if the imaging object is a plurality of types exist, differently types even transformation matrix H determined from the imaging subject as a common distance h, previously set the reference feature point coordinates X _T in each type.

  The predetermined point P3 is a predetermined point in the imaging target, but for example, the center of gravity may be used. If there are a plurality of imaging targets, any one (for example, one having the largest area in the captured image) may be set as a predetermined point. Alternatively, the point P3 may be determined as a predetermined point (for example, a center point) in the captured image instead of determining the point P3 as the predetermined point in the imaging target. For the predetermined distance h, a common constant is used for each captured image during a series of processes for creating map information.

  The creation unit 5 also compares the map information stored in the storage unit with the corrected image, synthesizes the corrected image and the map information, and updates the map information at a place where all or part of the map information matches. The matching is determined by template matching by scanning a partial area of the corrected image (all or a part is referred to as “partial area”) on map information, or feature points extracted from the partial area of the corrected image. Various conventional techniques such as feature point matching for searching the map information can be used. Here, the partial area used for matching may be a predetermined area (for example, a circumscribed rectangle) covering the entire imaging target, a part of the predetermined area, or an area including the periphery of the predetermined area.

  The “feature point” in the feature point matching is a well-known SIFT feature amount or SURF feature amount point and belongs to a concept different from the feature point coordinates extracted together with the boundary by the extraction unit 3 of the present invention. .

  In addition, as shown in the conceptual example of updating the map information in FIG. 9, there is map information M (t) at a certain time t, and when a corrected image A (t) is newly synthesized, a matching portion R1 ( t) (= M (t) ∩ A (t)) already contains the contents of the map information obtained in the past, so the corrected image A (t) The map information may be overwritten sequentially, or the map information may be retained and not overwritten. On the other hand, other than the coincidence portion R1 (t), correction is made for the portion R2 (t) (= A (t) \ R1 (t)) where the corrected image A (t) deviates from the area of the map information M (t). The content of the image is newly taken into the map information, and the map information area grows sequentially. That is, the map information M (t + 1) at the next time point t + 1 is obtained as M (t + 1) = M (t) ∪A (t).

  In addition, when it is known that the imaging target is regularly arranged as shown in FIG. 2 or FIG. 3, the arrangement structure is stored in advance as skeleton information when creating map information and extracted. By quantizing the feature point coordinates of the imaging target information into the reference feature point coordinates of the same arrangement structure, it is possible to create map information with higher speed and higher accuracy.

  That is, if it is known that square blocks are arranged in a grid pattern, for example, as in the example of FIG. 3, only the position of each rectangle obtained from the captured image is the time block. The tracking may be performed on a series of captured images on the series. Thus, as shown in FIG. 4, when the region R1 is obtained in the corrected image, the region R1 such as the block B1 at the position (1, 1) or the block B2 at the position (4, 4). Each block is identified with its quantized position (n, m) (where n and m are integers).

  Thereafter, when the region R2 is obtained, the quantization position of each block in the region R2 can be identified by the already identified block B2 or the like. Furthermore, when the region R3 is obtained after that, it is possible to identify that the position of the block B3 is (2, 6), for example, by the information of the quantized positions that are sequentially identified. As described above, highly accurate map information can be created by assigning a corrected image to the identified quantized position.

  The quantization can also be used in the estimation unit 4. That is, in the example of FIGS. 3 and 4, assuming that all of the series of rectangular vertices extracted from FIG. 3 are the vertices of the series of grids, Equation (1) May be calculated.

  As described above, the corrected image obtained by the creation unit 5 is output to the positioning unit 7, and the newly synthesized map information is output to the storage unit 6 as updated map information. In this way, the storage unit 6 continues to update and hold the latest map information, and provides the latest past map information for reference to the creation unit 5 so that the update is possible.

  Note that the map information is empty data at an initial time t = 0 when the imaging unit 2 starts imaging. Referring to the symbols in FIG. 9, the entire corrected image A (0) obtained at the initial time t = 0 is taken in the map information M (0) = φ as the empty data, and the next time t = The map data M (1) used in 1 is constructed as M (1) = A (0).

  The positioning unit 7 calculates the coordinates of the imaging unit 2 in the map information from the map information accumulated in the storage unit 6, the relative position information estimated by the estimation unit 4, and the corrected image converted by the creation unit 5. taking measurement. Specifically, the coordinates are obtained by reversely applying the conversion formula of the estimation unit 4 with reference to the place where the corrected image is combined with the map information. In the example of FIG. 8, when the map information is composed of (x, y) coordinates on the plane P, the coordinates of the imaging unit 2 obtained are (p, q).

  As described above, according to the present invention, the position of the imaging unit 2 can be measured by the positioning unit 7 simply by imaging the imaging target with the imaging unit 2. In addition, positioning can be realized by software because it analyzes the image captured by the imaging unit 2 and generates map information based on changes in the relative position of the imaging target with respect to the imaging unit 2. It is not necessary to incorporate simple hardware into the information terminal device 1.

  Furthermore, since the map information is generated by analyzing an image captured by the imaging unit 2 and collating it with past map information, it is not necessary to incorporate complete map information in advance. In addition, the map information is generated based on past map information and prior knowledge (the shape of the imaging target and that the imaging target is arranged on a plane), so that the processing load can be reduced. Become.

  Note that the present invention may be provided as a program that causes a computer to function as each unit illustrated in FIG. 1 and causes the computer to function as the information terminal device 1 by being read and executed by the computer. In this case, as a function of the imaging unit 2, a function of continuously reading captured images acquired by the imaging unit 2 as an external device of the computer instead of a function of actually performing imaging is performed by executing the program. May be provided.

  Further, the present invention may be implemented on the server side or the like by a part of each part of the information terminal device 1 shown in FIG. 1 communicating with the server or the like on the network. For example, the imaging unit 2 may be performed on the information terminal device 1 side owned by the user, and the processing after the extraction unit 3 may be performed on the server side by transmitting the captured image to the server. Then, the result of the creation unit 5 and / or the positioning unit 7 implemented on the server side may be received and displayed by the information terminal device 1.

  As a supplementary matter in the present invention, it is possible to additionally acquire information as to whether or not the image is on a plane when the corrected image is generated. Specifically, it is as follows. When a corrected image is generated by applying the matrix H to the captured image, feature points that are on the same plane as the imaging target exist as feature points on the corrected image, but feature points that are not on the same plane as the imaging target are It can be used that it does not necessarily exist as a feature point on the corrected image.

  For example, when the coordinates of all the feature points obtained from the captured image are transformed by H, it is evaluated whether or not the feature points exist in the vicinity of the same coordinates in the corrected image. If there is a feature point, it is determined to be a plane, and if there is no feature point, it is determined to be other than a plane. Information for determining whether or not there is a feature point is described on the map information by distinguishing colors, for example, so that the user viewing the map information can distinguish between a flat part and a non-planar part It becomes.

  DESCRIPTION OF SYMBOLS 1 ... Information terminal device, 2 ... Imaging part, 3 ... Extraction part, 4 ... Estimation part, 5 ... Creation part, 6 ... Memory | storage part, 7 ... Positioning part

Claims (13)

An information terminal device having an imaging unit that continuously performs imaging,
An extraction unit that extracts an imaging target known to be arranged on a shape and a plane from a captured image captured by the imaging unit, based on the shape;
An estimation unit that estimates a positional relationship between the extracted imaging target and the imaging unit based on the known shape;
A storage unit for holding map information;
Based on the estimated positional relationship, the captured image is converted into a corrected image viewed from a predetermined direction with respect to the imaging target, and the converted corrected image at the present time is converted to the map information held at the current time. An information terminal device comprising: a creation unit that synthesizes and updates the held map information.   The extraction unit extracts an imaging target by specifying a boundary by area division based on a known color feature of the imaging target and / or area division based on line segment detection from the captured image. The information terminal device described.   3. The extraction unit according to claim 2, wherein the extraction unit identifies a boundary of the imaging target by the region division, and further extracts an imaging target by applying an approximate shape based on the known shape to the boundary. Information terminal equipment.   The extraction unit limits a range to which the region division is applied to a range based on the extracted imaging target with respect to a past time point with respect to the applied current time, or an inclination of the imaging unit further provided in the information terminal device The information terminal device according to claim 2, wherein the information terminal device is limited to a range based on the inclination acquired by the acquiring unit.   The information terminal apparatus according to claim 1, wherein the estimation unit estimates the positional relationship as a transformation matrix for planar projective transformation. The extraction unit extracts the first and second imaging objects as the same kind of imaging objects in time or space,
The estimation unit obtains a first transformation matrix from the first imaging target, obtains a transformation relationship between the first imaging subject and the second imaging subject, and obtains a second transformation matrix from the second imaging subject. 6. The information terminal device according to claim 5, wherein the first transformation matrix and the transformation relationship are used when obtaining the information. The extraction unit extracts a plurality of imaging targets as the same type of imaging targets,
The estimation unit estimates a positional relationship for each extracted imaging target,
The creation unit generates a partial correction image for each estimated positional relationship in the imaging target, combines the partial correction image over the extracted imaging target, and performs the conversion. 7. The information terminal device according to claim 1, wherein a corrected image is obtained. It is known that a plurality of the imaging objects are regularly arranged in a planar shape,
The information terminal apparatus according to claim 1, wherein the estimation unit estimates a positional relationship based on the regularity.   9. The composition according to claim 1, wherein the creating unit performs the synthesis by performing template matching or feature point matching in a map image using a part or all of the corrected image. Information terminal device.   The information terminal device according to claim 9, wherein the creation unit employs a range including the extracted imaging target as a part of the correction image to be used. The imaging object is known to be regularly arranged in a grid pattern on a plane,
11. The information terminal device according to claim 1, wherein the creation unit quantizes the extracted imaging target in the corrected image into a grid-like arrangement position of the grid.   The positioning device according to claim 1, further comprising a positioning unit that determines a position of the imaging unit on the map information based on the corrected image, the estimated positional relationship, and the map information. The information terminal device according to any one of the above.   A program causing a computer to function as the information terminal device according to any one of claims 1 to 12.