JP2012048523A - Association device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an association device that can accurately associate image information and map information with each other.SOLUTION: The association device includes an input unit 101 into which an image is input; a map input unit 102 into which a map is input; an association input unit 103 which inputs association point information representing an association relation between points on the image and points on the map; a calculation unit 105 which calculates conversion parameters for each of divided areas of the image based upon the association point information; and a conversion unit 107 which converts predetermined points on the image to positions on the map using the conversion parameters for the each divided area.

Description

  Embodiments of the present invention relate to an associating device.

  Conventionally, there is a system for editing map information by matching aerial images and map information.

Japanese Patent No. 3622094

  In a system for editing a map as described above, map information conforming to the projection projection method must be prepared in order to correct the map information by projecting the map information onto an aerial image. However, these acquisition and creation operations take a lot of time and become a major obstacle to system introduction. In addition, if map information according to a map development method such as a projection method or a projection method cannot be obtained, a large error occurs in the conversion result. Therefore, there is a problem that an object in the aerial image is associated with an incorrect position on the map.

  In view of the above problems, an embodiment of the present invention aims to provide an associating device capable of accurately associating image information with map information.

  According to an embodiment of the present invention, an image input unit to which an image is input, a map input unit to which a map is input, and corresponding point information indicating a correspondence relationship between a point on the image and a point on the map are input. Based on the corresponding input unit, the calculation unit that calculates a conversion parameter for each divided region of the image based on the corresponding point information, and the conversion parameter for each divided region, the predetermined position on the image is And a conversion unit that converts the map to a position on the map.

The figure which shows matching of the position of an image and the person of a map. 1 is a block diagram of an associating device according to Embodiment 1. FIG. (A) is an image, (b) is a map, and the figure where the position was matched. (A) in Example 2 is an image, (b) is a map, The figure by which the incorrect position was matched. 10 is a flowchart of the association apparatus according to the second embodiment. (A) is an image, (b) is information stored in the parameter storage unit. An image divided by partial areas. (A) in Example 2 is an image, (b) is a map, Comprising: The figure with which many positions were matched. FIG. 9 is a block diagram of an associating device according to a third embodiment. FIG. 9 is a block diagram of an associating device according to a fourth embodiment.

  Hereinafter, the association apparatus 100 according to an embodiment of the present invention will be described. For example, as shown in FIG. 1, the associating apparatus 100 is a desk layout in which the positions of persons detected on the image of FIG. 1A are represented by a grid on the map of FIG. The position in the figure (one large desk and four small desks in FIG. 1B) is determined. Then, based on the position of the person obtained by the associating device 100, an energy saving control system can be realized that operates by limiting the lighting and air conditioning only to the area where the person exists. A non-existence recognition system can be realized.

  In this specification, “map” means not only the ground surface but also the interior of a building (for example, the layout of seats in a theater, the layout of exhibits in museums and museums, the layout of machinery in a factory), a room Interiors (for example, layouts of desks, chairs, sofas, etc. indoors), interiors of vehicles (for example, layouts of rooms and seats inside ships, cars, trains, airplanes), building sites (for example, parking lots, etc.) ) Objects (for example, desks, exhibits, chairs, seats, sofas, cars, etc.) or phenomena (temperature distribution, lighting ON / OFF, etc.) are scaled at a certain ratio, and the objects and phenomena are latticed. This means a figure expressed on a plane using a figure, symbol, and character.

  An association apparatus 100 according to Embodiment 1 of the present invention will be described with reference to FIGS.

  The configuration of the associating device 100 of the present embodiment will be described based on the block diagram of FIG.

  As shown in FIG. 2, the associating device 100 includes an image input unit 101, a map input unit 102, a corresponding input unit 103, an information storage unit 104, a calculation unit 105, a parameter storage unit 106, a conversion unit 107, a display unit 108, Have

  As shown in FIG. 3A, the image input unit 101 captures an entire room in which a worker works with a camera from above, and the captured still image is input. The camera need only be installed at a position where the state of the room can be grasped, or may be an image taken from above. In addition, it is preferable that the entire room is photographed because the correspondence with the map is easy, but an image in which a part of the room is photographed may be used. Note that this image may use one frame of a moving image taken by a video camera, or, as will be described later, a single still image after detecting a person from a moving image or a still image. May be. As shown in FIG. 3 (b), the map input unit 102 receives a map representing the arrangement state of the desks arranged in the room, each corner of the desk is indicated by a white circle, The shape of the desk is represented by a grid-like figure connecting the white circles with straight lines. FIG. 3B shows one large desk and four small desks combined. As an input method of this map, for example, it is input with a diagram creation software.

  The correspondence input unit 103 is input with corresponding point information indicating the correspondence between each point of the input image and the map. Specifically, the point 301 on the image is changed to a point 311 on the map. Corresponding, the point 302 corresponds to the point 312,..., The point 304 corresponds to the point 314. However, it is assumed that the point 301, the point 302,..., The point 304, and the point 311, the point 312, ..., the point 314 are on the same plane (for example, a desk surface). As a method for inputting the corresponding point information, for example, there is a method in which the user arranges an image and a map and clicks a corresponding point with a mouse. The input corresponding point information is stored in the information storage unit 104.

The calculation unit 105 uses the processing described in Non-Patent Document 1 (Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision (Second Edition). Cambridge University Press, 2003.). A transformation matrix H and its inverse transformation matrix H ₋₁ are obtained. By this transformation matrix H, it is calculated from the following equation (1) that an arbitrary point v = (x, y) on the image corresponds to a point v ′ = (x ′, y ′) on the map. it can.

Further, it is also possible that an arbitrary point v ′ = (x ′, y ′) on the map corresponds to a point of v = (x, y) on the image by the inverse transformation matrix H _−1. ).

  Further, information on the correspondence between the points 305 and 315 and the correspondence between the points 306 and 316 in FIG. 3 is also input, and when there is corresponding point information on n points (n> 4), various information among the n points can be obtained. By performing RANSAC processing using a set of four points, even if there is a position error in a part of the corresponding point information, an appropriate transformation matrix H that is less affected by the error can be obtained. Here, the RANSAC process is a concept of a general process in which the process is repeated and stabilized, and is represented by the process described in Non-Patent Document 1 described above.

The transformation parameters calculation unit 105 calculates, for example, the elements _{h ij} of these transformation matrices H, or, may be used an inverse transformation matrix _{H -1} thereof, the parameter storage unit 106 of these elements _{h ij,} or , Each element h ′ _ij is stored as a conversion parameter.

  The conversion unit 107 converts information on the image and information on the map from each other based on the conversion parameter stored in the parameter storage unit 106. In this conversion, for example, the position of the person detected on the image may be converted on the map, the information on the map may be converted and superimposed on the image, or the image may be converted on the map. It may be superimposed on.

  The display unit 108 displays the converted information.

  Note that only the rectangle 320 in FIG. 3A and the rectangle 321 in FIG. 3B are shaded, but in reality, all points on the image including the outside of the rectangle are converted on the map by the same conversion parameter. Or is associated.

  From the association information obtained by the association device 100, air conditioning, lighting control, and presence determination can be performed. For example, as shown in FIG. 1, when a person is detected at a point (x, y) on the image, the conversion unit 107 uses a point (x ′) that is a person position on the map based on the equation (1). , Y ′). Then, the display unit 108 displays the conversion result as the position of the person, or another means performs air conditioning or lighting control based on the position of the person or makes a presence determination.

  Note that when the associating device 100 directly performs air conditioning or lighting control, the display unit 108 is not an essential component.

  In addition, for example, Non-Patent Document 2 (Tomoki Watanabe, Satosi Ito, and Kentaro Yokoi. Co-occurrence Histograms of Oriented Gradients for Pedestrain Detection. In Transaction on Computer Vision and Apprication, Vol. 39-47, March 2010.).

  Further, since it is assumed that the points 301 to 306 on the image are on the same plane (for example, a desk surface), the coordinates of the person detection position are also the same plane (desk surface) as those points. It is necessary to use person position coordinates (x, y) at a height corresponding to.

  When performing person detection by upper body detection, for example, if the lower end coordinates of the upper body are used, the coordinates of a point on the same plane as the desk surface are obtained.

  When performing human detection by whole body detection, for example, if the coordinates of the center of the detection rectangle are used, the coordinates near the waist can be obtained, which are also the coordinates of points on the same plane as the desk surface.

  When a person is sitting, for example, if a coordinate of 1/3 from the lower part of the upper body region is used, the point is almost on the same plane as the desk surface.

  For identification of a standing person and a sitting person, for example, it is possible to identify the standing person and the sitting person by learning individual detectors, A person who is standing may be identified as a standing person, or whether the person is standing or sitting may be identified from a history of vertical movement.

  Note that the method for determining the detection position coordinates of the person is not necessarily limited to the above method.

  An association apparatus 100 according to Embodiment 2 of the present invention will be described with reference to FIGS.

  In the present embodiment, a description will be given of an association apparatus 100 that can accurately identify the position of a person even when the map input to the map input unit 102 is an inaccurate map that has a deviation from the actual map.

  As shown in FIG. 4B, on the map, the position that should be the point 420 is correctly input as the point 421, the position that should be the point 422 is erroneously input as the point 423, and It is assumed that the position to be is erroneously input as a point 425, and as a result, the desk 440 in the lower right on the image shown in FIG.

  At this time, as in the first embodiment, the calculation unit 105 determines from the correspondence between the points 301 and 311, the correspondence between the points 302 and 312, the correspondence between the points 303 and 313, and the correspondence between the points 304 and 314. A transformation matrix H is calculated. When the conversion unit 107 uses the conversion parameters of the conversion matrix H to convert the position 430 of the person on the border (dotted line portion) of the desk 440 on the image shown in FIG. In the above, it is converted to the shifted position 432 instead of the actual position 431. Therefore, the correct person position cannot be obtained.

  This is because the points 420, 422, and 424 are input to the right of the map on the map, and the corresponding points 313 and 314 are also input to the right of the map. This is because a conversion matrix H that is converted to the right is calculated.

  If the position of the person is erroneously determined in this way, it may not be possible to correctly determine which position of lighting or air conditioning should be controlled for the person, or erroneous presence determination may be performed. This can be solved by providing an accurate map, but inputting an accurate map can be cumbersome.

  Therefore, in this embodiment, the associating device 100 is provided so that a sufficiently accurate person position can be determined without inputting an accurate map.

  The difference between the present embodiment and the first embodiment is that, in the associating device 100 of the present embodiment, the calculation unit 105, the parameter storage unit 106, and the conversion unit 107 convert the conversion parameters for each partial region or each pixel in the image. Calculation, storage, and conversion. Thereby, the shift of the position of the person when the inaccurate map is used can be reduced, and the position of the person can be determined sufficiently accurately without inputting the accurate map. Here, an inaccurate map means that if the desk scale on the map is wrong, the desk's aspect ratio is wrong, the desk's shape is distorted, or the desk's position is misaligned. Refers to cases.

  The processing of the association apparatus 100 of the present embodiment will be described based on the flowchart of FIG. An example of information stored in the parameter storage unit 106 at this time is shown in FIG. Specifically, based on the input corresponding point information (corresponding to points 301 to 306 in FIG. 4A and points 311 to 316 in FIG. 4B), the image in FIG. The conversion parameter is obtained for each partial area or for each pixel. In the following description, a partial area on an image or a single pixel is collectively referred to as a “divided area”.

  Hereinafter, a method for calculating a conversion parameter for each point (x, y) corresponding to one pixel will be described.

  First, in step S101, the calculation unit 105 selects a point (x, y) on the image for which no conversion parameter has been calculated. As a selection method, in FIG. 6A, a point (pixel) that does not have an index to the conversion parameter is selected, that is, when the conversion parameter can be called for a certain pixel, an index is given and the conversion parameter is called. If it is not possible, the index is not assigned. For example, if the index C is assigned to the position of the point (x, y), the conversion parameter H3 can be called, so that it is not selected. This index need not be given for each pixel, but may be given for each partial region.

  Next, in step S102, the information storage unit 104 selects corresponding point information of n points (n> = 4) near the selected point (x, y) from the stored corresponding point information. The points in the vicinity will be described in detail later.

  Next, in step S103, the information storage unit 104 checks whether this n point is a combination selected in the past. That is, the determination is made from the data of corresponding points used in the calculation of the conversion parameter shown in FIG. If the point has not been selected in the past, the process proceeds to step S104 (in the case of NO), and if the point has been selected, the process proceeds to step S105 (in the case of YES).

  Next, in step S104, the calculation unit 105 calculates a conversion parameter from the corresponding point information of n points (n> = 4) by the method described in the first embodiment, and the process proceeds to step S105.

  Next, in step S105, the parameter storage unit 106 stores the conversion parameter. The data to be stored are, for example, the index shown in FIG. 6B, the corresponding points used for calculating the conversion parameter, and the conversion parameter (conversion matrix). Then, the parameter storage unit 106 adds an index to the newly added conversion parameter with respect to the index of the point (x, y). When the combination is selected in the past in step S103, the parameter storage unit 106 corrects and stores the index as shown in FIG. 6A so as to indicate the conversion parameter.

  Next, in step S106, it is checked whether or not conversion parameters have been calculated for all pixels on the image, and if calculated, the process ends (in the case of YES), and if not, the process returns to step S101 (in the case of NO).

  Through the above processing, conversion parameters H1, H2, H3,... Different for each pixel are obtained as shown in FIG. The effect of using different conversion parameters will be described in detail later.

  As the definition of the n points in the vicinity in step S102, for example, a set of n points that minimizes the sum of the distances from the point (x, y) to the n point may be selected. A set of n points that are close to the point (x, y) may be selected.

  In addition, in calculating the conversion parameter, it is preferable that the n points are not too close to each other. Therefore, when two points n1 and n2 are selected from the neighboring n points, the distance between n1 and n2 exceeds a certain threshold value. You may add the restriction | limiting of selecting so that it may become.

  In calculating the conversion parameter, it is preferable that the n points are not on a straight line. Therefore, when three points n1, n2, and n3 are selected from the neighboring n points, the angle formed by these three points exceeds a certain threshold value. You may add the restriction of selecting so that it becomes.

  In the above description, the method of calculating the conversion parameter for each point (x, y) corresponding to one pixel has been described. However, instead, the same processing can be performed for each partial region. That is, the same conversion parameter is calculated for a partial region (a set of a plurality of pixels) having a certain size such that the center coordinate is the point (x, y). It is assumed that it is applied in common to all the pixels in the partial area.

  Further, as the partial area, for example, a small area obtained by dividing an image into a lattice shape as shown in FIG. 7 may be used.

  In addition, since a distant area appears to be small in the upper area of the image, a small area obtained by dividing the upper area smaller may be used.

  Next, detection of a person will be described.

  Conversion is performed from the position of the person detected on the image to the position on the map by the conversion parameter of one divided area obtained above. In the description here, the case of one pixel as a divided region will be described.

  First, it is assumed that a person is detected at a point (x, y) on the image.

  Next, the conversion unit 107 acquires the conversion parameter of the point (x, y) on the image based on the information of FIG. 6B stored in the parameter storage unit 106, and uses the equation (1) to deal with the conversion parameter. A point (x ′, y ′) on the map to be calculated is calculated.

  Next, the display unit 108 displays an image, a point (x, y) that is a person position on the image, a map, and a point (x ′, y ′) that is a person position on the map.

  The effect of using the conversion parameter for each divided area as described above will be described in more detail with reference to FIG. FIG. 8A shows an image, and FIG. 8B shows a map.

  As shown in FIGS. 8A and 8B, it is assumed that points 601 and 611, points 602 and 612,..., Points 609 and 619 are associated with each other. On the map, it is assumed that the position of the point 620 is correctly input to the point 621, the point 622 to the point 623, the point 624 to the point 625,..., And the point 632 to the point 633.

  Here, in the case of the first embodiment, conversion parameters that are accurate to some extent are estimated by RANSAC processing or the like based on correct corresponding point information of other regions or a correct map of other regions. Then, the whole is converted with the conversion parameter. That is, the conversion is performed to some extent as a whole, but the area around the desk 630 where an incorrect map is input is converted to an incorrect position on the map.

  For example, the person 640 at the right end of the desk 630 is converted into a point 641 that is the right end of the rectangle surrounded by the points 626, 628, 630, and 632 that are the true positions of the desk 630 in FIG. . However, on the map, the desk 640 is erroneously input as a rectangle surrounded by the points 627, 629, 631, and 633. Converted to position.

  However, when converting with different conversion parameters for each divided region as in this embodiment, for example, the position of the person 640 is point 608, point 606, point 607, and point 602 on the image in FIG. 8B, the map is converted with the conversion parameters calculated by the correspondence of the four pairs (or four or more points) of points 618, 616, 617, and 612 on the map in FIG. Here, the point 629 at the corner of the desk in the map is input to be shifted to the upper right from the point 628 at the true position, but the corresponding point 617 is also input to be shifted to the upper right accordingly.

  Therefore, the conversion parameters calculated at the four points of point 608, point 606, point 607, and point 602 are converted on the map in such a way that the points on the surrounding images are shifted to the upper right from the actual position. become. The position of the person 640 is also converted into a point 642 that is shifted to the upper right from the point 641 of the position when the correct map is input. As a result, the person 640 is correctly determined to be present next to the desk 630 surrounded by the points 627, 619, 631, and 633.

  According to the present embodiment, even if there are various shifts in the map position, the conversion parameters calculated at corresponding points in the vicinity perform position conversion in accordance with the shift, so the position on the map is correctly confirmed. it can. Such locally distorted transformation applied to points around the points 608, 606, 607, and 602 on the image does not affect other points on the screen. Correctly input map information around the desk 650 is converted to a correct position without being affected by the distorted conversion.

  In the above description, the person is detected by using the conversion parameter of one pixel at the point (x, y) on the person image. However, the present invention is not limited to this, and there is the following method.

  For example, a conversion parameter of a point (pixel) in the vicinity of the point (x, y) on the person image may be used. For example, when the conversion parameter at the point (x, y) is extremely different from the conversion parameter at a nearby point, it is considered that the conversion parameter at the nearby point is inaccurate. In this case, it is preferable to use the conversion parameter of the nearest calculated point (pixel).

  In the above description, the detection of the person is performed using the conversion parameter of one pixel at the point (x, y) on the person image, but the same can be applied to the partial area.

  For example, the conversion parameter of the partial region including the point (x, y) on the person image is used.

  Alternatively, the conversion parameter of the partial region closest to the point (x, y) on the person image may be used. In this case, for example, when the conversion parameter is extremely different from the conversion parameter at a nearby point in the partial region including the point (x, y), the conversion parameter of the calculated partial region closest to the point is used.

  By the way, when such locally distorted transformation is used, the transformation parameters are different at the boundaries of the divided regions, so that the coordinate transformation becomes discontinuous. Therefore, at the boundary of the divided area, the coordinates calculated by the conversion parameters of the surrounding divided areas are also taken into consideration, and for example, the coordinate conversion is performed by using the average value of the coordinates calculated by the conversion parameters of the surrounding divided areas. It is possible to prevent discontinuity.

  A matching apparatus 100 according to a third embodiment of the present invention will be described with reference to FIG.

  As shown in FIG. 9, the configuration different from the present embodiment and the second embodiment is provided with a plurality of corresponding input units 103 and a display unit 108, which include an information storage unit 104 and a communication unit 110 (for example, LAN), and each user inputs corresponding point information.

  First, each user inputs the association only for his / her surroundings to the correspondence input unit 103.

  The corresponding point information input by a plurality of users is collected from the corresponding input unit 103 into the information storage unit 104 by the communication unit 110, and the same processing as in the first and second embodiments is performed.

  According to the present embodiment, the association input work by the administrator can be reduced by a plurality of users performing association input in a distributed manner.

  In addition, a plurality of inputs by a plurality of users can eliminate the input having a large error from other inputs as an erroneous input to stabilize the processing result, and use a large number of corresponding points to stabilize the RANSAC processing. The division area can be set finely, and information on the appropriate image and information on the map can be converted.

  A matching apparatus 100 according to a fourth embodiment of the present invention will be described with reference to FIG.

  The difference between the present embodiment and the third embodiment is that a plurality of map input sections 102 are provided in addition to a plurality of corresponding input sections 103 and a display section 108 as shown in FIG.

  Since the display unit 108 can also superimpose and display a map (information such as desk layout) on the image, the user can correct a map shift based on the superimposition information.

  According to the present embodiment, not only corresponding point information but also a map can be input and corrected by a plurality of users in a distributed manner, so that the map input work by the administrator can be reduced.

Example of change

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  The association apparatus 100 according to the present embodiment has been described with the map relating to the arrangement of the desks in the room in each of the above embodiments, but can be applied to the following instead.

  A map of theater seats is input as a map, and the positions of the people in the image are associated.

  The layout of museums and museum exhibits is input as a map, and the position of the person in the image is associated.

  The layout of the machines installed in the factory is input as a map, and the positions of workers in the image are associated.

  The bed layout in the hospital room is input as a map, and the position of the person in the image is associated.

  The layout of chairs and sofas in the waiting room of the hospital is input as a map, and the position of the person in the image is associated.

  Enter the layout of the sofa in the hotel lobby as a map, and associate the positions of the people in the image.

  Input the layout of desks, tables, chairs, sofas, toys, etc. in the room of school, kindergarten, nursery school, etc. as a map, and associate the positions of children, preschoolers, teachers, etc. in the image.

  The arrangement map of the installations in the station is input as a map, and the position of the person in the image is associated.

  The layout of the ship, car, train, airplane interior room, compartments, and seats is input as a map, and the positions of passengers and passengers in the image are associated.

  A map representing the parking area of each car in the parking lot is input as a map and the position of the car in the image is associated.

DESCRIPTION OF SYMBOLS 100 ... Association apparatus, 101 ... Image input part, 102 ... Map input part, 103 ... Corresponding input part, 104 ... Information storage part, 105 ... Calculation part, 106 ... Parameter storage unit 107 ... Conversion unit 108 ... Display unit

Claims (6)

An image input unit for inputting an image;
A map input unit for inputting a map;
A corresponding input unit that inputs corresponding point information indicating a corresponding relationship between a point on the image and a point on the map;
A calculation unit that calculates a conversion parameter for each divided region of the image based on the corresponding point information;
A conversion unit that converts a predetermined position on the image into a position on the map using the conversion parameter for each divided region;
An associating device characterized by comprising: When the calculation unit calculates the conversion parameter related to a certain divided area, in addition to the corresponding point information in the divided area, the corresponding point information of the plurality of divided areas existing in the vicinity of the divided area is also used.
The associating device according to claim 1. The divided area is a single area on the image or a partial area that is a set of a plurality of pixels on the image.
The associating device according to claim 2. The conversion unit uses the conversion parameter of the partial region including the one pixel corresponding to a predetermined position on the image or the predetermined position on the image,
The associating device according to claim 3. A plurality of sets of the corresponding input unit and the display unit;
A communication unit that connects the corresponding input units, the display units, and the calculation unit;
The associating device according to claim 4. A plurality of sets of the corresponding input unit, the display unit, and the map input unit;
A communication unit that connects the corresponding input unit, the display unit, the map input unit, and the calculation unit;
The associating device according to claim 4.

Images