JP2013257349A - Camera calibration result verification device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To verify whether or not camera calibration is accurately performed.SOLUTION: An appropriate position Qi on a photographic image is specified by the operation of a mouse 18 for instance. Then, an image coordinate value (u, v) of the specified position Qi is converted to a coordinate value of world coordinates set to a real space. At the time, it is converted to the world coordinate value on a plane where the height in the real space is zero. Further, the world coordinate value is converted to a two-dimensional coordinate value (α, β) on a map image which is a simulated image of the plane. Then, a marker 46 is displayed at a position Qj according to the two-dimensional coordinate value (α, β) on the map image. When the position Qj of the marker 46 corresponds to the specified position Qi on the photographic image, it is defined that the camera calibration is accurately performed.

Description

  The present invention relates to a camera calibration result verification apparatus and method, and more particularly to a two-dimensional image coordinate set on a captured image by a camera with a fixed shooting area and a three-dimensional set in a real space including the shooting area. The present invention relates to an apparatus and a method for verifying a result of camera calibration for obtaining a camera parameter that correlates with world coordinates.

  A so-called panning and tilting turning camera is particularly suitable for surveillance applications. In the monitoring application, when a moving body enters the monitoring area, there is a lot of demand for so-called automatic tracking that controls the turning camera so as to automatically track (capture) the moving body. In order to meet this requirement, there is a conventional one disclosed in Patent Document 1, for example. According to the prior art disclosed in Patent Document 1, in addition to the turning type camera, a wide-angle fixed camera capable of photographing the entire monitoring area is provided. When a moving object appears in the image captured by the fixed camera, the position of the moving object in the captured image, that is, the coordinate value of the moving object in the two-dimensional image coordinates of the captured image is obtained. Further, the two-dimensional coordinate value is converted into a coordinate value in a three-dimensional world coordinate called real space. Then, the turning camera is controlled so as to face the position (point) according to the converted world coordinate value.

JP-A-2005-3377

  Incidentally, in order to control the revolving camera on the world coordinates as described above, naturally, the installation position of the revolving camera on the world coordinates, that is, the coordinate value must be specified. In addition, a swivel camera generally has a certain reference direction, and given a declination with respect to this reference direction, in other words, a pan angle and a tilt angle, a direction corresponding to the pan angle and the tilt angle is given. Turn to face. Therefore, this reference direction must also be specified in world coordinates.

  For this reason, conventionally, the installation position of the swivel camera is measured by a surveying device (not specified in Patent Document 1) or by using a tool such as a measure, and the coordinate values in the world coordinates are identified. It was. Also, the reference direction has been specified by an extreme eye measurement using a tool such as a magnetic compass.

  Note that, as described above, in order to convert the coordinate value of the moving body in the two-dimensional image coordinates into the coordinate value in the three-dimensional world coordinates, it is assumed that camera calibration is performed.

  An object of the present invention is to provide a camera calibration result verification apparatus and method capable of verifying whether or not the camera calibration is accurately performed.

  In order to achieve this object, the camera calibration result verification apparatus according to the present invention is set in a real space including a two-dimensional image coordinate set on a photographed image by a camera having a photographing area fixed and the photographing area. An apparatus for verifying a result of camera calibration for obtaining a camera parameter that associates a three-dimensional world coordinate with a predetermined coordinate axis that is one of the three coordinate axes constituting the world coordinate. Simulated image display means for simulating a real space and displaying a simulated image in which two-dimensional coordinates correlating with world coordinates representing the plane are set. Further, a designation unit for designating an arbitrary position on the captured image is provided. Furthermore, a first conversion means for converting the coordinate value on the image coordinate of the designated position by the designation means into the coordinate value of the world coordinate on the plane based on the camera parameter is provided. In addition, a second conversion unit is provided for converting the coordinate value of the world coordinate converted by the first conversion unit into the coordinate value of the two-dimensional coordinate based on the correlation between the world coordinate and the two-dimensional coordinate. In addition, marker display means for displaying a marker at a position on the simulated image according to the coordinate values of the two-dimensional coordinates converted by the second conversion means is provided.

  In the present invention, the predetermined coordinate axis may define the height direction of the real space, and the simulated image may be an image simulating a plane in which the height of the real space is zero.

  The camera calibration result verification method according to the present invention includes a two-dimensional image coordinate set on a photographed image obtained by a camera with a fixed photographing region and a three-dimensional world coordinate set in a real space including the photographing region. Is a method for verifying the result of camera calibration for obtaining a camera parameter, and simulating a real space in a plane in which the value of a predetermined coordinate axis that is one of the three coordinate axes constituting the world coordinate is constant A simulated image display step of displaying a simulated image in which two-dimensional coordinates correlated with world coordinates representing the plane are set; In addition, a designation process for designating an arbitrary position on the captured image is provided. Furthermore, a first conversion process for converting the coordinate value on the image coordinate of the specified position in the specifying process into the coordinate value of the world coordinate on the plane based on the camera parameter is provided. In addition, there is provided a second conversion step of converting the coordinate value of the world coordinate converted by the first conversion step into the coordinate value of the two-dimensional coordinate based on the correlation between the world coordinate and the two-dimensional coordinate. In addition, a marker display process of displaying a marker at a position on the simulated image according to the coordinate value of the two-dimensional coordinate converted by the second conversion process is provided.

It is a block diagram which shows schematic structure of the surveillance camera system which concerns on one Embodiment of this invention. It is a schematic plan view of the real space which shows the installation condition of each camera in the embodiment. It is an illustration figure which shows an example of the image image | photographed with a certain fixed camera in the embodiment. FIG. 3 is an illustrative view showing a map image corresponding to the schematic plan view of FIG. 2. FIG. 4 is an illustrative view showing a state in which a moving body appears in the captured image of FIG. 3. It is an illustration figure which shows the state of the map image when it exists in the state of FIG. It is an illustration figure which shows an example of the picked-up image with a certain turning type camera in the state of FIG. It is an illustration figure which shows the state by which the scaler was set to the map image of FIG. It is an illustration figure which shows the relationship between the map image of FIG. 8, and real space. It is an illustration figure which shows one state of real space when camera calibration is performed in the embodiment. It is an illustration figure which shows the state of the map image when it exists in the state of FIG. It is an illustration figure which shows the state of the picked-up image by a fixed camera in the state of FIG. It is an illustration figure which shows the state different from FIG. 12 of the picked-up image by a fixed camera in the state of FIG. It is an illustration figure for demonstrating the verification procedure of the result in which the camera calibration was performed in the embodiment. It is an illustration figure which shows an example of the verification result when the camera calibration is not performed correctly. It is an illustration figure for demonstrating the verification procedure different from FIG. It is an illustration figure which shows one state of real space when specifying the installation conditions of a turning type camera in the embodiment. It is an illustration figure which shows the state of a map image when it exists in the state of FIG. It is an illustration figure for demonstrating the verification procedure of the result in which the installation conditions of the turning type camera were specified in the embodiment. It is an illustration figure for demonstrating the map point view function as another example of control of the turning type camera in the embodiment. It is an illustration figure for demonstrating the advance preparation in the map point view function. It is an illustration figure for demonstrating the principle of the map point view function. It is an illustration figure which shows an example of the picked-up image with a certain turning type camera in the map point view function. It is an illustration figure which shows an example different from FIG. It is an illustration figure which shows the example of a display of the icon in the same map point view function. It is an illustration figure which shows the example of a display of the map image in the map point view function.

  An embodiment of the present invention will be described by taking the surveillance camera system 10 shown in FIG. 1 as an example.

  As shown in the figure, the surveillance camera system 10 according to the present embodiment has a plurality of fixed cameras 12, 12,... With fixed shooting areas and variable shooting areas, specifically in the pan and tilt directions. A plurality of revolving cameras 14, 14,... Each capable of revolving and having a zoom lens. These fixed cameras 12, 12,... And turning cameras 14, 14,... Are connected to a personal computer (hereinafter referred to as PC) 16.

  The PC 16 functions as a camera control device that controls the revolving cameras 14, 14,... Based on the images taken by the fixed cameras 12, 12,. Each fixed camera 12 also functions as a camera calibration device that performs camera calibration described later. Further, each turning camera 14 also functions as an installation condition specifying device for specifying an installation condition described later. The PC 16 has input means for inputting various commands thereto, for example, an operation device 22 including a mouse 18 and a keyboard 20, and information output means for outputting various information including processing results by the PC 16. Are connected to the display 24.

  The surveillance camera system 10 is used to monitor a site including a building such as a building shown in FIG. In the case of the figure, one fixed camera 12, 12,..., Is installed in each of the four corners A, B, C and D in the generally rectangular site. In the vicinity of these four fixed cameras 12, 12,..., One, that is, a total of four swivel cameras 14, 14,. The fixed cameras 12, 12,... Are installed so as to photograph different areas. For example, the fixed camera 12 installed in the southwest corner (the lower left corner in the figure) A of the site may take a picture of the south area on the south side of the building on the site, as indicated by an arrow 12a in the figure. Is installed. Then, the fixed camera 12 installed in the southeast corner (lower right corner in the figure) B in the site captures the east region on the east side of the building in the site, as indicated by an arrow 12b in the figure. Is installed. Furthermore, the fixed camera 12 installed in the northeast corner (upper right corner in the figure) C of the site captures the north region on the north side of the building in the site, as indicated by an arrow 12c in the figure. Is installed. Then, the fixed camera 12 installed in the northwest corner (upper left corner in the figure) D in the site captures the west region on the west side of the building in the site as indicated by an arrow 12d in the figure. Is installed.

  Images taken by the fixed cameras 12, 12,... Are displayed on the display 24 one by one according to a predetermined order or simultaneously by a divided screen. For example, an image taken by the fixed camera 12 installed in the southwest corner A of the site is displayed as shown in FIG. In addition, a map image as a simulated image as shown in FIG. 4 is displayed on the display 24. This map image is a faithful representation of the floor plan of the site (the shape and dimensions of each part of the building, etc.), and can be represented by appropriate image files such as bitmap files and JPEG (Joint Photographic Experts Group) files. Is displayed. Moreover, an architectural drawing such as a building can be used as the map image. That is, since the shape and dimensions of the building and the like are accurately described in the architectural drawing, it is possible to save the trouble of newly creating a map image by using this (making it into an image file).

  Here, for example, it is assumed that a moving body such as a human has entered the shooting area (that is, the south area) of the fixed camera 12 installed in the southwest corner A of the site. Then, an image captured by the fixed camera 12 is as shown in FIG. That is, the moving body (strictly speaking, an image of the moving body) 30 is displayed on the captured image. The moving body 30 is detected by a known moving body detection technique such as a frame difference method or a background difference method, and a rectangular frame 32 is displayed so as to surround (inscribe) the moving body 30. . Further, a caution mark 34 for calling attention is displayed at a predetermined position on the photographed image, for example, near the upper left corner. Upon receiving the display of the rectangular frame 32 and the caution mark 34, the operator (monitoring person) can recognize that the moving body 30 has entered. The display of the rectangular frame 32 and the caution mark 34 is controlled by the PC 16.

  In addition, as shown in FIG. 6, a moving body mark 36 indicating the current position of the moving body 30 and a locus line 38 indicating the moving path of the moving body 30 are displayed on the map image shown in FIG. . Upon receiving this display, the operator can recognize the current position and moving path of the moving body 30 in the site. The display of the mobile object mark 36 and the trajectory line 38 is also controlled by the PC 16.

  Further, according to the current position of the moving body 30, an appropriate turning type camera 14, for example, the turning type camera 14 at the southwest corner a turns to face the moving body 30, and the moving body 30 is enlarged. And zoom up to shoot. Then, an image captured by the turning camera 14 is displayed on the display 24 as shown in FIG. Then, the turning operation and zoom operation of the turning camera 14 are controlled so as to automatically track the moving body 30. Note that the turning operation and zooming operation of the turning camera 14 and the display of the captured image by the turning camera 14 on the display 24 are also controlled by the PC 16. In addition to the swivel camera 14 in the southwest corner a, another swivel camera 14 that can capture the moving body 30, for example, the swivel camera 14 in the southeast corner b, is automatically tracked and the captured image thereof. May be displayed on the display 24.

  The above is the same when the moving body 30 enters the shooting area of each of the other fixed cameras 12.

  In this manner, the moving body 30 is displayed by displaying the moving body mark 36 and the trajectory line 38 on the map image, automatically tracking by the appropriate turning type camera 14, and displaying the captured image by the turning type camera 14 that is automatically tracking. It is possible to deal with the intrusion more quickly and appropriately, so that integrated monitoring is realized.

  By the way, in order to realize such integrated monitoring, when the moving body 30 appears in the captured images of the respective fixed cameras 12, this is detected, and the position of the moving body 30 in the real space is determined. It is necessary to recognize from an image captured by the fixed camera 12. Among these, for the former moving body detection, known techniques such as the frame difference method and the background difference method are employed as described above. On the other hand, for the latter of recognizing the position of the moving body 30 in the real space from an image taken by the fixed camera 12, it is assumed that camera calibration is performed. In the present embodiment, the following device is devised in order to realize this camera calibration very easily.

  That is, in the map image shown in FIG. 4, a linear scaler 40 is set as shown in FIG. Specifically, the above-described operation of the mouse 18 (drag and drop) is performed on a part where the actual size is known in the real space, for example, the part from the northwest corner Qa to the southwest corner Qb of the rectangular building in FIG. The scaler 40 is set by the operation. Then, the actual dimension (actual dimension) Lw of the part corresponding to the scaler 40 in the real space is input by, for example, the operation of the keyboard 20 described above. The unit of the actual size Lw is, for example, [mm] (millimeters), and the length dimension Lm of the scaler 40 on the map image is a unit of pixels constituting the map image, so-called [pixel] (pixel). . Then, by dividing the actual dimension Lw in the real space by the length dimension Lm of the scaler 40 on the map image, that is, by the following equation 1, how long 1 [pixel] on the map image is in the real space A coefficient k indicating whether it corresponds to the height dimension is obtained.

  Further, as shown in FIG. 9A, the lower left corner of the map image is set as the origin Om in the map image. A straight line passing through the origin Om and extending along the horizontal direction (extending toward the right side in the figure) is the α axis, passing through the origin Om and extending along the vertical direction (extending toward the upper side in the figure). The straight line is the β axis. As a result, two-dimensional orthogonal coordinates composed of an α axis and a β axis are set on the map image. The unit of two-dimensional coordinates on this map image is [pixel] described above.

  In addition, as shown in FIG. 9B, in the real space, the position corresponding to the origin Om in the map image is set as the origin Ow, and the x axis is set so as to correspond to the α axis and the β axis in the map image. And the y-axis is set. Then, a z-axis that defines the height direction is set so as to pass through the origin Ow and to be orthogonal to each of the x-axis and the y-axis (so as to extend upward in the figure). As a result, three-dimensional world coordinates including the x-axis, y-axis, and z-axis are set in the real space. The unit of world coordinates set in this real space is the above-mentioned [mm].

  That is, the map image of FIG. 9A is a simulation of a plane whose height z is zero (z = 0) in the real space of FIG. 9B. Therefore, the relationship between the coordinate values (α, β) and (x, y, 0) of both of these is expressed by the following formula 2 where the coefficient k is the conversion coefficient.

  Therefore, for example, if the two-dimensional coordinate value at an arbitrary position Qc in the map image of FIG. 9A is (α ′, β ′), the position Qd corresponding to the position Qc in the real space of FIG. The world coordinate values (x ′, y ′, z ′) of (x ′, y ′, z ′) = (k · α ′, k · β ′, 0) due to the relationship of Equation 2.

  Then, camera calibration is performed in the following procedure using the map image calibrated by setting the scaler 40 in this way.

  That is, as a conversion algorithm for the camera calibration, a known projective transformation matrix, particularly an eleven-variable projective transformation matrix is used. In this 11-variable projective transformation matrix, assuming that the coordinate value of the two-dimensional captured image by the fixed camera 12 is represented by (u, v), this image coordinate value (u, v) and the absolute coordinate value in the real space. The relationship with (x, y, z) is as shown in the following equation (3).

In Equation 3, s is a scale factor. _Then, p 1 _{~p 11} is a camera parameter. Further, as shown in FIG. 12 described later, the coordinate value (u, v) of the captured image by the fixed camera 12 is set to the upper left corner of the captured image as an origin Oi, and passes through the origin Oi and extends in the horizontal direction (same as above). A two-dimensional image in which a straight line extending to the right in the figure is the u-axis, and a straight line passing through the origin Oi and extending along the vertical direction (extending downward in the figure) is the v-axis. It is a coordinate value.

  In this equation 3, when the scale factor s is eliminated, simultaneous equations consisting of the following equations 4 and 5 are derived.

According to these equations 4 and 5, if at least six combinations of image coordinate values (u, v) and world coordinate values (x, y, z) are configured, simultaneous equations composed of these six combinations. Thus, a total of 11 camera parameters of p _{1 to} p ₁₁ are obtained. However, when all combinations are on the same plane, for example, on a plane where z = 0, camera parameters are not obtained. Therefore, the combination is required to be on at least two different planes.

  In order to realize this, in the present embodiment, first, as shown in FIG. 10, a person 42 as an index stands at an appropriate position Qe in the imaging region of the fixed camera 12 to be subjected to camera calibration. It is assumed that the height of the human 42 is known. Further, it is preferable that the standing position Qe of the human 42 is easy to understand on the map image.

  Next, as shown in FIG. 11, on the map image, a position Qf corresponding to the standing position Qe of the person 42 in FIG. 10 is designated by an operation (click operation) of the mouse 18, for example. Thereby, the two-dimensional coordinate value (α, β) of the designated position Qf is specified. Further, the two-dimensional coordinate value (α, β) is converted into the world coordinate value (x, y, 0) by the above-described formula 2. This world coordinate value (x, y, 0) corresponds to the foot position Qe of the human 42 in FIG.

  Subsequently, the height (unit [mm]) of the human 42 is input by operating the keyboard 20, for example. The input height corresponds to the position z of the head of the human 42 in the real space. That is, the height of the human 42 is added to the world coordinate value (x, y, 0) converted by the above-described formula 2, so that the world coordinate value (x, y) of the head of the human 42 in the real space. , Z).

  Further, as shown in FIG. 12, the foot Qg of a human (strictly speaking, a human image) 42 is designated by, for example, an operation of the mouse 18 on an image captured by the fixed camera 12. Thereby, the image coordinate value (u, v) of the foot Qg in the captured image is specified.

  Similarly, as shown in FIG. 13, the head Qh of the human 42 is designated by the operation of the mouse 18 on the image taken by the fixed camera 12. Thereby, the image coordinate value (u, v) of the head Qh in the captured image is specified.

Through this series of operations, combinations of world coordinate values (x, y, 0) and (x, y, z) existing on different planes and image coordinate values (u, v) corresponding to the respective coordinate values are obtained. Two sets are obtained. Accordingly, the same work is performed after the human 42 moves to another position Qe in the real space shown in FIG. 10, and this work is performed three times in total, thereby obtaining the image coordinate values (u, v) and A total of 6 combinations with world coordinate values (x, y, z) (including (x, y, 0)) are obtained. Then, camera parameters p _{1 to} p ₁₁ are obtained based on simultaneous equations composed of these six combinations, that is, camera calibration is realized.

  As described above, according to the present embodiment, the position Qf of the foot of the human 42 as an index is designated as shown in FIG. 11 on the map image simulating the z = 0 plane in the real space. The two-dimensional coordinate value (α, β) of the foot position Qf thus obtained is converted into the world coordinate value (x, y, 0) in the real space by the above-described formula 2. Thereby, the world coordinate value (x, y, 0) of the foot position Qe of the person 42 in the real space shown in FIG. 10 is obtained. Further, by adding the height of the person 42 to the world coordinate value (x, y, 0) of the foot position Qe, the world coordinate value (x, y, z) of the head of the person 42 is obtained. Therefore, compared to the conventional technique of measuring the world coordinates (Xw, Yw, Zw) of the index point in real space by triangulation or the like, the world coordinate values (x, y, z) ((x , Y, 0).)), And thus camera calibration can be performed. This is more conspicuous as the imaging area of the fixed camera 12 is wider.

  Note that the camera calibration can be performed using the same map image for each of the other fixed cameras 12. By sharing the map image in this way, it is possible to further save the trouble of camera calibration.

  Now, by performing the camera calibration in this way, a two-dimensional image coordinate value (u, v) on an image captured by an arbitrary fixed camera 12 is converted into a three-dimensional world coordinate value (x, y) in real space. , Z). For example, the image coordinate values (u, v) at the foot of the human 30 shown in FIG. 5 are converted into world coordinate values (x, y, z) as follows.

  That is, first, when the above-described expression 4 is transformed, the following expression 6 is derived. Similarly, when Equation 5 is transformed, Equation 7 is derived.

  Here, since the foot of the human 30 in the real space is in contact with the ground, and the height dimension z in the world coordinates of the ground can be regarded as z = 0, the condition that z = 0 is expressed by Equation 6 below. And the following equations (8) and (9) are derived.

  That is, the coordinate value of the foot on the x-axis of the world coordinates is obtained by Equation 8, and the coordinate value of the foot on the y-axis is obtained by Equation 9. That is, the world coordinate value (x, y, z) (= (x, y, 0)) corresponding to the image coordinate value (u, v) of the foot is obtained.

  Then, this world coordinate value (x, y, z) (= (x, y, 0)) is substituted into the following equation 10 which is the above-described transformation equation of equation 2, so that the human on the map image Thirty foot two-dimensional coordinate values (α, β) are obtained.

  Thus, by obtaining the two-dimensional coordinate values (α, β) of the foot of the person 30 on the map image, it is possible to display the moving object mark 36 and the locus line 38 as shown in FIG.

  Further, the coordinate value on the x-axis obtained by Equation 8 and the coordinate value on the y-axis obtained by Equation 9 are substituted into Equation 6, so that the head height z of the human 30 in real space is obtained. The following number 11 is derived for obtaining.

  Similarly, by substituting the coordinate value on the x-axis obtained by Equation 8 and the coordinate value on the y-axis obtained by Equation 9 into Equation 7, the height of the head of the human 30 in the real space can also be calculated. The following formula 12 is derived for determining the length dimension z.

  In addition, by obtaining the height dimension z of the head of the human 30 in this way, the height of the human 30 can also be known. Moreover, the approximate center position of the person 30 can be obtained by halving the height dimension z.

  Further, whether or not the camera calibration is accurately performed can be verified as follows.

  That is, as shown in FIG. 14 (a), on the image taken by the fixed camera 12, any position that satisfies the condition of zero height, preferably the real space and the map image, is easy to understand (in other words, it becomes a mark). ) The position Qi is designated by operating the mouse 18, for example. This figure shows a state in which the boundary portion between the southeast corner of the building and the ground is designated as the position Qi. Then, the image coordinate value (u, v) on the photographed image at the designated position Qi is specified. Then, the specified image coordinate value (u, v) is substituted into the above-described formula 8, whereby the coordinate value of the world coordinate on the x-axis is obtained. At the same time, the image coordinate (u, v) is substituted into the above-mentioned equation 9, whereby the coordinate value of the world coordinate on the y-axis is obtained.

  Then, the two-dimensional coordinate values (α, β) in the map image are obtained by substituting the coordinate values in the x-axis and y-axis of the world coordinates obtained in this way into the above-described Expression 10. . And as shown in FIG.14 (b), the marker 46 is displayed on the position Qj according to the said two-dimensional coordinate value ((alpha), (beta)) on a map image. If the position Qj of the marker 46 corresponds to the designated position Qi in the captured image of FIG. 14A, the camera calibration has been performed accurately.

On the other hand, for example, as shown in FIG. 15, when the position Qj of the marker 46 on the map image does not correspond to the position Qi in the captured image of FIG. 14A, the camera calibration is not accurately performed. become. In this case, for example, by operating the mouse 18, the original position Qj shown in FIG. 14B is designated again on the map image. Then, the two-dimensional coordinate value (α, β) at the position Qj is converted into the world coordinate value (x, y, 0) by the above-described formula 2. Based on the combination of the converted world coordinate value (x, y, 0) and the image coordinate value (u, v) of the position Qi in the captured image of FIG. Is called. Specifically, the above-described simultaneous equations of Equations 4 and 5 are solved based on this new combination and the previously obtained combination. In this case, for each of the camera parameters p ₁ ~p _11, since a plurality of solutions are determined, for example, by least squares regression analysis, such as, the best value is determined as the respective camera parameter p ₁ ~p ₁₁ It is done. By making such correction, the accuracy of camera calibration is improved.

  In order to verify the accuracy of the camera calibration, for example, as shown in FIG. 16, an appropriate scale, for example, a grid grid 48 can be displayed on the image captured by the fixed camera 12. That is, a plurality of world coordinate values (x, y, 0) are set at regular intervals under the condition that z = 0 in the imaging region of the fixed camera 12 on the world coordinates. Then, by substituting each world coordinate value (x, y, 0) into the above-described equations 4 and 5, the image coordinate values (u, v) corresponding to each are obtained. Then, the positions (coordinate points) according to these image coordinate values (u, v) are connected to each other by straight lines 50, 50,..., So that a grid 48 composed of these straight lines 50, 50,. Is displayed. In FIG. 16, since the grid 48 extends along the fence and the wall of the building parallel to the x axis in the real space, it is recognized that the camera calibration is accurately performed.

  Furthermore, in the present embodiment, the installation conditions of the respective revolving cameras 14, specifically, the world coordinate values (x ′, y ′, z ′) of the installation position Qk of the revolving camera 14 in the real space illustrated in FIG. ) And the reference direction E can be easily specified by using the map image described above. In other words, the revolving camera 14 is installed on a pole (not shown) or a wall of a building, but is installed without taking into consideration the installation position Qk, particularly the world coordinate values (x ′, y ′, z ′). Sometimes. Further, in the turning camera 14, a predetermined reference direction E is determined, and the direction of the turning camera 14 is controlled by the pan angle θ and the tilt angle ρ with respect to the reference direction E. However, the reference direction E may be installed without any consideration. For example, in order to perform the above-described automatic tracking using the world coordinates (x, y, z) in the real space, the world coordinate values (x ′, y ′, z ′) and the reference direction E of these installation positions Qk are Must be known. In other words, if the world coordinate values (x ′, y ′, z ′) and the reference direction E of these installation positions Qk are known, automatic tracking using the world coordinates (x, y, z) can be performed. it can. An example of realizing automatic tracking using the world coordinates (x, y, z) is disclosed in, for example, Japanese Patent Application Laid-Open No. 2005-3377. Therefore, in the present embodiment, a method of easily specifying the world coordinate values (x ′, y ′, z ′) and the reference direction E of the installation position Qk of the turning camera 14 by using a map image. Will also be described.

Referring to FIG. 17 again, for example, now the turning camera 14 turns in the pan direction around a straight line parallel to the z-axis in the world coordinates, and is parallel to the x-axis / y-axis plane of the world coordinates. It is assumed that it is installed so as to turn in the tilt direction around a straight line. In addition, the reference direction E is parallel to the x-axis / y-axis plane of the world coordinates, and is offset by a certain angle, that is, an offset with respect to the direction along the x-axis in a plane parallel to the x-axis / y-axis plane. Assume that the angle θ ₀ is formed. The unit of the pan angle θ and the tilt angle ρ is [rad] (radian). As for the pan angle θ, the counterclockwise direction is plus (+) when the revolving camera 14 is viewed from above, and the tilt angle ρ is plus (+) below the horizontal. Suppose there is. The pan angle θ and the tilt angle ρ can be obtained from the turning camera 14.

  In such a state, it is assumed that the turning camera 14 is gazing at a certain position Qm in the real space, and the world coordinate value of the gazing point Qm is (x ″, y ″, z ″). The following equations 13 and 14 are established.

According to these equations 13 and 14, when the turning camera 14 is facing the world coordinate value (x ″, y ″, z ″) of the gazing point Qm and the gazing point Qm, the gazing point Qm is described in detail. When the pan angle θ and tilt angle ρ of the swivel camera 14 are given when the is present on the optical axis of the swivel camera 14, the world coordinate values (x ′, y ′, z ′) and the above-described offset angle θ ₀ are used as two unknowns, and therefore, the world coordinate values (x ″, y ″, z ″) of the gazing point Qm and the turning camera If at least two or more combinations of 14 pan angles θ and tilt angles ρ are obtained, the four unknowns x ′, y ′, z ′ and θ ₀ can be obtained.

  Therefore, first, as shown in FIG. 18, on the map image described above, for example, a position Qn that is easy to understand at any position by the operation of the mouse 18, preferably on the real space and the image taken by the turning camera 14, Designated as a point of interest. Then, the two-dimensional coordinate value (α, β) on the map image at the gazing point Qn is specified, and the two-dimensional coordinate value (α, β) is substituted into the above-described formula 2, so that FIG. The world coordinate value (x ″, y ″, z ″) of the gazing point Qm in the real space shown in FIG. 17 is obtained. The height z ′ of the gazing point Qm in the real space is zero. If it is not zero, the measured value of the height z ′ is set by operating the keyboard 20, for example.

Subsequently, the turning camera 14 is directed to the gazing point Qm in the real space by manual control. At this time, the pan angle θ and the tilt angle ρ of the turning camera 14 are acquired. By this series of operations, a combination of the world coordinate value (x ″, y ″, z ″) of the gazing point Qm and the pan angle θ and the tilt angle ρ of the turning camera 14 is obtained. By performing the same operation twice in total, a total of two combinations of the world coordinate values (x ″, y ″, z ″) of the gazing point Qm and the pan angle θ and the tilt angle ρ of the turning camera 14 can be obtained. It is done. Then, based on the simultaneous equations composed of the two sets of combinations, the world coordinate values (x ′, y ′, z ′) and the offset angle θ ₀ of the installation position Qk of the turning camera 14 are obtained.

Thus, by obtaining the world coordinate value (x ′, y ′, z ′) of the installation position Qk of the turning camera 14 and the offset angle θ ₀ , any world coordinate value (x, y in real space) is obtained. , Z), a pan angle θ and a tilt angle ρ for directing the turning camera 14 to a position corresponding to the world coordinate value (x, y, z) are obtained. In other words, the pan angle θ is obtained by the following equation 15 which is a modified equation of the above equation 13, and the tilt angle ρ is obtained by equation 16 which is the modified equation of the equation 14.

  In addition, it can be verified as follows whether or not the installation conditions of the revolving camera 14 are specified accurately.

  That is, as shown in FIG. 19A, an arbitrary position on the map image, preferably a position Qo that is easy to understand in any of the real space and the image taken by the turning camera 14 is designated by operating the mouse 18, for example. Is done. This figure shows a state where the boundary portion between the southeast corner of the building and the ground is designated as the position Qo. Then, the two-dimensional coordinate value (α, β) on the map image of the designated position Qo is specified, and further, the specified two-dimensional coordinate value (α, β) is converted into a world coordinate value ( x, y, 0). Then, the converted world coordinate values (x, y, 0) are expressed as the world coordinate values (x ″, y ″, z ″) of the gazing point Qm shown in FIG. By substituting, the pan angle θ and the tilt angle ρ for directing the turning camera 14 to the gazing point Qm are obtained, and the turning camera 14 is based on the obtained pan angle θ and tilt angle ρ. 19B, the position Qp corresponding to the gazing point Qm is displayed at the center of the captured image as shown in Fig. 19B, and thus the position Qp is displayed at the approximate center of the captured image. It is confirmed that the installation conditions of the revolving camera 14 have been specified accurately.

On the other hand, when the position Qp is not projected at the approximate center on the image captured by the revolving camera 14, the installation conditions for the revolving camera 14 are not accurately specified. In this case, for example, the orientation of the swivel camera 14 is changed by manual control so that the position Qp is projected at the approximate center on the image captured by the swivel camera 14. Then, the pan angle θ and the tilt angle ρ of the revolving camera 14 after the change are acquired. As a result, a combination of the current world coordinate value (x ″, y ″, z ″) of the gazing point Qm and the pan angle θ and tilt angle ρ of the turning camera 14 is obtained. Based on the previously obtained combination, the world coordinate value (x ′, y ′, z ′) of the installation position Qk of the turning camera 14 and the offset angle θ ₀ are obtained again. In this correction, a regression analysis method such as a least square method may be used as in the above-described correction of the camera calibration.

Thus, according to the present embodiment, the gazing point Qm is set at an arbitrary position in the real space, and the world coordinate value (x ″, y ″, z ″) at the gazing point Qm and the gazing point Qm are set. Based on the pan angle α and the tilt angle ρ when the turning camera 14 is pointed, that is, based on a geometric calculation, the world coordinate value (x ′ of the installation position Qk of the turning camera 14). , Y ′, z ′) and the offset angle θ _0. Therefore, the swivel camera is determined by a surveying device, by using a manual tool such as a measure or a magnetic compass, or by visual measurement. Compared with the above-described prior art in which the installation condition is specified, the specification is performed more easily and accurately, and the world coordinate values (x ″, y ″, z ″) of the gazing point Qm in the real space are Required using map image Therefore, the efficiency of the work for the identification is further improved.

  Even without using a map image, the world coordinate values (x ″, y ″, z ″) of the gazing point Qm in the real space can be directly obtained, and thus the position condition of the turning camera 14 can be specified. However, it goes without saying that using map images is more efficient.

In the above-described camera calibration, the projective transformation matrix is used, but the present invention is not limited to this. For example, the known literature [RYTsai, “A versatile camera calibration technique for
high-accuracy 3D machine vision metrology using off-the shelf TV cameras and
lenses, ”IEEE Journal of Robotics and Automation, vol.RA-3, no.4, pp.323-331,
Aug. 1987] may be employed.

  Further, in the camera calibration, a human having a known height is used as an index. However, an index other than a human, for example, a rod-shaped object or a cone-shaped object may be used. Further, without preparing a special index, an existing one (part) having an appearance characteristic such as a corner or a gate of a building may be used as the index. However, also in this case, it is necessary that the dimension of the index (particularly the height dimension) is known.

  And the scaler 40 shown in FIG. 8 is one setting example, and is not limited to this. That is, in FIG. 8, the scaler 40 extends along the vertical direction, but may extend along other directions such as a horizontal direction and an oblique direction. However, as described above, it is necessary that the actual size of the portion corresponding to the scaler 40 in the real space is accurately known. In other words, as long as the actual size is accurately known, the scaler 40 is not limited to a straight line but may be a curved line. In short, the scaler 40 and the portion corresponding to the scaler 40 in the real space are geometrically similar, and the respective length dimensions Lm and Lw may be accurately known.

  Moreover, in this embodiment, although this invention was applied to the monitoring use in the site including a building, it is not restricted to this. For example, the present invention may be applied to monitoring in a building, or the present invention may be applied to a purpose other than monitoring.

  The above-described camera calibration and specifying the installation conditions of the turning camera 14 are realized by the PC 16, but may be realized by a dedicated device, for example.

  Further, for example, as shown in FIG. 6, the function of displaying the moving object mark 36 and the locus line 38 and the automatic tracking function may be realized by separate devices. Further, even if not all devices are separate, for example, only the display 24 displays a map image including the moving object mark 36 and the locus line 38 and an image captured by the revolving camera 14 performing automatic tracking. You may provide separately what makes it display, and what displays the picked-up image by the fixed camera 12. FIG.

  Note that the automatic tracking function, including the above-described moving body detection technology, is relatively complicated and expensive to implement this, and instead of this, it is a kind of manual operation called the following map point view function. A tracking function may be provided.

  That is, according to the map point view function, a position that the operator wants to pay attention to on the map image, for example, an arbitrary position Qq as shown in FIG. 20A is specified by operating the mouse 18 or the keyboard 20. Then, the turning camera 14 capable of photographing the position Qs (strictly, the position Qr) toward the position Qs obtained by adding a height dimension γ described later to the position Qr in the real space corresponding to the designated position Qq. For example, the turning camera 14 in the southwest corner a shown in FIG. 2 is automatically turned. As a result, from the turning camera 14 in the southwest corner a, a captured image as shown in FIG. 20B, specifically, the position Qs is projected in the center, and the subject interface (equivalent imaging) including the position Qs is displayed. A captured image having a constant width dimension δ of (surface) is obtained. Note that the captured image in FIG. 20B shows a state in which the person 60 exists at the position Qs in real space (that is, the position Qr). In addition, although not shown, a similar captured image, that is, a position Qs is projected in the center from the revolving camera 14 in the southeast corner b, and a captured image in which the width dimension δ of the subject interface including the position Qs is constant. Is obtained.

  In order to realize this map point view function, a shootable area is set in advance on the map image for each of the revolving cameras 14. For example, for the revolving camera 14 in the southwest corner a, a polygonal photographable region including a south region and a west region is set as shown by the hatched pattern 62 in FIG. Specifically, when each vertex of the polygon is designated by the operation of the mouse 18 or the keyboard 20, a closed region connecting these vertices is set as the imageable region 62. Similarly, for each of the other revolving cameras 14, the imageable area 62 is appropriately set.

  Further, referring to FIG. 22, the height dimension γ from the position Qr to the position Qs is arbitrarily set in units of [mm] by the operation of the keyboard 20, for example. In addition, the width dimension δ of the subject interface including the position Qs is also arbitrarily set in [mm] units by the operation of the keyboard 20. Note that the width dimension δ of the subject interface including the position Qs is between the distance ε from the turning camera 14 forming the subject interface to the position Qs and the angle of view φ of the turning camera 14. , Which is expressed by the following equation (17).

  The height dimension γ and the width dimension δ of the object interface are appropriately set on the assumption that the person 60 is the main object to be photographed, as shown in FIG. For example, as shown in FIG. 20B, it is convenient to grasp the overall appearance of the person 60 if the whole body of the person 60 is projected almost completely in the captured image. In order to realize this, it is appropriate that the width dimension δ of the subject interface is δ = 3000 [mm] and the height dimension γ is γ = 1000 [mm]. Further, for example, when the width dimension δ of the subject interface is δ = 1000 [mm] and the height dimension γ is γ = 1500 [mm], the face of the person 60 is photographed as shown in FIG. The image is projected relatively large in the approximate center of the image. This aspect is suitable for grasping the facial features and expressions of the human 60. Further, for example, when the width dimension δ of the subject interface is δ = 5000 [mm] and the height dimension γ is γ = 800 [mm], as shown in FIG. A relatively small image appears in the center of the image. This aspect is suitable for sensibly recognizing the overall appearance (atmosphere) of the person 60 and predicting the action of the person 60. Note that the height γ and the visual field size δ in the horizontal direction can be arbitrarily set for each turning camera 14.

  As described above, for each of the swivel cameras 14, the image-capable area 62 is set, and the height dimension γ and the width dimension δ of the object interface are set, thereby enabling the map point view function. That is, when an arbitrary position Qq on the map image is specified as described above, the two-dimensional coordinate values (α, β) on the map image at the specified position Qq are specified. Then, by substituting the two-dimensional coordinate value (α, β) of the specified position Qq into the above-described equation 2, the world coordinate value (x, y, x) of the position Qr in the real space corresponding to the position Qq. 0) is required. Further, by adding the height dimension γ to the world coordinate value (x, y, 0) of the position Qr, the world coordinate value (x, y, γ) of the position Qs serving as the target point is obtained.

On the other hand, from the two-dimensional coordinate values (α, β) of the position Qq on the map image, the turning camera 14 that includes the position Qq in the imageable region 62 is specified. Then, the world coordinate value (x ′, y ′, z ′) of the specified installation position Qk of the turning camera 14 and the offset angle θ ₀ are substituted into the above-described Expressions 15 and 16, and the target point By substituting the world coordinate values (x, y, γ) of the position Qs to (x ″, y ″, z ″) in the equations (15) and (16), the identified turning type camera 14 is set to the target point. The pan angle θ and the tilt angle ρ to be directed to the position Qs to be obtained are obtained, and the turning camera 14 turns based on the obtained pan angle θ and tilt angle ρ. Is directed to a position Qs that is a target point.

  Furthermore, the world coordinate value (x ′, y ′, z ′) of the installation position Qk of the turning camera 14 directed to the position Qs serving as the target point, and the world coordinate value (x, y, γ) of the position Qs. ) And the distance ε between the two is obtained. Then, the distance ε and the preset width dimension δ of the subject interface are substituted into the following equation 18 which is the above-described deformation equation 17 to be directed to the position Qs as the target point. The angle of view φ of the revolving camera 14 is obtained, and the angle of view φ (zoom magnification) of the revolving camera 14 is controlled accordingly.

As a result, a photographed image having the constant width dimension δ of the subject interface including the position Qs and the position Qs as a center is obtained from the turning camera 14 directed to the position Qs as the target point. That is, a series of map point view functions are realized. As with the automatic tracking function described above, the map point view function has the installation conditions (the world coordinate values (x ′, y ′, z ′) and the offset angle θ _{0 of the} installation position Qk) of each turning camera 14. Only when it is accurately identified can it be used.

  In this map point view function, several combinations of the height dimension γ and the width dimension δ of the object interface described above are preset (strictly, a memory circuit (not shown) in the PC 16 or the like). And a desired one of these combinations may be selected. In this case, in particular, as shown in FIG. 25, a plurality of types of icons 70, 72 and 74, for example, “A”, “B” and “C”, are displayed on the display 24. When any one of 72 and 74 is selected (clicked) on the display 24, the height dimension γ and the width dimension δ of the subject interface corresponding thereto may be automatically set. Specifically, when the icon 70 “A” is selected, the mode shown in FIG. 23 is obtained. When the icon 72 “B” is selected, the mode shown in FIG. When the icon 74 is selected, the mode shown in FIG. 24 may be obtained.

  Each icon 70, 72, and 74 shown in FIG. 25 is an example as an operator, and is not limited to this. For example, in FIG. 25, these icons 70, 72, and 74 are provided with symbols (symbols) simulating a human figure. Instead, other icons simulating the outer shape of a loupe or a building are used. You may adopt a design. In addition, instead of the software-like operation elements referred to as icons 70, 72, and 74, hardware-like operation elements (electronic parts) such as push buttons and switches may be employed, and appropriate symbols may be attached thereto. .

  Further, as shown in FIG. 26, marks 80, 80,... Representing the revolving cameras 14, 14,... May be displayed on the map image. Further, in this case, a substantially fan-shaped separate so as to connect the position Qq designated on the map image and the mark 80 representing the turning camera 14 directed to the position Qs as the target point corresponding to the position Qq. The mark 82 may be displayed on the map image. This general fan-shaped mark 82 represents in a simulated manner and in what form (particularly the angle of view φ) the position Qs as the target point is captured by which turning type camera 14. The spread angle (center angle) φ ′ from the mark 80 corresponding to the turning camera 14 toward the designated position Qq varies depending on the actual angle of view φ of the turning camera 14. FIG. 26 shows a state in which the position Qs as the target point corresponding to the designated position Qq is photographed with the angle of view φ corresponding to the spread angle φ ′ by the turning camera 14 in the southwest corner a. . Further, although not shown, a substantially fan-shaped mark 82 that connects the designated position Qq to the mark 80 representing the turning camera 14 in the southeast corner b is also displayed.

Claims (3)

A camera calibration result for obtaining a camera parameter for associating a two-dimensional image coordinate set on a photographed image by a camera with a fixed photographing region and a three-dimensional world coordinate set in a real space including the photographing region. A verification device,
Simulating the real space in a plane in which the value of a predetermined coordinate axis, which is one of the three coordinate axes constituting the world coordinate, is constant, and setting two-dimensional coordinates correlated with the world coordinate representing the plane Simulated image display means for displaying an image;
A designation means for designating an arbitrary position on the captured image;
First conversion means for converting a coordinate value on the image coordinates of the designated position by the designation means into coordinate values of the world coordinates on the plane based on the camera parameters;
Second conversion means for converting the coordinate value of the world coordinate converted by the first conversion means into the coordinate value of the two-dimensional coordinate based on the correlation between the world coordinate and the two-dimensional coordinate;
Marker display means for displaying a marker at a position on the simulated image according to the coordinate value of the two-dimensional coordinate converted by the second conversion means;
A camera calibration result verification device comprising: The predetermined coordinate axis defines the height direction of the real space,
The simulated image is an image simulating the plane where the height of the real space is zero.
The camera calibration result verification apparatus according to claim 1. A camera calibration result for obtaining a camera parameter for associating a two-dimensional image coordinate set on a photographed image by a camera with a fixed photographing region and a three-dimensional world coordinate set in a real space including the photographing region. A method of verifying,
Simulating the real space in a plane in which the value of a predetermined coordinate axis, which is one of the three coordinate axes constituting the world coordinate, is constant, and setting two-dimensional coordinates correlated with the world coordinate representing the plane A simulated image display process for displaying an image;
A designation process for designating an arbitrary position on the photographed image,
A first conversion step of converting a coordinate value on the image coordinates of the designated position in the designation step into a coordinate value of the world coordinate on the plane based on the camera parameter;
A second conversion step of converting the coordinate value of the world coordinate converted by the first conversion step into a coordinate value of the two-dimensional coordinate based on a correlation between the world coordinate and the two-dimensional coordinate;
A marker display step of displaying a marker at a position on the simulated image according to the coordinate value of the two-dimensional coordinate converted by the second conversion step;
A camera calibration result verification method comprising:

Images