JPWO2019193859A1 - Camera calibration method, camera calibration device, camera calibration system and camera calibration program - Google Patents

Abstract

In the moving image acquisition step, the moving image is acquired by moving the moving camera so as to overlap the field of view of the surveillance camera. In the procedure for deriving the calibration parameters for the moving camera, the calibration parameters for the moving camera are derived for each frame of the moving image. In the three-dimensional locus acquisition step, the three-dimensional movement locus of the moving camera in the reference coordinate system is obtained by using the calibration parameters for the moving camera. In the two-dimensional locus detection step, at least a part of the three-dimensional movement locus of the moving camera is detected as the two-dimensional movement locus on the captured image of the surveillance camera. In the surveillance camera calibration parameter derivation step, the surveillance camera calibration parameters are derived based on at least a part of the three-dimensional movement locus and the two-dimensional movement locus.

Description

The present invention relates to a camera calibration method, a camera calibration device, a camera calibration system and a camera calibration program for deriving calibration parameters for one or more surveillance cameras.

Conventionally, a distributed monitoring system has been widely used in which information recognized from captured images is shared among cameras distributed in a wide area environment, and the shared information is used for monitoring, crime prevention, or marketing. In the distributed monitoring system, in order to share the information recognized for each camera, after calibrating the parameters of each camera, the information recognized for each camera is expressed on the same reference coordinate system (or map), and the user Need to provide to. The parameters of the camera include, for example, the absolute position and orientation of the camera, the focal length of the lens, the distortion coefficient of the lens, and the time (shooting timing). In the following, calibrating the camera parameters (deriving the calibration parameters) is also simply referred to as camera calibration.

Here, as one of the methods for calibrating the camera, there is a method for manually calibrating the camera based on triangulation. However, since manual calibration requires a large human cost, automatic calibration is often used in recent years (see, for example, Patent Documents 1 to 6 and Non-Patent Document 1).

JP-A-2015-106287 Japanese Unexamined Patent Publication No. 2014-14912 Japanese Unexamined Patent Publication No. 2011-69996 Special Table 2009-509778 Japanese Unexamined Patent Publication No. 2010-210570 Japanese Unexamined Patent Publication No. 2005-241323

Hidehiko Shishido et al., "Calibration Method for Sparsely Arranged Multiview Cameras", 20th Image Recognition and Understanding Symposium, August 10, 2017

In each of the automatic calibration techniques of Patent Documents 1 to 4, one of the images to be photographed by at least one camera and the camera is moved relative to the other, and the image of the object to be photographed is acquired by the camera. It is a technique to calibrate the camera based on the created image. When using such a calibration technique, the moving object must be pre-calibrated relative to the camera to be calibrated (if the object is not calibrated, the camera is calibrated based on the image of the object to be calibrated). Because it cannot be done).

For example, as in Patent Document 1, when two cameras (stereo cameras) are moved relative to an object, the reference coordinate system (world coordinate system) of the object that moves relative to the two cameras. ) Must be pre-calibrated. Further, as in Patent Document 2, when a movable robot is photographed with a camera and the parameters of the camera are calibrated, the position of the robot moving relative to the camera in the reference coordinate system must be calibrated in advance. .. Similarly, in Patent Documents 3 and 4, the position of the moving body photographed by the camera in the reference coordinate system must be calibrated in advance.

As described above, in the automatic calibration techniques of Patent Documents 1 to 4, before calibrating the camera, it is necessary to prepare in advance for calibrating the object to be photographed that moves relative to the camera. Cannot calibrate. Therefore, it is inconvenient because the camera cannot be calibrated immediately when it is needed. This problem also arises in Patent Documents 5 and 6.

That is, in Patent Document 5, a GPS (Global Positioning System) device is mounted on a subject (moving object) to be photographed by two cameras, the subject is moved in a predetermined space, and a predetermined subject is moved. A technique for stopping at a position and acquiring calibration data based on the stop position of the subject and the captured image of the subject by two cameras is disclosed. However, also in this technique, it is necessary to calibrate the coordinate system of the GPS device in advance with respect to the reference coordinate system.

In Patent Document 6, a plurality of markers (three-dimensional coordinates are known) in space are photographed by a plurality of fixed cameras, and a plurality of markers are photographed from the observed coordinates on the captured image of the markers and the known three-dimensional coordinates of the markers. A technique for calculating the projection conversion matrix of a fixed camera of the above is disclosed. However, also in this technique, it is necessary to obtain the three-dimensional coordinates of the marker in advance, that is, to calibrate the position of the marker in advance with respect to the reference coordinate system.

Further, in Non-Patent Document 1, a dense multi-viewpoint image group is constructed by integrating images taken by a moving camera and images taken by a plurality of sparsely arranged still cameras, and the multi-viewpoint images are used as each other. A technique for calibrating a still camera from information on corresponding points (weak calibration technique) is disclosed. However, in the technique of Non-Patent Document 1, since the still camera is calibrated by finding the corresponding point by comparing the images, high accuracy is required for the detection of the corresponding point in order to improve the calibration accuracy. The accuracy of calibration tends to be lower than that of the techniques 1 to 6.

The above problem may occur not only in a distributed monitoring system in which a plurality of cameras to be calibrated are provided, but also in a system in which only one camera (surveillance camera) is installed in the environment.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to reduce the accuracy of calibration of a surveillance camera without pre-calibrating a moving object to be imaged relative to the surveillance camera. To provide a camera calibration method, a camera calibration device, a camera calibration system and a camera calibration program capable of calibrating a surveillance camera while avoiding it.

The camera calibration method according to one aspect of the present invention is a camera calibration method for deriving the calibration parameters of one or more surveillance cameras as calibration parameters for the surveillance camera, and the moving camera is placed so as to overlap the field of view of the surveillance camera. A moving image acquisition step of acquiring a moving image by moving the moving image, and a calibration parameter for the moving camera including the position and orientation of the moving camera in the reference coordinate system for each frame of the moving image based on the moving image. A three-dimensional locus acquisition step for obtaining a three-dimensional movement locus of the mobile camera in the reference coordinate system using the derivation step of the calibration parameter for the mobile camera to be derived, and the above three of the mobile camera. A two-dimensional locus detection step that detects at least a part of a dimensional movement locus as a two-dimensional movement locus on an image captured by the surveillance camera, at least a part of the three-dimensional movement locus, and the two-dimensional movement locus. It includes a step of deriving the calibration parameter for the surveillance camera, which derives the calibration parameter for the surveillance camera based on the movement locus.

The camera calibration program according to another aspect of the present invention is a program that causes a computer to execute the above camera calibration method.

The camera calibrator according to still another aspect of the present invention is based on the moving image of the moving camera that moves so as to overlap the field of view of one or more surveillance cameras and acquires the moving image, for each frame of the moving image. In addition, using the moving camera calibration parameter deriving unit for deriving the moving camera calibration parameters including the position and orientation of the moving camera in the reference coordinate system and the moving camera calibration parameters, the moving camera in the reference coordinate system. A two-dimensional movement locus acquisition unit that obtains the three-dimensional movement locus of the above, and a two-dimensional movement locus that detects at least a part of the three-dimensional movement locus of the moving camera as a two-dimensional movement locus on a captured image of the surveillance camera. Derivation of the calibration parameter for the surveillance camera that derives the calibration parameter for the surveillance camera, which is the calibration parameter of the surveillance camera, based on the trajectory detection unit, at least a part of the three-dimensional movement trajectory, and the two-dimensional movement trajectory. Including the part.

A camera calibration system according to still another aspect of the present invention includes the camera calibration device, one or more fixed cameras, and the mobile camera.

It is possible to calibrate the surveillance camera while avoiding a decrease in the accuracy of the calibration of the surveillance camera without pre-calibrating the image target moving relative to the surveillance camera (deriving the calibration parameters for the surveillance camera). Can be done).

It is a block diagram which shows the schematic structure of the camera calibration system of embodiment of this invention. It is explanatory drawing which shows typically an example of the movement locus of the moving camera included in the said camera calibration system. It is a flowchart which shows the process flow of the camera calibration method by the said camera calibration system. It is explanatory drawing which shows typically how the said moving camera moves in an environment. It is explanatory drawing which shows typically the image taken by the said moving camera at the position A of FIG. It is explanatory drawing which shows the example of the marker installed in the said environment. It is explanatory drawing which shows typically the correspondence relationship between the coordinate system of the moving camera and the reference coordinate system. It is explanatory drawing which shows typically the process of finding the position (absolute coordinate) in the reference coordinate system of the marker attached to the moving camera in order for each frame of the moving camera. It is explanatory drawing which shows the 3D movement locus in the reference coordinate system of the said marker which moves with the movement of the said moving camera. It is explanatory drawing which shows typically how the moving camera moves together with the said marker overlapping with the field of view of the surveillance camera included in the said camera calibration system. It is explanatory drawing which shows an example of the photographed image of the said surveillance camera schematically. It is a block diagram which shows the other configuration of the said camera calibration system. It is a flowchart which shows the process flow of the camera calibration method by the camera calibration system of FIG. It is explanatory drawing which shows the other structure of the marker attached to the moving camera schematically. It is a three-dimensional movement locus of the moving camera, and is explanatory drawing which shows the moving locus of a quadrangular shape. It is explanatory drawing which shows the 2D movement locus of the said moving camera on the photographed image of the said surveillance camera.

An embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to the following contents.

[Camera calibration system]
FIG. 1 is a block diagram showing a schematic configuration of the camera calibration system 1 of the present embodiment. The camera calibration system 1 is a system that calibrates the surveillance camera 2, that is, derives the calibration parameters of the surveillance camera 2 as calibration parameters for the surveillance camera, and includes the surveillance camera 2, the mobile camera 3, and the camera calibrator 4. have. The surveillance camera 2 and the mobile camera 3 are communicably connected to the camera calibrator 4 via a communication line (whether wired or wireless). Although FIG. 1 shows an example in which a plurality of surveillance cameras 2 are connected to the camera calibration device 4, only one surveillance camera 2 may be used.

The surveillance camera 2 is an imaging device that is installed in an environment to be monitored (for example, a store, a factory, or a room) and captures an image in the environment. The surveillance camera 2 is, for example, a fixed camera, but may be a camera capable of slight movement (for example, shift, rotation, rotation).

The mobile camera 3 is an imaging device that captures an image (moving image) by photographing the surroundings while moving, and includes a moving means. The moving means includes wheels, a shaft connected to the wheels, a motor for rotating the shaft, and the like. In the present embodiment, the moving camera 3 moves so as to overlap at least the field of view (shooting range) of the surveillance camera 2. FIG. 2 schematically shows an example of the movement locus of the moving camera 3. If the moving camera 3 moves so as to overlap the field of view of the surveillance camera 2, it may pass through the field of view of one surveillance camera 2 from the inside to the outside, or may include the field of view and the field of view of a plurality of surveillance cameras 2. You may move between. Such movement of the moving camera 3 is controlled by, for example, a movement control unit 5b described later of the camera calibration device 4.

Further, in the present embodiment, the marker M is attached to the moving camera 3. The marker M is, for example, an index M including feature points P (for example, corners) that can be recognized by the camera calibration device 4 (particularly the calibration processing unit 10) by the image recognition technique based on the image obtained by the surveillance camera 2. _Consists of 1. It is assumed that the relative position (coordinates) of the marker M with respect to the moving camera 3 is known.

The moving camera 3 may be moved by a user (for example, a person who calibrates the surveillance camera 2) while holding the moving camera 3. Even when the user's operation is involved in this way, the user only needs to perform the operation of moving the moving camera 3. That is, the user does not manually calibrate the surveillance camera 2 by performing triangulation using the instrument. Therefore, even when the user moves the moving camera 3, it can be said that the increase in human cost does not occur as in the case of manual calibration.

The camera calibrator 4 is a device that calibrates the surveillance camera 2 based on the captured image of the surveillance camera 2 and the captured image of the moving camera 3, and can be configured by, for example, a personal computer. The camera calibration device 4 includes a control unit 5, a storage unit 6, an input unit 7, a display unit 8, a communication unit 9, and a calibration processing unit 10.

The control unit 5 is composed of, for example, a central processing unit (CPU), and operates according to an operation program stored in the storage unit 6. The control unit 5 has a function as a main control unit 5a for controlling the operation of each unit of the camera calibration device 4 and a function as a movement control unit 5b for controlling the movement of the moving camera 3.

The storage unit 6 is a memory that stores an operation program for operating each part of the camera calibration device 4, data of each captured image acquired by the surveillance camera 2 and the mobile camera 3, and the like. The storage unit 6 is composed of, for example, a hard disk, but may be appropriately selected from recording media such as a RAM (Random Access Memory), a ROM (Read Only Memory), an optical disk, a magneto-optical disk, and a non-volatile memory. ..

The input unit 7 is composed of, for example, a keyboard, a mouse, a touch pad, a touch panel, and the like, and receives various instruction inputs by the user. The display unit 8 is a device that displays various information including each captured image acquired by the surveillance camera 2 and the mobile camera 3, and is composed of, for example, a liquid crystal display device. The communication unit 9 is an interface for communicating with the surveillance camera 2 and the mobile camera 3, and includes input / output terminals and the like. When, for example, at least one of the surveillance camera 2 and the mobile camera 3 and the camera calibrator 4 wirelessly communicate with each other (for example, transmission / reception of image data), the communication unit 9 includes an antenna, a transmission / reception circuit, a modulation circuit, and a demodulation circuit. Etc. may be included in the configuration.

The calibration processing unit 10 is a block that performs processing related to calibration of the surveillance camera 3, and is a calibration parameter derivation unit 10a for a mobile camera, a three-dimensional trajectory acquisition unit 10b, a two-dimensional trajectory detection unit 10c, and a calibration parameter for a surveillance camera. It is configured to include a derivation unit 10d. The calibration processing unit 10 is composed of, for example, a GPU (Graphics Processing Unit) which is an arithmetic unit specialized in real-time image processing, but may be composed of the same CPU as the control unit 5 or a separate CPU, and other CPUs. It may be composed of an arithmetic unit. The details of each part of the calibration processing unit 10 will be described in the following description of the camera calibration method.

[Camera calibration method]
Next, the camera calibration method of the present embodiment will be described. FIG. 3 is a flowchart showing a processing flow of the camera calibration method by the camera calibration system 1 of FIG. The camera calibration method of the present embodiment includes a moving image acquisition step (S1), a calibration parameter derivation step for a moving camera (S2), a three-dimensional trajectory acquisition step (S3), and a two-dimensional trajectory detection step (S4). It includes a calibration parameter derivation step (S5) for a surveillance camera. Hereinafter, a more detailed description will be given.

As described above, it is assumed that the moving camera 3 is attached with _{the index M 1 as the marker M.} Further, it is assumed that the time (shooting timing) is calibrated in advance here. That is, it is assumed that the surveillance camera 2 and the mobile camera 3 have the same shutter timing at the time of shooting.

(S1; moving image acquisition step)
In S1, the movement control unit 5b moves the moving camera 3 so as to overlap the field of view of the surveillance camera 2, and the moving camera 3 captures the environment to acquire an image (moving image). The data of the moving image acquired by the moving camera 3 is output to the camera calibration device 4 and stored in the storage unit 6 via the communication unit 9.

(S2; Calibration parameter derivation step for mobile camera)
In S2, the calibration parameter derivation unit 10a for the moving camera is based on the moving image acquired by the moving camera 3 (stored in the storage unit 6) for each frame of the moving image (corresponding to a different shooting timing or a different position). In, the calibration parameters including the position and orientation of the moving camera 3 in the reference coordinate system are derived. The calibration parameter of the mobile camera 3 is referred to as a calibration parameter for the mobile camera here. The calibration parameters for the mobile camera can be derived by using, for example, a known Visual-SLAM (Simultaneous Localization And Mapping). Visual-SLAM is a technology for estimating the three-dimensional shape of an environment or an object and the absolute or relative position / orientation of the camera at each frame from the moving image of the camera.

For example, FIG. 4 schematically shows how the moving camera 3 moves in the environment (room) along the path of the thick line, and FIG. 5 shows an image taken by the moving camera 3 at the position A in FIG. Is schematically shown. By using Visual-SLAM, from the moving image of the moving camera 3 (including the captured image of FIG. 5), the three-dimensional shape of the environment in which the moving camera 3 moves (the room shown in FIG. 4) and the moving camera 3 The position and orientation in the reference coordinate system (XYZ coordinate system) of can be obtained.

Here, the reference coordinate system shown in FIG. 4 may be a coordinate system whose origin is an arbitrary point of the three-dimensional shape of the environment obtained (restored) by Visual-SLAM. In addition, the obtained 3D shape of the environment is designed or set in advance on the reference coordinate system, and the environment data (index, landmark (geographical feature that serves as a mark), 3D point cloud, CAD data). , Wireframe model, etc.) and may be converted as a three-dimensional shape on the reference coordinate system. For example, as shown in FIG. 6, a marker M'(index) designed to correspond to the reference coordinate system is installed in the environment, and the marker M'is photographed by the moving camera 3 to recognize the position (coordinate system). By doing so, the obtained three-dimensional shape of the environment may be converted as a three-dimensional shape on the reference coordinate system. At this time, it is not necessary to photograph the marker M'with the surveillance camera 2.

(S3; 3D trajectory acquisition step)
In S3, the three-dimensional locus acquisition unit 10b obtains the three-dimensional movement locus of the moving camera in the reference coordinate system by using the calibration parameter for the moving camera obtained by the calibration parameter deriving unit 10a for the moving camera. In particular, in the present embodiment, the three-dimensional locus acquisition unit 10b uses the three-dimensional movement locus of the marker M, which moves with the movement of the moving camera 3, in the reference coordinate system as the three-dimensional movement locus of the moving camera 3. Ask. The three-dimensional movement locus refers to a set of three-dimensional position coordinates (points) in the reference coordinate system in each frame of the moving camera 3 (marker M).

FIG. 7 schematically shows the correspondence between the coordinate system of the moving camera 3 (moving camera coordinate system) and the reference coordinate system. In the drawing, for convenience, the three axes orthogonal to each other in the moving camera coordinate system are shown by the x-axis, y-axis, and z-axis, and the three axes (X-axis, Y-axis, and Z-axis) of the reference coordinate system are defined. Distinguish. As described above, the mobile camera calibration parameter derivation unit 10a uses Visual-SLAM to obtain the absolute position / orientation (in the reference coordinate system) of the mobile camera 3 in the time series (reference coordinate system and mobile camera coordinates). Coordinate transformation to and from the system is known). Therefore, if the relative position of the marker M (considering the feature point here) with respect to the moving camera 3 is known, that is, if the three-dimensional position of the marker M on the moving camera coordinate system is known, the position of the marker M is set. It can be mapped on the reference coordinate system. That is, the absolute coordinates of the marker M on the reference coordinate system can be calculated by coordinate transformation.

Therefore, as shown in FIG. 8, the three-dimensional locus acquisition unit 10b repeats the above coordinate conversion for each frame of the moving camera 3 with respect to the marker M moving together with the moving camera 3, and on the reference coordinate system of the marker M. By calculating the absolute coordinates of, as shown in FIG. 9, it is possible to obtain the three-dimensional movement locus T1 of the marker M that moves with the movement of the moving camera 3 in the reference coordinate system. Further, since the relative positional relationship between the marker M and the moving camera 3 is known, the moving locus T1 of the marker M can be considered to be equivalent to the three-dimensional moving locus of the moving camera 3. As a result, the three-dimensional locus acquisition unit 10b can obtain the movement locus T1 of the marker M as the three-dimensional movement locus of the moving camera 3.

(S4; 2D trajectory detection step)
In S4, the two-dimensional locus detection unit 10c detects at least a part of the three-dimensional movement locus of the moving camera 3 as a two-dimensional movement locus on the captured image of the surveillance camera 2. In particular, in the present embodiment, the two-dimensional locus detection unit 10c detects at least a part of the three-dimensional movement locus of the marker M (feature point) as the two-dimensional movement locus on the captured image of the surveillance camera 2. .. The two-dimensional movement locus refers to a set of two-dimensional position coordinates (points) of the moving camera 3 or the marker M on the captured image of each frame of the surveillance camera 2. Further, the data of the captured image acquired by the surveillance camera 2 is output to the camera calibration device 4 and stored in the storage unit 6 via the communication unit 9. As a result, the two-dimensional locus detection unit 10c can grasp the captured image of the surveillance camera 2 with reference to the storage unit 6.

The “at least a part of the three-dimensional movement locus of the moving camera 3” refers to a portion of the three-dimensional movement locus of the moving camera 3 that overlaps the field of view of the surveillance camera 2. Therefore, when the moving camera 3 moves inside and outside the field of view of the surveillance camera 2, the two-dimensional locus detection unit 10c captures a part of the three-dimensional movement locus of the moving camera 3 (marker M) by the surveillance camera 2. It will be detected as a two-dimensional movement trajectory on the image. When the moving camera 3 moves only within the field of view of the surveillance camera 2, the two-dimensional locus detection unit 10c captures the entire three-dimensional movement locus of the moving camera 3 (marker M) as an image captured by the surveillance camera 2. It will be detected as a two-dimensional movement locus above.

FIG. 10 schematically shows how the moving camera 3 moves together with the marker M so as to overlap the field of view of the surveillance camera 2. Further, FIG. 11 schematically shows an example of the captured image 2a of the surveillance camera 2. By photographing the moving camera 3 (marker M) by the surveillance camera 2, the position coordinates of the moving camera 3 (marker M) on the captured image 2a of the surveillance camera 2 are in chronological order as the moving camera 3 moves. can get. Therefore, the two-dimensional locus detection unit 10c maps the time-series position coordinates of the moving camera 3 (marker M) in the captured image 2a of the surveillance camera 2 to map the moving camera 3 (marker M) on the captured image 2a. The two-dimensional movement locus T2 can be detected. That is, the two-dimensional locus detection unit 10c can detect the three-dimensional movement locus T1 of the moving camera 3 as the two-dimensional movement locus T2 on the captured image 2a of the surveillance camera 2.

(S5; Calibration parameter derivation step for surveillance cameras)
In S5, the calibration parameter derivation unit 10d for the surveillance camera has at least a part of the three-dimensional movement locus of the moving camera 3 (three-dimensional movement locus T1 of the marker M) and the two-dimensional movement locus of the moving camera 3 (marker). Based on the two-dimensional movement locus T2) of M, the calibration parameter for the surveillance camera, which is the calibration parameter of the surveillance camera 2, is derived. The calibration parameters for the surveillance camera include external parameters (for example, position and orientation) and internal parameters (for example, focal length of the lens and distortion of the lens) of the surveillance camera 2.

Here, the problem of estimating the parameters of the camera from the set of the three-dimensional coordinates of the plurality of reference markers and the two-dimensional coordinates on the corresponding image is called the PnP (Perspective n-Point) problem. How to solve the problem has been known in the past (see, for example, "Masatoshi Okutomi," Digital Image Processing ", Revised New Edition, Image Information Education Promotion Association, March 2006, p328-331"), and also in this embodiment. This solution can be adopted. That is, the calibration parameter derivation unit 10d for the surveillance camera uses at least a part of the three-dimensional movement locus of the mobile camera 3 and the two-dimensional movement locus of the mobile camera 3 to solve the PnP problem for the surveillance camera. Calibration parameters can be derived.

〔effect〕
As described above, the camera correction method of the present embodiment is a method of deriving the calibration parameters of one or more surveillance cameras 2 as the calibration parameters for the surveillance cameras, and includes the above-mentioned moving image acquisition step (S1) and the above-mentioned moving image acquisition step (S1). The movement camera calibration parameter derivation step (S2), the three-dimensional locus acquisition step (S3), the two-dimensional locus detection step (S4), and the surveillance camera calibration parameter derivation step (S5) are included.

In this method, it is not necessary to calibrate the imaging target (moving camera 3) that moves relative to the surveillance camera 2 in advance with respect to the reference coordinate system. That is, in the present embodiment, the moving camera calibration parameter is derived (moving camera) in the process (S2) in the middle of starting the movement of the moving camera 3 in S1 and deriving the surveillance camera calibration parameter in S5. 3), it is not necessary to calibrate the position of the moving camera 3 with reference to the reference coordinate system before S1 to start the movement of the moving camera 3. Therefore, when calibrating the surveillance camera 2, the moving camera 3 can be immediately moved to calibrate the surveillance camera 2, and the convenience can be improved.

Further, as compared with the method of calibrating the camera by finding the corresponding points by comparing the images (weak calibration), it is not necessary to find the corresponding points, so that the accuracy of calibration can be improved. That is, the surveillance camera 2 can be calibrated while avoiding a decrease in the accuracy of the calibration of the surveillance camera 2 (the calibration parameter for the surveillance camera can be derived).

Further, in S3, the three-dimensional locus acquisition unit 10b obtains the three-dimensional movement locus T1 in the reference coordinate system of the marker M that moves with the movement of the moving camera 3 as the three-dimensional movement locus of the moving camera 3. In S4, the two-dimensional locus detection unit 10c detects at least a part of the three-dimensional movement locus T1 of the marker M as the two-dimensional movement locus T2 on the captured image 2a of the surveillance camera 2. By using the marker M, the two-dimensional locus detection unit 10c can easily recognize the marker M from the captured image 2a of the surveillance camera 2, so that at least a part of the three-dimensional movement locus T1 of the marker M can be obtained. It becomes easy to detect it as a two-dimensional movement locus T2 on the captured image 2a. As a result, the surveillance camera calibration parameter derivation unit 10d can accurately derive the surveillance camera calibration parameter based on at least a part of the three-dimensional movement locus T1 and the two-dimensional movement locus T2.

Further, for example, when a human holds and moves the moving camera 3, the marker M is not attached to the moving camera 3, and the human shape is recognized by image recognition on the captured image 2a of the surveillance camera 2 and the moving camera 3 is used. It is also possible to estimate the position and the like. However, since it is difficult to recognize a shape or feature point peculiar to a human on an image, there is a possibility of erroneously recognizing a human. In this regard, by attaching the marker M to the moving camera 3 and moving it, the recognition accuracy of the marker M on the image can be improved, and in this respect as well, the accuracy of deriving the calibration parameters for the surveillance camera can be improved. it can.

_{Further, the marker M is an index M 1} including a feature point P (see FIG. 2) that can be recognized (by the two-dimensional locus detecting unit 10c) based on an image taken by the surveillance camera 2. Since the marker M is the index M ₁ , the two-dimensional locus detection unit 10c can easily recognize the feature point P from the captured image 2a of the surveillance camera 2 by image recognition, and can easily recognize the marker M. It becomes. As a result, the effect of improving the accuracy of deriving the calibration parameters for the surveillance camera described above can be surely obtained.

[Other configurations of camera calibration system]
FIG. 12 is a block diagram showing another configuration of the camera calibration system 1 of the present embodiment. The camera calibration system 1 of FIG. 12 has the same configuration as the camera calibration system 1 of FIG. 1 except that the calibration processing unit 10 of the camera calibration device 4 further includes a two-dimensional point group detection unit 10e. The details of the two-dimensional point group detection unit 10e will be described in the following description of the camera calibration method.

FIG. 13 is a flowchart showing a processing flow of the camera calibration method by the camera calibration system of FIG. In the above camera calibration method, S2'is performed instead of S2, S5'is performed instead of S5, and the two-dimensional point group detection unit step (S4) by the two-dimensional point group detection unit 10e is performed between S4 and S5'. -1) is the same as the flow of the camera calibration method shown in FIG. 3 except that the procedure is further performed. Hereinafter, the points different from FIG. 3 will be described.

In S2', the mobile camera calibration parameter derivation unit 10a derives the mobile camera calibration parameters described above using Visual-SLAM based on the moving image acquired by the mobile camera 3, and in addition, the surveillance camera. Estimate a group of three-dimensional points indicating the shape (natural feature points) of an object or environment in the shooting range of 2. For example, the calibration parameter derivation unit 10a for a mobile camera uses Visual-SLAM to estimate a three-dimensional point cloud indicating the three-dimensional shape of a landmark within the shooting range of the surveillance camera 2 from the moving image. In S4-1, the two-dimensional point cloud detection unit 10e sets the position of the three-dimensional point cloud estimated by the mobile camera calibration parameter derivation unit 10a on the captured image 2a of the surveillance camera 2 as the two-dimensional point cloud. To detect.

Then, in S5', the surveillance camera calibration parameter derivation unit 10d derives the surveillance camera calibration parameter by solving the PnP problem using the three-dimensional information and the two-dimensional information. Here, the above three-dimensional information is the three-dimensional movement locus of the moving camera 3 (or marker M) acquired in S3, and the information of the three-dimensional point group estimated in S2', and is two-dimensional information. Is the information of the two-dimensional movement locus on the captured image 2a of the moving camera 3 detected in S4 and the information of the two-dimensional point group on the captured image 2a detected in S4-1.

As described above, by deriving the calibration parameters for the surveillance camera by using the information of the 3D and 2D point groups of the natural feature points in addition to the 3D and 2D movement loci of the moving camera 3. The amount of information used to derive the calibration parameters (PnP problem) compared to the case of deriving the calibration parameters for the surveillance camera based only on the movement locus of the moving camera 3 or based only on the information of the point group of the natural feature points. Since the number of points (the number of n) used when solving is increased, the accuracy of calibration of the surveillance camera 2 can be further improved.

[Other configurations of markers]
FIG. 14 schematically shows another configuration of the marker M attached to the mobile camera 3. As the marker M, a light source M ₂ that emits light may be used _{instead of the index M 1 described above.} As the light source M ₂ , for example, a set of a plurality of light emitting diodes (LEDs) that emit different colors (for example, red (R), green (G), and blue (B)) can be used. Light emission of the light source M _2, that is, the light source M blinking period of the light emitted from the ₂ (ON / each time off), color (selection of LED to emit light), and brightness (current supplied to each LED) Is controlled by, for example, a control unit 5 (for example, a main control unit 5a). In this case, the control unit 5 (for example, the main control unit 5a) functions as a light source control unit that controls the emission of light from the light _{source M 2.} The light emission from the light source M ₂ may be controlled by another control device (not shown) outside the camera calibration device 4.

_{As described above, even when the light source M 2} is used as the marker M, the above-mentioned two-dimensional locus detection unit 10c easily recognizes the position of the _{light source M 2} from the captured image 2a of the surveillance camera 2 by image recognition, and the marker. It becomes possible to easily recognize M. As a result, the effect of improving the accuracy of deriving the calibration parameters for the surveillance camera described above can be surely obtained.

When the light source M ₂ is used as the marker M, in S5 of FIG. 3 or S5'of FIG. 13, the calibration parameter derivation unit 10d for the surveillance camera uses the light _{emitted from the light source M 2} by the control unit 5 as a moving camera. When the appearance is changed to a unique appearance according to the shooting timing of 3 or the location in the environment, the shooting timing of the surveillance camera 2 or the shooting timing of the surveillance camera 2 or the image showing the appearance of the light acquired by the shooting by the surveillance camera 2 is used. Calibration parameters for surveillance cameras with respect to location may be derived. At this time, the control unit 5 _{changes, for example, the blinking cycle of the light emitted from the light source M 2} according to the shooting timing of the moving camera 3 (for example, changes so as to be synchronized with the shooting timing), thereby causing the light source. The _{light emitted from M 2} can be changed to a unique appearance according to the shooting timing of the moving camera 3. Further, the control unit 5 _{determines the color or brightness of the light emitted from the light source M 2 according} to the location in the environment, that is, the position of the mobile camera 3 derived by the calibration parameter derivation unit 10a for the mobile camera. By changing the light, the light _{emitted from the light source M 2} can be changed to a unique appearance depending on the location.

When the lighting timing of the light source M ₂ and the shooting timing of the surveillance camera 2 do not match, the calibration parameter derivation unit 10d for the surveillance camera recognizes the _{marker M (light source M 2) on the captured image 2a of the surveillance camera 2. (Because the light source M 2} is turned off when the shutter of the surveillance camera 2 is opened). Therefore, for example, the control unit 5 changes _{the blinking cycle of the light source M 2} according to the shooting timing of the moving camera 3, and the light _{emitted from the light source M 2} has a unique appearance according to the shooting timing of the moving camera 3. By changing to, the calibration parameter derivation unit 10d for the surveillance camera matches the blinking cycle of the _{light source M 2} based on the image showing the appearance of the light emitted from _{the light source M 2 (photographed image 2a of the surveillance camera 2).} That is, it is possible to calibrate the shooting timing of the surveillance camera 2 (obtain the calibration parameter related to the shooting timing) so as to match the shooting timing of the moving camera 3. As a result, even if the shooting timing of the moving camera 3 and the shooting timing of the surveillance camera 2 do not match before moving the moving camera 3, it is possible to accurately calibrate the time at which they are adjusted.

Further, in the above-mentioned S5 or S5', the surveillance camera calibration parameter derivation unit 10d obtains the calibration parameter for the position of the surveillance camera 2 by solving the PnP problem, but when a plurality of candidates for the position can be obtained. (You may get candidates for different positions). In such a case, the case control unit 5, the appearance of the light emitted from the light source M ₂ (color or brightness), to position the light source M ₂ before the first landmark in the environment, another Derivation of calibration parameters for surveillance cameras by changing the light emitted from _{the light source M 2} to a unique appearance depending on the location, different from when the _{light source M 2} is located in front of the second landmark of. Based on the image showing the appearance of the light emitted from the light source M ₂ (photographed image 2a of the surveillance camera 2), the unit 10d determines the location (position) of the surveillance camera 2 in the environment to some extent from the plurality of candidates. Can be squeezed. This makes it possible to further improve the accuracy of deriving the calibration parameters for the position of the surveillance camera 2.

That is, the time or place of the surveillance camera 2 is calibrated by using the light _{source M 2} as the marker M _{and changing the light emitted from the light source M 2} to a unique appearance according to the shooting timing or place of the moving camera 3. Can be performed accurately.

[Movement control of moving camera]
In the moving image acquisition step of S1 described above, the movement control unit 5b of the camera calibration device 4 may stop the moving camera 3 at least once when passing through the field of view of the surveillance camera 2. By controlling the movement of the moving camera 3 in this way, even if the time of the surveillance camera 2 is not calibrated, the three-dimensional position of the moving camera 3 at the same time as the shooting timing of the surveillance camera 2 can be known. It is possible to specify the three-dimensional position and the corresponding position on the captured image 2a of the surveillance camera 2. Thereby, in the calibration parameter derivation step for the surveillance camera of S5 or S5', the position of the surveillance camera 2 and the position of the surveillance camera 2 and the position of the surveillance camera 2 and the position of the surveillance camera 2 by solving the PnP problem using the above two points of information (coordinates of each position in the three dimensions and the two dimensions). It is possible to derive the calibration parameters for the surveillance camera including the posture. That is, even if the time of the surveillance camera 2 is not calibrated, the position and orientation of the surveillance camera 2 can be calibrated.

Further, in the moving image acquisition step of S1, when the movement control unit 5b of the camera calibrator 4 passes the moving camera 3 through the field of view of the surveillance camera 2, any arbitrary movement locus in the three-dimensional movement locus of the moving camera 3 The position and the corresponding position on the captured image 2a of the surveillance camera 2 may be moved so as to draw a identifiable locus. As the locus, for example, a graphic shape having a protruding portion can be considered. At this time, as the protruding portion, the corner portion of the polygon, both ends in the major axis direction of the ellipse, and the like can be considered, and therefore, the graphic shape includes a polygonal shape such as a triangle and a quadrangle, and the like. You can think of an elliptical shape.

FIG. 15 shows the movement locus when the three-dimensional movement locus of the moving camera 3 is rectangular, and FIG. 16 shows the two-dimensional movement locus of the moving camera 3 on the captured image 2a of the surveillance camera 2. Is shown. As shown in FIG. 15, when the moving camera 3 draws a quadrangular movement locus in a three-dimensional space, the protruding portion (corner portion) E included in the quadrangle suddenly or clearly moves in the moving direction of the moving camera 3. As shown in FIG. 16, even on the captured image 2a of the surveillance camera 2, the protrusion E corresponding to the three-dimensional movement locus of the moving camera 3 and the two-dimensional movement locus correspond to each other. Part E'appears clearly. Thus, the position E ₁ of the protrusion E of the three-dimensional movement locus, the 'position E ₁ of the' corresponding protruding portion E can be clearly identified on the captured image 2a. Therefore, in the step of deriving the calibration parameter for the surveillance camera in S5 or S5', even if the time of the surveillance camera 2 is not calibrated, the three-dimensional position information (position) of the moving camera 3 at the same time as the shooting timing of the surveillance camera 2 by solving the PnP problem by using the position including coordinates) and two-dimensional position information of the E ₁ (including position coordinates of the position E ₁ '), the calibration parameters for a monitoring camera including a position and orientation of the surveillance camera 2 It becomes possible to derive.

In particular, by possible trajectories identified the corresponding position E ₁ 'on an arbitrary position E ₁ and the photographed image 2a in the three-dimensional movement locus of the moving camera 3 is a figure shape having protrusions E, projecting and position E ₁ of part E, since the corresponding position E ₁ 'can be clearly identified on the captured image 2a, even if no time surveillance camera 2 is calibrated, it can be derived surveillance camera calibration parameters The effect can be surely obtained.

As long as it can identify the corresponding position E ₁ 'on an arbitrary position E ₁ and the photographed image 2a in the three-dimensional movement locus of the moving camera 3, three-dimensional movement locus of the moving camera 3, the projecting portion It is not limited to the figure shape having E. For example, even when the moving camera 3 is moved in a shape that is a combination of two circles (trajectory of "figure 8"), the contact points of the two circles constituting "figure 8" and the captured image 2a Since the corresponding position can be specified, it is possible to derive the calibration parameters for the surveillance camera using the three-dimensional position information and the two-dimensional position information even if the time of the surveillance camera 2 is not calibrated. Is.

[Programs and recording media]
The camera calibration device 4 described in the present embodiment can be configured by, for example, a computer (PC) in which a predetermined program (application software) is installed. By reading and executing the above program by a computer (for example, the control unit 5 as a CPU), each part of the camera calibration device 4 can be operated to execute each of the above-mentioned processes (each step). Such a program is acquired by downloading from the outside via a network, for example, and is stored in the storage unit 6. Further, the program is recorded on a computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory), and the computer reads the program from the recording medium and stores it in the program storage unit 12a. There may be.

[Other]
The camera calibration method, camera calibration device, camera calibration system, and camera calibration program of the present embodiment described above may be expressed as follows. In addition, the content described in this embodiment also includes a recording medium expressed as follows.

1. 1. A camera calibration method in which the calibration parameters of one or more surveillance cameras are derived as calibration parameters for surveillance cameras.
A moving image acquisition step of acquiring a moving image by moving the moving camera so as to overlap the field of view of the surveillance camera.
Based on the moving image, for each frame of the moving image, a procedure for deriving the calibration parameters for the moving camera including the position and orientation of the moving camera in the reference coordinate system, and a step for deriving the calibration parameters for the moving camera.
Using the calibration parameters for the moving camera, a three-dimensional locus acquisition step for obtaining a three-dimensional movement locus of the moving camera in the reference coordinate system, and a three-dimensional locus acquisition step.
A two-dimensional locus detection step of detecting at least a part of the three-dimensional movement locus of the moving camera as a two-dimensional movement locus on an image captured by the surveillance camera.
A camera calibration method including at least a part of the three-dimensional movement locus and a procedure for deriving a calibration parameter for a surveillance camera based on the two-dimensional movement locus. ..

2. In the procedure for deriving the calibration parameters for the mobile camera, in addition to deriving the calibration parameters for the mobile camera based on the moving image, a three-dimensional point cloud indicating the shape of an object or environment within the shooting range of the surveillance camera is estimated. ,
The camera calibration method is
Further including a two-dimensional point cloud detection step of detecting the position of the three-dimensional point cloud on the captured image of the surveillance camera as a two-dimensional point cloud.
In the surveillance camera calibration parameter derivation step, the surveillance camera calibration parameter is derived using the three-dimensional movement locus and the three-dimensional point group, and the two-dimensional movement locus and the two-dimensional point group. The camera calibration method according to 1 above.

3. 3. In the three-dimensional locus acquisition step, the three-dimensional movement locus of the marker attached to the moving camera, which moves with the movement of the moving camera, in the reference coordinate system is used as the three-dimensional movement locus of the moving camera. Ask,
The second-dimensional locus detection step according to the above 1 or 2, wherein at least a part of the three-dimensional movement locus of the marker is detected as a two-dimensional movement locus on the captured image of the surveillance camera. Camera calibration method.

4. The camera calibration method according to 3, wherein the marker is an index including feature points that can be recognized based on an image taken by the surveillance camera.

5. The camera calibration method according to 3, wherein the marker is a light source.

6. In the step of deriving the calibration parameter for a surveillance camera, when the light emitted from the light source is changed to a unique appearance according to the shooting timing or location of the moving camera, it is acquired by shooting with the surveillance camera. 5. The camera calibration method according to 5, wherein the surveillance camera calibration parameters relating to the imaging timing or location of the surveillance camera are derived using an image showing the appearance of the light.

7. In the step of deriving the calibration parameter for the surveillance camera, the light is converted into the moving camera by changing the blinking cycle, color, or brightness of the light emitted from the light source according to the shooting timing or location of the moving camera. The camera calibration method according to 6 above, characterized in that the appearance is changed to a unique appearance according to the shooting timing or location of the camera.

8. In the moving image acquisition step, the moving camera is stopped at least once when passing through the field of view of the surveillance camera, or an arbitrary position in the three-dimensional movement locus of the moving camera and the surveillance camera. The camera calibration method according to any one of 1 to 5 above, wherein the corresponding position on the captured image is moved so as to draw a identifiable locus.

9. The camera calibration method according to 8 above, wherein the locus has a graphic shape having a protruding portion.

10. A camera calibration program that causes a computer to perform the camera calibration method according to any one of 1 to 9.

11. Based on the moving image of the moving camera that moves so as to overlap the field of view of one or more surveillance cameras and acquires the moving image, the position and orientation of the moving camera in the reference coordinate system are set for each frame of the moving image. A mobile camera calibration parameter derivation unit that derives the mobile camera calibration parameters including
Using the calibration parameters for the moving camera, a three-dimensional locus acquisition unit for obtaining a three-dimensional movement locus of the moving camera in the reference coordinate system, and a three-dimensional locus acquisition unit.
A two-dimensional locus detection unit that detects at least a part of the three-dimensional movement locus of the moving camera as a two-dimensional movement locus on a captured image of the surveillance camera.
Includes at least a part of the three-dimensional movement locus and a surveillance camera calibration parameter derivation unit that derives the surveillance camera calibration parameter, which is the surveillance camera calibration parameter, based on the two-dimensional movement locus. A camera calibrator featuring.

12. Based on the moving image, the moving camera calibration parameter deriving unit estimates, in addition to deriving the moving camera calibration parameters, a three-dimensional point cloud indicating the shape of an object or environment within the shooting range of the surveillance camera. ,
The camera calibrator
A two-dimensional point cloud detection unit that detects the position of the three-dimensional point cloud on the captured image of the surveillance camera as a two-dimensional point cloud is further included.
The surveillance camera calibration parameter derivation unit derives the surveillance camera calibration parameter using the three-dimensional movement locus and the three-dimensional point group, and the two-dimensional movement locus and the two-dimensional point group. 11. The camera calibrator according to 11.

13. The three-dimensional locus acquisition unit uses the three-dimensional movement locus of the marker attached to the moving camera in the reference coordinate system, which moves with the movement of the moving camera, as the three-dimensional movement locus of the moving camera. Ask,
11 or 12, wherein the two-dimensional locus detection unit detects at least a part of the three-dimensional movement locus of the marker as a two-dimensional movement locus on an image captured by the surveillance camera. Camera calibrator.

14. The camera calibrator according to 13 above, wherein the marker is an index including feature points that can be recognized based on an image taken by the surveillance camera.

15. The camera calibrator according to 13 above, wherein the marker is a light source.

16. It further includes a light source control unit that controls the emission of light from the light source.
The calibration parameter derivation unit for a surveillance camera is used in the surveillance camera when the light emitted from the light source by the light source control unit is changed to a unique appearance according to the shooting timing or location of the moving camera. 15. The camera calibrator according to 15, wherein the surveillance camera calibration parameters relating to the imaging timing or location of the surveillance camera are derived using an image showing the appearance of the light acquired by imaging.

17. The light source control unit changes the blinking cycle, color, or brightness of the light emitted from the light source according to the shooting timing or location of the mobile camera, thereby converting the light into the shooting timing or the shooting timing of the mobile camera. The camera calibrator according to 16 above, wherein the camera calibrator is changed to a unique appearance depending on the location.

18. With the camera calibrator according to any one of 11 to 17 above.
With one or more of the fixed cameras
A camera calibration system comprising the mobile camera.

19. The camera calibrator may stop the moving camera at least once when passing through the field of view of the surveillance camera, or take an image of the moving camera at an arbitrary position in the three-dimensional movement locus of the moving camera and the surveillance camera. The camera calibration system according to 18, wherein the camera calibration system further includes a movement control unit that moves a corresponding position on an image so as to draw a identifiable locus.

20. The camera calibration system according to 19 above, wherein the locus has a graphic shape having a protrusion.

21. A computer-readable recording medium on which the camera calibration program according to 10 is recorded.

Although the embodiments of the present invention have been described above, the scope of the present invention is not limited to this, and can be extended or modified within a range that does not deviate from the gist of the invention.

The present invention can be used in a surveillance system including one surveillance camera or a distributed surveillance system in which two or more surveillance cameras are distributed and arranged.

1 Camera calibration system 2 Surveillance camera 2a Captured image 3 Mobile camera 4 Camera calibration device 5a Main control unit (light source control unit)
5b movement controller 10a for moving camera calibration parameter derivation portion 10b 3-dimensional trajectory obtaining unit 10c 2 dimensional trajectory detection section 10d surveillance camera calibration parameter derivation portion 10e 2-dimensional point group detecting unit E protrusion E ₁ position E ₁ 'position M Marker M ₁ Index M ₂ Light source P Feature point

Claims (20)

A camera calibration method in which the calibration parameters of one or more surveillance cameras are derived as calibration parameters for surveillance cameras.
A moving image acquisition step of acquiring a moving image by moving the moving camera so as to overlap the field of view of the surveillance camera.
Based on the moving image, for each frame of the moving image, a procedure for deriving the calibration parameters for the moving camera including the position and orientation of the moving camera in the reference coordinate system, and a step for deriving the calibration parameters for the moving camera.
Using the calibration parameters for the moving camera, a three-dimensional locus acquisition step for obtaining a three-dimensional movement locus of the moving camera in the reference coordinate system, and a three-dimensional locus acquisition step.
A two-dimensional locus detection step of detecting at least a part of the three-dimensional movement locus of the moving camera as a two-dimensional movement locus on an image captured by the surveillance camera.
A camera calibration method including at least a part of the three-dimensional movement locus and a procedure for deriving a calibration parameter for a surveillance camera that derives a calibration parameter for the surveillance camera based on the two-dimensional movement locus. In the procedure for deriving the calibration parameters for the mobile camera, in addition to deriving the calibration parameters for the mobile camera based on the moving image, a three-dimensional point cloud indicating the shape of an object or environment within the shooting range of the surveillance camera is estimated. ,
The camera calibration method is
Further including a two-dimensional point cloud detection step of detecting the position of the three-dimensional point cloud on the captured image of the surveillance camera as a two-dimensional point cloud.
In the surveillance camera calibration parameter derivation step, the surveillance camera calibration parameter is derived using the three-dimensional movement locus and the three-dimensional point group, and the two-dimensional movement locus and the two-dimensional point group. , The camera calibration method according to claim 1. In the three-dimensional locus acquisition step, the three-dimensional movement locus of the marker attached to the moving camera, which moves with the movement of the moving camera, in the reference coordinate system is used as the three-dimensional movement locus of the moving camera. Ask,
The camera calibration according to claim 1 or 2, wherein in the two-dimensional locus detection step, at least a part of the three-dimensional movement locus of the marker is detected as a two-dimensional movement locus on the captured image of the surveillance camera. Method. The camera calibration method according to claim 3, wherein the marker is an index including feature points that can be recognized based on an image taken by the surveillance camera. The camera calibration method according to claim 3, wherein the marker is a light source. In the step of deriving the calibration parameter for a surveillance camera, when the light emitted from the light source is changed to a unique appearance according to the shooting timing or location of the moving camera, it is acquired by shooting with the surveillance camera. The camera calibration method according to claim 5, wherein the surveillance camera calibration parameters relating to the imaging timing or location of the surveillance camera are derived using an image showing the appearance of the light. In the step of deriving the calibration parameter for the surveillance camera, the light is converted into the moving camera by changing the blinking cycle, color, or brightness of the light emitted from the light source according to the shooting timing or location of the moving camera. The camera calibration method according to claim 6, wherein the appearance is changed to a unique appearance depending on the shooting timing or location of the camera. In the moving image acquisition step, the moving camera is stopped at least once when passing through the field of view of the surveillance camera, or an arbitrary position in the three-dimensional movement locus of the moving camera and the surveillance camera. The camera calibration method according to any one of claims 1 to 5, wherein the corresponding position on the captured image is moved so as to draw a identifiable locus. The camera calibration method according to claim 8, wherein the locus is a graphic shape having a protruding portion. A camera calibration program that causes a computer to perform the camera calibration method according to any one of claims 1 to 9. Based on the moving image of the moving camera that moves so as to overlap the field of view of one or more surveillance cameras and acquires the moving image, the position and orientation of the moving camera in the reference coordinate system are set for each frame of the moving image. A mobile camera calibration parameter derivation unit that derives the mobile camera calibration parameters including
Using the calibration parameters for the moving camera, a three-dimensional locus acquisition unit for obtaining a three-dimensional movement locus of the moving camera in the reference coordinate system, and a three-dimensional locus acquisition unit.
A two-dimensional locus detection unit that detects at least a part of the three-dimensional movement locus of the moving camera as a two-dimensional movement locus on a captured image of the surveillance camera.
A surveillance camera calibration parameter derivation unit that derives a surveillance camera calibration parameter, which is a surveillance camera calibration parameter, based on at least a part of the three-dimensional movement locus and the two-dimensional movement locus. Camera calibrator. Based on the moving image, the moving camera calibration parameter deriving unit estimates, in addition to deriving the moving camera calibration parameters, a three-dimensional point cloud indicating the shape of an object or environment within the shooting range of the surveillance camera. ,
The camera calibrator
A two-dimensional point cloud detection unit that detects the position of the three-dimensional point cloud on the captured image of the surveillance camera as a two-dimensional point cloud is further included.
The surveillance camera calibration parameter derivation unit derives the surveillance camera calibration parameter using the three-dimensional movement locus and the three-dimensional point group, and the two-dimensional movement locus and the two-dimensional point group. The camera calibrator according to claim 11. The three-dimensional locus acquisition unit uses the three-dimensional movement locus of the marker attached to the moving camera in the reference coordinate system, which moves with the movement of the moving camera, as the three-dimensional movement locus of the moving camera. Ask,
The camera calibration according to claim 11 or 12, wherein the two-dimensional locus detection unit detects at least a part of the three-dimensional movement locus of the marker as a two-dimensional movement locus on an image captured by the surveillance camera. apparatus. The camera calibrator according to claim 13, wherein the marker is an index including feature points that can be recognized based on an image taken by the surveillance camera. The camera calibrator according to claim 13, wherein the marker is a light source. It further includes a light source control unit that controls the emission of light from the light source.
The calibration parameter derivation unit for a surveillance camera is used in the surveillance camera when the light emitted from the light source by the light source control unit is changed to a unique appearance according to the shooting timing or location of the moving camera. The camera calibrator according to claim 15, wherein the surveillance camera calibration parameters relating to the imaging timing or location of the surveillance camera are derived using an image showing the appearance of the light acquired by imaging. The light source control unit changes the blinking cycle, color, or brightness of the light emitted from the light source according to the shooting timing or location of the mobile camera, thereby converting the light into the shooting timing or the shooting timing of the mobile camera. The camera calibrator according to claim 16, which changes its appearance to a unique appearance depending on the location. The camera calibrator according to any one of claims 11 to 17.
With one or more of the fixed cameras
A camera calibration system including the mobile camera. The camera calibrator may stop the moving camera at least once when passing through the field of view of the surveillance camera, or take an image of the moving camera at an arbitrary position in the three-dimensional movement locus of the moving camera and the surveillance camera. The camera calibration system according to claim 18, further comprising a movement control unit that moves the corresponding position on the image so as to draw a identifiable trajectory. The camera calibration system according to claim 19, wherein the locus is a graphic shape having a protrusion.

Images