JP4980606B2 - Mobile automatic monitoring device - Google Patents

Description

  The present invention relates to a monitoring camera device for automatically monitoring a desired monitoring target using a photographing device such as a video camera mounted on a vehicle or a laser scanner, and more particularly to automatically detect and automatically recognize an obstacle or a suspicious object. The present invention relates to a mobile automatic monitoring apparatus capable of displaying a corresponding video and automatically monitoring and displaying the movement or the like of the monitored object when the monitored object is determined.

In general, as a security system and a security system, a surveillance camera system for unattended and automatic surveillance of a desired surveillance object by a photographing device such as a video camera is widely used.
This type of surveillance camera system, for example, installs surveillance cameras on predetermined surveillance objects and surveillance areas, such as in banks and convenience stores, cash corners, airport terminals and runways, and shoots with surveillance cameras. By monitoring the recorded image by remote operation, the suspicious person is discovered, a predetermined monitoring object is tracked and monitored, and the like.

In such a conventional surveillance camera system, a narrow-angle camera equipped with a zoom function is mounted on a pan head and is rotated in the viewing angle direction so that a single camera can shoot and monitor a certain range. It was. Then, according to the number and size of objects to be monitored, the size of the area to be monitored, etc., one or a plurality of monitoring cameras are arranged and installed at appropriate locations, and desired monitoring is performed. It was.
As a technique related to such a conventional surveillance camera system, for example, “all-around device” described in Japanese Patent Laid-Open No. 7-283379 (Patent Document 1) and Japanese Patent Laid-Open No. 8-194809 (Patent Document 2). A “360-degree monitoring system” is known.

Japanese Patent Laid-Open No. 7-283379 (page 2-3, FIG. 1) JP-A-8-194809 (page 2-3, FIG. 1)

However, in conventional surveillance camera systems, the surveillance camera can be rotated in the viewing angle direction by the pan head, but there is a limit to the rotational drive by the pan head, and the target surveillance range is limited. It was a thing.
In addition, a blind spot that is not always monitored is produced by monitoring with a rotating camera, and even if the camera is rotated 360 degrees, the entire circumference of the camera cannot be imaged and monitored simultaneously.

Furthermore, in conventional surveillance camera systems, the monitoring effect is greatly affected by the surrounding conditions and environment, especially in the case of monitoring at night or when there is fresh snow, etc. It was extremely difficult to identify and detect objects. Moreover, since surveillance cameras for night use are usually fixedly installed, blind spots are generated, and it is impossible to shoot and monitor the surveillance area evenly.
Thus, the conventional surveillance camera system is not perfect as a security system and a security system.

  As a result of earnest research, the inventor of the present application links the all-around camera with a fixed viewing angle and a narrow-angle camera capable of controlling the viewpoint direction to expand the monitoring range to the entire visible range, while monitoring the whole, At the same time, it is possible to obtain an enlarged image for the monitoring target, eliminate the blind spot, display the entire monitoring target area as a single image, and increase both the resolution of the image and the widening of the monitoring range. Inventing a surveillance camera system that can be realized simultaneously, Japanese Patent Application No. 2005-218053 proposes a "surveillance camera system in which a camera with a fixed viewing angle and a narrow-angle camera that can control the viewpoint direction are linked". .

  The present invention further improves and develops the monitoring camera system according to the above-mentioned proposal by the present inventor, and loads a measuring instrument such as a camera for monitoring and photographing a monitoring target on a moving body such as a vehicle. By acquiring the video of the monitoring target while moving the monitor, the monitoring range is expanded to the entire visible range with a camera that moves with the moving body, etc., and the whole is monitored, and the monitoring target is expanded in detail It is possible to obtain images, eliminate blind spots, display the entire monitored area as a single image, and realize both high resolution of the camera and widening of the monitoring range at the same time. The purpose is to provide a mobile automatic monitoring device that can realize a highly reliable monitoring system and security system.

In order to achieve the above object, a mobile automatic monitoring device according to the present invention is mounted on a mobile body and moves together with the mobile body to acquire spatial information around the mobile body. An acquisition unit, a position coordinate acquisition unit that acquires position coordinate information indicating the three-dimensional position and orientation of a moving moving body, a monitoring area where the moving body moves, and the spatial information acquisition unit acquires spatial information from the monitoring area 3D spatial information around the moving object based on the 3D map part with a 3D map of the possible area, the spatial information around the moving object, the position coordinate information of the moving object, and the 3D map of the monitoring area An integrated spatial information unit for generating integrated spatial information, an automatic detection unit for detecting and identifying a monitoring object in a monitoring area by associating the integrated spatial information around the moving object with a 3D map, and , Supervision identified by the automatic detection unit An automatic recognition unit for recognizing predetermined attribute information of an object, image information necessary for moving a moving object, a video of a monitoring target specified by an automatic detection unit, and a monitoring target whose attribute is recognized by an automatic recognition unit A monitor display unit that synthesizes and displays a CG or video of an object in a real video, a live-action video, or a three-dimensional map in accordance with a viewpoint observed from a moving object, and a monitor displayed via the monitor display unit And a recording unit that records predetermined data of the object.

  In addition, as described in claim 2, the mobile automatic monitoring device of the present invention detects a change in the size, shape, position, etc. of the object specified as the monitoring object by the automatic detection unit. The map update unit is configured to reflect the change in the three-dimensional map information of the three-dimensional map unit and / or the predetermined data of the recording unit.

  According to a third aspect of the present invention, the mobile automatic monitoring device of the present invention includes a three-dimensional coordinate of the monitoring object specified by the automatic detection unit and a moving object acquired by the position coordinate acquisition unit. A viewpoint direction control unit that drives and controls an observation device provided in the spatial information acquisition unit is set in the direction of the monitoring target determined based on the three-dimensional coordinates.

  According to another aspect of the mobile automatic monitoring device of the present invention, as described in claim 4, the spatial information acquisition unit is an all-around camera that acquires a wide range of images around the moving body, and a specified direction camera that directs the viewpoint in a specified direction. , Equipped with a part or all of each device such as a laser scanning device that scans laser light to measure the direction and distance, a radar device that emits radar waves and measures the direction and distance of reflected waves, day and night, season, According to the situation such as the weather, these devices are configured to acquire spatial information around the moving body either alone or in any combination.

  According to a fifth aspect of the present invention, there is provided the mobile automatic monitoring apparatus according to claim 5, wherein the monitoring target specified by the automatic detection unit and the attribute information of the monitoring target recognized by the automatic recognition unit are included. An automatic determination unit that determines the next action of the moving body, and an automatic control unit that controls the moving body according to the next action of the moving body determined by the automatic determination unit. is there.

  Further, according to the mobile automatic monitoring device of the present invention, as described in claim 6, the integrated spatial information unit includes spatial information around the moving body acquired by the spatial information acquisition unit, and the position coordinate acquisition unit. Based on only the position coordinate information of the moving body acquired in step 3, the three-dimensional space information around the moving body is integrated to generate integrated space information, and the three-dimensional map unit is generated based on the integrated space information. It is as a structure which produces | generates the provided three-dimensional map.

According to the mobile automatic monitoring device of the present invention having the above-described configuration, first, in the video of the all-around camera mounted on a moving body such as a vehicle, the monitoring object (predicted object, obstacle, A suspicious object etc.) can be detected and the zoom camera can zoom up there. Thereby, a suspicious object is discovered with the all-around camera, the three-dimensional position is calculated | required, and a zoom camera can zoom up with respect to the coordinate. In addition, for existing objects, attribute registration, attribute update, and attribute search attribute display can be performed by clicking on the object.
This is proposed in Japanese Patent Application No. 2005-218053 filed earlier by the applicant of the present application, "Surveillance Camera System Linking a Peripheral Camera with Fixed Viewing Angle and a Narrow Angle Camera whose View Direction can be Controlled". The invention also partially follows the technology.

In the present invention, the monitoring object and the monitoring object (predicted object, obstacle, suspicious object, etc.) are monitored by the equipment loaded on the moving vehicle, the moving vehicle is stopped, If the device is operated in a stopped state, some techniques relating to image processing become the invention described in the above-mentioned prior application (Japanese Patent Application No. 2005-218053) of the present applicant. In this case, the narrow-angle camera in the prior application corresponds to the camera installed in the vehicle of the present invention or the existing camera.
However, the above-mentioned prior application is intended for the case where the monitoring camera is fixedly installed at a predetermined position, and the three-dimensional measurement of the object by the moving camera and the monitoring object Detection, recognition, etc. of objects, obstacles, suspicious objects, etc.) are not proposed.

Therefore, the mobile automatic monitoring device of the present invention is based on the monitoring area acquired in advance and the surrounding three-dimensional map as a reference, and compared with the moving image captured by the camera mounted on the moving body, The original position coordinates and orientation are obtained with high accuracy.
In addition, the mobile automatic monitoring device of the present invention automatically detects a sufficient number of feature points from a plurality of frame images of a moving image captured by a camera mounted on a moving body, and automatically tracks the feature points between frames. By doing this, it is possible to obtain the CV data indicating the camera position and the rotation angle with high accuracy by performing overlap calculation on a large number of feature points, and thereby the high-precision 3D position coordinates of the camera moving with the moving body It is something to get to.
As a result, it is possible to perform three-dimensional measurement of an object to be monitored by a camera mounted on a moving object such as a vehicle, and detection / recognition of an arbitrary object to be monitored. The entire image is monitored while being spread over the entire area, and a detailed enlarged image can be obtained for the monitoring target.

Therefore, according to the present invention, in a monitoring area, a blind spot is eliminated, and a road surface shape, a monitoring object (a predicted object, an obstacle, a suspicious object, etc.) and the like are reliably detected even at night. It is possible to detect the three-dimensional shape of an arbitrary monitoring target while traveling or stopping even in the snow condition of fresh snow having the worst conditions.
As a result, it is possible to realize a complete and highly reliable monitoring system and security system.

According to the mobile automatic monitoring apparatus of the present invention, a CV that indicates a camera position, a camera rotation angle, and the like with high accuracy from a plurality of frame images of a moving image while using a 3D map as a reference or generating a 3D map. (Camera vector) can be obtained automatically by calculation, and high-accuracy position information of the moving camera can be generated and acquired in real time, and the obtained high-accuracy position information is used to capture and monitor the camera. 3D measurement of monitored objects and detection / recognition of arbitrary monitored objects.
Thereby, while expanding the monitoring range to the entire visible range by a camera or the like that moves with the moving body, it is possible to monitor the whole and obtain a detailed enlarged image for the monitoring target, eliminate the blind spot, A complete and reliable monitoring system and security system that can display the entire monitored area as a single image and can simultaneously achieve both high resolution and wide monitoring range of the camera. Can be provided.

Hereinafter, a preferred embodiment of a mobile automatic monitoring apparatus according to the present invention will be described with reference to the drawings.
Here, the mobile automatic monitoring apparatus of the present invention described below is realized by processing, means, and functions executed by a computer according to instructions of a program (software). The program sends commands to each component of the computer, and performs the following predetermined processing and functions such as automatic extraction of reference points and feature points from the video, automatic tracking of the extracted reference points, Calculation of 3D coordinates, comparison of video and 3D map, calculation of camera vectors, automatic detection / recognition of monitoring objects, calculation of 3D coordinates of monitoring objects, CG conversion of monitoring object images, etc. Make it. Thus, each process and means in the present invention are realized by specific means in which the program and the computer cooperate.
Note that all or part of the program is provided by, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and the program read from the recording medium is installed in the computer and executed. The The program can also be loaded and executed directly on a computer through a communication line without using a recording medium.

[Basic configuration]
First, a basic configuration of a mobile automatic monitoring apparatus according to an embodiment of the present invention will be described with reference to FIG.
FIG. 1 is a block diagram showing a basic configuration of a mobile automatic monitoring apparatus according to an embodiment of the present invention.
The mobile automatic monitoring apparatus 1 of this embodiment shown in the figure includes a spatial information acquisition unit 10, a position coordinate acquisition unit 20, an integrated spatial information unit 30, a three-dimensional map unit 40, an automatic detection unit 50, A map update unit 60, a viewpoint direction control unit 70, an automatic recognition unit 80, a monitoring display unit 90, a data recording unit 100, an automatic determination unit 110, and an automatic control unit 120 are provided.

The spatial information acquisition unit 10 includes devices such as cameras, laser scanners, radars, and the like that are mounted on a moving body such as a vehicle (see the moving body 10a shown in FIGS. 3 to 4 described later). By traveling in the monitoring area and using three or more devices in combination or in combination, the three-dimensional spatial information around the vehicle is grasped over a wide range.
Specifically, as the measurement equipment provided in the spatial information acquisition unit 10, an all-around camera (see FIGS. 3 to 4 to be described later) that acquires a wide range of images around the vehicle, or designation that directs the viewpoint in a specified direction There are directional cameras, laser scanning devices that scan laser light to measure the direction and distance, radar devices that emit radar waves and measure the direction and distance of reflected waves, etc. It is mounted on a moving body and acquires spatial information such as a video of a monitoring area.

These devices can acquire spatial information by selecting an appropriate device by using any combination of the devices according to the conditions such as day and night, season, weather, etc., or a combination thereof. In the unit 40, the three-dimensional spatial information can be integrated and generated from the spatial information obtained by appropriate selection depending on the situation.
Therefore, a vehicle is loaded with a plurality of devices, and these devices are switched or used in combination so that spatial information can be acquired under various conditions such as daytime, nighttime, sunny, rain, snow, etc. It is preferable to obtain ambient spatial information.

Spatial information is basically acquired while the vehicle is running, but if conditions such as weather and road surface are bad, the vehicle is stopped at an appropriate point and the spatial information is taken over time. You may make it acquire.
In the daytime when the field of view is good, it is possible to acquire good spatial information while traveling with only a wide-angle camera or an all-around camera.
At night, spatial information is obtained by arranging a large number of infrared cameras.
In particular, when there is snow, it is difficult to analyze the image, so a laser scanner is appropriate.
In addition, since the spatial information can be acquired by analyzing the image of the irradiation pattern on the snow surface by the laser scanner, the laser scanner is effective during snowfall and at night.

The position coordinate acquisition unit 20 acquires the three-dimensional coordinates and posture of the traveling vehicle using known means such as GPS, IMU, and gyro.
In addition, the position coordinate acquisition unit 20 uses the CV image obtained by the CV calculation using the all-around video from the output from the camera (the all-around camera unit 11 to be dictated) provided in the spatial information acquisition unit 10, and Generate / acquire CV values indicating 3D coordinates and orientation. Details of the CV calculation for obtaining the CV value will be described later (see FIGS. 2 to 14).

  Furthermore, the position coordinate acquisition unit 20 can continuously take corresponding points between designated points in the 3D map provided in the 3D map unit 40 to be described later and the spatial information. Compared with the case of performing a feature point tracking CV calculation described later, position information can be acquired at a higher speed. In particular, the latter using a three-dimensional map is more advantageous for real-time processing. Further, by using both the CV calculation and the three-dimensional map, errors can be compensated for and the CV value can be acquired stably.

Here, the correspondence between the 3D map and the video acquired in real time may be to automatically extract the feature points on the video, but the specified points specified in advance and the markers prepared for the specified purpose are added to the video of the 3D map. By imprinting, it is possible to acquire the position coordinates and posture of the vehicle by comparison with the designated point and the same marker acquired when acquiring the spatial information. Here, it is important that the designated point has three-dimensional coordinates, which means that if the designated point can be detected in the video acquired in real time, the three-dimensional coordinates of the point can be obtained.
As the designated point marker prepared for the purpose of designation, a simple geometric pattern such as a circle, a triangle, or a square, or a combination thereof is preferable, and these geometric patterns are placed at positions that enter the field of view.

The vehicle position and orientation acquired by the correspondence between the designated point of the live-action image and the designated point marker and the three-dimensional map are the CV values themselves, which will be described later, and completely the same as the CV values acquired by the CV calculation by the feature point tracking method described later. Have the same meaning.
That is, a real-time video acquired as spatial information can acquire a CV value by comparing with a three-dimensional map. In addition, since the CV value in this case is a value that accurately indicates the camera position, calibration with the vehicle position is required. However, as long as the relationship between the two is maintained, it is only necessary to calibrate once.

The three-dimensional map unit 30 is previously provided with a CV video three-dimensional map related to the inside of the monitoring area, which is generated by CV video, laser scan data, or the like.
The three-dimensional map is preferably prepared in advance by accurately surveying the surrounding area and its surroundings to generate a highly accurate three-dimensional map. Can be generated, and although the accuracy is lower than that by surveying, it can be sufficiently used for simplicity. Therefore, even if there is no 3D map prepared in advance, the monitoring purpose can be sufficiently achieved if the position accuracy is not a problem.

That is, in the mobile automatic monitoring apparatus 1 of the present embodiment, integrated spatial information can be acquired only by the spatial information acquisition unit 10 and the position coordinate acquisition unit 20 without the 3D map unit 30. Compared to a case where a highly accurate 3D map is provided, the accuracy is lowered, but it is possible to acquire integrated spatial information and generate a 3D map therefrom.
Therefore, it is possible to perform monitoring while initially generating the 3D map from the acquired integrated spatial information without having the 3D map unit 30. In this case, a 3D map is newly generated by the map update unit 80 described later, and the 3D map generated here functions as the 3D map unit 40 from the next time onward. From the next time, the monitoring process can be performed by using the previously generated 3D map as the 3D map unit 40.

The integrated spatial information unit 40 comprehensively determines and integrates various types of spatial information obtained by measuring with various types of measuring devices, generates integrated three-dimensional spatial information, Integrated spatial information is generated from the comparison between the original position and orientation and the 3D map.
The automatic detection unit 50 compares the three-dimensional map with the integrated spatial information to correspond to each other, confirms the presence of the monitoring target (predicted target) described in the map in the integrated spatial information, and determines the position in the map. Or find a monitoring object (obstacle, suspicious object) in the monitoring area that does not exist in the 3D map, and determine and specify the position in the map.
Here, “detection” means finding a monitoring object (predicted object, obstacle, suspicious object, etc.), and “specific” means a monitoring object registered in advance in the 3D map. Confirm the presence of the object in the general space information, or discover the monitoring object (predicted object, obstacle, suspicious object, etc.) in the general space information, and its position and approximate size It means to decide.
Further, as will be described later, “recognition” means that the attribute of the object is confirmed.

The viewpoint direction control unit 60 is determined by calculation from the three-dimensional coordinates of the monitoring object (predicted object, obstacle, suspicious object, etc.) specified by the automatic detection unit 50 and the three-dimensional coordinates of the vehicle. It is a control means for controlling the observation equipment to be directed in order to examine details with respect to the direction of the monitoring object (predicted object, obstacle, suspicious object, etc.). For example, a video camera mounted on a vehicle and set to a narrow angle drives and controls the camera platform and zoom so that it is directed to the target monitoring object (predicted object, obstacle, suspicious object, etc.) To do.
In addition, when the monitoring target object (predicted target object, obstacle, suspicious object, etc.) moves, the viewpoint direction control unit 60 tracks and controls the camera in the moving direction.

The automatic recognition unit 70 recognizes the attribute of the specified monitoring target (predicted target, obstacle, suspicious object, etc.) information.
Here, “recognition” of the attribute information of the monitoring target means that the attribute can be confirmed against the memory registered in the database.
In the present embodiment, as the automatic recognition unit 70, the object attribute recognition process is performed by using the PRM technology (see the PRM recognition unit 71 in FIG. 19 described later).

  Here, PRM is an abbreviation for Parts Reconstruction Method (3D space recognition method), and is a technique for recognizing an object developed by the present inventor (see International Application PCT / JP01 / 05387). Specifically, the PRM technology prepares all the shapes and attributes of the object to be predicted in advance as parts (operator parts), compares these parts with actual live-action images, and selects matching parts. Technology that recognizes objects. For example, “parts” of objects required for traveling vehicles and the like are lanes as road markings, white lines, yellow lines, crossing roads, speed signs as road signs, guide signs, etc. Therefore, the recognition can be easily performed by the PRM technology. In addition, even when searching for an object in a CV video, it is possible to limit the expected three-dimensional space in which the object exists to a narrow range, thereby improving recognition efficiency.

Specifically, if it is a traffic light, it is “detection” only by acquiring its three-dimensional shape, and it is “specific” if the coordinates and orientation are known. It turns into blue-yellow, and it can be said that the attributes were understood (recognized) while understanding the meaning of each traffic.
Note that not all monitored objects (predicted objects, obstacles, suspicious objects, etc.) are recognized by PRM recognition. That is, an object whose information does not exist in the database to be stored cannot be recognized. However, this is not a problem. Even if it is a human being, it is the same as not being able to recognize what is not in memory. This is because recognition is obtained by comparing itself with memory of something close to itself.
However, when the automatic recognition unit 70 cannot recognize it, it is of course possible for the operator (human) of the monitoring apparatus 1 to recognize it and register / store it in the monitoring apparatus 1. Thereafter, the object is automatically recognized by the automatic recognition unit 70.

The map update unit 80 has a monitored object (predicted object, obstacle, suspicious object, etc.) disappeared, changed in shape, or identified as a monitored object, or registered in advance. When a change in size, shape, position, or the like is detected in a target object, it is newly registered and reflected in the map of the 3D map unit 30.
Further, the map update unit 80 may not only register “specification” as described above but also newly register an object that has advanced to “recognition”.

The monitoring display unit 90 is a display means for displaying the detected and specified monitored objects (predicted objects, obstacles, suspicious objects, etc.), and includes a liquid crystal display or other displays.
Specifically, the monitoring display unit 90 converts the video information necessary for travel guidance to a vehicle on which the monitoring device 1 is mounted, and further detects, identifies and recognizes the monitoring target detected by the monitoring device 1. Display objects and their attributes. As display methods, there are methods such as (1) displaying alone in a real image, (2) displaying in a real image, and (3) displaying in a three-dimensional map.

Hereinafter, the display methods (1) to (3) will be specifically described.
First, here, the image information necessary for traveling of the vehicle of the monitoring device 1 is, for example, a daytime image in driving at night, and a summer image in driving in winter snow, for example. It is.
Furthermore, it is an image with a good visibility when the visibility is poor due to fog, for example.

In the method (1) for displaying alone in the actual video, the image information necessary for traveling is converted to CG, and further, the monitoring object detected and identified by the monitoring device 1 is simplified to a simple figure, Alternatively, it is cut out from an image acquired at a high resolution by a camera in a specified direction, and the recognized monitoring object is converted into a CG, and is synthesized and displayed in the field of view of the actual image (actual landscape) of the observer (or driver).
The observer will synthesize and observe the CG with the actual video that he / she is viewing without any contradiction in the viewpoint.
As a matter of course, the movement of the vehicle in this case can synthesize the position coordinates and the posture without any contradiction with the actual video, for example, by the CV value (described later) acquired by the CV calculation.

In the method (2), the image information necessary for traveling is converted into CG, and the monitoring object detected and identified by the monitoring device 1 is simplified in figure or designated direction camera. Is cut out from the video acquired at a high resolution, and the recognized monitoring object is converted into a CG, combined with a live-action video, and displayed.
The observer (driver) sees the CG image synthesized with the real image on the monitor screen.
As in the case of (1), the movement of the vehicle in this case can synthesize the position coordinates and the posture without contradiction with the actual video, for example, by the CV value acquired by the CV calculation.

The method of displaying in the three-dimensional map of (3) is to CG the image information necessary for traveling, and further to make the monitored object detected and identified by the monitoring device 1 into a simplified figure or designated direction The image is cut out from the video acquired with high resolution by the camera, and the recognized monitoring object is converted into a CG, combined with the actual video and displayed.
An observer (driver) sees a CG image combined with a three-dimensional map on the monitor screen.
The movement of the vehicle in this case can also synthesize the position coordinates and the posture without any contradiction, for example, from the CV value acquired by the CV calculation.

The data recording unit 100 is a recording medium such as a hard disk that records data on a monitoring target, a predicted target, an obstacle, a suspicious object, and the like for later analysis.
As data recorded in the data recording unit 100, in addition to image data and the like displayed via the monitoring display unit 90, determination accuracy and determination time are improved as reference information for later determination. Therefore, it is desirable to simultaneously record various conditions that have been used as a basis for judgment.

With the configuration from the spatial information acquisition unit 10 to the data recording unit 100 as described above, the purpose of automatic monitoring by the monitoring device 1 can be achieved.
By the way, although it is common for a vehicle to be driven by a human being, the mobile automatic monitoring device 1 according to the present embodiment can accurately detect, identify and recognize the traveling path of the vehicle. Can be automated.
That is, a vehicle equipped with the monitoring device 1 can be driven automatically by judging the condition of the traveling road by itself without driving by a human.
Hereinafter, a configuration for automatically driving the vehicle by the monitoring apparatus 1 will be described.

As shown in FIG. 1, the mobile automatic monitoring device 1 according to this embodiment includes an automatic determination unit 110 and an automatic control unit 120 as a configuration necessary for automatic driving.
The automatic determination unit 110 is a determination unit that determines the next action of the vehicle by determining from the result specified by the automatic detection unit 50 and the result recognized by the automatic recognition unit 70.
The automatic control unit 120 is drive control means for controlling the vehicle in accordance with the next action of the vehicle determined by the automatic determination unit 110.
The monitoring purpose of the spatial information acquisition unit 10 is to acquire spatial information of monitoring objects (predicted objects, obstacles, suspicious objects, etc.) around the vehicle. Is also monitored as a monitoring target, monitoring the road surface conditions of vehicle travel, and monitoring whether there are monitoring objects (predicted objects, obstacles, suspicious objects, etc.) on the road, It is necessary to ensure driving safety.

The automatic determination unit 110 grasps the three-dimensional shape of the travel path based on the result specified by the automatic detection unit 50. Further, from the result recognized by the automatic recognizing unit 70, the attribute can be understood by the object, so that the importance and the handling method can be understood.
Therefore, the monitoring apparatus 1 detects a monitoring object (predicted object, obstacle, suspicious object, etc.) while monitoring a predetermined monitoring object, and further uses the same equipment and method to safely drive the vehicle. Drive while automatically judging the necessary road surface conditions.
Then, the automatic determination unit 110 makes a determination based on these pieces of information, understands the situation around the vehicle, confirms the planned traveling path, and checks whether there are objects to be monitored (predicted objects, obstacles, suspicious objects, etc.). The next action of the vehicle is determined from the above.
For example, the object to be monitored is determined to be a dangerous object, and it is determined whether to move away from the object to be monitored quickly or to approach further for detailed examination and to observe or trace the surroundings from each direction.

The automatic control unit 120 controls the steering, accelerator, brake, etc. of the vehicle in accordance with the next action of the vehicle determined by the automatic determination unit to perform automatic driving.
Note that the spatial information acquisition unit 10, the position coordinate acquisition unit 20, the three-dimensional map unit 30, the integrated spatial information unit 40, the automatic detection unit 50, the viewpoint direction control unit 60, and the automatic recognition unit 70 for automatic driving as described above. Each part of the map updating unit 80 may be dedicated for the purpose of automatic driving. However, since the number of devices increases by separating the operation purpose and the monitoring purpose into a dedicated device, the cost, and further, the automatic operation requires almost the same function as the monitoring device 1, It is efficient and preferable to share the purpose of monitoring and the purpose of automatic driving.
Therefore, in this embodiment and each embodiment described below, the spatial information acquisition unit 10, the position coordinate acquisition unit 20, the three-dimensional map unit 30, the integrated spatial information unit 40, the automatic detection unit 50, the viewpoint direction control unit 60, the automatic The recognition unit 70 and the map update unit 80 are described as being shared without being distinguished from those for automatic driving.

[Feature point tracking CV calculation]
Next, details of the CV calculation (feature point tracking method CV calculation) used in the position coordinate acquisition process of the camera in the position coordinate acquisition unit 20 of the mobile automatic monitoring apparatus 1 as described above will be described with reference to FIGS. I will explain.
Here, the CV calculation means obtaining a CV value, and the obtained result is referred to as CV value or CV data. The notation CV is an abbreviation for camera vector, and the camera vector (CV) is a value indicating a three-dimensional position and a three-axis rotation posture of a camera such as a video camera that acquires an image for measurement or the like. .
The CV calculation acquires a moving image (video image), detects a feature point in the image, tracks it in a plurality of adjacent frames, and creates a triangle formed by the camera position and the tracking locus of the feature point in the image. The three-dimensional position of the camera and the three-axis rotational posture of the camera are obtained by generating a large number of images and analyzing the triangle.

In the CV calculation of the feature point tracking method, it is an important characteristic that, in the process of obtaining the CV value, simultaneously, the three-dimensional coordinates of the relative value are simultaneously obtained for the feature point (reference point) in the video.
Further, the CV value obtained by calculation from the moving image is obtained simultaneously with the three-dimensional camera position and the three-dimensional camera posture corresponding to each frame of the moving image. Moreover, in principle, the characteristic that a CV value is obtained corresponding to an image with a single camera is an excellent feature that can be realized only by CV calculation.
For example, in other measurement means (such as GPS and IMU), in order to simultaneously acquire each frame of a moving image and its three-dimensional camera position and three-dimensional camera posture, an image frame and a measurement sampling time are used. Since it has to be highly accurate and completely synchronized, it becomes a huge device, which is practically difficult to realize.

CV data obtained by calculation from a moving image is a relative value when not processed, but if it is a short section, three-dimensional position information and three-axis rotation angle information can be acquired with high accuracy.
In addition, since the CV data is acquired from the image, the acquired data is a relative value, but it can be realized by other methods that can measure the positional relationship with an arbitrary object in the image. With characteristics.
Further, since the CV value corresponding to the image can be obtained, the CV calculation capable of obtaining the camera position and its three-axis rotation posture directly from the image in the in-image measurement or survey is suitable for the in-image measurement or in-image survey. .
The mobile automatic monitoring apparatus of the present invention generates / acquires position coordinate information of the monitoring camera mounted on the moving body based on the CV value data obtained by the CV calculation.

[(Feature point tracking method) CV calculation unit]
The feature point tracking method CV calculation is performed by the position coordinate acquisition unit 20 (see FIG. 1) or the CV calculation unit functioning as the all-around video CV calculation unit 20a (see FIG. 19) to be described later. Done.
FIG. 2 is a block diagram showing a basic configuration of an embodiment of the CV calculation unit 20 that functions as a position coordinate acquisition unit of the mobile automatic monitoring apparatus of the present invention.
As shown in FIG. 2, the CV calculation unit 20 performs a predetermined CV calculation process on the video image or the like input from the spatial information acquisition unit 10. Specifically, the CV calculation unit 20 and the feature point extraction unit 21 , A feature point correspondence processing unit 22, a camera vector calculation unit 23, an error minimization unit 24, a three-dimensional information tracking unit 25, and a high-precision camera vector calculation unit 26.
Then, the CV value obtained here is input to the integrated space information unit 30.

First, any video may be used for the CV calculation. However, in the video with a limited angle of view, the video is interrupted when the viewpoint direction is moved. ) Is desirable. Note that a moving image is similar to a continuous still image, and a continuous still image can be handled in the same manner as a moving image.
In addition, as the video, a video captured in real time according to the movement of the vehicle acquired by the spatial information acquisition unit 10 of the mobile automatic monitoring device 1 is used, but when real-time monitoring is not necessary, Of course, it is also possible to use a moving image in which the monitoring area is previously captured and recorded.

In the present embodiment, as an image used for CV calculation, an all-around image (see FIGS. 3 to 4) obtained by photographing the entire 360 ° circumference of a moving body such as a vehicle, or a wide-angle image close to the all-around image is used. The CV calculation is performed by treating the image as a part of the all-around video and performing the CV calculation, or by developing the entire circumference video in a plane direction in the viewpoint direction.
The planar development of the all-around video is to express the all-around video as a normal image in perspective. Here, the term "perspective method" is used because the entire circumference image itself is displayed by a method different from the perspective method, such as the Mercator projection method and the spherical projection method (see FIG. 5C). This is because it can be converted and displayed as a normal perspective image by displaying it in a plane.

In order to generate an omnidirectional video in the spatial information acquisition unit 10 of the mobile automatic monitoring device 1, first, as shown in FIGS. 3 and 4, the omnidirectional video camera 11 is used to acquire CV value data. For the purpose, the entire periphery video camera 11 fixed to the moving body 10a such as a traveling vehicle photographs the periphery of the moving body as the moving body 10a moves.
Note that the moving body 10a may be provided with a position measuring device or the like constituted by, for example, a GPS device alone or an IMU device to which absolute coordinates are acquired for the purpose of acquiring the position coordinates.
Further, the all-round video camera 11 mounted on the moving body 10a may have any configuration as long as it captures and acquires a wide range of images. For example, a wide-angle lens, a camera with a fisheye lens, a moving camera, There are a fixed camera, a camera in which a plurality of cameras are fixed, a camera that can rotate around 360 degrees, and the like. In this embodiment, as shown in FIGS. 3 and 4, a plurality of cameras are integrally fixed to the vehicle, and an all-round video camera 11 that captures a wide range image as the moving body 10 a moves is used. .

Then, according to the all-around video camera 11 as described above, as shown in FIG. 4, 360-degree all-around video of the camera is simultaneously captured by a plurality of cameras by being installed on the ceiling of the moving body 10a. The moving body 10a moves, and a wide range video can be acquired as moving image data.
Here, the omnidirectional video camera 11 is a video camera that can directly acquire the omnidirectional video of the camera, but can be used as the omnidirectional video if more than half of the entire circumference of the camera can be acquired as video.
Even in the case of a normal camera with a limited angle of view, the accuracy of the CV calculation is reduced, but it can be handled as a part of the entire peripheral video.

Note that the wide-range video imaged by the all-around video camera 11 can be pasted as a single image on a virtual spherical surface that matches the angle of view at the time of imaging.
The spherical image data pasted on the virtual spherical surface is stored and output as spherical image (360 degree image) data pasted on the virtual spherical surface. The virtual spherical surface can be set to an arbitrary spherical shape centered on the camera unit that acquires a wide range image.
FIG. 5A is an appearance image of a virtual spherical surface to which a spherical image is pasted, and FIG. 5B is an example of a spherical image pasted to the virtual spherical surface. FIG. 4C shows an example of an image obtained by developing the spherical image of FIG.

Then, the all-round video image generated / acquired as described above is input to the CV calculation unit 20 to obtain CV value data (see FIG. 2).
In the CV calculation unit 20, first, the feature point extraction unit 21 has a sufficient number of feature points (references) from the moving image data that is captured and temporarily recorded by the all-around video camera 11 of the all-around video image unit 10. Point) is automatically extracted.
The feature point correspondence processing unit 22 automatically obtains the correspondence relationship by automatically tracking the feature points automatically extracted in each frame image between the frames.
The camera vector calculation unit 23 automatically calculates a camera vector corresponding to each frame image from the three-dimensional position coordinates of the feature points for which the correspondence relationship has been determined.
The error minimizing unit 24 performs statistical processing so that the solution distribution of each camera vector is minimized by overlapping calculation of a plurality of camera positions, and automatically determines the camera position direction subjected to the error minimizing process. .

The three-dimensional information tracking unit 25 positions the camera vector obtained by the camera vector calculation unit 23 as an approximate camera vector, and based on the three-dimensional information sequentially obtained as part of the image in the subsequent process, a plurality of frame images 3D information is automatically tracked along an image of an adjacent frame. Here, three-dimensional information (three-dimensional shape) is mainly three-dimensional distribution information of feature points, that is, a collection of three-dimensional points, and this collection of three-dimensional points constitutes a three-dimensional shape. To do.
The high-precision camera vector calculation unit 26 generates and outputs a higher-precision camera vector than the camera vector obtained by the camera vector calculation unit 23 based on the tracking data obtained by the three-dimensional information tracking unit 25.
Then, the camera vector obtained as described above is input to the mobile automatic monitoring apparatus 1 to be described later and used for the position coordinate acquisition process and the integrated space information process.

There are several methods for detecting a camera vector from feature points of a plurality of images (moving images or continuous still images). In the CV calculation unit 20 of the present embodiment shown in FIG. By automatically extracting a number of feature points and automatically tracking them, a three-dimensional vector and a three-axis rotation vector of the camera are obtained by epipolar geometry.
By taking a sufficient number of feature points, camera vector information is duplicated, and an error can be minimized from the duplicated information to obtain a more accurate camera vector.

A camera vector is a vector of degrees of freedom possessed by a camera.
In general, a stationary three-dimensional object has six degrees of freedom of position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) of the respective coordinate axes. Therefore, the camera vector refers to a vector of six degrees of freedom of the camera position coordinates (X, Y, Z) and the rotation angles (Φx, Φy, Φz) of the respective coordinate axes. When the camera moves, the direction of movement also enters the degree of freedom, which can be derived by differentiation from the above six degrees of freedom.
Thus, the detection of the camera vector in the present embodiment means that the camera takes six degrees of freedom values for each frame and determines six different degrees of freedom for each frame.

Hereinafter, a specific camera vector detection method in the CV calculation unit 20 will be described with reference to FIG.
First, the image data acquired by the all-round video camera 11 of the spatial information acquisition unit 10 described above is input to the feature point extraction unit 21 of the CV calculation unit 20 indirectly or directly. A point or a small area image that should be a feature point is automatically extracted from a frame image that is appropriately sampled, and a feature point correspondence processing unit 22 automatically obtains a correspondence relationship between feature points among a plurality of frame images. .
Specifically, more than a sufficient number of feature points that are used as a reference for detecting a camera vector are obtained. Examples of feature points between images and their corresponding relationships are shown in FIGS. In the figure, “+” is a feature point that is automatically extracted, and the correspondence is automatically tracked between a plurality of frame images (see correspondence points 1 to 4 shown in FIG. 8).
Here, for extracting feature points, as shown in FIG. 9, it is desirable to specify and extract a sufficiently large number of feature points in each image (see circles in FIG. 9). For example, about 100 feature points are extracted. Extract points.

Subsequently, the camera vector calculation unit 23 calculates the three-dimensional coordinates of the extracted feature points, and calculates the camera vector based on the three-dimensional coordinates. Specifically, the camera vector calculation unit 23 includes a sufficient number of feature positions that exist between successive frames, a position vector between moving cameras, a three-axis rotation vector of the camera, and each camera position and feature. Relative values of various three-dimensional vectors such as vectors connecting points are continuously calculated by calculation.
In this embodiment, for example, camera motion (camera position and camera rotation) is calculated by solving an epipolar equation from the epipolar geometry of a 360-degree all-round image.

Images 1 and 2 shown in FIG. 8 are images obtained by Mercator expansion of 360-degree all-round images. When latitude φ and light θ are assumed, points on image 1 are (θ1, φ1) and points on image 2 are ( θ2, φ2). The spatial coordinates of each camera are z1 = (cos φ1 cos θ1, cos φ1 sin θ1, sin φ1), z2 = (cos φ2 cos θ2, cos φ2 sin θ2, sin φ2). If the camera movement vector is t and the camera rotation matrix is R, z1T [t] × Rz2 = 0 is the epipolar equation.
By providing a sufficient number of feature points, t and R can be calculated as a solution by the method of least squares by linear algebra calculation. This calculation is applied to a plurality of corresponding frames.

Here, it is preferable to use a 360-degree all-round image as an image used for the calculation of the camera vector.
The image used for the camera vector calculation may be any image in principle, but a wide-angle image such as a 360-degree all-round image shown in FIG. 8 makes it easier to select many feature points. Therefore, in this embodiment, a 360-degree all-round image is used for the CV calculation, whereby the tracking distance of the feature points can be increased, and a sufficiently large number of feature points can be selected. It becomes possible to select a feature point convenient for each short distance. In addition, when correcting the rotation vector, the calculation process can be easily performed by adding the polar rotation conversion process. As a result, a calculation result with higher accuracy can be obtained.
In FIG. 8, in order to make it easy to understand the processing in the CV calculation unit 20, a 360-degree all-round spherical image obtained by synthesizing images taken by one or a plurality of cameras is developed by a Mercator projection called map projection. However, in an actual CV calculation, it is not necessarily a developed image by the Mercator projection.

Next, the error minimizing unit 24 calculates a plurality of vectors based on each feature point according to a plurality of calculation equations based on the plurality of camera positions and the number of feature points corresponding to each frame, Statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized to obtain a final vector. For example, the optimal solution of the least square method is estimated by the Levenberg-Marquardt method for multiple frames of camera position, camera rotation, and multiple feature points, and the error is converged to obtain the camera position, camera rotation matrix, and feature point coordinates. .
Further, feature points having a large error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of computation at each feature point and camera position.
In this way, the position of the feature point and the camera vector can be obtained with high accuracy.

10 to 12 show examples of the three-dimensional coordinates of feature points and camera vectors obtained by CV calculation. 10 to 12 are explanatory diagrams showing the vector detection method by the CV calculation according to the present embodiment, and showing the relative positional relationship between the camera and the object obtained from a plurality of frame images acquired by the moving camera. FIG.
FIG. 10 shows the three-dimensional coordinates of the feature points 1 to 4 shown in the images 1 and 2 in FIG. 8 and the camera vector (X, Y, Z) that moves between the images 1 and 2.
11 and 12 show a sufficiently large number of feature points, the positions of the feature points obtained from the frame image, and the position of the moving camera. In the figure, a circle mark that continues in a straight line at the center of the graph is the camera position, and a circle mark located around the circle indicates the position and height of the feature point.

Here, in the CV calculation in the CV calculation unit 20, in order to obtain more accurate three-dimensional information of feature points and camera positions at a high speed, as shown in FIG. Set feature points and repeat multiple operations.
Specifically, the CV calculation unit 20 automatically detects a feature point having a video feature in an image, and obtains a corresponding point of the feature point in each frame image. And the n + m-th two frame images Fn and Fn + m are used as unit calculations, and unit calculations with n and m appropriately set can be repeated.
m is the frame interval, and the feature points are classified into a plurality of stages according to the distance from the camera to the feature point in the image. The distance from the camera to the feature point is set so that m becomes larger. It is set so that m is smaller as the distance to is shorter. This is because the change in position between images is less as the distance from the camera to the feature point is longer.

Then, while sufficiently overlapping the classification of the feature points by the m value, a plurality of stages of m are set, and as n progresses continuously with the progress of the image, the calculation proceeds continuously. Then, the overlap calculation is performed a plurality of times for the same feature point in each step of n and m.
In this way, by performing unit calculation focusing on the frame images Fn and Fn + m, a precise camera vector is calculated over a long time between frames sampled every m frames (frames are dropped). However, in m frames (minimum unit frames) between the frame images Fn and Fn + m, a simple calculation that can be performed in a short time can be performed.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames overlap with the Fn and Fn + m camera vectors that have been subjected to the high precision calculation. Accordingly, m minimum unit frames between Fn and Fn + m are obtained by a simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by the simple calculation are Fn and Fn + m obtained by high precision calculation. The scale adjustment of m consecutive camera vectors can be made to match the camera vectors.
In this way, as n progresses continuously with the progress of the image, the scale adjustment is performed so that the error of each camera vector obtained by calculating a plurality of times for the same feature point is minimized, and integration is performed. A camera vector can be determined. Accordingly, it is possible to speed up the arithmetic processing by combining simple arithmetic operations while obtaining a highly accurate camera vector having no error.

Here, there are various simple calculation methods depending on the accuracy. For example, when (1) many feature points of 100 or more are used in high-precision calculation, the minimum number of simple calculation is about ten. (2) Even if the number of the same feature points is the same as the feature points and camera positions, innumerable triangles are established there, and equations for that number are established. By reducing the number of equations, it can be simplified.
In this way, integration is performed by adjusting the scale so that the error of each feature point and camera position is minimized, distance calculation is performed, and feature points with large error distribution are deleted, and other features are added as necessary. By recalculating the points, the calculation accuracy at each feature point and camera position can be improved.

  In addition, by performing high-speed simple calculation in this way, camera vector real-time processing becomes possible. In real-time processing of camera vectors, calculation is performed with the minimum number of frames that can achieve the target accuracy and the minimum number of feature points that are automatically extracted, the approximate value of the camera vector is obtained and displayed in real time, and then the image is accumulated. Accordingly, the number of frames can be increased, the number of feature points can be increased, camera vector calculation with higher accuracy can be performed, and approximate values can be replaced with camera vector values with higher accuracy for display.

Furthermore, in this embodiment, it is possible to track three-dimensional information (three-dimensional shape) in order to obtain a more accurate camera vector.
Specifically, first, the three-dimensional information tracking unit 25 positions the camera vector obtained through the camera vector calculation unit 23 and the error minimization unit 24 as an approximate camera vector, and the image generated in the subsequent process. Based on the three-dimensional information (three-dimensional shape) obtained as a part, partial three-dimensional information included in a plurality of frame images is continuously tracked between adjacent frames to automatically track the three-dimensional shape.
Then, from the tracking result of the three-dimensional information obtained by the three-dimensional information tracking unit 25, a high-precision camera vector is obtained by the high-precision camera vector calculation unit 26.

In the feature point extraction unit 21 and the feature point correspondence processing unit 22 described above, feature points are automatically tracked in a plurality of inter-frame images, but the number of feature point tracking frames is limited due to disappearance of feature points. May come. In addition, since the image is two-dimensional and the shape changes during tracking, there is a certain limit in tracking accuracy.
Therefore, the camera vector obtained by the feature point tracking is regarded as an approximate value, and the three-dimensional information (three-dimensional shape) obtained in the subsequent process is traced on each frame image, and a high-precision camera vector is obtained from the trajectory. Can do.
The tracking of 3D shapes is easy to obtain matching and correlation accuracy, and the 3D shape does not change in size and size depending on the frame image, so it can be tracked over many frames. The accuracy of the camera vector calculation can be improved. This is possible because the approximate camera vector is already known by the camera vector calculation unit 23 and the three-dimensional shape is already known.

  When the camera vector is an approximate value, the error of 3D coordinates over a very large number of frames is small because there are few frames related to each frame by feature point tracking. The error of the three-dimensional shape when a part of the image is cut is relatively small, and the influence on the change and size of the shape is considerably small. For this reason, the comparison and tracking in the three-dimensional shape is extremely advantageous over the two-dimensional shape tracking. In tracking, when tracking with 2D shape, tracking changes in shape and size in multiple frames are unavoidable, so there are problems such as large errors and missing corresponding points. However, in tracking with a three-dimensional shape, there is very little change in shape, and in principle there is no change in size, so accurate tracking is possible.

Here, as the three-dimensional shape data to be tracked, there are, for example, a three-dimensional distribution shape of feature points, a polygon surface obtained from the three-dimensional distribution shape of feature points, and the like.
It is also possible to convert the obtained three-dimensional shape from a camera position into a two-dimensional image and track it as a two-dimensional image. Since the approximate value of the camera vector is known, projection conversion can be performed on a two-dimensional image from the camera viewpoint, and it is also possible to follow a change in the shape of the object due to movement of the camera viewpoint.

The camera vector obtained as described above can be displayed in an overlapping manner in the video image shot by the all-round video camera 11.
For example, as shown in FIG. 14, the image from the in-vehicle camera is developed in a plane, the corresponding points on the target plane in each frame image are automatically searched, and the corresponding points are combined to match the target plane. A combined image is generated and displayed in the same coordinate system.
Furthermore, the camera position and the camera direction can be detected one after another in the common coordinate system, and the position, direction, and locus can be plotted. The CV data indicates the three-dimensional position and the three-axis rotation, and the CV value can be observed simultaneously in each frame of the video image by displaying it over the video image. An example of an image displayed by superimposing CV data on a video image is shown in FIG.
If the camera position is correctly displayed in the video image, the position in the video image indicated by the CV value is the center of the image, and if the camera movement is close to a straight line, the CV values of all frames are displayed overlapping. Therefore, for example, as shown in FIG. 14, it is appropriate to display a position of 1 meter right below the camera position. Alternatively, it is more appropriate to display the CV value at the height of the road surface based on the distance to the road surface.

[Three-dimensional designated point correspondence method CV calculation]
Next, the three-dimensional designated point corresponding method CV calculation, which is an application of the above feature point tracking method CV calculation, will be described in detail with reference to FIGS.
In the following description of the CV calculation for the three-dimensional designated point correspondence method, a CV video three-dimensional map with a known CV value is used as a reference video, and a video acquired in real time with an unknown CV value is used as a comparative video. I use it.

First, with reference to FIG. 15 to FIG. 17, the three-dimensional designated point corresponding CV calculation for performing the high-speed CV calculation in the integrated space information unit 10 of the mobile automatic monitoring apparatus 1 described above will be described.
FIG. 15 is a block diagram showing a basic configuration of the CV calculation unit 100 that performs CV calculation of the three-dimensional designated point correspondence method. 16 and 17 are block diagrams showing details of the CV calculation unit 100 of the three-dimensional designated point correspondence method shown in FIG.

[(Three-dimensional designated point correspondence method) CV calculation]
As shown in FIG. 15, the CV calculation unit 100 of the three-dimensional designated point correspondence method is a device for integrating the real-time video into the coordinates of the CV video three-dimensional map (three-dimensional map). Video three-dimensional map unit 101, CV adding unit 102, three-dimensional designated point designating unit 103, real-time video unit 104, initial position setting unit 105, three-dimensional designated point correspondence unit 106, correspondence designated point tracking unit 107, a CV acquisition unit 108, and a coordinate integration unit 109.

The CV video three-dimensional map unit 101 holds a CV video three-dimensional map (three-dimensional map) made up of a captured video image as a reference video.
The CV adding unit 102 performs the above-described CV calculation (feature point tracking method CV calculation) on each frame of the CV video three-dimensional map input from the CV video three-dimensional map unit 101, and is calibrated with absolute coordinates. A CV value (a three-dimensional value of the camera position and orientation) is added. This is preferably calculated and added in advance when the CV video 3D map part is created.
The three-dimensional designated point designating unit 103 designates an arbitrary three-dimensional designated point automatically or manually in the CV video three-dimensional map to which the CV value having the absolute coordinates is added by the CV adding unit 102 and added. Based on the CV value, the three-dimensional coordinates of an arbitrary designated three-dimensional designated point are acquired.

The real-time video unit 104 holds, as a comparison video, a real-time video composed of a moving video to be compared with the CV video 3D map held in the CV video 3D map unit 101.
The initial position setting unit 105 automatically or manually initializes the correspondence between common points in the CV video 3D map and the real-time video.
The three-dimensional designated point corresponding unit 105 includes a CV video three-dimensional map specified by the three-dimensional designated point specifying unit 103 over each frame of a moving image starting from the frame initially set by the initial position setting unit 105. The part corresponding to the three-dimensional designated point is automatically associated in the real-time video, and the corresponding designated point is continuously searched over each frame.
The correspondence designated point tracking unit 107 tracks the correspondence designated points associated by the three-dimensional designated point correspondence unit 106 over each frame in which the real-time video proceeds.

The CV acquisition unit 108 transplants the three-dimensional coordinates of the three-dimensional designated point of the CV video three-dimensional map to the correspondence designated point of the real-time video based on the correspondence result of the correspondence designated point. The CV value is obtained by the above-described CV calculation (feature point tracking method CV calculation), and the CV value is obtained over each frame by tracking. That is, the CV acquisition unit 108 is configured such that the correspondence between the CV video three-dimensional map and the real-time video is obtained by the three-dimensional designated point correspondence unit and the correspondence designated point tracking unit. If the coordinates are transplanted to the real-time video and there are three or more points, the camera position and orientation of the real-time video can be obtained, and the CV value can be acquired. In practice, it is desirable to increase the accuracy by obtaining a response of several tens of points.
The coordinate integration unit 109 coordinate-integrates the CV video 3D map to which the CV value is added by the CV addition unit 102 and the real-time video from which the CV value is obtained by the CV acquisition unit 108 in the same coordinate system.

[Coordinate integration processing]
Hereinafter, the projection coordinate integration process (coordinate integration method) of the CV video three-dimensional map and the real-time video in the CV calculation unit 100 of the three-dimensional designated point correspondence method will be described.
First, considering the efficiency of the entire system of the CV calculation unit 100 of the three-dimensional designated point correspondence method, even if a burden is imposed on the CV video three-dimensional map (three-dimensional map) that can be calculated in advance, real-time processing is performed. It is necessary to reduce the burden of computation on the real-time video (real-time video) side to be performed.
Therefore, in order to enable real-time calculation by the real-time video unit 104, the initial position setting unit 105, and the three-dimensional designated point correspondence unit 106, the calculation on the CV video three-dimensional map side is performed as much as possible. It is preferable to perform the process in advance.

In the three-dimensional designated point correspondence unit 106, the correspondence relationship between the correspondence designated points is searched over all frames, so that the three-dimensional coordinates of the correspondence designated points can be transplanted to the real-time video side. With respect to the real-time video, the CV value can be acquired for each frame.
From the principle of triangle, if there is a transplant of at least three 3D coordinates per frame, the CV value can be obtained by calculation, but in order to ensure accuracy, transplantation of several tens or more is necessary. It is.
In addition, since the three-dimensional designated point has three-dimensional coordinates and the viewpoint can be changed to the same CV value as the CV value of the real-time video, the correspondence between the CV video three-dimensional map and the real-time video becomes easy.

Further, in the three-dimensional designated point correspondence unit 106, correspondence is taken intermittently in the representative frame, not in all frames, and for other frames between the representative frames, the correspondence designated point is tracked in the real-time video. The CV value can also be obtained by obtaining the CV value.
In addition, CV value acquisition by all frames and CV value acquisition by tracking corresponding designated points are combined to obtain corresponding designated points over all frames, transplantation of three-dimensional coordinates, and further from the tracking results of corresponding designated points. By obtaining the CV value, a more accurate CV value can be obtained.

Here, it is important that the three-dimensional designated point is not two-dimensional but three-dimensional. Using a plurality of three-dimensional designated points whose three-dimensional coordinates are known in advance, tracking in the CV video three-dimensional map in advance, and corresponding the three-dimensional shape in the real-time video, or a plurality of tertiary By tracking the original designated point in the CV video 3D map, the real-time video, and both videos at the same time, it is possible to take a 3D correspondence between the videos.
Therefore, the following embodiments are characterized by providing a three-dimensional designated point in a CV video three-dimensional map, taking correspondence with a real-time video, and tracking the three-dimensional designated point in the real-time video.
However, if a three-dimensional designated point is used, three-dimensional coordinates are transplanted, so that the CV value can be obtained for only one frame on the real-time video side. This is a significant difference compared to the case of the CV calculation (see FIGS. 2 to 14) of the feature point tracking method that requires tracking results over a plurality of frames.

Note that there is a premise that the correspondence between the CV video three-dimensional map and the real-time video in the three-dimensional designated point correspondence unit 106 is considerably more difficult than the correspondence (tracking) in the same image.
Therefore, in the process of the three-dimensional designated point corresponding unit 106, it is fully possible that a frame that fails to acquire the corresponding point is generated. It can also be assumed that three or more corresponding points cannot be obtained. Therefore, the three-dimensional designated point corresponding unit 106 obtains corresponding points as much as possible, and even if a frame that fails to correspond is generated, that portion is tracked on the real-time video side only by the real-time video side and the feature point tracking method CV It is practical to calculate and supplement the CV value of the missing part.
The correspondence processing in the three-dimensional designated point correspondence unit 106 will be further described.

[Three-dimensional designated point correspondence processing]
Hereinafter, the correspondence processing in the three-dimensional designated point correspondence unit 106 will be described more specifically with reference to FIGS. 16 and 17.
In general, a CV video three-dimensional map and a real-time video are different from each other in camera, image quality, and shooting conditions. Therefore, it is difficult to establish a correspondence between the CV video three-dimensional map and the real-time video. In this respect, as a conventional method, there has been performed a method of tracking a texture of a minute region constituting a three-dimensional point by matching or correlation. However, with such a conventional method, it has been extremely difficult to accurately obtain the correspondence because of different image quality images between different cameras and differences in shooting conditions.

Therefore, in this embodiment, the correspondence between the CV video 3D map and the real-time video is performed by a new method as shown in FIGS. 16 and 17 without using the conventional 2D small area matching.
In the three-dimensional designated point correspondence method CV calculation unit 100 shown in FIG. 16, in correspondence with the three-dimensional designated point on the CV video three-dimensional map side, the three-dimensional designated designated point is made to correspond on the real-time video side, and the three-dimensional coordinates are transplanted. Thus, the CV calculation is performed.
Further, in the three-dimensional designated point correspondence method CV calculation unit 100 shown in FIG. 17, since the three-dimensional shape is known, the direction on the real-time video side is specified, and the CV value is obtained from the direction. Yes.

[3D shape support]
In the three-dimensional designated point correspondence method CV computing unit 100 shown in FIG. 16, the three-dimensional designated point correspondence unit 106 shown in FIG. 15 includes a three-dimensional designated point shape correspondence unit 106a, and the correspondence designated point tracking unit 107 has a three-dimensional design. A designated point shape tracking unit 107a is provided.
The three-dimensional designated point shape correspondence unit 106a focuses on a three-dimensional shape composed of a plurality of three-dimensional designated points in the CV video three-dimensional map in the three-dimensional designated point correspondence unit 106, and The correspondence of the three-dimensional designated point is obtained by obtaining a CV value such that the three-dimensional corresponding point having the same coordinates as the designated point matches the three-dimensional shape.
Further, the 3D designated point shape tracking unit 107a causes the corresponding designated point tracking unit 107 to track the corresponding designated point over a plurality of adjacent frames in the real-time video, so that a known tertiary in the CV video 3D map is obtained. Implant the 3D coordinates of the original shape directly to the tracking point in the real-time video.

In general, in a two-dimensional image, a corresponding point is a two-dimensional image of a small area, and the corresponding point is obtained by matching and correlation by comparing textures. Comparison between maps and real-time video is extremely difficult.
Therefore, in the three-dimensional designated point correspondence method CV calculation unit 100 shown in FIG. 16, the designated point correspondence to be compared is not a two-dimensional texture, but a three-dimensional texture, and a three-dimensional coordinate in the image, that is, a three-dimensional shape. By using the 3D coordinate relationship itself, it is possible to compare these CV video 3D maps and real-time video even if there are various physical condition differences.

For example, considering a tetrahedron composed of four 3D designated points in a CV video 3D map, this is the same in real-time video only in 3D measurement information in the real-time video regardless of the difference in image quality. You can search for a tetrahedron of shape.
This is because the three-dimensional shape and its coordinates can be obtained in the real-time video by assuming the CV value in the real-time video, and the correspondence as the coordinates can be obtained instead of the correspondence as the image. Means. Therefore, since the 3D shape does not change depending on the distance from the camera, regardless of the image quality due to the difference in the camera, the 3D shape does not change depending on the distance from the camera. Is possible, which is extremely advantageous in terms of accuracy.

Even in this method, it is necessary to search for a three-dimensional shape in which the projections form a solid and each three-dimensional designated point has an appropriate distribution. Even if a large number of three-dimensional designated points are taken on the surface, an appropriate tertiary can be obtained. The original shape cannot be constructed.
In the present embodiment, an optimal CV value is searched for while making a minute change (meaning a viewpoint change) using the CV value as a variable in the real-time video.
The correct CV value can be obtained by continuing to check whether or not it exists at the position obtained by calculation from the CV value. Alternatively, it is possible to search while changing the viewpoint using the CV value as a variable on the CV video three-dimensional map side.
In other words, it is free to change the viewpoint on the CV video 3D map side to match the real time video, or to change the viewpoint on the real time video side to match the real time video.

Furthermore, it is also possible to take a correspondence by focusing on only the three-dimensional coordinates that do not change depending on the viewpoint without changing the viewpoint, that is, without changing the apparent shape of the three-dimensional shape.
Also, a plurality of CV value acquisition methods can be used in combination, the approximate value of the CV value and the viewpoint change by it can be sufficiently performed, and the CV value can be asymptotically obtained in the vicinity thereof.
If the correspondence of the three-dimensional designated point fails by one method, another CV value acquisition method may be used. Even if all correspondence fails, the initial value setting is repeated to continuously CV value can be obtained.

[Directional support]
In the three-dimensional designated point correspondence method CV computing unit 100 shown in FIG. 17, the three-dimensional designated point correspondence unit 106 shown in FIG. 15 includes a three-dimensional designated point direction correspondence unit 106b, and the correspondence designated point tracking unit 107 is three-dimensional. A designated point direction tracking unit 107b is provided.
The three-dimensional designated point direction correspondence unit 106b pays attention to the azimuth elevation angle from the camera position of the plurality of three-dimensional designated points in the three-dimensional designated point correspondence unit 106, and the three-dimensional designated point in the real-time video assumes a CV value. The corresponding point in the real-time video is made to correspond to the three-dimensional designated point by confirming that it exists at the azimuth elevation angle expected by the calculation in this case.
In the corresponding designated point tracking unit 107, the three-dimensional designated point direction tracking unit 107b converts the three-dimensional designated point in the three-dimensional map of the CV video into the three-dimensional shape whose three-dimensional coordinates are known in the real-time video. Are tracked over a plurality of adjacent frames.

In the three-dimensional designated point correspondence method CV calculation unit 100 shown in FIG. 17, a three-dimensional designated point having a known three-dimensional coordinate is used as the three-dimensional designated point in the CV video three-dimensional map. The corresponding point is not a shape but only its direction, and it can be compared with the viewpoint direction from the camera of the three-dimensional coordinate of the three-dimensional designated point that can be predicted by the calculation when the CV value is assumed.
That is, if the direction in the real-time video of the three-dimensional designated point whose three-dimensional coordinates are known can be acquired, the CV value corresponding to the direction can be obtained by calculation.
As described above, various methods have been described for the correspondence method in the three-dimensional designated point correspondence unit 106 and the tracking method in the correspondence designated point tracking unit 107, but these can be arbitrarily combined or used together. By using properly, the CV value can be acquired easily, accurately, and quickly for an image of any condition.

[Other Embodiments of Three-dimensional Designated Point Corresponding Method CV Calculation Unit]
Next, with reference to FIG. 18, another embodiment of the above-described three-dimensional designated point correspondence method CV calculation unit 100 will be described.
FIG. 18 is a block diagram showing a basic configuration of a modified embodiment of the three-dimensional designated point corresponding method CV calculation unit shown in FIG.
As shown in the figure, the present embodiment is a modified embodiment of the above-described three-dimensional designated point correspondence method CV calculation unit 100 (see FIG. 15). Specifically, as shown in FIG. In addition to the configuration of the three-dimensional designated point correspondence method CV calculation unit 100, the general reference space configuration unit 110, the general CV acquisition unit 111, the general comparison space configuration unit 112, the CV calibration unit 113, and the space correspondence unit 114 are further included. It has each part.

The general reference space configuration unit 110 generates a general 3D space of the CV video 3D map from the CV value acquired by the CV adding unit 102 with respect to the CV video 3D map.
On the other hand, the general CV acquisition unit 111 detects and tracks feature points on the real-time video side, independently of the CV video three-dimensional map, and independently performs real-time video (real-time video). An approximate CV value indicating the acquired position and orientation of the camera is acquired.

The approximate comparison space configuration unit 112 generates an approximate comparison space for real-time video from the approximate CV value acquired by the approximate CV acquisition unit 111 and the like.
The CV calibration unit 113 calibrates the relative CV value of the real-time video with the CV value obtained by the three-dimensional designated point corresponding unit 106 and the corresponding designated point tracking unit 107, and converts it into the coordinate system of the CV video three-dimensional map. To do.
The space correspondence unit 114 compares and associates the approximate three-dimensional space of the CV video three-dimensional map with the approximate three-dimensional space of the real-time video, roughly matches each space, and the corresponding region in the three-dimensional designated point corresponding unit 1106 Narrow down.

The CV video 3D map and the real-time video are originally composed of close positions in the same space, and there is only a slight difference in viewpoint due to a slightly different camera position.
In the rough reference space constructing unit 110 of the present embodiment, a rough three-dimensional shape of a CV video three-dimensional map is generated. Thus, it is possible to roughly compare with real-time video.
In addition, based on the three-dimensional coordinates of many feature points acquired when obtaining the CV value, the approximate reference space configuration unit 110, the approximate comparison space configuration unit 112, and the spatial correspondence unit 114 compare the approximate space from the distribution thereof. Is possible.

In addition, many feature points of CV video 3D map and real-time video are detected in the calculation process, and further, they are tracked to obtain many 3D feature points. Accordingly, a three-dimensional distribution of these three-dimensional feature points is generated, and the space correspondence unit 114 compares the three-dimensional shapes of the respective spaces, and searches for the positions where the respective spaces are matched. The video can be roughly compared.
That is, the approximate positional relationship can be obtained.

On the other hand, as described above, the CV calculation unit 100 shown in FIG. 15 can transplant the 3D coordinates as they are by tracking the 3D specified points in the real-time video. Corresponding points in the real-time video with respect to the three-dimensional designated point in the map (three-dimensional map) are directly obtained as three-dimensional coordinates, and can be directly coordinated with the CV video three-dimensional map.
Therefore, in the embodiment shown in FIG. 18, the feature point tracking CV calculation (see FIGS. 2 to 14) is used on the real-time video side in the three-dimensional designated point corresponding method CV calculation 100 that can be expected to speed up the calculation time. By doing so, it is possible to calibrate the CV calculation while increasing the speed.

In addition, when the CV calculation of the feature point tracking method and the CV calculation of the three-dimensional point tracking are used in combination, the three-dimensional point tracking does not necessarily have to be performed over the entire frame, and a necessary part requiring calibration is not necessary. Only frames will suffice, enabling faster and more efficient arithmetic processing.
Thus, according to the present embodiment, the CV value can be obtained with higher accuracy by properly using the method of the first embodiment and the method of the present embodiment according to the properties of the image. Become.

As described above, according to the mobile automatic monitoring apparatus 1 according to the present embodiment, the monitoring object (predicted object, obstacle, A suspicious object etc.) can be detected and the zoom camera can zoom up there. Thereby, a suspicious object is discovered with the all-around camera, the three-dimensional position is calculated | required, and a zoom camera can zoom up with respect to the coordinate. In addition, for existing objects, attribute registration, attribute update, and attribute search attribute display can be performed by clicking on the object.
In the mobile automatic monitoring apparatus 1 of the present embodiment, a sufficient number of feature points are automatically detected from a plurality of frame images of a moving image captured by a camera mounted on the vehicle, and feature points are automatically detected between frames. By tracking, it is possible to calculate CV data indicating the camera position and the rotation angle with high accuracy by performing overlapping calculation on a large number of feature points, and thereby to accurately calculate the three-dimensional position coordinates of the camera moving with the vehicle. To get to.

As a result, it is possible to perform three-dimensional measurement of an object to be monitored by a camera mounted on a vehicle and detection / recognition of an arbitrary object to be monitored. The entire image is monitored, and a detailed enlarged image can be obtained for the monitoring target.
Therefore, according to the mobile automatic monitoring apparatus 1 of the present embodiment, the road surface shape and the monitoring object (predicted object, obstacle, suspicious object) are eliminated in the monitoring area even at night or the like. It is possible to detect the three-dimensional shape of any monitoring object while traveling or stopping even in the snow conditions of fresh snow with the worst conditions. A complete and reliable monitoring system and security system can be realized.

Next, a more specific embodiment of the mobile automatic monitoring apparatus according to the present invention for automatically monitoring an object by a camera mounted on a vehicle based on the CV value obtained as described above will be described. This will be specifically described with reference to FIG.
[Mobile automatic monitoring equipment in airports]
First, an embodiment of an in-airport mobile automatic monitoring apparatus that automatically monitors the inside of an airport using the mobile automatic monitoring apparatus of the present invention will be described with reference to FIGS.
FIG. 19 is a functional block diagram showing a schematic configuration of the in-airport mobile automatic monitoring apparatus according to the present embodiment.
20 and 21 are explanatory diagrams schematically showing a vehicle equipped with the in-airport mobile type automatic monitoring device shown in FIG. 19, where FIG. 20 shows a case when the weather is good, and FIG. Is the case.
FIG. 22 is a plan view schematically showing the entire airport monitored by the in-airport mobile automatic monitoring apparatus shown in FIG.

The mobile automatic monitoring apparatus 1A shown in these figures is a dedicated vehicle loaded with the mobile automatic monitoring apparatus 1 according to the present invention as a start-up inspection in the airport. This is a device that discovers objects to be monitored (predicted objects, obstacles, suspicious objects, etc.) and also finds damage to existing facilities.
First, the dedicated vehicle according to the present embodiment detects an abnormality in a monitored object while monitoring a predetermined course simultaneously with normal manual operation, or detects a monitored object (a predicted object, an obstacle, a suspicious object). If an object is discovered, it will be possible to perform an automatic monitoring operation by an automatic driving function such as observing close to the target object, making self-judgment, and sending data to the central monitoring unit if self-judgment is not possible It has become.

In general, at an airport, monitoring is necessary regardless of whether it is daytime or nighttime, and it is necessary to monitor not only when the weather is fine, but also when it is raining or snowing. And when it snows, it is necessary to acquire the shape etc. of the snowbank which snowed, to detect the object (predicted object, an obstruction, a suspicious object, etc.), and to recognize. In this embodiment, an automatic monitoring operation can be performed even during such snowfall.
A vehicle equipped with the in-airport mobile type automatic monitoring device 1A of the present embodiment is loaded with a set of dedicated monitoring devices on the vehicle and travels in the airport area. It is assumed that the driver can perform both normal driving and automatic driving.
As shown by the two-dot chain line in FIG. 22, the vehicle travels throughout the monitoring area while performing monitoring work along a course in which the monitoring area in the airport is determined.

The components of the in-airport mobile automatic monitoring device 1A mounted on the vehicle have substantially the same configuration as that of the mobile automatic monitoring device 1 shown in FIG. 1, and specifically, as shown in FIG. . In the figure, the same components as those of the mobile automatic monitoring apparatus 1 shown in FIG.
The spatial information acquisition unit 10 includes a plurality of devices that spatially grasp the surroundings of the vehicle. Specifically, an omnidirectional camera or wide-angle camera, a specified direction camera that captures an object in a specified direction, a traveling road surface camera that captures a road surface in the traveling direction, a laser scanner, and a radar are provided.
The all-around camera unit 11 is provided with a plurality of wide-angle lens cameras or all-around cameras (see FIGS. 20 to 21) in order to secure a field of view around the entire vehicle. In order to acquire detailed information of a portion that becomes a shadow of the vehicle, it is desirable to install a plurality of wide-angle cameras or a plurality of all-around cameras.
In addition, it is desirable that the all-around camera for snow accumulation be installed at a position higher than normal (see FIG. 21).

The traveling road surface camera unit 12 is a camera that captures the road surface in the traveling direction of the vehicle (see FIGS. 20 to 21), and it is desirable to capture the road surface with a plurality of cameras. By capturing the same road surface portion with a plurality of cameras, it can be captured as a three-dimensional image.
At night, the sensitivity of the all-around camera is lowered, so it is desirable to shoot with a lighting device at a short distance, or to install a reflective object, a small light, or the like in the vicinity. If 100 or more reflective objects or small lights are reflected in the field of view, spatial information can be acquired by the all-around camera.

The designated direction camera unit 13 is a camera mounted on a traveling vehicle, and has a function of acquiring an image in a designated direction according to a command from a designated direction control unit 60 described later.
The designated direction camera is preferably a camera with a zoom function that directs the camera in an arbitrary designated direction and captures an object at a designated angle of view, but may be a narrow angle camera.
If there are multiple objects that need to be monitored at the same time and multiple objects to be monitored (predicted objects, obstacles, suspicious objects, etc.) are found, it is difficult to handle them all with one camera. Become. Therefore, it is desirable to use a plurality of designated direction cameras.
Specifically, it is desirable to install a plurality of vehicles on the roof of the vehicle so as to surround the vehicle (see a plurality of “cameras” shown in FIG. 20A).

Here, if a monitored object or a monitored object (a predicted object, an obstacle, a suspicious object, etc.) is found, the designated direction camera is directed in that direction and automatically captured as an image. A plurality of cameras can be designated to capture the same object, and each camera can be designated to capture a different object.
In addition, by photographing the same object from different directions with a plurality of cameras, an image having parallax information can be acquired, analyzed in three dimensions.
Note that the target object may be an existing monitoring target, may be a discovered monitoring target (a predicted target, an obstacle, a suspicious object, etc.), or is still unidentified. It may be an object. Further, it may be a crack on the road surface or a monitoring target (predicted target, obstacle, suspicious object, etc.) that cannot be captured by the traveling road surface camera.

In addition, it is possible to obtain a high-resolution image having a parallax in a partial space or the entire space by partially overlapping the visual fields of the plurality of designated direction cameras and scanning the range or the entire area having the visual field. Since multiple cameras have parallax, they can be three-dimensionalized as they are. However, the all-around camera can acquire the entire image in an instant, but the method using a plurality of cameras takes time, so it is desirable to use them properly depending on the situation.
Furthermore, while monitoring during the day can be performed with a video camera in a specified direction, it is switched to an infrared camera at night. In order to use the infrared camera more effectively, it is effective to use an infrared illumination device together.

The laser scanner unit 14 emits a laser, scans it, and generates a three-dimensional shape of the scanning range. The laser scanner can be operated two-dimensionally by the device itself to generate the target three-dimensional shape. However, even if the scanning is only in one direction, the laser scanner can be operated in a plane. Is possible. Although it can be used in either case, a surface scanning type laser profiler is preferable.
In particular, during fresh snow, it is often subject to image analysis by laser scanning and laser pattern irradiation. Since it is assumed that only the image processing cannot cope with the snow accumulation situation of fresh snow, the laser scanner is particularly effective for detecting a snow bank at the time of snow accumulation.

The laser itself generates a three-dimensional shape in the scanning direction of the irradiated object, but it is possible to irradiate the object to be monitored with the irradiation pattern by the laser to make image analysis advantageous.
In a snow bank when there is a new snow cover, the snow surface cannot be captured effectively by video analysis alone, making analysis difficult. For this reason, especially when there is snow on fresh snow, the image of the image itself can be easily analyzed by irradiating the snow bank surface with a laser irradiation pattern and capturing it with a specified direction camera or all-around camera, and with multiple cameras. The following effects can be expected.
Moreover, nighttime accuracy is improved by attaching a plurality of laser scanning devices in a plurality of directions and acquiring a three-dimensional shape.
As the laser profiler device, LMS-210 manufactured by RIEGL of Austria is suitable.

The radar unit 15 emits a low-power radar radio wave and is effective in capturing a relatively large moving object such as an aircraft or a vehicle. If the moving object is a monitoring object, the whole image can be grasped, and even if the moving object is other than the monitoring object, it is effective for obtaining spatial information around the vehicle. This is particularly effective when the omni-directional camera such as nighttime is not sufficiently effective. However, the use of radar in the airport has a risk of adversely affecting control, aircraft, etc., so the use must be minimized.
It should be noted that monitoring at night or during snowfall is desirably handled by using a video scanner (all-around camera unit, traveling road surface camera unit, and designated direction camera unit) in combination with a laser scanner (laser scanner unit).

The position coordinate acquisition unit 20 is a part that acquires the position coordinates of the vehicle.
As shown in FIG. 19, first, the omnidirectional video CV calculation unit 20a acquires the omnidirectional video from the on-vehicle omnidirectional camera, and performs the above-described feature point tracking CV calculation (see FIGS. 2 to 14) and real time. This is a part having a function of acquiring the relationship between the three-dimensional position around the vehicle and the three-dimensional position and posture of the traveling vehicle by the three-dimensional designated point correspondence method CV calculation by comparing the video and the three-dimensional map. In this embodiment, since the video of the omnidirectional camera 11 of the spatial information acquisition unit 10 can be used, there is no need to newly install an omnidirectional camera for CV calculation.
In addition, during daytime, it can be obtained by shooting various surrounding objects with an omnidirectional camera, but nighttime spatial information is obtained by acquiring all illuminants installed in the surveillance area as an omnidirectional image. Become. In addition, in order to perform the CV calculation of the three-dimensional designated point correspondence method at night, the image and coordinates of the light emitter are registered in the three-dimensional map.

That is, each camera of the spatial information acquisition unit 10 is originally intended to shoot a monitoring object, but it can acquire not only spatial information but also three-dimensional coordinates of the vehicle position from the data. .
By capturing existing objects that can be photographed by the infrared camera of the designated direction camera unit 13, not only nighttime spatial information but also light with known three-dimensional coordinates can be obtained at night when there is little information. This contributes to the acquisition of position coordinates.

Furthermore, according to the 3D designated point correspondence method CV calculation, by continuously taking corresponding points between the video (spatial information) acquired in real time and the designated point in the 3D map and the spatial information, Compared with the feature point tracking CV calculation, position information can be acquired at a higher speed, which is particularly advantageous for real-time processing.
The correspondence between the 3D map and the video acquired in real time can be specified in the 3D map by designating various signs and marks in the airport surveillance area as designated points. May be searched for in the video acquired in real time as spatial information.

This method is effective because there are many signs and marks suitable for designated points at airports. An example that can be designated points is shown in FIG. In the figure, “▲” indicates a road surface that can be a designated point, “+” indicates a sign that can be a designated point, and “•” indicates a light that can be a designated point.
In areas where there are few signs and marks in the airport, a new designated mark prepared for designated purposes is installed at the site, and it is captured on the 3D map image, which is obtained when acquiring spatial information. The position coordinates and posture of the vehicle can be acquired by comparison with the designated point.

In order to obtain the camera position, that is, the vehicle position from the corresponding designated point, in principle, if there are three to four designated points, the three-dimensional coordinates and posture of the camera can be detected. Actually, it is sufficient if correspondence of about 15 to 20 designated points can be obtained, and if it is further taken near 100 points, highly accurate three-dimensional coordinates and postures can be acquired.
As a result, the CV value obtained by the CV calculation of the three-dimensional designated point correspondence method is the same as the CV value obtained by the CV calculation of the feature point tracking method, and without comparison with the three-dimensional map, It has the same meaning as the CV value of the feature point tracking method obtained by CV calculation by feature point tracking alone. That is, a real-time video acquired as spatial information can acquire a CV value by comparing it with a three-dimensional map. In other words, it cannot be distinguished whether the CV value obtained as a result is based on the feature point tracking method CV calculation or the three-dimensional designated point correspondence method CV calculation.
In the present embodiment, it is assumed that the three-dimensional designated point correspondence method CV calculation is performed by the integrated space information unit 40 shown in FIG.

The GPS / IMU / gyro unit 20b is a part that carries a GPS and an IMU and has a function of acquiring the three-dimensional coordinates and posture of the vehicle position (see FIGS. 20 and 21). A gyro can be substituted for the IMU.
The GPS and the IMU cannot acquire the surrounding spatial information, but operate regardless of day and night, and can acquire data indicating the vehicle position.
In addition, in the case where nothing can be seen due to fog, it is only possible to rely on the IMU, but it is usually advantageous to obtain the approximate position by GPS and obtain the CV value by comparing the 3D map and the real image.
Thus, it is not necessary for GPS and IMU to use all devices at the same time, and these devices can be switched and used as necessary. For example, in order to compare with a video, a CV value obtained from an all-round video that is absolute calibrated with GPS is accurate and useful.

The integrated spatial information unit 40 is a part having a function of integrating the three-dimensional spatial information acquired by the spatial information acquisition unit 10.
Specifically, the coordinate integration unit 41 of the integrated space information unit 40 includes an omnidirectional camera unit 11, a traveling road surface camera unit 12, a designated direction camera unit 13, a laser scanner unit 14, and a radar unit 15 of the spatial information acquisition unit 10. Spatial information is integrated by integrating data obtained from any one or a combination thereof and vehicle position data obtained by the position coordinate acquisition unit 20.
As described above, the spatial information is obtained not by a single device but by a combination of a plurality of pieces of information from various devices. In addition, spatial information can be acquired with a single device, but more precise and detailed spatial information can be acquired by using a plurality of spatial information. It is.
Then, the plurality of acquired spatial information is integrated by the coordinate integration unit 41.

The vehicle position / orientation specifying unit 42 acquires the coordinates and posture of the vehicle based on the CV value obtained from the all-around video in the position coordinate acquisition unit 20 and the position information from the GPS and IMU. The vehicle position is specified in the three-dimensional map based on the coordinates and posture obtained by the vehicle position / posture specifying unit 42.
Then, the vehicle position obtained in the integrated space information unit 40, for example, the vehicle position obtained by CV calculation and GPS is marked on the coordinates on the 3D map, thereby matching the integrated space information unit and the 3D map, It is possible to integrate and further specify the vehicle position in the 3D map.
As a result, the three-dimensional map and the spatial information are integrated.

The three-dimensional map unit 30 records a three-dimensional map generated in advance in and around the airport monitoring area. Specifically, a 3D shape map 31, a daytime 3D video map unit 32, and a nighttime 3D video map unit 33 are provided.
Here, the three-dimensional map can be combined with surveying in advance to generate a highly accurate three-dimensional map. That is, the 3D shape map 31 can be accurately made into a three-dimensional map by surveying or the like.
In addition, a 3D shape map 31 can be obtained by superimposing a three-dimensional shape by a laser scanner that measures the details in the monitoring area in advance.

As described above, it is desirable that the 3D map is obtained by photographing the entire monitoring area in detail in the daytime in the daytime in detail and recording the CV-calculated all-around video (CV video) as the daytime 3D video map unit 32. .
In particular, it is desirable to add to the 3D map unit 30 a 3D shape map 31 that is calibrated with high accuracy by combining high-accuracy GPS and actual surveying in addition to CV images.
It is also desirable to add a small 3D video map unit 33 in which a plurality of small lights whose coordinates are known at night and markers that are reflected by light from a vehicle are installed in the monitoring space and the coordinates are registered.

In this way, since the images are different at night and daytime, in this embodiment, two types of video maps at night and daytime are provided.
As a special case, the accuracy is reduced, but a three-dimensional map can be generated only by an on-board device. In this case, the monitoring action is performed for the first time while generating a map without a map, and from the next time, the 3D map generated at the first time is treated as data of the 3D map part and the map acquired at the first time is further updated. Monitoring activities can be performed.

The automatic detection unit 50 includes a comparison correspondence unit 51 and a monitoring target (predicted target, obstacle, suspicious object, etc.) automatic detection unit 52.
The comparison corresponding unit 51 compares the three-dimensional map representing the three-dimensional coordinates in the monitoring area with the integrated spatial information, and detects the monitoring object (predicted object, obstacle, suspicious object, etc.). Check the monitoring object.
Here, the automatic detection unit 50 determines that the object in the integrated spatial information existing at the same position on the reference three-dimensional map is a predicted object and is not an obstacle or a suspicious object.
On the other hand, if there is an object that does not exist in the three-dimensional map and exists in the acquired integrated spatial information, it is determined as an obstacle or a suspicious object, not a predicted object.
Furthermore, when it exists in the three-dimensional map and does not exist in the acquired integrated spatial information, it is determined to be examined or confirmed.

The monitoring object (predicted object, obstacle, suspicious object, etc.) automatic detection unit 52 compares the three-dimensional map with the integrated spatial information, and determines a matching part and a mismatching part that does not correspond.
Specifically, the monitoring object automatic detection unit 52 separates and extracts the mismatched portion between the three-dimensional map and the integrated spatial information, and monitors the monitoring object (predicted object, obstacle, suspicious object) in the monitoring area. ).
At this stage, the three-dimensional coordinates of the inconsistent part can be specified and it can be determined as an obstacle or a suspicious object, but it is not recognized, and it remains unknown what the specified object is. is there.

The map update unit 80 has a function of registering the confirmation result of the monitoring object and the monitoring object (predicted object, obstacle, suspicious object, etc.) in the 3D map of the 3D map unit 30.
Specifically, if there is a change in the object as a result of comparison with the three-dimensional map, the map update unit 80 updates and registers it in the three-dimensional map.
In addition, the map update unit 80 also temporarily registers a monitoring object (predicted object, obstacle, suspicious object, etc.) for reconfirmation at the next monitoring.
Furthermore, the map update unit 80 reflects and registers the recognition result if it is necessary to reflect it in the three-dimensional map.
As described above, when the update of the 3D map is continued by the map update unit 80, the 3D map provided in the 3D map unit 30 is always updated with new information.

The viewpoint direction control unit 60 is configured to calculate a monitoring target (predicted object determined by calculation from the three-dimensional coordinates of the specified monitoring target (predicted target, obstacle, suspicious object, etc.) and the three-dimensional coordinates of the vehicle. (Object, obstacle, suspicious object, etc.) having a function of controlling a device such as a camera.
Specifically, the viewpoint control unit 61 can control the direction of the laser scanner device as a movable system.
Since laser devices are generally expensive, it is difficult to mount a plurality of laser devices. Therefore, it is desirable to use one laser scanner device in a movable type.
In addition, it is advantageous in terms of cost if the viewpoint control unit related to the laser is not installed, and the laser is fixed and the designated direction is observed only with the camera.

In addition, the viewpoint control unit 61 directs the camera in the designated direction to the monitoring target (predicted target, obstacle, suspicious object, etc.), so that the monitoring target (predicted target, obstacle, suspicious) It is also possible to confirm whether or not the existing monitoring target is present in the same posture at the position marked on the three-dimensional map.
For this confirmation, the direction from the vehicle position is specified, so if there is an existing monitoring target in the specified direction as planned, it can be confirmed at the same time as the monitoring target as well as the vehicle position. is important.
The correct three-dimensional trajectory of the camera can be acquired by confirming that existing monitoring objects are present in the designated direction one after another.

The moving body tracking unit 62 extracts only the moving object from the moving body CV value obtained from the image of a single omnidirectional camera or from the images of a plurality of synchronized omnidirectional cameras.
In other words, the moving object tracking unit 62 detects the moving object and its approximate size on the assumption that the monitoring object (predicted object, obstacle, suspicious object, etc.) also exists in the moving object. Next, the detected moving body is tracked and its three-dimensional position and direction are tracked.

The automatic recognizing unit 70 comprehensively combines the acquired information, and compares the attribute information predicted as the monitoring target (predicted target, obstacle, suspicious object, etc.) with the monitoring target (predicted). (Objects, obstacles, suspicious objects, etc.).
Here, “recognition” refers to a state in which the attribute of the monitored object can be called out from the record. For example, if a "tire" has fallen in the monitoring area, it is detected and specified at the stage where it is determined as a monitoring object and its three-dimensional coordinates are acquired, but it is still "recognized". Is not in the situation. To determine that the tire is falling on the road, the monitored object (predicted object, obstacle, suspicious object, etc.) was compared with the database in the recognition part and matched with the attribute of the tire Sometimes it was recognized as a tire.

In the present embodiment, a PRM recognition unit 71 that automatically recognizes compared with the recorded attribute is provided.
The PRM recognizing unit 71 uses the above-described PRM technology for recognizing an object, and prepares all the shapes and attributes of the object to be predicted in advance as parts (operator parts). Compare the actual video and select the matching parts to recognize the object.

The monitoring display unit 90 is a display unit that displays a recognized monitoring object (a predicted object, an obstacle, a suspicious object, or the like).
Specifically, the monitoring display unit 90 alone recognizes the monitored object (predicted object, obstacle, suspicious object, etc.) with its attributes and further with its three-dimensional position. Display in a live-action video or in a 3D map.
It should be noted that the automatic recognition unit 70 cannot recognize an object identified as a monitoring target (predicted target, obstacle, suspicious object, etc.) unless it exists in the record. The monitoring display unit 90 can display the shape of the monitoring object (predicted object, obstacle, suspicious object, etc.) as an image.
In addition, attributes are registered in advance for some of the objects to be monitored, updated as necessary, searched, and if called automatically or manually in the image, the called attributes are displayed together with the video. be able to.

For example, an aircraft moving in an airport is an object to be monitored, but it is monitored and detected while the vehicle is stopped or moved by the monitoring apparatus 1A of the present embodiment, and the name of the aircraft is identified from its shape and mark, From the other attribute information to the flight name can be recognized and displayed on the monitoring display unit 90.
Further, it is possible to specify a monitoring object (a predicted object, an obstacle, a suspicious object, etc.) and emit an alarm sound. Further, if attribute information can be acquired from the storage device, it is possible to automatically recognize and display the attribute or the like, or to issue an alarm or the like.

Furthermore, it can be CG-combined with a real real image and displayed on the monitor display unit 90. For example, in a night-time monitoring operation, a three-dimensional video in the daytime or a two-dimensional video moved in the viewpoint is converted into a night video. Can be synthesized.
In the case of nighttime monitoring, the actual real image is completely dark, but by matching the vehicle real-time image with the three-dimensional position and posture of the vehicle, that is, by obtaining the CV value of the nighttime image, the driver's viewpoint and CG The daytime video can be combined with the nighttime video by matching the viewpoints.
Thus, the operator can drive while watching the daytime video displayed on the monitoring display unit 90 at night, and can continue the monitoring action.

The data recording unit 100 is a recording unit that records the result of automatic detection by the automatic detection unit 50 and the result of automatic recognition by the automatic recognition unit 70 described above.
Here, it is desirable to record all devices and the history of each unit in the data recording unit 100 at the same time.
The data to be recorded in the data recording unit 100 can also be recorded for a long period of time as an automatically detected monitoring object and an automatically recognized monitoring object as an image or simultaneously CG.

The automatic determination unit 110 grasps and determines the situation in which the vehicle is placed from the automatic detection result of the automatic detection unit 50 and the output of the automatic recognition result of the automatic recognition unit 70, and determines the next action to be taken by the vehicle. decide. In other words, the automatic determination unit 110 performs determination according to the purpose of monitoring as a monitoring device.
The automatic control unit 120 is a control unit that automatically controls each part of the vehicle based on the output of the automatic determination unit 110 and performs automatic driving.
In addition, the automatic control unit 120 performs control of each device necessary for monitoring, automatic tracking of a moving monitoring object, generation of an alarm sound, automatic communication with related departments, and the like.

The vehicle unit 130 is a vehicle itself that is controlled by the automatic control unit 120.
Various types of equipment of the automatic monitoring apparatus 1A as described above are mounted on the vehicle, and automatic driving is possible. When automatic driving is not performed, normal driving by an operator is possible.
Further, the vehicle can automatically track a moving suspicious object by a tracking function and an automatic driving function of a monitoring object (predicted object, obstacle, suspicious object, etc.).

The communication line unit 140 transfers the viewpoint direction data of the viewpoint direction control unit 60 to the central monitoring unit 150 installed in the terminal or the like via the communication line or the fixed installation video camera unit 160 installed along the runway or the like. Send.
As a result, the central monitoring unit 150 can point the existing fixed camera to the designated coordinates manually or automatically. ) Can be confirmed on the video.

As mentioned above, although the monitoring apparatus in an airport was shown as one Embodiment of the mobile automatic monitoring apparatus of this invention, various applications other than the monitoring objective are possible for this invention.
In other words, if a 3D map is provided or a 3D map is not provided, a 3D map can be generated in real time so that it can be used for normal driving as an automatic driving or automatic guidance device. . It can also be applied to aircraft, submarines, space exploration devices, etc.
Hereinafter, an embodiment in which the mobile automatic monitoring apparatus of the present invention is applied for purposes other than monitoring purposes will be described.

[Work operation guidance display device]
Hereinafter, an embodiment of a work operation guidance display device for carrying goods in a warehouse using the mobile automatic monitoring device of the present invention will be described with reference to FIGS.
FIG. 23 is a functional block diagram illustrating a schematic configuration of the work operation guidance display device according to the present embodiment.
FIG. 24 is an explanatory view schematically showing a work vehicle equipped with the work operation guidance display device shown in FIG.
FIG. 25 is an example of a display image displayed on the monitoring display unit of the work operation guidance display device shown in FIG.

As described above, the mobile automatic monitoring device according to the present invention, for example, at the time of takeoff and landing of an aircraft, adds a target guide marker or a monitoring object to a real image of an airport, a real image, or a three-dimensional image prepared in advance. Duplicate display is possible.
In this embodiment, by utilizing such a feature of the present invention, when an operator or an operator works using a work vehicle, a construction machine, or the like, a guide mark is displayed to assist the work operation. And a work operation guidance display method and apparatus by combining real and real images,
Specifically, the work operation guidance display device 1B of the present embodiment is an actual real image or a real image when working using a general work-dedicated vehicle or heavy equipment such as a Yumbo, a backfor, a bulldozer, a crane, and a lift. This is a guidance display device that displays and confirms a 3D construction work plan by superimposing it on a live-action video, and overlaying the design map with the actual real image of the site.

The work operation guidance display device 1B of the present embodiment is characterized by the monitoring display unit 90 in the mobile automatic monitoring device 1 described above, and the other parts are the mobile automatic monitoring device 1 (and in the airport mobile type). The configuration is the same as that of the automatic monitoring apparatus 1A).
Specifically, it is as shown in FIG. 23. In FIG. 23, the same components (same as the mobile automatic monitoring apparatus 1 shown in FIG. 1 (and the in-airport mobile automatic monitoring apparatus 1A shown in FIG. 19)). The operation and function are performed in the same manner. For example, even in a work vehicle equipped with the proposed display device 1B of the present embodiment, automatic driving and automatic work can be performed as in the case of the monitoring vehicle of the in-airport mobile automatic monitoring device 1A described above.

Hereinafter, the configuration and operation of the monitoring display unit 90 when a lift is used as a work vehicle will be described in detail with reference to the drawings.
As shown in FIG. 23, in this embodiment, the monitoring object (predicted object, which is captured and monitored by the spatial information acquisition unit 10, detected by the automatic detection unit 50, and recognized by the automatic recognition unit 70). A video (detected video) of an obstacle, a suspicious object, etc.) is once recorded in the composite target video material part 91 of the monitoring display part 90.
Further, in the synthesis target video material unit 91, a recognition CG component (recognition video, recognition CG) of an object to be monitored, an existing CG (design drawing, plan diagram), and the like are recorded as the synthesis target video material.
Then, as shown in FIG. 23, these composite target video materials are output and displayed as a real real image 92, a real video 93 acquired by the spatial information acquisition unit 10, and an on-site 3D map 94 cut out from the 3D map. In addition to these images, a componentized CG, an existing three-dimensional map, a design drawing showing a completed form, and the like are output and displayed.
The case where CG is synthesized and displayed on the actual real image 92 as a video material for synthesis will be described below.

FIG. 25A is a real real image. A real real image is not an image captured by a camera or the like, but the reality itself, which means the situation in the real world.
FIG. 25B is a work completion schedule diagram of the completed shape of the final stacking of the containers that are the monitoring objects identified by the guidance display device 1B.
Then, as shown in FIG. 24, these real images (FIG. 25 (a)) and composite purpose video materials (FIG. 25 (b)) are provided by a half mirror and a video projection device provided in front of the lift operator. Is displayed as an optically synthesized visual field composite diagram (see FIG. 25C).
In this embodiment, as shown in FIG. 25 (c), the visual field composite diagram shows a vehicle guidance marker that automatically guides the direction in which the vehicle moves while avoiding obstacles (see FIG. 24 (b)). It is supposed to be. Further, as shown in FIG. 25 (c), in the visual field composite diagram, the moving direction when each container is stacked is also indicated by an arrow as a work guide marker.

As a result, the real image of the real world (see FIG. 25 (a)) and the completed shape of the final stacked container (see FIG. 25 (b)), which is the identified monitoring target, are combined and displayed. (Refer to FIG. 25 (c)) In the work of organizing and stacking containers using a lift, the operator (driver) is combined with the real real image and the final form of the work through the front half mirror. You can observe the video and work on it.
As described above, in the present embodiment, the real scale image material recorded in the composite purpose video material unit 91 is overlapped with the real video field of view by the video projection device and the half mirror shown in FIG. Thus, it is possible to observe from the operator without any contradiction between the viewpoint and the distance, and the operator can observe the visual field composite diagram in the composite visual field.

It should be noted that a monitoring object such as a container that is necessary for the traveling operation of the lift is converted into a CG in advance and stored in the database of the automatic recognition unit 70.
The container of the monitoring object detected and identified / recognized by the work guidance display device 1B is obtained by extracting CG data from the database of the automatic recognition unit 70 and combining it with the field of view of the actual real image of the worker. Is done. Since the CG is synthesized into a real real image on a three-dimensional real scale, the operator synthesizes and observes the CG with the real real image he / she sees without any contradiction in viewpoint. In addition, even if the operator moves the head or changes the viewpoint direction, the viewpoint relationship between the real real image and the CG is maintained because the three-dimensionally consistent composition is achieved.
As a matter of course, the movement of the vehicle in this case is determined by comparing the image of the wide-angle camera (see FIG. 24A) attached to the roof of the lift and the three-dimensional map, and the actual real image is obtained. Can be synthesized without contradiction.

In order to monitor the work status of the worker, the work manager can automatically detect and automatically recognize the live-action video by the wide-angle camera of the spatial information acquisition unit 10 attached to the vehicle sent to the management room wirelessly. What is necessary is just to synthesize | combine and observe with the monitoring target object (container).
In this case, a real image or a three-dimensional map is synthesized instead of an actual real image. Also in this case, since the CV value is obtained by comparing the wide-angle camera image with the three-dimensional map, three-dimensional composition without contradiction is possible.

As described above, the lift has been described as the vehicle on which the work guidance display device 1B according to the present embodiment is mounted. However, it is obvious that the mounted vehicle is effective as a heavy vehicle such as another general vehicle or a jumbo, a backhoe, a bulldozer, or a crane. It is.
For each vehicle's movement and work, the completed form is designed as a design drawing, displayed in a three-dimensional map, in a live-action image, or in a real real image. Management will be appropriate and construction records etc. will be taken accurately.
Furthermore, in addition to those shown in this embodiment, as an example of using a 3D map as a composition target video material, a real image and a detected and recognized monitoring object are synthesized as a viewpoint without contradiction. Therefore, there are various applications. For example, it is also effective in providing a device that can display an airport image on a three-dimensional map or display a take-off / landing guide marker by CG synthesis and can take off and land while checking the image even in fog.

[Real-time vehicle position and orientation detection device]
Furthermore, as an application example of the mobile automatic monitoring device of the present invention, an embodiment of a real-time vehicle position / orientation detection device functioning as a single function will be described with reference to FIG.
As described above, according to the mobile automatic monitoring device of the present invention, it is possible to identify the position and posture of the vehicle on the three-dimensional map by detecting the position and posture of the vehicle in real time from the moving vehicle. From here, various application devices and products can be used.

In that case, since it is necessary to detect the vehicle position and orientation in any application, first, a wide-angle camera attached to the vehicle is used as the spatial information acquisition unit 10 and compared with a three-dimensional map in real time. A vehicle position / orientation detection apparatus as a single function for performing vehicle position / orientation detection can be configured.
As a matter of course, by using the vehicle position / orientation detection device as a single function, the above-described in-airport mobile automatic monitoring device 1A and work operation guidance display device 1B can be realized at low cost. .

FIG. 26 is a functional block diagram showing a schematic configuration of the real-time vehicle position / orientation detection apparatus 1C according to the present embodiment.
As shown in the figure, the real-time vehicle position / orientation detection device 1C according to the present embodiment includes the above-described mobile automatic monitoring device 1 (see FIG. 1), airport mobile automatic monitoring device 1A (see FIG. 19), work Similar to the operation guidance display device 1B (see FIG. 23), a spatial information acquisition unit 10, a position coordinate acquisition unit 20, a three-dimensional map unit 30, an integrated spatial information unit 40, and a monitoring display unit 90 are provided.

The spatial information acquisition unit 10 includes a wide-angle camera 11.
From the standpoint of accuracy, it is desirable that a plurality of wide-angle cameras 11 is more than one. Although an all-around camera is most desirable, a wide-angle camera is employed in this embodiment in order to keep costs low. By treating the wide-angle camera as a part of the all-round image, the same handling as the all-round camera is possible. Then, the real-time video 11a acquired by the wide-angle camera 11 is sent to the position coordinate acquisition unit 20 for three types of CV calculations described later.
The position coordinate acquisition unit 20 includes a GPS 20b for acquiring approximate positions. Since it is difficult to detect a position with high accuracy from the viewpoint of accuracy, the GPS 20b is simply used as approximate positioning in this embodiment.

The three-dimensional map unit 30 includes a CV video map DB 31 having absolute coordinates, and stores a database of all-around video converted to CV video.
Here, each frame of the all-round image converted into the CV image originally has the camera position and orientation at the time of shooting as a CV value of 6 degrees of freedom, but the image correction is performed so that the camera rotation component becomes zero. It is preferable to use a CV image without shaking. In this way, the rotation component of the image is 0, the CV value is only the three-dimensional coordinates of the camera, and the data is substantially three-degree-of-freedom, so the amount of data is reduced and the handling becomes extremely easy.
The monitoring display unit 90 includes a display monitor 91 that displays a vehicle position and a CV video map, which are output results of the real-time vehicle position / orientation detection apparatus 1C according to the present embodiment.

In the real-time vehicle position / orientation detection apparatus 1C configured as described above, vehicle position / orientation detection is performed as follows (see FIG. 26).
First, the approximate area where the vehicle is located is acquired by the GPS 20b of the position coordinate acquisition unit 20, and the CV image map of the CV image map DB 31 of the three-dimensional map unit 30 is selected based on the approximate area information (S301).
In this CV video map selection (S301), the CV video around the approximate area acquired by the GPS 20b is selected. This start setting can be manually aligned. More precisely, it is preferable to use a function of designated point correspondence (S201) described later, but the approximate position of the start point may be set to GPS and manual.

Next, a corresponding point is obtained in the real-time video later from the selected CV video in accordance with the designated point in the map (S302) in the three-dimensional map unit 30, and the CV of the above-described three-dimensional specified point corresponding method is used. Select and specify the designated point for the operation.
Although this designated point can be automatically detected when the location is designated, it is desirable to detect it beforehand in the CV video.
Next, the designated point correspondence (S201) in the position coordinate acquisition unit 20 indicates the three-dimensional coordinates of the detected designated point, and the designated point in the CV video is included in the video of the real-time video 11a obtained by the wide-angle camera. Find the point corresponding to.
Next, by tracking the designated point (S202), the 3D designated point is tracked in the real-time video while matching the corresponding point in the real-time video.

Next, a designated point CV calculation (S203) is performed.
In the designated point CV calculation (S203), a three-dimensional designated point correspondence method CV calculation is performed, and a plurality of designated points (preferably several tens from the viewpoint of accuracy) are obtained in each tracked frame. Since the three-dimensional coordinates of the point are known by being transferred to the video, if correspondence of four or more designated points is required, the wide-angle camera 11 that has acquired the CV value in each frame, that is, the real-time video 11a. The three-dimensional position and orientation of each frame are immediately and independently determined as frame-specific values regardless of other frames.
Further, in this three-dimensional designated point correspondence method CV calculation, if the designated point of the CV image as a three-dimensional map has absolute coordinates, the CV value obtained by the three-dimensional designated point correspondence method CV computation also has absolute coordinates. become. This is important.
The three-dimensional designated point correspondence method CV computation is distinguished from the above-described feature point tracking method CV computation (see FIGS. 2 to 14). The CV calculation described above may be referred to as a feature point tracking method CV calculation (abbreviated as “feature point CV calculation”) in order to distinguish from the specified point CV calculation. However, the calculated results are the same for both CV values.

In addition, the designated point is tracked in the real-time video 11a by correspondence tracking (S204), and the traced designated point is regarded as a feature point by the feature point handling CV calculation (S205), and the relationship between the frames is traced. The feature point CV calculation is performed. At this time, since the designated point has three-dimensional coordinates, it is important that the above-described feature point CV calculation has not the relative coordinates but the same absolute coordinates as the three-dimensional map.
On the other hand, in the real-time video 11a, feature point extraction is automatically performed by feature point extraction (S206) by feature point CV calculation, feature point tracking is performed by short frame feature point tracking (S207), and feature point CV calculation is performed. By (S208), the feature point CV calculation between short frames is performed. The reason why the CV calculation is performed with the short frame simplification is to reduce the calculation time for real-time processing.
Here, if organized, there are three types of CV operations. There are three types: (1) feature point tracking method CV computation (feature point CV computation), (2) three-dimensional designated point correspondence method CV computation (designated point CV computation), and (3) feature point handling CV computation.

Then, the results of the three types of CV computations, the specified point CV computation (S203), the feature point handling CV computation (S205), and the feature point CV computation (S208), are sent to the integrated space information unit 40 for coordinate integration ( S401) integrates three types of CV values.
At this time, the errors cancel each other, and in the camera position and orientation detection (S402), a more accurate camera position and orientation (CV value) are obtained.
Since the camera position and the vehicle position are determined by the relationship with the camera mounting position, it is finally converted into the vehicle position by the vehicle position and orientation conversion (S403), and is used for the monitoring display unit 90 and other purposes. Is output.
The output here can be used for various applications.

In the monitor display unit 90, the vehicle position is displayed on the display monitor 91 alone or in combination with a three-dimensional map (CV video map).
As described above, in the real-time vehicle position / orientation detection apparatus 1C according to the present embodiment, the designated point CV calculation (S203), the feature point handling CV calculation (S205), and the feature point CV calculation (S208) are performed in real time. By performing three types of CV operations, the accuracy can be improved.
Then, by adding various other functions based on the real-time vehicle position / orientation detection apparatus 1C according to the present embodiment, it is possible to assemble apparatuses suitable for various applications.

The mobile automatic monitoring apparatus of the present invention has been described with reference to the preferred embodiment, but the mobile automatic monitoring apparatus according to the present invention is not limited to the above-described embodiment, and the scope of the present invention. Needless to say, various modifications can be made.
For example, in the above-described embodiment, the airfield runway and the terminal are described as examples of the monitoring target / monitoring area monitored by the mobile automatic monitoring device of the present invention. However, the application target of the present invention is limited to these. Instead, the present invention can be applied to any object, place, or space as long as it is a monitoring object or a monitoring object area that requires or is suitable for remote monitoring by a monitoring camera.

  The mobile automatic monitoring device of the present invention is a monitoring system / security system for monitoring a predetermined monitoring target or monitoring target area, such as an airport terminal, runway, warehouse, large store, etc., automatic driving system / automatic work, etc. It can be suitably used as a system or the like.

1 is a block diagram showing a schematic basic configuration of a mobile automatic monitoring apparatus according to an embodiment of the present invention. It is a block diagram which shows the basic composition of one Embodiment of the CV calculating part of the feature point tracking system which functions as a position coordinate acquisition part of the mobile automatic monitoring apparatus of this invention. It is the schematic which shows the means to image | photograph the perimeter video image | video used by the CV calculating part shown in FIG. 2, and is a perspective view of the vehicle which mounts the perimeter camera in the roof part. It is the schematic which shows the means which image | photographs the perimeter video image | video used by the CV calculating part shown in FIG. 2, (a) is a front view of the vehicle which mounts a perimeter camera in a roof part, (b) is a top view similarly. It is. It is explanatory drawing which shows the conversion image obtained from the image | video image | photographed with a omnidirectional camera, (a) is a virtual spherical surface to which a spherical image is affixed, (b) is an example of a spherical image affixed to a virtual spherical surface , (C) shows an image obtained by developing the spherical image shown in (b) on a plane according to the Mercator projection. It is explanatory drawing which shows the specific detection method of a camera vector in the CV calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the detection method of the specific camera vector in the CV calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the detection method of the specific camera vector in the CV calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the designation | designated aspect of the desirable feature point in the detection method of the camera vector by the CV calculating part which concerns on one Embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the CV calculating part which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the CV calculating part which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the CV calculating part which concerns on one Embodiment of this invention, and a camera vector. In the CV calculating part which concerns on one Embodiment of this invention, it is explanatory drawing which shows the case where a some feature point is set according to the distance of a feature point from a camera, and a some calculation is repeated. It is a figure at the time of displaying the locus | trajectory of the camera vector calculated | required by the CV calculating part which concerns on one Embodiment of this invention in a video image | video. It is a block diagram which shows the basic composition of the CV calculating part of the three-dimensional designated point corresponding | compatible system which concerns on one Embodiment of this invention. It is a block diagram which shows the detail of the reference point corresponding | compatible part of the CV calculating part shown in FIG. It is a block diagram which shows the detail of the reference point corresponding | compatible part of the CV calculating part shown in FIG. It is a block diagram which shows the basic composition of the CV calculating part of the three-dimensional designated point corresponding | compatible system which concerns on other embodiment of this invention. It is a functional block diagram which shows schematic structure of the mobile type | mold automatic monitoring apparatus in the airport which concerns on one Embodiment of this invention. BRIEF DESCRIPTION OF THE DRAWINGS It is explanatory drawing which shows typically the vehicle carrying the in-airport movement type automatic monitoring apparatus which concerns on one Embodiment of this invention, (a) is a top view of a vehicle, (b) is a side view similarly. It is explanatory drawing which shows typically the vehicle carrying the in-airport movement type | mold automatic monitoring apparatus which concerns on one Embodiment of this invention, and is a side view of the vehicle carrying the monitoring apparatus for the time of snow. It is the top view which showed typically the whole airport monitored by the in-airport mobile automatic monitoring apparatus which concerns on one Embodiment of this invention. It is a functional block diagram showing a schematic structure of a work operation guidance display device according to an embodiment of the present invention. It is explanatory drawing which shows typically the work vehicle carrying the work operation guidance display apparatus which concerns on one Embodiment of this invention. It is an example of a display image displayed on the monitoring display unit of the work operation guidance display device according to the embodiment of the present invention. It is a functional block diagram which shows schematic structure of the real-time vehicle position and orientation detection apparatus which concerns on one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Mobile type | mold automatic monitoring apparatus 1A Airport movement type | mold automatic monitoring apparatus 1B Work operation guidance display apparatus 1C Real-time vehicle position and orientation detection apparatus 10 Spatial information acquisition part 20 Position coordinate acquisition part 30 Three-dimensional map part 40 Integrated space information part 50 Automatic detection Unit 60 View direction control unit 70 Automatic recognition unit 80 Map update unit 90 Monitoring display unit 100 Data recording unit 110 Automatic determination unit 120 Automatic control unit

Claims (6)

A spatial information acquisition unit that is mounted on a mobile body and acquires spatial information around the mobile body by moving with the mobile body;
A position coordinate acquisition unit that acquires position coordinate information indicating the three-dimensional position and posture of the moving moving body;
A three-dimensional map unit provided with a three-dimensional map of a monitoring region where the moving body moves and a region where the spatial information acquisition unit can acquire spatial information from the monitoring region;
Based on the spatial information around the moving body, the position coordinate information of the moving body, and the three-dimensional map of the monitoring area, an integrated spatial information unit that generates integrated spatial information by integrating the three-dimensional spatial information around the moving body;
An automatic detection unit that associates the integrated spatial information around the moving body with a three-dimensional map, and detects and identifies a monitoring object in the monitoring area;
An automatic recognition unit for recognizing predetermined attribute information of the monitoring object identified by the automatic detection unit;
Image information necessary for moving the moving body, a video of the monitoring target identified by the automatic detection unit, and a viewpoint from which the CG or video of the monitoring target identified by the automatic recognition unit is observed from the moving body A monitoring display unit that matches and displays the actual video, the live-action video, or the three-dimensional map,
A recording unit for recording predetermined data of the monitoring object displayed via the monitoring display unit;
A mobile automatic monitoring device comprising:   When a change in the size, shape, position, etc. of an object specified as a monitoring object is detected by the automatic detection unit, the change is detected by the 3D map information of the 3D map unit and / or the recording unit. The mobile automatic monitoring apparatus according to claim 1, further comprising a map updating unit that reflects the predetermined data.   The space toward the direction of the monitoring object determined based on the three-dimensional coordinates of the monitoring object specified by the automatic detection unit and the three-dimensional coordinates of the moving object acquired by the position coordinate acquisition unit. The mobile automatic monitoring apparatus according to claim 1, further comprising a viewpoint direction control unit that drives and controls an observation device provided in the information acquisition unit. Spatial information acquisition unit
An all-round camera that captures a wide range of images around a moving object, a specified direction camera that directs the viewpoint in a specified direction, a laser scanning device that scans laser light to measure direction and distance, and a direction of reflected waves by emitting radar Information on the surroundings of the moving object, either alone or in any combination according to the situation such as day and night, season, weather, etc. The mobile automatic monitoring device according to any one of claims 1 to 3, which acquires An automatic determination unit that determines the next action of the moving object based on the monitoring target identified by the automatic detection unit and the attribute information of the monitoring target recognized by the automatic recognition unit;
An automatic control unit that controls the moving body according to the next action of the moving body determined by the automatic determination unit;
The mobile automatic monitoring apparatus according to any one of claims 1 to 4, further comprising: The integrated spatial information part is
Based on only the spatial information around the moving body acquired by the spatial information acquisition unit and the positional coordinate information of the moving body acquired by the position coordinate acquisition unit, the three-dimensional spatial information around the moving body is integrated. While generating integrated spatial information,
The mobile automatic monitoring device according to any one of claims 1 to 5, wherein a three-dimensional map provided in the three-dimensional map unit is generated based on the integrated spatial information.