JP2009077092A - Multi-camera system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a multi-camera system which can change camera direction and focal distance corresponding to a position of a detection target object, switches image processing corresponding to the camera direction, and solves conventional technical problems that a camera structure and control processing content are determined according to particular purposes. <P>SOLUTION: Comparing to conventional multi-camera systems, the embodied multi-camera system obtains functions that applications are adaptively switched if arrangement of the respective cameras and camera optical axis directions are changed in the multi-camera system. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a multi-camera system that detects, tracks, and analyzes an object using at least two imaging devices.

  Conventionally, in a monitoring / monitoring system, an object to be monitored, a person to be monitored, or an object is imaged with a single camera, and the image is accumulated and distributed. In addition, a method has been adopted that performs image processing and image recognition as necessary, issues warnings and cautions to monitoring personnel and those who are monitoring, and reduces the human burden caused by monitoring and monitoring. . However, in the case of monitoring / monitoring with a single camera, the image recognition performance may deteriorate due to concealment of the detection target. In addition, when a single camera performs image recognition, false alarms due to the influence of shadows and light frequently occur, and the image recognition algorithm for monitoring / monitoring may not operate properly in an actual environment.

  In recent years, in order to solve these problems, there are increasing cases of using a multi-camera imaging device or system including a stereo camera. For example, in [Patent Document 1], whether a subject photographed by reducing the false alarm due to illumination fluctuations such as shadows and light by using a stereo camera or measuring the size and volume of the subject to be photographed is a person. To determine whether it is a vehicle. In addition, a technique for enabling target object shooting without concealment using images from a plurality of cameras and improving the accuracy of a face authentication system, for example, is shown.

Japanese Patent Laid-Open No. 10-134187 US Patent 6,711,293 Sato, Satoshi, “Computer Vision-Geometry of Vision”, Corona, 1994

  In the conventional multi-camera imaging technique, the camera configuration is uniquely determined according to a specific purpose. For example, when distance measurement is performed, two cameras having the same characteristics are fixed in parallel. Alternatively, when detecting and tracking a moving object, the moving object is detected and tracked with a wide-angle camera, and a detailed image of the moving object is captured with a telephoto camera.

  That is, the camera configuration and application are determined according to a specific purpose, and the application is not adaptively switched when the camera arrangement or the camera optical axis direction in the multi-camera system changes dynamically.

  In order to solve such a problem, the present invention provides a multi-camera system that is more adaptive even when the camera arrangement and the direction of the camera optical axis are dynamically changed.

  In order to achieve the above object, the multi-camera system of the present invention includes at least two imaging devices, means for switching image recognition processing according to the degree of overlap of images obtained from the two imaging devices, The image pickup apparatus includes an electric head that can move the direction of the optical axis, and the electric head is dynamically controlled according to the result of the image recognition process.

  Furthermore, the multi-camera system of the present invention switches the image recognition processing according to the overlapping state of the field of view of the imaging device. If the field of view of the two imaging devices overlaps, the stereo distance measurement function and the three-dimensional object shape measurement are performed. If at least one of the functions, panorama image generation function, blind spot shooting function, and human behavior analysis function is executed and the field of view of the two imaging devices does not overlap, detection of the detection target by the two cameras The function is performed, and the function is switched according to the position of the moving object to be detected and the angle between the optical axes of the two imaging devices.

  Furthermore, the multi-camera system of the present invention is characterized in that a combination of at least the two or more functions is performed in accordance with the output of the image recognition process.

  Furthermore, the multi-camera system according to the present invention is a processing method for determining the degree of overlap of the images, detecting feature points that appear simultaneously in two images by image processing, performing matching, and calculating epipolar geometry from the matching results. A function for determining the degree of image overlap, and by superimposing images while shifting the other image with respect to one image, the amount of image shift when the overlay error is minimized A function to output as the degree of overlap and a detection target to be detected by image processing, specifically, a geometric structure existing in a person or vehicle, a shooting scene, or a specific marker defined for a multi-camera system And at least one of the functions for calculating the degree of image overlap from the positional relationship between the two images. It is characterized in that to obtain.

  Furthermore, the multi-camera system of the present invention has two image pickup devices with different characteristics, specifically, resolution, frame rate, shutter speed, aperture, lens brightness, lens diameter, filter attached to the lens, and image pickup wavelength. Even in a combination of imaging devices in which at least one of the regions is different, the degree of image overlap is calculated between two images.

  Furthermore, in the multi-camera system of the present invention, the orientation of the optical axis of the imaging device is fixed without using an electric camera platform, and at least two or more angles formed by the optical axes between the two imaging devices are different. A pair of imaging devices is provided.

  In order to achieve the above object, a multi-camera system according to the present invention includes at least two imaging devices, means for determining an overlap of images obtained from the two imaging devices, and an image overlapping determination unit. Means for switching an image recognition process in accordance with an output, and controls an imaging state of the imaging device.

  Furthermore, the multi-camera system of the present invention performs processing related to the object in the field of view when the field of view of the two imaging devices overlaps as the output of the image overlap determination means. When the field of view does not overlap, the object is detected by two imaging devices.

  Furthermore, the multi-camera system of the present invention includes an electric camera platform that can move the direction of the optical axis of the image device, and the electric camera platform is controlled according to the output of the image overlap determination means. Is.

  According to the multi-camera system of the present invention, the image recognition application can be switched in accordance with the arrangement of the imaging device and the change in the direction of the optical axis. Therefore, even if the imaging device is arbitrarily installed, the multi-camera system itself captures the image. Since an image recognition application most suitable for device installation can be selected, an easy-to-use system can be provided. Further, for example, by adding a cheaper imaging device to the existing imaging device, for example, the variation of the image recognition application can be increased easily and inexpensively.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows an example of a functional block diagram of a multi-camera system implementing the present invention. The main camera 101 and the sub camera 102 are installed on the electric camera platform 103 and 104, respectively. The main camera 101 and the sub camera 102 may have the same or different camera characteristics, specifically, image resolution, lens size, focal length, frame rate, and imageable wavelength region. In addition, both of the electric pan heads 103 and 104 are controlled manually or automatically.

  The visual field overlap determination unit 105 determines whether or not the visual fields of the images captured by the cameras 101 and 102 overlap, and passes the determination result to the image processing function selection unit 106. Based on the result, the image processing function selection unit 106 selects a function of object tracking, distance measurement, action determination, panorama generation, or a combination thereof.

  The video analysis unit 107 performs the image processing selected by the image processing function selection unit 106 and outputs the analysis result to the outside, or passes it to the pan head control unit 108 to control the operation of the electric pan heads 103 and 104. To do. There are cases where two electric pan heads are movable, one is fixed while the other is movable, and both are fixed. In addition, when the two pan heads move synchronously, for example, from the initial state in which the two camera optical axes are parallel, the two cameras always operate at the same angle and the respective cameras are at an arbitrary angle. When moving, for example, a moving object may be tracked. When the focal length of the camera is variable, it is possible to prevent the two pan / tilt heads from being controlled in conjunction with the value of the focal length.

  Next, a specific example of the operation of the electric pan heads 103 and 104 will be described using the embodiment of FIG. A left camera 201 and a right camera 202 are provided, and the relationship between the detection and tracking target object 203 is shown. In FIG. 2A, θ is the optical axis direction of the right camera 202 when the optical axis direction of the left camera 201 is used as a reference, and the counterclockwise direction is positive. The broken line indicates the field of view (view angle) of the two cameras. In the case of FIG. 2A, θ takes a negative value. However, it is assumed that the optical axes of the two cameras are calibrated so as to be in the same plane.

  First, when the moving object is far away as shown in FIG. 2A, the fields of view of the two cameras do not overlap. As shown in FIGS. 2B, 2C, and 2D, as the moving object approaches the camera side, the value of the angle θ between the two cameras increases, and the overlapping degree of the visual field increases. Finally, as shown in FIG. 2 (e), the two cameras face each other and photograph the moving object from the opposite direction. The visual field overlap determination unit 105 determines the states shown in FIGS. 2A to 2E and sends the determination results to the image processing function selection unit 106. The video analysis unit 107 performs video analysis according to the output result of the image processing function selection unit 106 and sends the analysis result to the pan head control unit 108.

  Details of the above processing will be described below with reference to FIG. The operation of the electric camera platform when the tracking target object moves from a distance toward two cameras 301 (for example, the left camera 201) and the camera 302 (for example, the right camera 202) will be described. The object detection unit 303 has a function of detecting a moving object. If there is no moving object in the field of view of the camera 301 or the camera 302, the object detection unit 303 continues processing until a moving object is found. If there is a moving object in the field of view of the camera 301 or the camera 302, the next step 305 calculates the overlapping degree of the two images, and the step 306 selects the next process to be performed. . The processing of step 305 and step 306 corresponds to the processing of the visual field overlap determination unit 105 in FIG.

  If the detected object is far away as shown in FIG. 2A and the fields of view of the two cameras do not overlap, in step 306 in FIG. 3, the object to be tracked is reflected only in the right camera. First, it is determined that the image is not captured by the left camera. In response to the determination result, the two cameras track the target object with the right camera until the moving object appears in the left camera. At the same time, the left camera rotates the electric pan head so that it faces the right camera. A series of operations of the left camera is passed through the visual field overlap determination unit 105, the image processing function selection unit 106, and the video analysis unit 107, and tells the pan head control unit how much the left camera is directed to the right and rotates the left camera. Let Both the left camera 201 and the right camera 202 rotate the motorized heads 103 and 104 so that the tracking target can always be seen near the center of the image. If the apparent size of the moving object does not change and it can be determined that the distance between the moving object and the camera does not change, the inter-camera moving object tracking process in step 308 of FIG. 3 is performed.

  In the state of FIG. 2A, the fields of view of the two cameras do not overlap, and in order to identify the same object between the cameras, the detection results of the moving object at different times are stored, Identify objects detected at different times between cameras. The detection results of the two camera moving objects are stored in the video analysis unit 107 in FIG.

  Here, if the cameras 301 and 302 are zoom cameras with variable focal lengths, the focal lengths are adjusted manually or automatically so that the detected object always looks the same size.

  Next, it is assumed that the moving object further approaches the camera side as shown in FIG. In this state, the moving object is still captured only by the right camera 202, but the fields of view of the two cameras are beginning to overlap. In step 306 in FIG. 3, it is determined that the images overlap, and in step 307, it is determined whether or not the angle θ between the two camera optical axes is θ> −Th. Here, ± Th is a threshold value of θ for determining whether or not the angle between the two camera optical axes is close to 0, that is, whether it can be regarded as parallel. Here, if the camera angle of view is 2θ0, and θ <2θ0 and θ> Th is not satisfied, the fields of view of the two cameras are overlapped with the camera optical axis being out of parallel. In this state, a panoramic image can be generated by removing the influence of distortion and scale variation between the two images and pasting the overlapping portions.

  Next, when the tracking target object approaches the camera and the optical axes of the two cameras 301 and 302 are nearly horizontal, and -Th ≦ θ ≦ Th in step 310 of FIG. The mode is determined to be almost parallel, and the mode is switched to the mode for calculating the distance to the tracking target object. This determination is performed by the image processing function selection unit 106 in response to the output of the visual field overlap determination unit 105. In the next video analysis unit, the distance to the tracking target object is calculated sequentially, and for example, an alarm is issued when the tracking target object passes through a virtual plane set at a certain distance, or it passes through the virtual plane. Run an application that counts the number of people At the same time, the video analysis unit sequentially tracks the moving object, and controls the pan head control unit 107 from the position information on the screen of the moving object so that the moving object always appears near the center of the image.

  Also in this case, the focal lengths of the cameras 301 and 302 can be adjusted so that the tracking target object always has the same size in the field of view. At this time, the focal lengths of the two cameras are synchronized and always have the same focal length value so that the distance can be easily measured even when the focal length is changed.

  Next, when the moving object approaches the camera as shown in FIG. 2D, the same feature point of the moving object may not be visible from the two cameras. The visual field overlap determination unit 105 detects this state and sends the information to the image processing function selection unit 106. The image processing function selection unit 106 can select, for example, blind spot imaging in which the same object is viewed from different viewpoints. If it demonstrates with the flow of FIG. 3, the blind spot imaging | photography 311 will be performed through the determination result of the determination parts 309 and 310. FIG. After any of the inter-camera moving object tracking 308, panoramic image generation 309, distance measurement 310, and blind spot photographing 311 is performed, in step 312, the moving object is tracked. If the detected moving object is visible near the center of the image as determined in step 313, the object is continuously tracked, and if the moving object is off from the vicinity of the center of the image or disappears from the field of view. In step 314, the amount and direction of movement are predicted from the tracking result up to the current frame. In step 314, the two cameras 301 and 302 are controlled, and the object detection unit 303 performs object detection again, and the above processing is repeated. .

  The camera faces the camera according to the movement of the object, and when the moving object passes further, the process is performed in the reverse order until the fields of view no longer overlap each other. As a result, an all-week image of the moving object is finally obtained.

  Here, the details of the portion for calculating the overlapping degree of the visual field in FIG. 3 will be described with reference to FIG. An image 1 (403) and an image 2 (404) are captured by the camera 401 and the camera 402, respectively. In step 405, a variable T that controls the number of times the subsequent processing is repeated is initialized to T = 0.

  Next, in step 406, the feature point correspondence calculation of the two images is started. As a feature point extracted from an image, for example, a corner point is detected from an image pattern. As means for detecting corner points and matching between images, for example, SIFT (Scale Invariant Feature Tracking) shown in [Patent Document 2] is used. In step 407, the number of matching between two images is determined. If the number of matching N is N ≧ 7, the feature point correspondence calculation is terminated in step 408, and the F matrix is calculated in step 409. The F matrix is also called a basic matrix, and is a matrix that defines epipolar geometric constraints between cameras when a scene is shot with two cameras. That is, it is a matrix that determines on which line on the other image the feature points on one image are located, and the straight line is called an epipolar line (see, for example, [Non-Patent Document 1]).

  If the epipolar lines of the two images are parallel and the height coordinates are the same, for example, the feature points of the right image correspond to the left image by scanning on the straight line of the same height of the left image. You can find a spot. This is a camera state called parallel stereo in which the optical axes of two cameras are parallel, and the deviation of the feature point between the left and right images is parallax, and the feature point from the camera in real space to parallax. The distance to is calculated. Since the F matrix can be calculated if there are at least seven corresponding combinations of feature points, the lower limit threshold value of the number of matching is N = 7.

  In step 410, it is determined whether the F matrix has been calculated or not. The fact that the F matrix is calculated means that there is correspondence between the feature points between the two images, and that the number is 7 or more. In this case, the two images overlap. Judge that The result of the determination is output in step 411 if the fields of view overlap, and the process is terminated. If the F matrix is calculated in step 410 for some reason, for example, if the correspondence between the feature points of the two images is erroneously calculated and the F matrix is not calculated correctly, it is determined that the two images do not overlap. In step 415, the determination result that the visual fields do not overlap is output.

  In the determination unit in step 407, if the number N of feature points matching is 6 or less, the F matrix cannot be calculated. Therefore, the feature point correspondence calculation of two images in step 406 is repeated. If the corresponding number of feature points is 7 or more, the process proceeds to step 408 and subsequent steps to determine whether the F matrix can be calculated. If seven or more feature point correspondences are not obtained and the number of iterations exceeds the upper limit threshold value M, the iteration operation is terminated in step 414, and it is determined that the two images do not overlap. In 415, a determination result that the visual fields do not overlap is output.

  In the multi-camera system, the two cameras dynamically change the camera direction while tracking the detection target, and dynamically switch the image processing / image recognition application according to the state of the optical axis direction of each camera. Means are shown. When there are two or more cameras, the above means can be executed by combining any two of them.

  In addition, in FIG. 1, when the control information of the electric camera platform 103, 104, that is, the rotation angle and the rotation speed can be extracted as parameter information from the camera platform, the application function corresponding to the parameter information is stored in the lookup table, The image processing function selection unit 105 in FIG. 1 has a function for inputting the rotation angle to the lookup table when the rotation angle of the camera platform is controlled automatically or manually and selecting an image processing / image recognition application as the output. Can be made. In this case, the image processing application can be selected directly from the rotation angle of the camera platform without determining the overlapping degree of the visual field. In this case, since the image processing function selection unit 105 does not depend on the detection result of the moving object, by designating an arbitrary camera direction, for example, only a certain distance and a moving object that has entered each image are ambushed and imaged. Only specific image processing can be performed.

  Thus, in the case of a conventional multi-camera system, for example, in the case of a stereo camera, it is necessary to fix two imaging devices having the same characteristics to a robust housing in order to perform distance measurement. In addition to the cost, it could not be used for applications other than distance measurement, specifically for panoramic image generation and person tracking between multiple imaging devices. On the other hand, the present invention shown in the embodiment According to the multi-camera system, it is not necessary to limit the configuration of the imaging device according to a specific application, and it is possible to provide a plurality of image recognition applications.

  According to the multi-camera system of the present invention, it is possible to provide an easy-to-use video detection system because the multi-camera system itself can select an optimal image recognition application for installing the imaging device even if the imaging device is installed arbitrarily. realizable.

It is explanatory drawing which showed the structure of the multi camera system. It is explanatory drawing which showed the operation | movement form of the multi camera when tracking a moving object. It is the flowchart which showed the implementation method for switching the image processing function of a multicamera according to the detection result of an object. It is the flowchart which showed the implementation method for determining the overlapping degree of two camera visual fields.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Main camera 102 Sub camera 103,104 Electric pan head 105 View overlap judgment part 106 Image processing function selection part 107 Image | video analysis part 108 Pan head control part 201 Left camera 202 Right camera 203 Detection target object 301, 401 Camera 1
302, 402 Camera 2
303 Object Detection Unit 304 Object Presence Determining Unit 305 Image Overlap Degree Calculation Unit 306 2 Image Overlap Degree Determining Unit 307 Two Camera Optical Axis Angle Calculation Unit 308 Inter-Camera Object Tracking Unit 309 Panorama Mode Distance measurement mode determination unit 310 Distance measurement mode and blind spot imaging mode determination unit 311 Blind spot imaging unit 312 Target object tracking unit 313 Target object position determination unit 314 Camera platform control unit,
403 image data 1
404 Image data 2
405 Initial setting of number of repetitions 406 Feature point correspondence calculation start of two images 407 Matching number determination 408 Feature point correspondence calculation end 409 F matrix calculation unit 410 F matrix calculation end determination unit 411 Fields of view of two images overlap Flag 412 Update of the number of iterations 413 Upper limit of the number of iterations to update 414 Feature point correspondence calculation end 415 Flag that the fields of view of two images do not overlap

Claims (9)

At least two imaging devices;
Means for switching image recognition processing in accordance with the degree of overlap of images obtained from the two imaging devices;
An electric head that can move the direction of the optical axis of each imaging device;
The electric camera platform is dynamically controlled according to the result of the image recognition process. The multi-camera system according to claim 1, wherein
When switching the image recognition process according to the overlap of the field of view of the imaging device, if the field of view of the two imaging devices overlaps, stereo distance measurement function, three-dimensional object shape measurement function, panoramic image generation function, blind spot shooting function , Execute at least one of the human behavior analysis functions, and execute a function of detecting a detection target with two cameras when the fields of view of the two imaging devices do not overlap. The multi-camera system is characterized in that the function is switched according to the position of the moving object and the angle between the optical axes of the two imaging devices. The multi-camera system according to claim 2,
A multi-camera system comprising a combination of at least the two or more functions according to an output of the image recognition process. The multi-camera system according to one of claims 1 to 3,
As a processing method for determining the degree of image overlap, a function for detecting feature points that appear simultaneously in two images by image processing, performing matching, and calculating epipolar geometry from the matching result to determine the degree of image overlap A function that superimposes images while shifting the other image with respect to one image, and outputs the amount of image displacement as the degree of image overlap when the overlay error is minimized, and image processing Detects a detection target, specifically a person or a vehicle, a geometric structure existing in a shooting scene, or a specific marker defined for a multi-camera system, and a positional relationship between two images A multi-camera system comprising at least one of the functions for calculating the degree of image overlap from Beam. The multi-camera system according to one of claims 1 to 4,
Two imaging devices with different characteristics, specifically, at least one of resolution, frame rate, shutter speed, aperture, lens brightness, lens diameter, filter attached to the lens, and imageable wavelength range are different A multi-camera system that calculates the degree of image overlap between two images even in a combination of imaging devices. The multi-camera system according to claim 1, wherein
The direction of the optical axis of the imaging device is fixed without using an electric pan head, and at least two pairs of imaging devices having different angles formed by the optical axes between the two imaging devices are provided. A featured multi-camera system. At least two imaging devices;
Means for determining an overlap of images obtained from the two imaging devices;
Means for switching image recognition processing in accordance with the output of the image overlap determination means,
A multi-camera system that controls an imaging state of the imaging device. The multi-camera system according to claim 7.
When the field of view of the two imaging devices overlaps as an output of the image overlap determination means, processing related to the object in the field of view is performed, and when the field of view of the two imaging devices does not overlap, 2 A multi-camera system characterized in that an object is detected by one imaging device. The multi-camera system according to claim 7.
An electric pan head that moves the direction of the optical axis of the image equipment is provided,
The multi-camera system characterized in that the electric head is controlled in accordance with the output of the image overlap judging means.

Images